As cipher17 noted, use a hex editor. There are some excellent 'free' ones out there...
Many hex editors have two windows... one that shows the ASCII character and the other holding the hexidecimal value. Look to see if there is a pattern in the ASCII window; this may yield the length of whatever 'records' they have used. It really helps if you have one of the records converted to numeric value. It's a matter of looking at the hex value for the numeric value. You may also require determining if numeric variables are single or double or something else...
Start with the easy ones first and then fill in the blanks... good luck... I recently unravelled some climatic data, having the Fortran source for the 'encoding'... it was not pretty; it looks like the original record had been added to by various programmers over the years... but, by adding to the original source... after several iterations, it was really strange.