Reading certain values from binary files

Hi all.
I need to be able to obtain the value as an unsigned integer from a 4 byte word from binary files. Any advice?

Thanks in advance.

Comments

Reading 8 bytes as True Basic number

The True Basic built-in functions NUM and NUM$ might also be useful, if the Binary64 floating-point format is just byte-reversed from the NUM$ format. I haven't checked that, but I run True Basic 2.7.2 on MacOS 9 and version 5.5 on Windows XP, and they can read and write each other's binary numbers in byte files using READ and WRITE statements. Not only that, but they also work correctly cross-platform using PACKB and UNPACKB for two-byte integers, which I would have expected to be byte-swapped, but aren't. So kudos to True Basic for excellent cross-platform compatibility.

Can now convert 8 byte 64-bit words to decimal values!

Here you go, girls and boys. I hope this is useful to you.
Here is a TB function to derive 64-bit double-precision decimal values from 8-byte words from byte files.

Note that this is written to be used on a PC (Intel/Windows platform, and its Little Endian form of numeric storage). It is also known as the Binary64 version of 64-bit values. This is by far the most common way 64-bit decimals are stored in byte files on the PC platform. And it is NOT the same as the 64-bit values TB can read natively.

This function can be used by TB coders as the 'double' would be used by C coders.

I modified this from a similar function for reading 32-bit values (what C coders refer to as 'floats'). I got the original (float) function from someone years ago, and I don't have a record or a memory that lets me give that credit where it's due. Part of me wants to believe it was BigJohn. If so, or to anyone else who may recognize important parts of this, please take a bow.

DEF dbl$(cvt$)
LET a$=""
LET b$=""
LET b1$=""
LET dblt$=""
! We first reverse the order of the 8 bytes to comp for Little Endian form
FOR i=8 to 1 step -1
LET dblt$=dblt$&cvt$[i:i]
NEXT i
LET cvt$=dblt$
FOR mb=1 to 8
! convert the reversed string to a hex string
LET aa$=hex$(ord(cvt$[mb:mb]))
! pad individual hex value w/ a zero if needed
IF len(aa$)<2 then LET aa$="0"&aa$
LET a$=a$&aa$
NEXT mb
! add any zeros needed to padd out the overall hex value.
DO while len(a$)<16
LET a$=a$&"0"
LOOP
IF a$<>"00000000" then
!!! the following binary string equals "0123456789ABCDEF"
LET b$="0000000100100011010001010110011110001001101010111100110111101111"
LET c$=""
FOR mb=1 to len(a$)
LET h=ord(a$[mb:mb])
IF h>=48 and h<=57 then
LET b=h-47
ELSE IF h>=65 and h<=70 then
LET b=h-54
END IF
LET c=4*b-3
LET c$=c$ & b$[c:c+3]
NEXT mb
! calculate the sign from 1st bit
IF c$[1:1]="1" then LET s=-1 else LET s=1
! calculate the exponent bits
LET e=0
FOR mb=12 to 2 step -1
LET e=e+val(c$[mb:mb])*2^(12-mb)
NEXT mb
! calculate the mantissa bits
LET man=0
FOR mb=13 to 64
LET man=man+val(c$[mb:mb])*2^(12-mb)
NEXT mb
! calculate the final IEEE 754 decimal number
LET d=s*2^(e-1023)*(1+man)
PRINT "Decimal number: ";d
LET dbl$=trim$(str$(truncate(d,8)))
ELSE
LET dbl$="0"
END IF
END DEF

! Now, that we have our function defined, read your data into a string
READ #1, BYTES 8: MyVal$
! Then call the function to convert the string to a decimal
LET MyDecVal$=dbl$(MyVal$)
! And convert it to its numeric form if you like
LET MyDecVal=VAL(MyDecVal$)

Best regards.

re: reading binary files

Hmmm, I just now tried to use UNPACKB in accordance with the manual (p.301-302; 18-92) and now TB won't even recognize UNPACKB as a function: "Undefined routine unpackb in Main program"...!

And of course it was DECLAREd up at the top. It is supposed to be a "built-in" function. Now it doesn't work with either Silver 5.33 or the buggy Gold 5.5b19. Now what.
This basically from the manual:

option nolet
DECLARE DEF unpackb
s$=chr$(85)&chr$(170)
print,"s$="&s$,
get key zz
! s$="0101010110101010" ! the 16 bits in this 2-byte string; decimal 21930
lens=len(s$)
print "len(s$)=";lens
upb=UNPACKB(s$,1,16) ! according to p. 302 =21930, as it should be
print,"unpackb(s$,1,16)=";upb !!!!!!!! "Undefined routine unpackb in Main Program"
END

I can't get this to work. You might have to write something yourself. I had to do something once when dealing with a MIDI file.

Offhand, maybe you want something like the following, which should work for a word of any number of bytes:

option nolet
DECLARE DEF bin2dec, d2b$, Nbytes2dec
print
w$="abcd" ! some 4-byte word of interest
print, "word$=";w$;" Nbytes2dec(w$)=";Nbytes2dec(w$)
print
END

DEF bin2dec(a$) ! converts binary a$ to decimal number n
option nolet ! MSB on left
u$=a$
l=len(a$) ! highest power of 2 will be l-1
n=0
i=1
DO
p$=u$[i:i] ! may need error trap
if p$="1" then
n=n+2^(l-i)
end if
i=i+1
LOOP until i=l+1
bin2dec=n
END DEF

DEF d2b$(d,r) ! r is the radix: the base number
If d=0 then
d2b$="0"
EXIT DEF
End if
z$=""
w$="01"
u=d
log2=log(2)
p=int(log(u)/log2) ! max power of r involved
FOR i=p to 0 step -1
rp=r^i
q=u/rp
qq=int(q)
z$=z$&w$[qq+1:qq+1]
u=fp(q)*rp
NEXT i
lz=len(z$)
if lz<7 then z$=repeat$("0",7-lz)&z$
d2b$=z$
END DEF

DEF Nbytes2dec(b$)
DECLARE DEF d2b$, bin2dec !(d,r)
lenb=len(b$)
dim u(1)
mat redim u(lenb)
for i=1 to lenb
u(i)=ord(b$[i:i]) ! "U">>85
u$=d2b$(u(i),2)
if len(u$)<8 then u$=repeat$("0",8-len(u$))&u$
ubin$=ubin$&u$
u$=""
next i
Nbytes2dec=bin2dec(ubin$)
END DEF

Hope this is of some use.
Regards, Mike C.

option nolet ! DECLARE DEF

option nolet
! DECLARE DEF unpackb
s$=chr$(85)&chr$(170)
print,"s$="&s$,
get key zz
! s$="0101010110101010" ! the 16 bits in this 2-byte string; decimal 21930
lens=len(s$)
print "len(s$)=";lens
upb=UNPACKB(s$,1,16) ! according to p. 302 =21930, as it should be
print,"unpackb(s$,1,16)=";upb !!!!!!!! "Undefined routine unpackb in Main Program"
END

This runs--Gives s$ = U^a, len(s$)= 2, unpack(s$,1,16)= 21930

Not sure you can DECLARE DEF an internal function.

That's as far as I can go right now.

rwt

UNPACKB no longer gives unsigned integers

Thanks for trying rtarara. Much appreciated. But your test code doesn't give the desired result.

For example, the four byte word: 2C 82 01 00 should return 98860 as an unsigned integer.

I'm stumped.

By the way, you can probably get UNPACKB to at least do whatever it's doing these days by making sure you have the library loaded, for example:

LIBRARY "C:\somepath\HexLib.TRC"

Regards.

Convert 2C 82 01 00 to decimal

Aha, so this is HEX, not binary. Try this:

Put the bites into the form: s$="00" & "01" & "82" & "2C" & "h" , i.e., "0001822Ch", as follows:

LIBRARY "hexlib.trc"! "C:\tbv5\tblibs\hexlib.trc"

declare def convert

option nolet

! 2C 82 01 00 are your bytes, given in that order...

DIM b$(4)

MAT READ b$

DATA 2C, 82, 01, 00

FOR i=4 to 1 step -1

s$=s$&b$(i)

NEXT i

s$=s$&"h"

dec=convert(s$)

print

Print,"HEX """&s$&""" converts to decimal "&str$(dec)

END

Ref p. 414 of manual; CONVERT function in HEXLIB.

Hope helps.
Regards,
Mike C.

Can now convert to Unsigned Integers, but not Doubles?

mcc was quite correct. CONVERT enabled me to derive correct unsigned integers from the BYTE files I'm reading. However, when reading them in from the file, they need to be pre-processed before handing them off to CONVERT. Here is what must be done.
After this explanation, I then have an additional challenge - reading longer values which are stored in byte files as what are commonly known as 'doubles'.
First, here's the explanation of handling unsigned integers:

!First, declare the function named to handle the conversion.
!Note here that we first get the "ord" for each byte and then get their hex values in $tring form, and then concatenate them together in reverse order, and finally TB's "convert" function can give us the correct value in $tring form.
DEF ui$(uint$)
LET s$=hex$(ord(uint$[4:4]))&hex$(ord(uint$[3:3]))&hex$(ord(uint$[2:2]))&hex$(ord(uint$[1:1]))&"h"
LET ui$=str$(convert(s$))
END DEF

! Here's where we read in our 4 byte string from the byte file and pass it to the 'ui' function above, where it will convert the result. We'll then convert it to numeric form.
SET #1:RECORD knownlocation
READ #1, BYTES 4: MyValString$
LET MyUnsignedInt=val(ui$(uidblk$))

I hope that is useful to others. It was to me. And thanks to mcc for pointing me in the right direction.
Now, on to my next problem...

Unfortunately the same approach used above can't be used to derive double precision 8 bytes 64-bit values. Additionally frustrating is that the TB6 reference manual says on page 124 that a simple READ statement can be used to directly read such values (for example: READ #1: MyDouble ) But this doesn't give expected results.

Perhaps the manual is correct in some environments, but C programs running in a Windows (little endian) environment typically compose 64-bit values using the very common "double" function. This is my situation. And until I can find a way to use TB to read these doubles, I'm severely stuck!

So here are some specifics in case you have the ability to help with this issue:

8 hex bytes as read from the file: 72 50 C2 4C DB 17 46 40 (reversed due to their little endian form)
The expected decimal value: 44.18638

Any and all help will be greatly appreciated.
Best regards.
rrapp

re: reading binary files

I would refer you to the manual and BYTE files. You will need to open the file as a Byte file, then you can read the info byte by byte or in 4 byte chunks. However, I think you have to know the file structure of your binary file--any headers for example, and the byte format for storing the numerical information. If you have that, you then know at which byte the data begins and can determine how to interpret a string of 4 bytes to extract the integer value. The manual provides enough info for reading the bytes but you may need functions from the HEXLIB.TRC to extract the numbers. FWIW

rwt

re: reading binary files

Thanks but I know all of that. I know where to find the bytes I'm looking for. And I know how to use TB to read byte files, and I've done it quite a bit in years past using v5.5 an older versions. My question is how to use v6 to derive an unsigned integer (a 'uint' in the C language) from a four-byte word. This used to work well using UNPACKB. But UNPACKB isn't producing the same results as before..

re: reading bindary files

Version 6 is only a new Editor. The TBsystem.exe is version 5.5b19. You can use any previous TBsystem you want--just replace the one in the Version 6 folder.

I would first bind your program and try running it 'outside' the editor to see if that works. If not, rename the TBsystem.exe file in the Version 6 folder, replace it with whatever system file you had been using, and try again. If this still doesn't work, report back here. I haven't worked with the UNPACKB functions for a decade or more so I'm not sure if and when it might have stopped working, but I don't believe it is a version 6 problem.

The bottom line here is that ONLY the editor is new in version 6. The base language and libraries are unchanged (see the thread on updating the language about that). Most problems that people have recently had involve compatibility issues with Win 7 and Win 8 which _usually_ can be solved by setting the compatibility for Tbsystem.exe and any bound programs back to XP.

rwt

re: reading bindary files

Thanks. Good to know. I guess I may have to try that exe juggling act and see if it makes a difference. But making any compatibility setting changes won't suffice. I need to be able to provide a bound program to a customer knowing it will work as-is on a Win 7 64-bit OS.

I had hoped someone from the forum might have some known ways to derive an unsigned integer from a four byte word.

And I assumed the True BASIC folks could verify if UNPACKB can indeed do that - it did in the past.

UNPACKB

Hi,

Whatever UNPACKB could do in the past, it can still do. there has been no change to this function in version 6. In fact there have been no changes to any function or statement between version 5 and version 6.

In one of the above posts the code example declared unpackb. This is not necessary for built-in functions.
There is also another error in the same post where a string was created, e.g. LET s$="0101010110101010". This is not a bit string at all. The way to create a bit string is to use CALL packb.

In my tests, both packb and unpackb continue to work exactly as they have always worked.

Regards
Big John