Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Data Compression and transfer

Author
Message
Ben_UK78
16
Years of Service
User Offline
Joined: 10th Jul 2010
Location:
Posted: 8th Aug 2011 12:40
Link
Hi All,

I am using multisync to transmit a series of words, floats and integers over the internet. All is working OK.

However, I am wondering if there is a way to reduce the amount of data I am transmitting. I have already reduced the data types down to their minimum, i.e words not floats when necessary etc (or bytes if I can get away with it).

My question is: is there some theory I am missing as to how I can send my data using even fewer transmission bytes.

(off the top of my head I am struggling to think of a foolproof way of doing it).

Thanks for any help.
Back to top
Profile PM Email
TheComet
18
Years of Service
User Offline
Joined: 18th Oct 2007
Location: I`m under ur bridge eating ur goatz.
Posted: 8th Aug 2011 13:28 Edited at: 8th Aug 2011 13:29
Link
How much are you sending? You can always use masks to squeeze everything into one byte, let's say you have this:

IJustPwndHim = 0-1
HeJustPwndMe = 0-1
DaPwnCounter = 0-9
PONIES = 0-3

Instead of sending 4 bytes of data, you can press that all into one byte and send it:



Same goes for decoding it on the other side:



TheComet

Back to top
Profile PM Email Website
Ben_UK78
16
Years of Service
User Offline
Joined: 10th Jul 2010
Location:
Posted: 9th Aug 2011 15:50
Link
OK, i'll have to stare at that a little more.

The data I am sending is like this:

string (time of database record creation)
string (date of record creation)
float (0.0 - say 40000.0, but must be a float)
float (as above)
float (0.0 - around 300.0, but must be float)
float (0.0 - around 30.0, but must be a float)
word (around 20-450ish)
word (0-200ish)
float (0.0-100.0 ish)
float (0.0-25.0 ish)
float (0.0-1000.0 ish)
word (0-10000ish)
word (0-300ish)
word (0-150ish)
word (0-150ish)
float (0.0-100.0) (percentage, but must be a float)
float (0-10.0ish)

So that is the data structure I am transmitting.

Is your way still applicable?

Much thanks.
Back to top
Profile PM Email
WLGfx
18
Years of Service
User Offline
Joined: 1st Nov 2007
Location: NW United Kingdom
Posted: 9th Aug 2011 21:16 Edited at: 9th Aug 2011 21:17
Link
For char strings, the characters usually range from 32 to 127, so they would fit into just 7 bits instead of 8. You wouldn't save much but if you're transferring thousands of strings then you'd save a little.

For floats you might get away with converting them to fixed point floats and just use either 2 or three bytes. ie. A fixed float may look like 10099 in int format but divided by 100 it would be 100.99. Depending on the range of your float, 2 bytes can fit a range of -32k to +32767. ie 327.67

Otherwise a float will take 4 bytes.

Your other integer values can be transferred as bytes or words depending on their range.

Compressing integer values across the board of your integers is also possible using bits.

Warning! May contain Nuts!
Back to top
Profile PM Email Website
IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 10th Aug 2011 15:59
Link
For all number types, if your numbers are always positive, or always negative, you don't need the sign bit - 1 bit saved.

Floats use a sign/exponent/mantissa system of 1 bit + 8 bits + 23 bits - reduce the accuracy (number of bits in the mantissa), or the range (number of bits in the exponent), for example by converting to a 16 bit float manually.

Get the latest Matrix1Utility plug-ins (20-APR-2011)
Back to top
Profile PM Email Website
Ben_UK78
16
Years of Service
User Offline
Joined: 10th Jul 2010
Location:
Posted: 11th Aug 2011 15:45
Link
OK, silly question time.

What am I missing here?

How do I go about changing float types of data types/structures?

I could potentially save quite a bit since no values need to be negative and float accuracy does not need to go beyond 2 decimals.

(I suspect I'm at that line now of too much work for too little gain, apart from where the database's sync at start it only transmits say 100 bytes every second on average including control bytes etc)
Back to top
Profile PM Email
IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 12th Aug 2011 00:08
Link
100 bytes / sec == Don't worry about it

Get the latest Matrix1Utility plug-ins (20-APR-2011)
Back to top
Profile PM Email Website
TheComet
18
Years of Service
User Offline
Joined: 18th Oct 2007
Location: I`m under ur bridge eating ur goatz.
Posted: 12th Aug 2011 15:33
Link
A lot of words there can be easily stored in bytes.



When using multisync, you stack all of the values into one big packet anyway, you don't send them all individually. So you'll be sending 70 bytes + 42 byte header of the packet = 112 bytes 60 times a second. That equals 6.56 KiB/s. Don't worry

Also, have you ever noticed that those words are very food-related? The inventors must have been hungry fellows. nibble, byte, bit...

TheComet

Back to top
Profile PM Email Website
Back to top

Login to post a reply

Server time is: 2026-07-10 15:31:45
Your offset time is: 2026-07-10 15:31:45