Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Fixing small numbers when you divide

Author
Message
Hamish McHaggis
23
Years of Service
User Offline
Joined: 13th Dec 2002
Location: Modgnik Detinu
Posted: 11th Oct 2003 21:29
Link
When I do:

a#=1/10.0

a# gains the value 0.10000000149, which is wrong and also annoying when trying to display it in a compact space. Does anyone know how I could shorten it to 3 decimal places so I can get rid of the small number on the end? Thanks.

Brains are for idiots.

Athelon XP 1400 Plus - Nvidia Geforce MX400 - 256mb RAM
Back to top
Profile PM Email Website
heartbone
23
Years of Service
User Offline
Joined: 9th Nov 2002
Location:
Posted: 12th Oct 2003 18:06 Edited at: 12th Oct 2003 18:12
Link
a#= Int(a#*1000.0)/1000.0

However when I print the value I see 0.1 before and after. (patch 4)

Did something change in patch 5?

If the value still has the excess you could try to find the decimal in the string and truncate at the proper position.

The more you see, the more you know.
The more you know, the more you see.
Back to top
Profile PM Website
Hamish McHaggis
23
Years of Service
User Offline
Joined: 13th Dec 2002
Location: Modgnik Detinu
Posted: 12th Oct 2003 19:22
Link
I get the same before and after, obviously something has changed in patch 5.

Brains are for idiots.

Athelon XP 1400 Plus - Nvidia Geforce MX400 - 256mb RAM
Back to top
Profile PM Email Website
IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 12th Oct 2003 19:37
Link
Yep. The precision of printing numbers changed due to the fact that people were complaining they weren't printing accurately enough ...

Enough people have had problems with this that I might try and put a FORMAT type TPC function together.
Back to top
Profile PM Email Website
Hamish McHaggis
23
Years of Service
User Offline
Joined: 13th Dec 2002
Location: Modgnik Detinu
Posted: 12th Oct 2003 20:25
Link
OK, I just posted this in the bug reports, better go and withdraw it.

Brains are for idiots.

Athelon XP 1400 Plus - Nvidia Geforce MX400 - 256mb RAM
Back to top
Profile PM Email Website
David T
Retired Moderator
23
Years of Service
User Offline
Joined: 27th Aug 2002
Location: England
Posted: 12th Oct 2003 20:37
Link
I was just wondering how DBP gets the answer 0.10000000149

A computer program will always do what you tell it to do, but rarely what you want to do.
Back to top
Profile PM Email Website
Hamish McHaggis
23
Years of Service
User Offline
Joined: 13th Dec 2002
Location: Modgnik Detinu
Posted: 12th Oct 2003 20:41
Link
No, it isn't the print function, try this code, it should result in a# being 0, but it doesn't...



Brains are for idiots.

Athelon XP 1400 Plus - Nvidia Geforce MX400 - 256mb RAM
Back to top
Profile PM Email Website
IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 12th Oct 2003 20:43
Link
Have a look at the link posted by EC in Hamishs bug report. That explains it all - it even uses the same number (10) as an example.
Back to top
Profile PM Email Website
Hamish McHaggis
23
Years of Service
User Offline
Joined: 13th Dec 2002
Location: Modgnik Detinu
Posted: 12th Oct 2003 20:46
Link
OK, so you didnt mean it was a problem with the print function?

Brains are for idiots.

Athelon XP 1400 Plus - Nvidia Geforce MX400 - 256mb RAM
Back to top
Profile PM Email Website
DrakeX
23
Years of Service
User Offline
Joined: 26th Aug 2002
Location:
Posted: 12th Oct 2003 21:49
Link
it's not a problem -- and it's rather hard to explain and understand. someone told us that that's as accurate as it can get. it's something to do with how the floating point number is represented in binary. they said it's something like.. trying to represent 1/3 as a decimal. you can never be accurate enough. so, even though 0.1 is a simple terminating decimal, the computer has trouble with it. because, after all, it uses binary. which isn't very closely related to decimal at all.

stop looking at me!
Back to top
Profile PM Email
Back to top

Login to post a reply

Server time is: 2026-07-26 04:27:53
Your offset time is: 2026-07-26 04:27:53