Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Discussion / these numbers just wont add up

Author
Message
Little Joe
22
Years of Service
User Offline
Joined: 26th Aug 2002
Location:
Posted: 28th Feb 2003 23:26
Link
just to say i've already posted this on the realgametools.net site.

ok, so i'm trying to get db to do some adding and subtracting the a coupla big numbers, but it seems it doesnt like going to more than about 3 decimal places :s

The code (coming up) in the end works up untill the final calculation (hf#-workf#) where hf# is something to the power om minus 29, and workf# is something like 1.5

typical values for workfunction are anything between 1.5 and 4.5, and wavelength andything between about 300 and 700



can anyone give me a few pointers?
Back to top
Profile PM
Red Giant 2183
22
Years of Service
User Offline
Joined: 15th Dec 2002
Location:
Posted: 9th Mar 2003 10:14
Link
the problem lies in the first variables assignment (h#)
its well out of range of the variable type ! it would be like multiplying the 6.63 times .000000000000000000000000000000001
or the equivilant of
h#=6.63/100000000000000000000000000000000

RGteam~
Back to top
Profile PM
IanM
Retired Moderator
22
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 9th Mar 2003 14:37
Link
The floating point implementation within DB only *guarantees* 7 significant digits of storage.

Likewise, in DBPro, although DBPro does have a more precise floating point type as well - 15 significant digits guaranteed.
Back to top
Profile PM Email Website
QuothTheRaven
22
Years of Service
User Offline
Joined: 2nd Oct 2002
Location: United States
Posted: 9th Mar 2003 19:30
Link
not even the best graphing calculator is that precise

Darken the skies, we are god
Back to top
Profile PM
Back to top

Login to post a reply

Server time is: 2025-05-15 13:02:33
Your offset time is: 2025-05-15 13:02:33