Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Need Help with Reading Datafiles

Author
Message
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 4th Oct 2010 03:56 Edited at: 8th Oct 2010 07:29
EDIT 7 Oct 2010:

An attachment of all the code is attached.

Ok, so because a person on the second page asked me to keep updates here I will.

As for now if you skim through the posts you will be able to find some pretty decent code to graph plots (y can be given a log scale).

What's planned:
* Functionality for typing an X value and giving Y coordinate.
* Have it "guess" the largest slope within a range of points.
* Multiple graphs on the same coordinate system.
* More remarks
* Screen capture and Save functions.
* Record select points to txt document

Anyone can use the code. I ask that you use the same data only for purposes of testing the code (it's for a research project that I'm involved with at an university).

Other important info:
Format of Data:
in a txt document... currently you must place it in a folder (you can change its name in the code of course). The program supports only one column for data, the outputs. From here there's a constant that needs to be changed in order to support the size of the graph. (I'll get that more user friendly when I upload to this post.)

After that you're ready to graph stuff.

Keep in mind that the graph is as only accurate as floats, until double floats work, which I'm waiting on myself.


The code below is not code... it's just legacy text from what was originally here... yep even in cyberspace I'm a pack rat.

P.S. If a mod see's this it would be useful to rename this and place it in the WIP forum. Thanks!


EDIT: And of course some screens!






If at first you dont succeed, LOWER YOUR STANDARDS.
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 00:02
Figured it out how I can do it with the first array. Now I'm in a different predicament.



When ever I run this code it crashes immediately as it starts... Am I missing something?

If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 5th Oct 2010 00:22
With a normal .txt file the strings are at different lengths with datafiles you have to know exactly how many characters each string will contain. So if it's just a straight .txt file you don't want to load it as a datafile but with the normal Darkbasic file commands.



sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 00:25 Edited at: 5th Oct 2010 00:34
Ok, the txt file is actually data out from a machine recording echoes. It's formatted to where each number is on a separate line... is that what you mean?

EDIT: Also, the change still didn't fix the crash.

EDIT2: I've got the crash narrowed down to 5 possible lines:


If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 5th Oct 2010 00:43 Edited at: 5th Oct 2010 00:44
In a normal .txt file the string length is dynamic meaning it can be a small amount of characters or a large amount of characters. With datafiles to load it properly you have to specify how many characters you want to save for each string. Like if you opened a datafile and wrote 5 strings and each string is 20 characters in length if you want to load the second string you use SET DATAFILE POSITION 1,22*2 (22 characters * 2nd record ) then ycord$ = DATAFILE STRING$(1). Just loading it as a datafile blindly won't work. And because you cannot predict in a normal .txt file exactly the size of each string before loading it you can't use the datafile commands to load a normal .txt file.

Are you sure it's not crashing when you dimensionalize such a huge array? "Dim activeDatafile$(313508)" may be too big for your memory.

sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 00:51 Edited at: 5th Oct 2010 01:13
I thought it might be the array size, but I switched the array to a 1, and it still will crash.

Maybe it could be that I'm just using a 2.77MB (just looked it up) text file? I'll try and make another txt file with only a portion of the info.

Thanks for clarification on the datafile commands.

EDIT:
Not the size of the txt file either.
EDIT2:
More support against it being the array size:
http://forum.thegamecreators.com/?m=forum_view&t=91088&b=1
EDIT3:
I've got it narrowed down to writing to the array. I guess it doesn't like the strings? I've uploaded an excerpt from the text files... any other ideas how I should use it?

Notes about file: each number on a separate line. Comment lines start with a #. After each comment line the next set of data or "index" begins.

If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 5th Oct 2010 02:59 Edited at: 5th Oct 2010 03:01
It seems to work fine for me (see attached image). Try printing the data as it's loaded like this code snip with your huge .txt file. See how many it loads before it crashes.



sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 04:31 Edited at: 5th Oct 2010 06:07
Well, hmmm... depends on your definition of working.

It seems to get through about 10174 (+/- 2000) lines till it crashes (five tries). But, I ended up having to add another wait key command before the next command to even get that. Then after the first one printed I could press escape and it would start making the list. Pretty quickly I might add, as it only took 2-3 seconds till it crashed.

Maybe if there was a way to make it pace itself?

-- I changed the wait key function after the print to "wait 1" it then, after five minutes, it actually finished.

Is there a faster way to pace the loop so it isn't such a boring wait? lol

While this was doing its thing I turned on the performance monitor in the task manager.
CPU never went above 50% (probably averaged high 30s, and I noticed that one core was getting the crap beat out of it vs the rest of them... will DBPro not multithread applications?).
Memory went up to: 2.14GB. After I closed the program memory went down to: 2.11 GB after <1 min, so that's like 30MB.



Edit: probably a more accurate reading on the Mem usage... the processes tab states it's using 20,608K after 2 minutes of being finished.

Edit2:
So... I figured out that I could get to less than a millisecond wait period if I did the following code:


It's interesting, because it went from 5 minutes to about 3 minutes. Also, it was very evident when it was at the wait point because it was waiting to print too it seemed.

I'd still like to get the file loaded in about 30 secs or less if I can. If I set the pause timer to 10000 though it crashes. any idea?

EDIT 3:
Back in the main loop the code will still crash. Could this be because of syncing? or other things in loops? (2D graphics here fyi)

If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 5th Oct 2010 06:54
With all the edits it's a bit confusing... I just have questions for you with no answers right now.

Is it actually loading all the data then crashing?
Do you really need all that data loaded at once?
What are you actually doing with the data?
Have you tried using a float array instead (loading as a string and putting it into a float with the VAL() command)?

sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 07:06 Edited at: 5th Oct 2010 07:24
I'll try and include everything at once this time, and make it a bit neater. Sorry.

Is it actually loading all the data then crashing?

Do you really need all that data loaded at once?

What are you actually doing with the data?

Have you tried using a float array instead (loading as a string and putting it into a float with the VAL() command)?


If at first you dont succeed, LOWER YOUR STANDARDS.
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 07:38 Edited at: 5th Oct 2010 08:16
Answered my own question:


EDIT:

Ok, so I came up with some working code! Still doesn't work in the main loop, but I think it's going to be easier to work with.



Actual program (other then DLLS (BBB Gui and IanM's I believe) no external media):


If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 5th Oct 2010 08:35 Edited at: 5th Oct 2010 08:36
Quote: "Else it will crash if I don't have the wait in there."


It is weird that it crashes without a WAIT.

Quote: "Well, I need to get to it fairly quickly... Maybe only 3 or 4 indexes at a time; however, I'm not sure how else to do it."


You could have each index in their own file Index1.txt Index2.txt and so on. It would load faster if you did only do 4 at a time.

Quote: "See attached picture "


That's cool. Is that all the data at once?

Quote: "I was just thinking... Would it be easier on the computer to take a two dimension array?
ie (DIM activeDatafile$(indexnum,amountofcoordinates)

If so, how would I reference data? All programming I've done before I've always used one dimension, or just steered away from them."


I don't think the computer cares either way. It may be easier on you to use a multi-dimensional array. You've got the right variables in that DIM statement if you do want to make it that way.



sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 5th Oct 2010 15:09
Quote: "That's cool. Is that all the data at once?"

Nope. That's index 0. If I have it read only the first 2927 lines it works just fine, and I had it plot that.

The data is for astroparticle physics research that I'm involved in.

If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 5th Oct 2010 23:30
Yeah, just put each index in it's own file and load them one at at time (no need to make the array over 312,000). I still think it's taking up too much memory loading it all at once.

sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 6th Oct 2010 01:15
Quote: "Yeah, just put each index in it's own file and load them one at at time"


Do you mean into separate arrays? If so would I just keep clearing the same array out over and over?

If at first you dont succeed, LOWER YOUR STANDARDS.
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 6th Oct 2010 02:15 Edited at: 6th Oct 2010 02:16
I got it to work!!!! YAY! Exciting!

Here's the code:



Here's some media to try it with if you're interested enough. You need to rename the "CresisXPtScanOut - Copy" folder to "CresisXPtScanOut" to make it work. Also you need the BBB Gui plugin.

If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 6th Oct 2010 10:40
Quote: "I got it to work!!!! YAY! Exciting!"


I'm glad you got it to work. I like how you remarked your END just outside of the DO/LOOP with "//Freak accident?".

sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 6th Oct 2010 19:07
Quote: "I'm glad you got it to work. I like how you remarked your END just outside of the DO/LOOP with "//Freak accident?"."


Quite a few years ago I had a programming class (Quick Basic ), and he wanted every single entrance and exit remarked... That's usually what I would do for the main loop.

Now, I just need to figure out how to manipulate 2D graphics in DBPro without constantly redrawing them... if it's possible.

If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 7th Oct 2010 02:36 Edited at: 7th Oct 2010 02:37
Quote: "Quite a few years ago I had a programming class (Quick Basic ), and he wanted every single entrance and exit remarked... That's usually what I would do for the main loop. "


I used to program in QuickBasic too (many, many years ago) but I never had an official programming class in it. Back then we only had rotten Apples in school... I just learned at home. I only remark ENDs to tell newbies to always put an END before the first function just in case something horrible goes wrong.

Quote: "Now, I just need to figure out how to manipulate 2D graphics in DBPro without constantly redrawing them... if it's possible."


What kind of graphics do you mean? There are many options for graphic manipulation.

sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 7th Oct 2010 03:14 Edited at: 7th Oct 2010 04:02
Quote: "I used to program in QuickBasic too"

Whomever said I programed in Quick Basic? lol, I just said I took the class. And could make a comment in the program I didn't really learn much, other then keeping all the tabs nice and pretty, and how to make a random number generator (I think that's where I learned it).

Quote: "What kind of graphics do you mean? There are many options for graphic manipulation."


Several things I have planned for it, so it may sound a bit over the top for not having the best programming skills I'm sure.

Like if I wanted the For Next loop to draw the values between #xcord = 100 and #xcord = 2000
Ect Ect.... producing a zoom effect, and maybe with the mouse wheel.

Also, to get an accurate (to as many decimals as possible) reading of slopes on the lines, and lastly to get the MouseX position and it's calculated Y value (actual value... not the log value).

EDIT:
Can a Mod change the name of this thread to "DarkBasic Plotter"? It would be more appropriate. Granted it probably ought to be moved to the WIP Forum then.

If at first you dont succeed, LOWER YOUR STANDARDS.
SH4773R
16
Years of Service
User Offline
Joined: 18th Jan 2010
Location: AMERICA!!!
Posted: 7th Oct 2010 05:58
Well for zoom u could figure out how to save a image of the graph then use the mouses scrool wheel to increase the size of the image, that would be easy to move around 2, sorry for butting in u two seem to work pretty well
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 7th Oct 2010 06:09
Quote: "sorry for butting in u two seem to work pretty well "


Not a problem at all. I've seen Grog's posts around the forums and he is a very helpful member, but if I wanted it exclusively between me and him I would of just sent him a private message or something (not sure if I can now that I mention it).

Quote: "Well for zoom u could figure out how to save a image of the graph then use the mouses scrool wheel to increase the size of the image, that would be easy to move around 2"


I thought about that, but if I were to zoom out wouldn't it get distorted since it is just lines? I mean zooming in would be good though.

If at first you dont succeed, LOWER YOUR STANDARDS.
SH4773R
16
Years of Service
User Offline
Joined: 18th Jan 2010
Location: AMERICA!!!
Posted: 7th Oct 2010 06:53
I don't think it would get distorted because its just lines, I think it might make it look better even, that's the only piratical speedy way I can think of. Plus why would You want to zoom out anyway?
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 7th Oct 2010 07:29 Edited at: 7th Oct 2010 07:30
Quote: "I don't think it would get distorted because its just lines, I think it might make it look better even, that's the only piratical speedy way I can think of. Plus why would You want to zoom out anyway?"


I don't know if I would want to, but the ability to be able to would be nice. The next issue it would present is that the plot is graphed on a logarithmic scale... so simply cropping the image is going to really mess that up. The top of the graph in the picture a few posts up represents 10,000, where the middle of the graph is 1 and the bottom is .0001.
Basically as I scale it down I'm going to be having to find some way to adjust the scale too. I guess it could still be possible with the Get Image, but I'm just not necessarily sure it would be the most practical for its application.

however, would the following code work (can't test it at the moment, and writing from memory)?



If at first you dont succeed, LOWER YOUR STANDARDS.
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 7th Oct 2010 08:51
Quote: "Whomever said I programed in Quick Basic? lol, I just said I took the class. And could make a comment in the program I didn't really learn much, other then keeping all the tabs nice and pretty, and how to make a random number generator (I think that's where I learned it)."


Falling asleep in class will do that.

Quote: "Like if I wanted the For Next loop to draw the values between #xcord = 100 and #xcord = 2000 Ect Ect.... producing a zoom effect, and maybe with the mouse wheel."


There are a couple of ways to zoom. The one Commico Inc mentioned with images/sprites or you can just redraw it but use BOX instead of LINE to make the lines thicker. That's basically what a zoom is... taking a single pixel and making it a 2x2 box, a 3x3 box, a 4x4 box and so on. Sven B has a really cool plugin for Darkbasic Pro called Image Kit that has many graphic effects.

http://forum.thegamecreators.com/?m=forum_view&t=176270&b=8

Quote: "however, would the following code work (can't test it at the moment, and writing from memory)?"


It looks ok, but I'm not that good at math.

sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 7th Oct 2010 18:24 Edited at: 7th Oct 2010 20:26
Quote: " Sven B has a really cool plugin for Darkbasic Pro called Image Kit that has many graphic effects."


I'll look in to it.

Quote: "It looks ok, but I'm not that good at math."


Need help? (Mathematics major)

Quote: "Falling asleep in class will do that. "

That and not waking up for class


Edit3: I got the zoom to work! Grant it's only zooming to center, but that should be much of a problem.



Ignore the following section... it's trivial at this point (previous post that I figured out.


If at first you dont succeed, LOWER YOUR STANDARDS.
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 7th Oct 2010 22:25 Edited at: 7th Oct 2010 22:25
Updates:
* Plot onto linear y scale graph or log y scale graph
* Zooms (to center of graph) without having to repress plot
* X and Y axis are accurate to the Max and Min values of the graph



If at first you dont succeed, LOWER YOUR STANDARDS.
SH4773R
16
Years of Service
User Offline
Joined: 18th Jan 2010
Location: AMERICA!!!
Posted: 8th Oct 2010 04:21
Nice job, you might consiter using functions instead of subroutines then make a graphing library .dba file for all of us to use in the future, I would also advise editing your first post with updates aswell as pposting them.

I was just wondering how old are you? If your under 18(I am) don't post your real age just put "I'm under 18"
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 8th Oct 2010 06:10
Quote: " was just wondering how old are you?"

19, but why's this important(no offense just curious)?

Quote: "Nice job, you might consiter using functions instead of subroutines then make a graphing library .dba file for all of us to use in the future, I would also advise editing your first post with updates aswell as pposting them."


I've read somewhere on the forums that functions are slower, if by only a fraction, than subroutines. I could probably convert it to a function, but at the same time depending on the size of the graph those calculations might bog it down.

I'll go ahead and update the first post.

If at first you dont succeed, LOWER YOUR STANDARDS.
SH4773R
16
Years of Service
User Offline
Joined: 18th Jan 2010
Location: AMERICA!!!
Posted: 8th Oct 2010 07:56
Ok ill awnser in order cuz I can't use quotes on my phone,

Well I wanted to know because you seem to know your stuff prettey well,

Really? Interesting, I would advise running a time test with and without functions and see then the percentage of lag

Thanks! I don't really care personally but some people may still think its a question that has not been awnserd yet.

Also a idea... you could make a benchmarking tool with this!
sneaky smith12
20
Years of Service
User Offline
Joined: 30th Apr 2006
Location: Like id tell you, oh wait i just did
Posted: 8th Oct 2010 08:05
Quote: "Also a idea... you could make a benchmarking tool with this!"


As of right now, not the plan, but what do you mean? It would be hard to get a fair benchmark for anything because it's straining RAM, 2D Video processing, and your CPU all about the same time. Couldn't really test one thing on it's own I don't think.

Quote: "Thanks! I don't really care personally but some people may still think its a question that has not been awnserd yet."

Not a problem I think I mentioned on a post yesterday to have a mod change the name of the forum

Quote: "Really? Interesting, I would advise running a time test with and without functions and see then the percentage of lag"

Yeah, I found it again. TDK mentioned it, but apperently I missed the part where he said, "but not enough to be of any consequence"
http://forum.thegamecreators.com/?m=forum_view&t=171266&b=7

Quote: "Well I wanted to know because you seem to know your stuff prettey well"

This is just an illusion. I know math well, but I struggle with programming. I'm pretty good at knowing what to search for on the forums though.

If at first you dont succeed, LOWER YOUR STANDARDS.
SH4773R
16
Years of Service
User Offline
Joined: 18th Jan 2010
Location: AMERICA!!!
Posted: 8th Oct 2010 08:26
I guess I could be a full system benchmarker, idle just a thought out of a tiered mind lol. (Idle where u are but its 10:23 at night where I am)

Lol if a task is too much work.. search it

Well we seen to be just about opposite then
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 9th Oct 2010 05:36 Edited at: 9th Oct 2010 05:37
Quote: "Need help? (Mathematics major)"


Na, I'm fine. My friend Casey tells me "You're really good at math but don't know it.". I'm betting he means any calculations I do in programming.

My thoughts about functions vs subroutines is learn how to make subroutines but once you learn functions you should just use functions. I never use GOTO or even GOSUB anymore (unless helping a newbie with their own code). But it's up to you how you want to code.

Login to post a reply

Server time is: 2026-07-22 03:51:17
Your offset time is: 2026-07-22 03:51:17