Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Large database write theory - a question

Author
Message
Ben_UK78
16
Years of Service
User Offline
Joined: 10th Jul 2010
Location:
Posted: 2nd Jun 2011 11:51
Link
Hi All,

I am writing a program to capture data from an external device through the comm port. It is going to capture 15 or so variables every 4 seconds and may be required to run for several months at a time.

Obviously this will build up quite a bit of data.

I was just wondering how people would go about structuring this. I am tempted to stick it all in a single array since I will need to plot time based charts of the data using different time ranges and jumping anywhere in the dataset, but if it crashes I don't want to lose the data.

My main question is how would people would organise it, and how people would avoid losing data? It will be a large text file to save every few seconds!
Is there some way that I can just write the new data to the file on the hard disk? (just update the file every 4 seconds instead of writing it afresh)

As you can see, I am in a small pickle over the theory and I'm hoping someone has a bright idea.

Thanks in advance for any help.
Back to top
Profile PM Email
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 2nd Jun 2011 12:08 Edited at: 2nd Jun 2011 12:09
Link
You can use IanMs DATAFILE commands. His commands have the ability to open a file, write some data, and close the file... without having to delete the file first and rewrite it like we have to do using the native Darkbasic Pro file commands.

Here's his list of DATAFILE commands:
http://www.matrix1.demon.co.uk/Matrix1Utils_Help/Matrix1Util_22_index.html

And if you don't have IanMs Matrix 1 Utilities Plugin then what the heck have you been programming with?... a chisel and a tablet of rock?

Direct Download:
https://forumfiles.thegamecreators.com/download/2192430

Back to top
Profile PM Email
DVader
22
Years of Service
User Offline
Joined: 28th Jan 2004
Location:
Posted: 2nd Jun 2011 13:41
Link
lol Grog, some of us don't like plugins. I find them a pain in the behind when it comes to sending projects to other people to test, missing dlls and such, even if the plugin is not used , . I do have Ian's utils installed at present though, even then I only use any of the commands occasionally.
BTW you don't HAVE to delete your data with DB if you save the array rather than a file. I use them all the time for game data without ever having to delete them. But with the amount of data access required here you have probably suggested the best method

http://s6.bitefight.org/c.php?uid=103081
Back to top
Profile PM Website
KISTech
18
Years of Service
User Offline
Joined: 8th Feb 2008
Location: Aloha, Oregon
Posted: 2nd Jun 2011 18:26
Link
With that amount of data you may run into performance issues if you try and store it all in one file. Plus you may not be able to access the file while it's in use by the process.

Two possible solutions would be:

- Open a new file each day at midnight, spreading the data across multiple files.

- Use an MSSQL or MySQL database server. Which would give you complete access to up to the minute data, and good performance.

Out of curiosity, what's the data you are gathering?

Back to top
Profile PM Email Website
IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 2nd Jun 2011 23:57
Link
That's not so much data really - it comes out at around 160MB of disk storage assuming that the 15 variables are of the 4-byte variety and stored in their binary form.

The more difficult part is running a DBPro program on windows, non-stop, for 4 months.

I'd suggest following grog's suggestion to ensure that you aren't using up too many resources or hitting the system too hard.

If you are a little more adventurous, I'd also suggest looking at using a memory-mapped file to lower resource use further.

It's only when you need to analyse the data (if that's the reason for storing it), that you should translate it into readable text form and dump it into a database for querying.

Get the latest Matrix1Utility plug-ins (20-APR-2011)
Back to top
Profile PM Email Website
enderleit
19
Years of Service
User Offline
Joined: 30th May 2007
Location: Denmark
Posted: 3rd Jun 2011 00:49
Link
Quote: "I do have Ian's utils installed at present though, even then I only use any of the commands occasionally."

I can't live without the FIND FREE RESOURCE() commands and the HITIMER() commands...

Well... I can, but they are SO usefull.

matrix1utils is just about standard nowadays.

- Martin Enderleit
http://sites.google.com/site/menderleit
Back to top
Profile PM Email Website
Phaelax
DBPro Master
23
Years of Service
User Offline
Joined: 16th Apr 2003
Location: Metropia
Posted: 3rd Jun 2011 05:49
Link
Quote: "lol Grog, some of us don't like plugins. I find them a pain in the behind when it comes to sending projects to other people to test"

Then you really limit yourself without plugins. Besides, any plugin you use should be included in the compiled executable, I don't see how it would be an issue.

Back to top
Profile PM Email Website
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 3rd Jun 2011 05:57 Edited at: 3rd Jun 2011 05:59
Link
@ DVader:

With IanMs plugin it's worth the extra effort to make sure no .dlls are missing. I use several of his commands in every project I start so you can see my bias.

Like the lowly MID$() command. The native Darkbasic MID$() command only takes out one character (unlike normal Basic/GWBasic). I have no idea why it wasn't allowed multiple characters but thankfully IanM brought back the old MID$() we grew up on (thank God for function overloading).

True we don't have to delete the file when we use SAVE ARRAY but it basically does because it seems to rewrite the entire file. It may delete it automatically. I'm usually first one at the bat when it comes to suggesting IanMs DATAFILE commands I love those things because I hated Darkbasics "have to delete the file to write a new one" demand. Back before I used his plugin I felt like I was programming with my hands tied behind my back. I remember on here everybody was telling me to use MEMBLOCKS but it was basically the same thing (still had to rewrite the entire file). It wasn't until I learned of IanMs DATAFILE commands and their ability to open a file and write data without having to delete the file first that my hands were freed of their bonds. That's why I'm IanMs greatest fan.

Back to top
Profile PM Email
BatVink
Moderator
23
Years of Service
User Offline
Joined: 4th Apr 2003
Location: Gods own County, UK
Posted: 3rd Jun 2011 09:13
Link
You can also write the data in batches of 15 lines once a minute rather than once every 4 seconds, reducing your writing bursts by a factor of 15. If you're running this for months, then losing 15 data reads if it crashes is insignificant. I would probably even do it once every 5 minutes.

Back to top
Profile PM Email
Back to top

Login to post a reply

Server time is: 2026-07-11 02:14:13
Your offset time is: 2026-07-11 02:14:13