Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Read File / Write File slooooowwwww [help]

Author
Message
UncleRemus
14
Years of Service
User Offline
Joined: 19th Mar 2012
Location:
Posted: 19th Mar 2012 20:17
Below is the script I am using to read a file and write into another file...The reason I am doing this is to script a basic file joiner that joins multiple files into one large file.



Took it 7 minutes to write 20mb....this is way to slow lol
Brendy boy
20
Years of Service
User Offline
Joined: 17th Jul 2005
Location: Croatia
Posted: 20th Mar 2012 05:15 Edited at: 20th Mar 2012 05:15
remove the cls and print commands from the loop

Kevin Picone
23
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 20th Mar 2012 05:51


I'm not surprised that such a loop takes a long time to execute. When fetching or writing data to a drive, a better approach is to buffer in block in memory rather than it byte by byte, since drives have such large read/write latency.

So you'd create a small mem block buffer then step through the source file in chunks of the buffer size. SO reading a chunk, then writing a chunk.

(DBpro styled Example written in PB)




I still wouldn't recommended reading Byte by Byte though, use the biggest data type that dbpro allows you to read/write when accessing the data. Preferable read/writing the entire block in one hit. There's bound to be plug in that offers this.. (Read/Write Memory)


Another approach would be to do the entire merge in memory. First calc the total size of the output file and allocate an appropriately sized mem block. Then read the files into the buffer. When done, save the entire mem block..

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 20th Mar 2012 16:12
Quote: "Another approach would be to do the entire merge in memory. First calc the total size of the output file and allocate an appropriately sized mem block. Then read the files into the buffer. When done, save the entire mem block."


Would make memblock from file, make file from memblock and copy memblock be the commands to use?
UncleRemus
14
Years of Service
User Offline
Joined: 19th Mar 2012
Location:
Posted: 22nd Mar 2012 01:14 Edited at: 23rd Mar 2012 00:04
first off, sorry for the late response...now to the goods

Quote: "remove the cls and print commands from the loop"

The cls and print commands are not executed every read/write. It is only updated when the next file is loaded. It will only run a CLS command when it changes file for precisely this reason, and copies very slow byte by byte.

Quote: "Would make memblock from file, make file from memblock and copy memblock be the commands to use?"

Memblock, yes, that is what I was looking for...thats what you get when you are on 48 hours of caffiene hacking away in codesurge buzzing through the help file -.-

will let you know the results shortly via edit (20 mins tops?)

EDIT - Heres what I got to work, although it doesnt copy the file perfectly...it adds 4 bytes to the beginning of each file which can be worked around but its a pain

Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 23rd Mar 2012 13:48
Is it much faster?
IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 27th Mar 2012 20:30
No-one has mentioned it, so I'll raise my plug-ins as the answer:


On my system it executes in 63ms with a combined file size of 31.5MB. I think you'll agree that it's a little better than the original 7 minutes

Mage
Valued Member
19
Years of Service
User Offline
Joined: 3rd Feb 2007
Location:
Posted: 2nd Apr 2012 03:40 Edited at: 2nd Apr 2012 03:46
Original Method:
You need to do more than this.

It was suggested you read the file in blocks or "buffer" the transfer. This is crucial as mentioned above, due to drive latency. The advice is right, this is huge.

The End Of The File:
You will discover at the end of the file you don't fill the entire buffer. You will need to use a smaller buffer or end the file byte by byte, which is the sloppy slow way.

Buffer Size:
I recommend 20k bytes if the transfer is over a network, and up to 10MB buffer with disk to disk transfers. Bigger buffers stop getting faster. Making a large buffer increases delay due to transfer errors, and uses more memory.


IanM's Method:
Simple and effective. But caution, I suspect his code is loading the entire file into memory. If this is true then the code may fail on larger files where memory is an issue. Also this may introduce a long transfer period where the program becomes unresponsive during the transfer. Just imagine a 2GB file taking several minutes or more. Finally, because the transfer is handled with one command sequentially, you can't update the screen with progress indicators. You have to wait until it's done. If I'm right then small files or short transfers should be no issue.

Edit: UncleRemus's method probably has the same issues as IanM's.

Mage
Valued Member
19
Years of Service
User Offline
Joined: 3rd Feb 2007
Location:
Posted: 2nd Apr 2012 05:26 Edited at: 2nd Apr 2012 06:13
I made an attempt to solve this problem. See below.

You can't read or write in large enough chunks with DBPRO commands to properly buffer a file transfer. I thought I could buffer DWORDS with an array to compensate but it has little effect. You need to actually read/write in larger chunks.

DISREGARD--- The half completed solution is below if you wanted some idea how this would work. The code still has bugs that produce a damaged file at the ending.

I was able to transfer and view a 2 hour film using this code. There was no point in finishing since the transfer was about 1/10 what I was hoping for. (1MB/s Read/Write same drive.)

Edit: Finished Code Below
Decided to finish the code, performance still much slower then it really should be. It works, at least it seems to. I'm able to completely and perfectly transfer files byte for byte. Someone might be able to use a plugin to buffer the read/writes so i decided to finish.



IanM
Retired Moderator
23
Years of Service
User Offline
Joined: 11th Sep 2002
Location: In my moon base
Posted: 2nd Apr 2012 15:51 Edited at: 2nd Apr 2012 22:15
As the total size of the resulting file was only 20MB, I certainly didn't think about memory usage.

If I had to think about large files in this context, I'd still use my own file IO commands as they are far faster than DBPro's, and I'd read/write largish chunks of multi-megabytes at a time as you suggested.

[edit]


It's massively slower though - it now takes almost a second and a half to run for the same size files

Mage
Valued Member
19
Years of Service
User Offline
Joined: 3rd Feb 2007
Location:
Posted: 3rd Apr 2012 18:05
That's still awesome thanks.

Login to post a reply

Server time is: 2026-07-09 17:42:32
Your offset time is: 2026-07-09 17:42:32