Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

AppGameKit Classic Chat / CopyFile/ MoveFile is it possible with AGK ?

Author
Message
Freddix
AGK Developer
21
Years of Service
User Offline
Joined: 19th Sep 2002
Location: France
Posted: 20th Jan 2023 16:43 Edited at: 20th Jan 2023 16:48
Everything is in the question.

I want to build a small tool (Windows) that organize photos by date folders and I must be able to move files from source folder to destination one.
Is this possible ?

Regards,
Virtual Nomad
Moderator
18
Years of Service
Recently Online
Joined: 14th Dec 2005
Location: SF Bay Area, USA
Posted: 20th Jan 2023 17:08 Edited at: 20th Jan 2023 17:09
SFSW
21
Years of Service
User Offline
Joined: 9th Oct 2002
Location:
Posted: 20th Jan 2023 18:07
Yup, the memblock commands are handy for that. Here is a quick function I like to use:



Then you can optionally delete the original if you just want to move it (and put it into a 'MoveFile' function for easy/repeat implementation).
Freddix
AGK Developer
21
Years of Service
User Offline
Joined: 19th Sep 2002
Location: France
Posted: 20th Jan 2023 18:54
Memblocks force app to load the file and save it again. it slower than real file copy.
Memblocks are ok when you handle few files, but when you must handle thousands of files, it is not adapted.
A true "Move file" should be used.

Thank you for your answer showing me that AppGameKit does not own true copy/move files functions for Windows.
SFSW
21
Years of Service
User Offline
Joined: 9th Oct 2002
Location:
Posted: 21st Jan 2023 01:37 Edited at: 21st Jan 2023 01:41
Quote: "Memblocks force app to load the file and save it again."


Of course, but any full copy operation is a read/load + write/save operation. So you will always have to load data and then save it out again whenever anything copies data. Is there a reason you expect it wouldn't/shouldn't be that way? What do you mean by 'real file copy'?

I manage about 1800 files with one of my projects and it does quite well. What are your performance results? To compare, I ran a test copying 100 files with that function I posted at 18 MB in size each with a result of 1.67 second (that's moving about 1.8 GB, so <100ms per 100 MB). Performing a 'copy *' operation from the command line is/was about the same.

I also ran a test with 1000 files at 1.8 MB in size each with a result of 1.173 second. So unless you are working with many thousands of files that are enormous (bigger than high mega-pixel JPG images at least), I would not expect performance to be much of an issue for you. And if you were trying to manage that many massive files, you'd probably encounter problems with any file copy method.

Also, be sure to use the file-to-memblock-to-file commands suggested and don't load the file(s) into an image memblock before you save them out. For maximum performance, you want to perform a direct byte-for-byte copy operation from source to destination with no memblock conversions or operations in the middle.

Did you try MadBit's plugin for a third method to compare with? I'd be interested to learn what methods you've tried and what your results are for comparison. Perhaps there is another factor I/we might need to consider.
Virtual Nomad
Moderator
18
Years of Service
Recently Online
Joined: 14th Dec 2005
Location: SF Bay Area, USA
Posted: 21st Jan 2023 05:54 Edited at: 21st Jan 2023 07:25
knocked up some additional testing with madbit's FileExplore and Memblock methods (on my 2-3 yo lenovo laptop's internal SSD with only ~5GB free ).
2,262 Files @ 3.0 GB
files ranged from 1 KB .pngs to 44 MB .gifs (just noted a few ~400KB .pdfs in there if it matters).

FileExplore Copy: 39.58s
FileExplore Move: 0.92s

Memblock Copy: 44.94s
Memblock Move: 17.82s

WIN Copy/Paste ~ 13s ("details"/no thumbs)
WIN Delete ~7s

@SFSW,
can you run the (complete project + 64 bit exe) attached on your batch of files?
maybe the large gifs threw my results off?
ie, how did you achieve a much lower Elapsed avg with Memblocks than i?

otherwise, see anything wrong with my functions? and, there might be some more-useful functions in madbit's set vs manual copy method.

Attachments

Login to view attachments
SFSW
21
Years of Service
User Offline
Joined: 9th Oct 2002
Location:
Posted: 21st Jan 2023 07:01 Edited at: 21st Jan 2023 07:15
Cool test program you wrote there

I ran the test with the 100 large 18MB files (1.8 GB) and my results were:

'COPY' operation: 1.21 second
'COPY MEM operation: 2.02 seconds
'MOVE' operation: 0.02 second
'MOVE MEM' operation: 2.05 seconds

To get the 'MOVE MEM' operation to work, I just ported the copy function and added a delete command after the copy operation. I also changed the virtual button line, the updated code is as follows:



As for results, this is with my main development system, which is pretty powerful. I suspect the ability to read/write data to/from memory has a lot to do with overall performance, which the laptop may be somewhat slow with (the Windows operations may also be somewhat slow also at around ~20 seconds total, but probably within expected results for such a device). Plus, if you're trying to move/copy 3 GB of data with only 5 GB free on an SSD, that could be a significant limiting factor... not to mention very hard on an SSD when it needs some room to load spread (cell balancing). I'd free up maybe at least 1/3 of the storage on your SSD, then try it again. If you still get similar results, see if you get better results maybe on a different system with better performance (particularly memory).

Either way, those results seem to confirm that FileExplore's copy operation is similar to internal memblock copying. Move has a distinct advantage (likely as well with Windows) with your system, likely attributable to method. But memblock copying/moving is still pretty quick, even on possibly slower machines, relative to Windows. Image files on their own should do pretty well, even larger ones copied in fairly large sets.

Edit: Running the test again with a wide variety of files (types, sizes, names, etc) and over 2800 files, the results are 2.79 seconds for FE's copy method, 0.47 second for FE's move method, 2.4 seconds for 'Copy Mem', and 1.79 for 'Move Mem'. However, I suspect some caching was involved in the latter, so I wouldn't put too much weight in the improved time for secondary operations. But overall, still really fast with more files and a wider variety of file types (including faster than native Windows copy and move operations, although move was fairly close). Windows copy operation was nearly 10 seconds, move was quicker and closer to the copy operations of the other tests.
Virtual Nomad
Moderator
18
Years of Service
Recently Online
Joined: 14th Dec 2005
Location: SF Bay Area, USA
Posted: 21st Jan 2023 07:33
hmm. adding the MemMove() and Move < Copy:

Memblock Copy: 44.94s
Memblock Move: 17.82s

re-running some of the others and my speeds are all over the place (last "copy" < 15 secs)... so, my data is garbage.

i did buy a 6 TB external ~xmas that i need to set up

thanks for testing/finishing/fixing it. i might expand it "some day".

SFSW
21
Years of Service
User Offline
Joined: 9th Oct 2002
Location:
Posted: 21st Jan 2023 17:26
I think if the program created new test files (large text files with random characters in them) each time, then started the timer to monitor copy duration, it would probably provide more accurate consistent results without caching offsets. But it's still valuable to gauge overall results, even with caching involved.
SFSW
21
Years of Service
User Offline
Joined: 9th Oct 2002
Location:
Posted: 21st Jan 2023 20:42
Here is an update version of the program that creates 1000 files with random line lengths for a more consistent test result:



For me, I get pretty consistent times when running each method repeatedly. So this may be a better overall gauge of performance. I generally get in the 0.95 to 1.2 second range (few tenths slower for mem operations, so marginal difference). Caching may still get involved from time to time as occasionally, delays in retesting seem to slow down the initial delete operation significantly. But that would likely be true of any file management command, not just AGK's. Overall, all are comparatively very fast.
blink0k
Moderator
11
Years of Service
User Offline
Joined: 22nd Feb 2013
Location: the land of oz
Posted: 21st Jan 2023 22:42
You may run into issues when you copy a file > 4GB
fubarpk
Retired Moderator
19
Years of Service
User Offline
Joined: 11th Jan 2005
Playing: AGK is my friend
Posted: 28th Jan 2023 10:41 Edited at: 28th Jan 2023 23:52
Thanks for sharing SFSW and VN
@Blink0k I think windows always had a problem with large file copying. hence why programs like unstoppable copier etcetera were created in the first place
https://www.majorgeeks.com/files/details/unstoppable_copier.html there are several variants of this style of program out there, tier 2 might be the
better approach if you are after true file copying, doesnt mean it won't hang on certain unpredictable interruptions and we have the ability to error
trap interrupts I predict there will always be some type of issue as with 90 percent of software out there.
fubarpk
https://fubarpk.itch.io/

Login to post a reply

Server time is: 2024-04-19 07:43:27
Your offset time is: 2024-04-19 07:43:27