Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Re-processing identical array data takes longer and longer

Author
Message
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 26th Mar 2010 22:35
Link
This is a long shot, I know. Sorry folks, can't show code, hundreds of lines relating to thousands of others and kinda proprietary.

All I want is a push in the right direction - "Did you check this?"

I have a great big typed array "A"(~2500 lines). I copy (~500 line) chunks of it into a similarly typed array "B" starting from the top and work down progressively.

From the smaller array "B" I generate another differently typed array "C". Both "B" and "C" are used to create 3D objects. They are destroyed by redimming to (0) when the next chunk is copied.

When the 3D objects are created, global variables and several nested do loops are involved with exits from the loops.

The code runs great. Never failed. Never crashed. It just takes 25% to 70% longer each time a chunk is copied. To test, I even made the data chunks identical but it still takes longer and longer. Array sizes don't change. I've re-checked all my initializations. Almost feels like some stack keeps growing somewhere but I don't know about such things.

If anyone out there has had a similar problem, I'd be interested in hearing about how you fixed it.

Regards,
fh

Back to top
Profile PM Email
Hawkblood
16
Years of Service
User Offline
Joined: 5th Dec 2009
Location:
Posted: 26th Mar 2010 22:49
Link
Quote: "They are destroyed by redimming to (0) when the next chunk is copied."

"redimming" or undim? To release any memory used by a "dim" command, you use "undim".

The fastest code is the code never written.
Back to top
Profile PM Email Website
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 26th Mar 2010 22:59 Edited at: 26th Mar 2010 23:00
Link
I can't help you directly on this, but could you clarify what you mean by:

Quote: "I have a great big typed array "A"(~2500 lines)."


Do you mean the type consists of 2500 different things - or do you mean the array index goes up to 2500? If you mean the latter, how many different items does the type have?

A bit more detail and it might be worth us experimenting to see if we can reproduce the problem.

In the meantime, I wonder whether you've found a memory leak. You might be able to check this using Task Manager and watching the memory for your exe's process.

You also say:

Quote: "To test, I even made the data chunks identical but it still takes longer and longer."


In that particular case you can re-use the arrays B and C. Have you tried that? It might give a clue where the problem lies.

One possibility is that some part of memory is getting fragmented as you keep redimming but I'm not sure.

You say you are creating 3D objects. What happens when you move on to the next pair of B and C arrays? Do you delete the previous objects or is the total number of objects gradually increasing? How many objects are involved? Either way a slowdown would be involved.
Back to top
Profile PM Email
tiresius
23
Years of Service
User Offline
Joined: 13th Nov 2002
Location: MA USA
Posted: 27th Mar 2010 06:30
Link
Quote: "You say you are creating 3D objects. What happens when you move on to the next pair of B and C arrays? Do you delete the previous objects or is the total number of objects gradually increasing? How many objects are involved? Either way a slowdown would be involved. "

I second this as a possibility. I once had a loop that would build temporary objects while it was running and then after it was done I would delete all the temp objects that were made. Well it was slow, so I decided to try and delete the temp objects immediately after I didn't need them, within the loop. It ran much faster. So make sure you are deleting the objects created in a timely manner.


A 3D marble platformer using Newton physics.
Back to top
Profile PM Email
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 27th Mar 2010 15:38 Edited at: 30th Mar 2010 03:10
Link
Thanks guys.

If I preface my arrayname dim(0) statements with undim arrayname(0) statements, there is no improvement.

I am working with user-defined type structures with about 40 "fields". They are a mix of integers, floats, and strings. The array "A" index goes up to about 2500. I'll try the task manager thing.

I have timed various segments of my code and have isolated two still large segments with the "longer and longer" problem. The problem is not the 3D object generation - DBPro shines there. 6000 objects no problem.

I did a re-write of one of the problem segments. Faster, yes, and it does not seem to be in the "longer and longer" mode any longer. Part of the re-write was to use another "permanent" UDT to hold some of my interim variables. My thinking was to force them to have a memory "home" that wouldn't move (as a hobbyist that is about as technical as I get) and it seems to work. Not sure if I can do the same for the other segment.

[edit]

The one qualitative before and after difference in the code block I was able to improve (get rid of the longer and longer characteristic) is that the before code used thousands of exits from a while-endwhile loop but the after code does not. The remaining problem block of code also has a lot of whle-endwhile exits. If I can re-write this to eliminate the do while exits and the "longer and longer" goes away, I may be onto something.

[edit]

That's it! (I think) Repeated exits from while-endwhiles are bad mojo. I guess it remains for me to post some code demonstrating this. I'll try later. If someone beats me to it, great.

[edit] Spend an hour trying to mimic the code and structures with this problem, but unable to reproduce the effect. I may have to gradually de-construct 9,000+ lines of code to see exactly what is doing this.

Back to top
Profile PM Email
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 27th Mar 2010 18:10
Link
Sounds plausible. I wonder if you've uncovered a stack problem.
Back to top
Profile PM Email
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 14th Apr 2010 03:50
Link
Not really scientific, I know, but reproducing/isolating this behavior to post as a bug could take me a week, which I don't have.

I have to take back what I said above.

It now appears that when I put a (seemingly redundant) 'undim arrayname (0)' statement in front of all my large UDT dim(0) statements, the problem seems to go away (processing the same data takes the same amount of time).

Back to top
Profile PM Email
Kevin Picone
23
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 15th Apr 2010 16:28
Link
If you're doing lots of allocation/de-allocation (resizing arrays for examples) then this sounds suspiciously like memory fragmentation.


Learn to code for FREE with PlayBasic Learning Edition
Back to top
Profile PM Website
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 15th Apr 2010 20:50 Edited at: 15th Apr 2010 20:53
Link
I am. By memory fragmentation, you mean a large data structure gets spread out over non-contiguous portions of memory?

If so, how might it be dealt with? I suppose I could make giant permanent arrays big enough for any data set and just clean them out before I dumped new data into them . . . or would the cleanout shrink them back down and allow subsequent data structures to 'box them in'?

Any ideas from others who have encountered this would be appreciated.

Back to top
Profile PM Email
Hawkblood
16
Years of Service
User Offline
Joined: 5th Dec 2009
Location:
Posted: 15th Apr 2010 21:37
Link
For very large amounts of data, use memblocks. Arrays use up more memory than the amount of data you have stored in them. Memblocks have a small overhead and is a faster access datatype. And don't worry about cleaning it out, just write over the info.

The fastest code is the code never written.
Back to top
Profile PM Email Website
Green Gandalf
VIP Member
21
Years of Service
User Offline
Joined: 3rd Jan 2005
Playing: Malevolence:Sword of Ahkranox, Skyrim, Civ6.
Posted: 16th Apr 2010 00:21
Link
Quote: "If so, how might it be dealt with?"


I thought you provided the solution yourself in your previous post - which seems consistent with Kevin Picone's theory. Each time you free the array memory you presumably get rid of the defragmentation because you start from scratch each time. I'm guessing ...
Back to top
Profile PM Email
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 16th Apr 2010 16:36
Link
Thanks for replies. I'll have to look into memblocks (all the forum heavies seem to use it).

Suppose I have a large UDT A() at 100MB. Then large UDT B() is created later at 200MB. Then A() is destroyed but then re-created at 150MB. Does it start where it was and part of it linked over after B() or does the OS look for a continuous 150MB block and place it there? Does it matter?

Maybe related maybe not. I have an XP machine with 2GB of ram. I can only make about 53,000 primitive cubes before DBP just crashes silently. I'm not good/familiar with memory math. What is going on here? With all that memory is 53,000 a realistic limit or does DBP only have so much memory allocated to it?

Back to top
Profile PM Email
Hawkblood
16
Years of Service
User Offline
Joined: 5th Dec 2009
Location:
Posted: 16th Apr 2010 17:13
Link
53,000!!!! That's HUGE! Each cube has 8 verteces, each with 3 floats and then an index buffer of 3 DWORDs per face (math,math,math,math)..... Well, that's a lot of memory. Your doing great to get the 53,000.

BTW, what kind of game would take that many pollys anyway?

The fastest code is the code never written.
Back to top
Profile PM Email Website
basjak
16
Years of Service
User Offline
Joined: 16th Apr 2010
Location: feel like signing up for mars
Posted: 16th Apr 2010 18:55
Link
it's much faster to save those arrays on a temporary file, so you can keep the memory running free for other problems.
Back to top
Profile PM Email Website
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 17th Apr 2010 16:50
Link
I'm considering a "real world" app and the world is very big . . .

Windows Task Manager reports I have 1.029G of physical memory available when DBPro running 3D and no cubes loaded and 1.005G when with 5000 cubes loaded. There are my associated UDTs to consider but still that works out to about 4800 bytes 'consumed' per cube. I still have over a gig of physical memory left which should allow me another 200K cubes. I'm hitting a wall somewhere, DX? The graphics card?

Back to top
Profile PM Email
Hawkblood
16
Years of Service
User Offline
Joined: 5th Dec 2009
Location:
Posted: 19th Apr 2010 16:53
Link
There is a limit to the card. If memory serves, it's about 60,000 verteces. Each cube has 8 therefore you can have up to 7,500 cubes. I may be off on my max number, but you see the problem.

The fastest code is the code never written.
Back to top
Profile PM Email Website
BatVink
Moderator
23
Years of Service
User Offline
Joined: 4th Apr 2003
Location: Gods own County, UK
Posted: 19th Apr 2010 23:15
Link
Do you have any strings in your array? If so, then this will be your problem. You have to delete the strings before you undim the array.
Otherwise, they get left in memory and slowly grind your application to a halt.

Back to top
Profile PM Email
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 20th Apr 2010 07:24 Edited at: 20th Apr 2010 17:32
Link
Wow BatVink. That is news to me. I have tons of strings in UDTs. I just read the bug report.

So, since the last post in that thread (2009), when working with UDTs, you do not bother with free string$() or IanM's clear array command but 'delete' strings? Do you mean overwrite with ""?

Back to top
Profile PM Email
Diggsey
20
Years of Service
User Offline
Joined: 24th Apr 2006
Location: On this web page.
Posted: 20th Apr 2010 20:36
Link
@freight hopper
Use 'clear array' before you undim it By 'delete' batvink simply means that you should free the memory belonging to strings in your array. The 'clear array' function is the simplest way to do this.


New DBPro IDE + integrated debugger!
Back to top
Profile PM Email Website
freight hopper
22
Years of Service
User Offline
Joined: 26th Dec 2003
Location: Just beyond the Dunsmuir yard limits
Posted: 21st Apr 2010 17:57
Link
I've played with the bug report snippets and am encouraged. Thanks for all the suggestions. It's good to be aware.

Banner suggestion: DBPro . . . leakproof as an adult diaper . . . someday . . . .

Back to top
Profile PM Email
Back to top

Login to post a reply

Server time is: 2026-07-26 07:35:46
Your offset time is: 2026-07-26 07:35:46