Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / Advice on Low-Level Memory Allocation and Memory Access Functions

Author
Message
Sasuke
20
Years of Service
User Offline
Joined: 2nd Dec 2005
Location: Milton Keynes UK
Posted: 4th Jun 2014 15:07
Hi guys, I've seen people using the memory access functions like Peek and Poke ete... and I've never quite understood there advantages/disadvantage and how they work (I've the basic concept of how in my head). Memory access in this way has always eluded me but I'm want to start dipping my toes in the pool of data.

So can anyone offer some advice, little bit of knowledge or even sources on how to you them (doesn't have to be in DBP, I can get the gist reading from other languages even if I don't use them). Any would be greatly appreciated

"Get in the Van!" - Van B
Van B
Moderator
23
Years of Service
User Offline
Joined: 8th Oct 2002
Location: Sunnyvale
Posted: 4th Jun 2014 15:33
I'm not sure what people are using them for these days, other than memory pointers for faster array and variable access.

I mean, it used to be the case on old computers that we had to peek and poke to do lower level stuff, like setting sound chip registers for music or sound effects, that sort of thing. In DBPro's case, and probably PC development in general, they are probably used for lower level access to memory.

Say you had a structured array, like a typed array for example - it might be beneficial to access the data directly rather than going through each part of the structure. If you wanted to save or load the array quickly, you could get the pointer for the first index and save the data out as raw bytes, or load them back in again. Also for sending network data - it's faster to send a string of bytes than individual fields, some network plugins might even enforce that method, through a memblock for instance.

Personally I don't like the way DBPro handles pointers and I don't even know why - C++ does such a good job of it, maybe I'm just spoilt. It also relies on a fairly robust array structure, fixed width strings and all - which I prefer not to stick to in DBPro. Maybe it's something best left as an option for optimizing memory access, saving and loading later in development - unless you need it for a network data system of course.

I hope someone can provide a good example of usage in DBPro and explain the benefits better than I can.

I am the one who knocks...
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 4th Jun 2014 17:20 Edited at: 4th Jun 2014 21:05
I peek and poke daily in DBPro; I peek directly, and tend to encapsulate poke calls using helper functions so to speak.

I only use them to create dynamic lists because DBPro has no simple means of defining new array references at runtime, nor can I see any better way to process hierarchies of data at optimal speed. I do not use them for anything else; but can imagine them to be faster than bank or memblock calls and have seen people use them for all sorts of things.

I'll give an example of use

The following code demonstrates how I would list 5 different sound numbers, send the tuple of sound numbers to a random sound player; and have the player run one of the sounds in the tuple. To demonstrate iteration, I created a sound removal function which iterates and deletes them.

To reiterate; I use helper functions to create and manage the tuples, but I always use the peek command to access my values directly:



Note that this is not tested code because I wrote directly into my browser. I have also taken a primitive approach when working with the memory, I imagine you would handle the operations with helper functions and classes as I do.

TheComet
18
Years of Service
User Offline
Joined: 18th Oct 2007
Location: I`m under ur bridge eating ur goatz.
Posted: 4th Jun 2014 17:42 Edited at: 4th Jun 2014 17:42
Chris, did you just give me a glimpse on how to have arrays inside user defined types? God damn it I wish you showed up with this a year ago, when I was still developing PonyCraft in DBP.

Your dungeon has been arrested by a signature image because it tried to be a mod
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 4th Jun 2014 18:26 Edited at: 4th Jun 2014 18:28
I know.. I tried all sorts of hacks from linking arrays, co-routines, strings, link lists you name it; eventually I ended up finding this to be the most comfortable approach.

Hopefully we will have better arrays in any new DarkBASIC update so that UDTs can store them.

Zero G Scott
18
Years of Service
User Offline
Joined: 14th Dec 2007
Location: California, USA, Earth, Sol Sys
Posted: 4th Jun 2014 20:07
Wait, what?
Diggsey
20
Years of Service
User Offline
Joined: 24th Apr 2006
Location: On this web page.
Posted: 5th Jun 2014 03:26
I use it to run custom machine code from DBPro (don't use this unless you really know what you're doing!):


Also, I use it to support returning UDTs from functions and modifying passed in parameters:
http://forum.thegamecreators.com/?m=forum_view&t=191773&b=1

And also just to implement custom data structures that are not easy to represent in DBPro.

However, I would suggest using the link/unlink array functions if you want arrays in UDTs, it's safer and likely faster than peeking/poking directly.

[b]
Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 9th Jun 2014 13:46 Edited at: 9th Jun 2014 13:51
This is why I personally choose to pay no attention to speed optimization, speed logic, principle, behavior, ideas or past tendencies prior to completing the game engine.

I did not care whether peek or poke was faster than arrays; but it seems like there is a possibility that peek and poke is twice as fast at setting many lists:

6 ms vs 3 ms

And half as fast at setting a single list:

1 ms vs 2 ms, perhaps because no link/unlink is required

I do use link/unlink and poke, but it has proven comfortable for me to use poke where many link/unlink calls would be required to carry out a basic task such as populating a list box or calculating an aggregate. Additionally, it is desirable to have functions which can work on any data, than have a seperate function for each UDT.

Again, it is safer to use arrays; but I am not 100% sure if it is always faster; especially when networking or low level manipulation comes in to play. This is only a meaningful argument for a large scale project, since with a small solo project, peek and poke would require about 1000 to 2000 lines of management code to become to read, whereas you can manage arrays using plugin commands.

thenerd
17
Years of Service
User Offline
Joined: 9th Mar 2009
Location: Boston, USA
Posted: 9th Jun 2014 23:51 Edited at: 9th Jun 2014 23:56
Quote: "I only use them to create dynamic lists because DBPro has no simple means of defining new array references at runtime, nor can I see any better way to process hierarchies of data at optimal speed. I do not use them for anything else; but can imagine them to be faster than bank or memblock calls and have seen people use them for all sorts of things.
"


It's completely possible though! I use the matrix1 array commands to easily dynamically create arrays and then store a pointer to them that you can save inside another array, or... wait for it... a UDT The only problem with the system I use currently is that I haven't figured out how to dynamically change the array size so the arrays are usually a fixed size. That could be easily integrated into the system though by just storing a variable somewhere that stores the size of the array.

[EDIT] Actually, I'm wrong about the array being a fixed size. In my code I set the size to 32 but in reality you only have to CREATE an array of a fixed size. Once it's been created you can add and remove elements freely.

Essentially the block of code works like the following. This code is taken from my entity system module, so it saves the pointer for the attributes and values array and saves its pointer into a UDT value on another array. You could store this pointer anywhere, it could just be in a variable too. I also have all this stuff enclosed in functions so everything's self contained.

Creating an array


Deleting an array


Accessing the array data


This system is extremely fast, I haven't done any benchmarks without my ingame graphics but it's smoothly handled almost 1000 entities each with 10 or so attributes. That's ~2000 arrays being dynamically accessed each frame since each entity has 2 arrays (for attributes and values.) I'm not sure the pointer system works on UDT arrays, I haven't had the need to try it.


Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 10th Jun 2014 00:01 Edited at: 10th Jun 2014 00:11
Thanks for the info.

I am sticking with raw memory however so that my entity classes can be defined without having to recompile.

I used to use link array like crazy 5 years ago though; I just can't go back now.

thenerd
17
Years of Service
User Offline
Joined: 9th Mar 2009
Location: Boston, USA
Posted: 10th Jun 2014 00:12 Edited at: 10th Jun 2014 00:18
Ahh, I just saw your and Diggsey's posts above - I had the page open from before so I didn't see your posts about the commands! In my experience the link/unlink commands have still been pretty fast, of course the peek and poke commands are definitely faster but it's a matter of convenience. I find the array approach to be much more straightforward to code so for me it's worth a small loss in performance. I definitely need to experiment more with the raw memory.

However, I'm not sure that the link array command has a significant overhead to it. The command isn't copying any data or creating new data, it is simply changing what the array is pointing to - so that's likely quite fast. In addition, since you're creating an empty temporary array, that shouldn't take time either. So the system seems inefficient but the link command may be faster than we think.

Quote: "I am sticking with raw memory however so that my entity classes can be defined without having to recompile. "

That's actually what I use the dynamic array system for, I don't have defined classes - instead I have lists of attributes and their values. One of the attributes would be the entity type. But from my engine's perspective every entity is standard.


Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 10th Jun 2014 01:39 Edited at: 10th Jun 2014 02:39
I understand.

It may seem odd, but I could not do things any differently, probably someone could make links work for my project, but with my understanding alone, I'd have to change my organization name from 'Binary Modular' to Binary Singular' or 'Binary not-so-modular afterall'. My entity abstraction system would not be very abstract; as I will explain.

Quote: "of course the peek and poke commands are definitely faster"


I personally am not sure; that was my point. My point is that no program has been created the same, and no end user or system is identical. In some cases they might be faster, in other cases slower.

My test indicated on my machine, link array is faster with the same list; but not with many lists.

And my workflow is 'it is fast enough until proven too slow!'

Quote: "I find the array approach to be much more straightforward"


One aspect of my approach has not been revealed, the actual management code. I do not actually have to write poke commands and memory allocation and resizing of my lists; I have only shown the inner workings to demonstrate to Sasuke what I do.

In reality I just call Insert( number, listID ), Remove( number, listId ), AddListToList( list, iListId ).

That is not possible with array links; because the function has to know the UDT and its format. Where as with abstract data, the management functions need not be concerned with what type of data is being managed; nor is there a need for a new function for every UDT.

I do not have to dimension local lists, link them and unlink them like:
Local Dim TextMessage$(10) : For i = 1 to 10 : TextMessage$(i) = GetNetString$() : Next i : Player.TextMessages = Get ArrayPtr( TextMessage$() ) : Unlink Array TextMessage$()

I can just set something; and the system puts it in a list for me; even if the list did not exist:
For i = 1 to 10 : Player.TextMessages = AddString(Player.TextMessages, GetNetString$() ) : Next i

I do not need to link my list. The list is just there; I just access it
MyVariable = IntegerItem(MyList, Index)

I also do not need to recompile 90,000 lines of code, to add a new field, which would be the case if everything had its own UDT array

Quote: "I'm not sure that the link array command has a significant overhead to it. The command isn't copying any data or creating new data, it is simply changing what the array is pointing to"


It is like a reference to an area of memory containing the list; quite similar underneath the surface.

Quote: "That's actually what I use the dynamic array system for, I don't have defined classes - instead I have lists of attributes and their values. One of the attributes would be the entity type. But from my engine's perspective every entity is standard."


I am afraid this is where raw memory roars past like a Ferrari racing a push bike. Whilst what you have done is wickedly smart; it is also constrained by the array system.

I am not arguing, just stating why I quit using links:

UDT lists limit you to fields that only store data in a fixed format.

So entities ranging from short pieces of text to the entities that form a world must inherit from a 'superclass' and thus assign unnecessary bytes and more bytes for fields to declare simple entities, where as my raw entities can actually be 8 bytes long; an integer or a float along with the address.

As far as I know you could not attach events handlers to UDT fields,
- nor can you have variable number of fields in the same/similar class (See polymorphism)
- nor can you listen to changes to field values
- nor can you clone one UDT class into a new one.
- nor can you watch or edit any of your field values in your debug window for all of your instances
- nor can you combine two UDTs into a new one at runtime.
- nor can you get one UDT from one application and put it into another application over a network
- you could not set readonly functionality as easily
- UDT fields cannot easily share validation functions, you'd have to call string or number checks in hard code each time the field is assigned a value; it is tedious to add validation functions which can check input on unlimited integer, string or floating point fields unless the assignment function has an easy way of accessing any UDT it sees fit; a walk in the park with memory addresses.
- and no polymorphism, that is to interact with instances of similar classes. - A flight function could not make a UDT bird and a UDT aeroplane fly, unless they where the same type.

Arrays must all be the same class; unless you get into the world the Array Field Ptr, in which case you would end up using peek and poke commands anyway; but far worse because you have to figure out the location of fields based on the definition order. Whereas having the field locations defined in the first place means not having to figure out anything later on.

The only reason why I know this, is because of finding out the hard way.

It is extremely difficult for me to put into words; I think it will make more sense when I release my editor.

Finally I must clarify that my engine actually uses array UDTs everywhere; it is just link/unlink that I dropped.

thenerd
17
Years of Service
User Offline
Joined: 9th Mar 2009
Location: Boston, USA
Posted: 10th Jun 2014 03:38
Quote: "UDT lists limit you to fields that only store data in a fixed format.
"


This is the bottleneck with my system. I've temporarily just been saving the values as strings and converting them when needed which is really inefficient and causes the most overall delay in my system.

As far as your list of concerns with UDTs - I agree with all of that which is why I don't actually use UDTs for anything.

Quote: "Finally I must clarify that my engine actually uses array UDTs everywhere; it is just link/unlink that I dropped."


That's ironic I'm exactly the opposite - I never use UDTs with the exception of general engine management. Link and unlink, on the other hand, is the core of my engine.

I hope you don't think I'm arguing for arguing's sake, actually the opposite, I'm very interested in hearing the other side of this debate because it hasn't been something that I have ever bothered to put much thought into. I've had success with linking so I haven't experimented as much as I should with other methods. This kind of stuff is what helps keep Dark Basic alive because we are attempting to implement semi-modern features


Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 10th Jun 2014 03:59 Edited at: 10th Jun 2014 04:02
I can understand why you would not use UDTs based on your design principle. I am doing what you did here: Entities(i).values.ptr

Except that my field is a memory address rather than an array pointer, no doubt the same type of thing. I stuck with top level UDTs for syntactic sugar for encapsulating the lower levels.

Kevin Picone
23
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 10th Jun 2014 07:47 Edited at: 10th Jun 2014 07:49
In DBPRO when you call a function the compiler leaves a snail trial of overhead behind. I think that's why they added the C styled write pointer stuff to the parser. Which is solved in line and doesn't call a function. Doing this way is not only faster but less code is produced. It's not a lean and mean as it should be, but seems quicker from the stock DBpro FREE edition.

The code bellow benches the average performance between write pointer with write memblock with a bit stepping code for good measure



Chris Tate
DBPro Master
17
Years of Service
User Offline
Joined: 29th Aug 2008
Location: London, England
Posted: 10th Jun 2014 10:27 Edited at: 10th Jun 2014 10:28
Indeed; I also think that all memblock commands check whether the memblock exists and whether the address is in range. Direct access bypasses the check which you could validate yourself more efficiently when doing what you did with the memblock pointer.

thenerd
17
Years of Service
User Offline
Joined: 9th Mar 2009
Location: Boston, USA
Posted: 10th Jun 2014 21:33
Quote: "In DBPRO when you call a function the compiler leaves a snail trial of overhead behind. I think that's why they added the C styled write pointer stuff to the parser. Which is solved in line and doesn't call a function. Doing this way is not only faster but less code is produced. It's not a lean and mean as it should be, but seems quicker from the stock DBpro FREE edition.
"


Hmm... do you mean DBpro user defined functions? does this mean precompiling source code to eliminate functions would produce a faster executable?


Kevin Picone
23
Years of Service
User Offline
Joined: 27th Aug 2002
Location: Australia
Posted: 11th Jun 2014 14:38 Edited at: 11th Jun 2014 14:42
Quote: " Hmm... do you mean DBpro user defined functions? does this mean precompiling source code to eliminate functions would produce a faster executable? "


Yes, but that's well known, so it shouldn't come as any great surprise. Not really my point though.

If you look at the assembly output from DBpro (it's in your temp folder btw) when compiling the code fragment above, it demonstrates that the compiler can resolve the pointer write / reads in line. This negates the need to call a function that would perform the same action. Even in functions with only a single parameter, there's still a heavy overhead in just setting up the call process, without even executing the function code inside it. All this is peanuts for the low frequency calls, but death in anything that requires brute force.


Take a look a following snippet and DBPRO's output bellow and you'll soon get the idea of how much code is produced when calling internal command set functions and user defined functions..





Here's what DBPRO produces with some extra spacing/lines to make it more obvious.



Login to post a reply

Server time is: 2026-07-06 05:35:36
Your offset time is: 2026-07-06 05:35:36