Well first you need to have DBC enhanced, meaning you need DarkMatter. This gives you the ability to call .dlls from within your programs. These .dll's, written in lower level, more powerful languages contain the new commands. To write your own you need to be able to code in C, C++, pascal, VB etc.
[edit] you didnt specify what kind of commands. It is possible in a way to create commands by way of using DB's functions. That however acts less of custom commands and is really just a shortcut, allowing you to code a function once and call it as required. If you want to add extra funtionality to DB, ie like NuclearGlory has written a complete 3d collision engine to replace the native DB collision, then DLLs are the way to go.