Go into vertex mode, then use the "target weld" , wich appears in the right-click-menu. Then just click on the vertex you want to weld and then click on the other vertex you want to weld with the previus one.
Another way is to, go to edge mode, then select the edges you want to remove, but instead of pressing the delete key, open the right-click-menu and click on "Remove". This will remove the edges, BUT NOT THE VERTEXS, so you'll have to go to vertex mode and select the lonely vertexs and do the same, right-click-menu and "Remove".
Why first remove edges and then vertex and not the other way around?, well try it out and see for yourself, but basically i'd say that doing it that way you'll remove all edges that go to those vertex, and sometimes you'd like to keep 1 edge.
Further on my stuff at...