Thanks! In the meantime I found out. HexToBase64 takes pairs of hex characters.
My encryption function now looks like this:
function encryptString(input$ as string, key$ as string)
output$ = ""
// Get the encryption code length
codeLen = Len(key$)
//loop through the bytes and encrypt them
codeCounter = 0
for n = 1 to Len(input$)
inc codeCounter, 1
if codeCounter > codeLen then codeCounter = 1
coded = Asc(mid(input$, n, 1)) ~~ Asc(mid(key$, codeCounter, 1))
if Len(Hex(coded)) = 1
output$ = output$ + "0" + Hex(coded)
else
output$ = output$ + Hex(coded)
endif
next n
output$ = HexToBase64(output$)
endfunction output$
Works as intended.