Sorry your browser is not supported!

You are using an outdated browser that does not support modern web technologies, in order to use this site please update to a new browser.

Browsers supported include Chrome, FireFox, Safari, Opera, Internet Explorer 10+ or Microsoft Edge.

DarkBASIC Professional Discussion / non-latin characters in code

Author
Message
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 18th Nov 2013 00:20
Link
Hi, fellows!
Once I made a small text app on Turbo Basic. It was MS Dos version, afair. Also back then I worked with Visual Basic for MS Dos. Problem: when running compiled app on different machines, there is always a chance that all non-latin characters will be corrupted. Notice - in compiled app. Language code page issue? I don't know, I am not power user.

I think that this is very important moment as there are lot's of guys here using custom language characters in code (French, German etc...)

So:

1. Have you ever meet this problem on any language/platform?
1-a. Solution?

2. How does DBPro handles all non-latin characters:
*- inside compiled code in string values (like "КРАСНАЯ ПЛОЩАДЬ"
*- inside comments in code (I know there is a problem)
*- when using non-latin characters in file operations (reading, writing or comparing to string value hardcoded in app)

Anything is greatly appreciated.

Back to top
Profile PM
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 18th Nov 2013 22:43
Link
DBPro uses ASCII extended for 1-byte characters in its strings.
By changing the charset you can chose from pre-made packages of what the last 128 values will map to (the first 127 are always the same). This is enough for most any european language I believe, but it definitively falls short of things like chinese, japanese and maybe russian as well.
Apart from output, the charset is most likely irrelevant - a character that is displayed as '°' in one charset will be regarded as equivalent to 'ö' in another. Only the actual character values are used when comparing strings.


"Why do programmers get Halloween and Christmas mixed up?"
Back to top
Profile PM Email Website
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 18th Nov 2013 22:47
Link
I won't say how to do it but in fact DBP does support Chinese and Japanese...

Just buy the Dark Source pack to see how...

Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 18th Nov 2013 23:03
Link
Quote: "DBPro uses ASCII extended for 1-byte characters in its strings."

You mean that DBPro supports 100% only characters listed here?

Quote: "Only the actual character values are used when comparing strings."

Don't understand this

Personally I don't know anything about charsets or related stuff. So what should we do to get 100% compatibility? Let's shrink the problem to european characters only (asian characters is a whole another world).

Back to top
Profile PM
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 18th Nov 2013 23:04
Link
Quote: "Let's shrink the problem to european characters only (asian characters is a whole another world)."


Pretty sure Russian is classed as an Asian language.......

ICBW

Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 18th Nov 2013 23:11 Edited at: 18th Nov 2013 23:15
Link
Is this somehow helpful?

Quote: "Pretty sure Russian is classed as an Asian language"

I think it's classed as cyrillic. But, ICBW

Back to top
Profile PM
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 18th Nov 2013 23:50
Link
Quote: "I won't say how to do it but in fact DBP does support Chinese and Japanese..."

I'm not saying it can't be done, but since the basic implementation obviously doesn't use multi-byte character encoding, I doubt it would be as easy as myString$ = "insert-weird-unicode-characters-here".
And no, I'm still a poor student, so can't buy anything just to check up on that...


"Why do programmers get Halloween and Christmas mixed up?"
Back to top
Profile PM Email Website
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 18th Nov 2013 23:57
Link
I give up, I will end with this...

Quote: "set text font"


good luck, I am glad my new language of choice supports Unicode by default...



I was sad that there was no support whatsoever on how to obtain the strange characters...

Back to top
Profile PM
MrValentine
AGK Backer
15
Years of Service
User Offline
Joined: 5th Dec 2010
Playing: FFVII
Posted: 18th Nov 2013 23:57
Link
Quote: "I doubt it would be as easy as myString$ = "insert-weird-unicode-characters-here"."


Actually it is...

Back to top
Profile PM
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 19th Nov 2013 00:10
Link
Yes, but that's what I was talking about earlier. It changes the charset, but it cannot (or, maybe it can, but not bloody likely) change the whole encoding from single- to multi-byte. Supposedly it then gives you 128 (or possibly 256) generic characters.
Still, if you look for example here, that one, which is indeed on the "supported list" of DBP, does use two bytes per character.

It is possible that DBPro's strings are variable-width. If they are that would break lots of functionality for them that I have written, assuming they are strictly one byte / character though


"Why do programmers get Halloween and Christmas mixed up?"
Back to top
Profile PM Email Website
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 19th Nov 2013 07:20 Edited at: 19th Nov 2013 07:25
Link
Asian charset is a different thing, it needs special fonts and input tools.

I just thought - there are two types of string usage: internal and output. Output uses system fonts, so it depends on them, if they don't support 'Ä', obviously they can't use it. Internal usage means only internal strings***. Then if internal string contains character code, like 'chr187' or how it is there, will it work always in "read string from file and compare"? I believe it reads 'code' not depending on system fonts?

***edit: I mean internal in compiled app. Could compiled app have umm... internal charset?...and could code characters be corrupted during compilation?

Back to top
Profile PM
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 20th Nov 2013 14:50
Link
Please, anyone? It is still unclear to me.

Back to top
Profile PM
Grog Grueslayer
Valued Member
21
Years of Service
User Offline
Joined: 30th May 2005
Playing: Green Hell
Posted: 21st Nov 2013 17:40
Link
Hey this is my first post in a long, long time.

The only way to do it is the way Rudolpho has been saying. If you look at a Russian keyboard you can see that "КРАСНАЯ ЛПОЩАДЬ" is actually "RHFCYFZ KGJOFWM" so you keep the English letters in code but use a bitmap font that shows Russian letters on the screen instead... so every Z in strings will always be backwards R.



Back to top
Profile PM Email
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 21st Nov 2013 21:21
Link
Mmmm yeah, I think that it is truly the most efficient way - to encode with character substitution.

And the "last boss" - comments in code. I can't encode comments. Can they somehow spoil or affect compilation or compiled executable?

Back to top
Profile PM
Rudolpho
20
Years of Service
User Offline
Joined: 28th Dec 2005
Location: Sweden
Posted: 21st Nov 2013 23:30
Link
Comments should be removed from the source as a precompilation step.
I seem to recall this was probably not the first precompilation step a few years back however, which caused problems as if you typed 'as' in a comment, that would be matched against the next identifier and yield errors like 'xxx' is not a type. I believe this has since been fixed however; you really should assume comments are simply removed from the source alltogether before any actual compilation / substitutions are made.


"Why do programmers get Halloween and Christmas mixed up?"
Back to top
Profile PM Email Website
mr Handy
18
Years of Service
User Offline
Joined: 7th Sep 2007
Location: out of TGC
Posted: 22nd Nov 2013 11:56
Link
Thanks!

Back to top
Profile PM
Back to top

Login to post a reply

Server time is: 2026-07-07 08:11:34
Your offset time is: 2026-07-07 08:11:34