Author Topic: Zaurus And Unicode Characters.  (Read 3291 times)

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« on: June 01, 2006, 05:24:31 pm »
I am having a problem with displaying unicode characters on my zaurus with the japanese rom(and cacko as well).  I have a lot of Japanese music that just shows up as garbage on the zaurus.  In fact, it seems not limited to Japanese music.  It seems that any music that I have modified tags of will show like this on my zaurus:

[img]http://radnimax.com/misc/scrn003.jpg\" border=\"0\" class=\"linked-image\" /]
[img]http://radnimax.com/misc/scrn004.jpg\" border=\"0\" class=\"linked-image\" /]

Even in the music player it shows like that.  It reads the ID tags the same, garbage.


But they still show up just fine on my desktop while the zaurus is connected:

[img]http://radnimax.com/misc/music.jpg\" border=\"0\" class=\"linked-image\" /]
[img]http://radnimax.com/misc/music2.jpg\" border=\"0\" class=\"linked-image\" /]

Anyone have any ideas?  The following is the locale I am using on my desktop pc:
Code: [Select]
lain Music_Files # locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I'm not sure how to check what the zaurus uses as it has no "locale" command, but I thought I saw something that had "utf8" in it on boot up.
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« Reply #1 on: June 01, 2006, 06:19:03 pm »
hmm, I created a folder on the zaurus with a japanese file name.  Then I connected it to my PC and the folder shows up as all "????" marks.  So I guess obviously the locale used betreen the two computers is different.  But what does the Zaurus use?
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« Reply #2 on: June 01, 2006, 06:55:04 pm »
Now, the stuff that shows as garbage on the zaurus also shows as garbage when it is connected in windows.  And that folder I created on the zaurus also shows just fine in windows.  So I guess I need to know what locale windows (and the zaurus) uses and set my Linux to match.  Any ideas?  The reason I used utf8 in Linux was because I understood it to be the most supported.

Or maybe I don't understand how it works.
« Last Edit: June 01, 2006, 06:55:24 pm by xamindar »
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

ShiroiKuma

  • Hero Member
  • *****
  • Posts: 900
    • View Profile
Zaurus And Unicode Characters.
« Reply #3 on: June 02, 2006, 03:56:04 am »
Quote
Now, the stuff that shows as garbage on the zaurus also shows as garbage when it is connected in windows.  And that folder I created on the zaurus also shows just fine in windows.  So I guess I need to know what locale windows (and the zaurus) uses and set my Linux to match.  Any ideas?  The reason I used utf8 in Linux was because I understood it to be the most supported.

Or maybe I don't understand how it works.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=129373\"][{POST_SNAPBACK}][/a][/div]
Well, this is the punishment for listening to Morning Musume, 'nuff said  

Seriously though:

I'd bet that the disk is not mounted as UTF-8, locale has no effect on this.

Check /etc/fstab and make sure that the option to mount the disk where you have the files is UTF-8. It works, you'll then see the proper Japanese names.
[span style=\'font-size:8pt;line-height:100%\']Das ganze tschechische Volk ist eine Simulantenbande.[/font][/span]
Militäroberarzt Bautze

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« Reply #4 on: June 02, 2006, 04:42:47 am »
are you talking about the /etc/fstab on the zaurus?  Wouldn't it allready be set up sense it is from Japan?

I tried remounting it on my linux box but I don't see any change.  I typed this:
# mount -o remount utf8 /media/disk-2
and get no errors or anything.  Mount just reports:
/dev/sda on /media/disk-2 type vfat (rw)

Should it say utf-8 as well?  Funny thing is, if I create a japanese filename with the zaurus that file will show up as "?????" when viewed on my linux machine.  It will show just fine in windows.  So basicly:

Code: [Select]
File created with:          shows correct on:        mojibake(garbage):
zaurus                      zaurus, windows          linux
linux                       linux                    windows, zaurus
windows                     windows, zaurus          linux

So it must be something wrong with linux.  Gah!
Linux is using the nls_cp932 which I thought was the standard.  There is also nls_euc_jp but I don't know what they really mean.  I'm so confused.

I'm also trying to get help here:  http://forums.gentoo.org/viewtopic-p-3354140.html#3354140
But it isn't looking so good.
« Last Edit: June 02, 2006, 04:54:30 am by xamindar »
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

ShiroiKuma

  • Hero Member
  • *****
  • Posts: 900
    • View Profile
Zaurus And Unicode Characters.
« Reply #5 on: June 02, 2006, 04:52:32 am »
Quote
are you talking about the /etc/fstab on the zaurus?  Wouldn't it allready be set up sense it is from Japan?
Yeah. And it's not setup correctly. In Japan, they mostly use euc-JP for Linux Japanese, which is a problem, Windows can't work with it.

Unless, you specify UTF-8 everywhere it won't be mounted correctly. I've gone that route too. Once you do and reboot, then you'll be able to connect to a Win machine and copy back Japanese files and dirs no prob.
[span style=\'font-size:8pt;line-height:100%\']Das ganze tschechische Volk ist eine Simulantenbande.[/font][/span]
Militäroberarzt Bautze

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« Reply #6 on: June 02, 2006, 05:01:46 am »
Quote
I've gone that route too. Once you do and reboot, then you'll be able to connect to a Win machine and copy back Japanese files and dirs no prob.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=129422\"][{POST_SNAPBACK}][/a][/div]
But it works fine between windows already.  If I copy japanese files from windows to the zaurus everything is fine.  The problem is with my Linux machine.  It seems only the files made on my linux machine show as garbage in windows and on the zaurus.

I might as well give it a shot though.  Is it just "utf8" that I add into fstab?

EDIT:  The zaurus hard drive isn't even listed in fstab.  It is probably being mounted by a script somewhere.
« Last Edit: June 02, 2006, 05:04:20 am by xamindar »
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

Stubear

  • Hero Member
  • *****
  • Posts: 1164
    • View Profile
    • http://
Zaurus And Unicode Characters.
« Reply #7 on: June 03, 2006, 03:42:40 am »
When you "connect to" your PC, how are you doing so?

Are you using the "USB storage" setting or one of the PC link settings?

If you are using USB - TCP/IP (possibly USB - IO too but have never tried) then the zaurus is sharing it's drives via SAMBA. You will need to check the settings for "Character Set" and "Client Code Page" (in Cacko 1.23 they are set 866 - Cyrillic) in /home/etc/samba/smb.conf

With regards to Japanese showing as mojibake in the console, try changing the fonts (There is a Font menu in QKonsole) by default it was choosing VGA when I first installed it, I find that Unifont or LCFont are better for viewing Japanese.

Edit: You may find this document helpful. I haven't tried it yet but will later and report back

 [ Invalid Attachment ]


Hope this helps some

stu
« Last Edit: June 03, 2006, 03:54:28 am by Stubear »
SL-C1000, Hand converted to English with Japanese Input
Running X apps via X/Qt
iRiver USB host cable; Diatec P-Cord usb power cable (extendable); Acro's Reel Cable USB (A to A, B, Mini-B,  & Mini-B 8pin); GreenHouse 1Gb PicoDrive+; 2x256Mb Hagiwara SD cards; 128Mb Transcend CF card; 512Mb PQI CF card; AmbiCom WL1100C-CF 11B WLAN card

Meanie

  • Hero Member
  • *****
  • Posts: 2803
    • View Profile
    • http://www.users.on.net/~hluc/myZaurus/
Zaurus And Unicode Characters.
« Reply #8 on: June 03, 2006, 04:04:35 am »
Quote
Well, this is the punishment for listening to Morning Musume, 'nuff said 

how dare you !!! i listen to Morning Musume all the time

anyway, as stu mentions above, yu prolly need to install a samba codepage before you can do the mod to the samba conf file mentioned.
« Last Edit: June 03, 2006, 04:06:35 am by Meanie »
SL-C3000 - pdaXii13 build5.4.9 (based on pdaXrom beta3) / SL-C3100 - Sharp ROM 1.02 JP (heavily customised)
Netgear MA701 CF, SanDisk ConnectPlus CF, Socket Bluetooth CF, 4GB Kingston CF,  4GB pqi SD, 4GB ChoiceOnly SD, 2GB SanDisk SD USB Plus, 1GB SanDisk USB Plus, 1GB Transcend SD, 2GB SanDisk MicroSD with SD adaptor, Piel Frama Leather Case, GoldX 5-in-1 USB cable, USB hub, USB mouse, USB keyboard, USB ethernet, USB HDD, many other USB accessories...
(Zaurus SL-C3000 owner since March 14. 2005, Zaurus SL-C3100 owner since September 21. 2005)
http://members.iinet.net.au/~wyso/myZaurus - zBook3K

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« Reply #9 on: June 03, 2006, 04:13:44 am »
It's not connected as samba, it is connected as a usb storage device.  I'll try changing the font in qconsole though.
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

xamindar

  • Hero Member
  • *****
  • Posts: 803
    • View Profile
    • http://www.radnimax.com
Zaurus And Unicode Characters.
« Reply #10 on: June 03, 2006, 05:50:34 pm »
Quote
Quote
are you talking about the /etc/fstab on the zaurus?  Wouldn't it allready be set up sense it is from Japan?
Yeah. And it's not setup correctly. In Japan, they mostly use euc-JP for Linux Japanese, which is a problem, Windows can't work with it.

Unless, you specify UTF-8 everywhere it won't be mounted correctly. I've gone that route too. Once you do and reboot, then you'll be able to connect to a Win machine and copy back Japanese files and dirs no prob.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=129422\"][{POST_SNAPBACK}][/a][/div]


OK, you were right.  I finally got it to mount in Linux with utf-8.  It wasn't listed in the man page for mount but all I needed was "utf8=true" and it worked.  Now I can copy files over to the zaurus and they will show correctly in the *FILES TAB*.  But they still show all goofy in Music Player.  One or two files will show correctly, and I found that those files are ones that I had not modified ID tags for through Amarok (on my linux machine).  I tested it, I took one that showed correctly on the zaurus and opened it up in amarok, changed a character in it's tag, then saved it.  Sure enough, when I copied it over to the zaurus it saw all messed up in the music player.

Man, talk about neverending problems.  Do I need to mount the drive on the zaurus using utf or something?  But would that affect the ID tags of my mp3s?

Also, the qconsole font is already set to lcfont.  Every font setting shows the same garbage in the console.
« Last Edit: June 03, 2006, 05:53:23 pm by xamindar »
SL-C3100 happily Dualbooting Japanese Rom 1.02 and Debian Eabi
Replaced internal CF with 8gb seagate cf hard drive
Ambicom CF GPS
CyberPower battery powered USB hub
D-link DCF-650W (MAN THIS THING IS HUGE!!)

Meanie

  • Hero Member
  • *****
  • Posts: 2803
    • View Profile
    • http://www.users.on.net/~hluc/myZaurus/
Zaurus And Unicode Characters.
« Reply #11 on: June 03, 2006, 08:18:24 pm »
Quote
Quote
Quote
are you talking about the /etc/fstab on the zaurus?  Wouldn't it allready be set up sense it is from Japan?
Yeah. And it's not setup correctly. In Japan, they mostly use euc-JP for Linux Japanese, which is a problem, Windows can't work with it.

Unless, you specify UTF-8 everywhere it won't be mounted correctly. I've gone that route too. Once you do and reboot, then you'll be able to connect to a Win machine and copy back Japanese files and dirs no prob.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=129422\"][{POST_SNAPBACK}][/a][/div]


OK, you were right.  I finally got it to mount in Linux with utf-8.  It wasn't listed in the man page for mount but all I needed was "utf8=true" and it worked.  Now I can copy files over to the zaurus and they will show correctly in the *FILES TAB*.  But they still show all goofy in Music Player.  One or two files will show correctly, and I found that those files are ones that I had not modified ID tags for through Amarok (on my linux machine).  I tested it, I took one that showed correctly on the zaurus and opened it up in amarok, changed a character in it's tag, then saved it.  Sure enough, when I copied it over to the zaurus it saw all messed up in the music player.

Man, talk about neverending problems.  Do I need to mount the drive on the zaurus using utf or something?  But would that affect the ID tags of my mp3s?

Also, the qconsole font is already set to lcfont.  Every font setting shows the same garbage in the console.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=129652\"][{POST_SNAPBACK}][/a][/div]

this is now another problem.
probably your id tag editor is using another format to the musicplayer which is very likely to be eucjp and it was using shiftjis..
SL-C3000 - pdaXii13 build5.4.9 (based on pdaXrom beta3) / SL-C3100 - Sharp ROM 1.02 JP (heavily customised)
Netgear MA701 CF, SanDisk ConnectPlus CF, Socket Bluetooth CF, 4GB Kingston CF,  4GB pqi SD, 4GB ChoiceOnly SD, 2GB SanDisk SD USB Plus, 1GB SanDisk USB Plus, 1GB Transcend SD, 2GB SanDisk MicroSD with SD adaptor, Piel Frama Leather Case, GoldX 5-in-1 USB cable, USB hub, USB mouse, USB keyboard, USB ethernet, USB HDD, many other USB accessories...
(Zaurus SL-C3000 owner since March 14. 2005, Zaurus SL-C3100 owner since September 21. 2005)
http://members.iinet.net.au/~wyso/myZaurus - zBook3K

Snappy

  • Hero Member
  • *****
  • Posts: 793
    • View Profile
    • http://
Zaurus And Unicode Characters.
« Reply #12 on: June 13, 2006, 03:02:44 am »
I just tried out Samba for accessing my akita from WindowsXP and it works once I got it setup!

I am however, noticing the garbled characters when I look at the directory list of files (CF) from the Akita on my notebook.

Important point is that the same files on the CF appears right (Chinese characters) when I connect via usb using storage(CF) option instead of USB-TCPIP (advanced) option. I checked etc/hotplug/usbdstorage.agent and found it mounting the CF as UTF8 for iocharset.

Changing fstab to mount CF as UTF8 and smb.conf to specify UTF8 do not seem to change anything. What could be wrong?
Snappy!
------------------------
Akita (Daily use)
<span style='font-size:8pt;line-height:100%'>SL-C1000 with Cacko 1.23 full
Accessories: Wifi XI-825 CF  |  16GB (6) SDHC (Transcend) | 1GB 80x CF (RiDATA PRO-2) </span>

Collie (Sandbox)
<span style='font-size:8pt;line-height:100%'>SL-5500 with OZ/GPE 3.5.4.1 alpha3 build 2006-04-27
Accessories: 512MB A-Data SD | 64MB Toshiba SD</span>