Author Topic: Various Questions About Zbedic  (Read 26115 times)

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #15 on: December 20, 2005, 05:01:29 pm »
Quote
I can already answer some of my own questions.
I report my findings here, in case any one would be interested, though I'm sure most of you know it already, there may be newbies (as I am) interested.

First of all, the ZBedict home page:
http://bedic.sourceforge.net/index.html

To make a dictionary:
Very good explanations are given here, with step by step at the end of the page:
http://cvs.sourceforge.net/viewcvs.py/bedi...1.5&view=markup

The feature to edit an existing dictionary will be developped int the coming future.
See news:
http://bedic.sourceforge.net/index.html#news

Now, I need to find out how to create a Japanese dictionary, that seems to be more tricky.

Any help from any one?

Ludo
[div align=\"right\"][{POST_SNAPBACK}][/a][/div]


Hi, ludo!!! So glad you enjoy your 3000!

RAFM made it much easier to create new dictionaries with the mkbedic program.  

First, you need to create a dictionary source file in any text editor (it should be simple txt in unicode, I am not sure if UTF-8 will support Japanese though) using the Zbedic format (the link you provided above).  Please see this example dictionary:
[a href=\"http://cvs.sourceforge.net/viewcvs.py/bedic/libbedic/doc/example.dic?rev=1.2&view=markup]http://cvs.sourceforge.net/viewcvs.py/bedi...1.2&view=markup[/url]

The file may need to contain the char-precendent entry (it is not necessary under some circumstances).  However, with kanji I do not see it is possible.  Please see an example of char-precedence here:
http://cvs.sourceforge.net/viewcvs.py/bedi...1.1&view=markup

Second, you will need to compile mkbedic program on your linux system and see the man page for mkbedic on how to use it.  Mkbedic will tell you if there are some important errors with your dic source file.  Note that mkbedic may miss some problems that you will only discover when some of the entries are not displayed correctly (like, missing closing tags, etc.).

Third, after you run mkbedic on your dictionary source file, you will need to dictzip it.  See http://www.die.net/doc/linux/man/man1/dictzip.1.html for more info.  You will need to install dictzip on your system.  The resulting file will be dic.dz

I will add more details later.

I also believe that there is a way to unpack the old dic.dz files (they used a different form of compression and then modify the contents of the file to add new entries.  I have never done it though.  I always work with my source files and make any changes to the original source files.
« Last Edit: December 20, 2005, 05:01:55 pm by kurochka »
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

ludo

  • Jr. Member
  • **
  • Posts: 53
    • View Profile
Various Questions About Zbedic
« Reply #16 on: December 22, 2005, 10:42:54 pm »
Rafm,

is that possible to create an english-japanese-english dictionary to work on the zbedict? What would be the issue with caracter encoding? From what I understand, ZBEDic accepts only UTF-8 coding, and it seems that lot of Japanese is coded with EUC-JP. What do you think?

Thanks Kurochka.

In the meantime I have been investigating about the EPWING format, which seem dedicated to Japanese. But I come back to Zbedict anyway.

I have no linux machine now, do I need one to run mkbedic, or can I do it on the Z? What Linux distribution should I chose?

As for the japanese, I have seen there is a Japanese- english dictionary available for the Bedic. I am just downloading it now, will install it soon to see how it works.

The japanese english dictionary for the ZBEDic works all right, display is nice. Do you know how is the japanese encoded there?

Ludo
« Last Edit: December 23, 2005, 01:55:27 am by ludo »
C3000 sharp ROM

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #17 on: December 23, 2005, 11:58:41 am »
Quote
Rafm,

is that possible to create an english-japanese-english dictionary to work on the zbedict? What would be the issue with caracter encoding? From what I understand, ZBEDic accepts only UTF-8 coding, and it seems that lot of Japanese is coded with EUC-JP. What do you think?

Thanks Kurochka.

In the meantime I have been investigating about the EPWING format, which seem dedicated to Japanese. But I come back to Zbedict anyway.

I have no linux machine now, do I need one to run mkbedic, or can I do it on the Z? What Linux distribution should I chose?

As for the japanese, I have seen there is a Japanese- english dictionary available for the Bedic. I am just downloading it now, will install it soon to see how it works.

The japanese english dictionary for the ZBEDic works all right, display is nice. Do you know how is the japanese encoded there?

Ludo
[div align=\"right\"][a href=\"index.php?act=findpost&pid=108170\"][{POST_SNAPBACK}][/a][/div]

I don't know much about Japanese.  I know that Zten, Zdic and some other dictionaries (including the Zaurus built in dictionary) that are based on EPWING format are popular among Japanese learners.  Moreover, I understand that there are a bunch of dictionaries available in those formats.

I am not sure about the encoding for Japanese-English zbedic dictionary.  I suspect it is unicode of some sort (UTF-16?).  You may actually try different encodings on a test dictionary and see whether it displays properly using a font that supports that encoding.

mkbedic will compile and run only under Linux, as I understand (there are some special projects that try to port Linux programs to Windows API but I have not tried those).  I have never compiled anything on Zaurus but I've heard it is possible.  Given that you need to jump through hoops to install the development environment and the fact that Zaurus is much slower than a desktop, you are better off compiling dictionaries on a desktop.  If your dictionary is not too big you could make the txt file on the Zaurus first and then mkbedic and dictzip it on a desktop.

You could look at knoppix.net for a simple LiveCD Linux that you don't have to install on your Windows computer.  You just boot your computer from the CD without installation.  When you are done using Linux you are back to your Windows without any modifications.  I hope that knoppix has all the necessary libraries and programs to compile mkbedic.  It must have.

I use Suse Linux 10 and Windows XP on the same system (dual boot).  All Linux flavors can be downloaded for free and burned to a cd or dvd.
« Last Edit: December 23, 2005, 11:59:49 am by kurochka »
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #18 on: December 24, 2005, 09:40:22 am »
Quote
I will keep this thread going by asking other questions and making suggestions and comments:

1. Emphasis Tag. I've noticed that {em} tag is hardly seen on the screen.  I understand it makes letters bold but somehow they are almost indistinguishable from normal letters.  I wonder if {em} could instead make the tagged text a different color to make it stand out.  I am using this tag for showing accent in the words.  BTW, if the full text search is implemented, will the {de} tags within the word break the search?  Any other useful tags that can work for showing word accent (stress)?

2. Ways to Show Word Accent/Stress.  As indicated earlier, I use {em} tags to show word stress in the words in the translation portion.  This seems somewhat awkward now but it works for the translation portion.  How can I show word accent for words for the keywords without ruining the search mechanism?  One solution that might work (I am not sure though) --  I could probably use the Unicode stress symbol (I think there is a special symbol in Unicode for word accent) and put it in the ignore char list.  Will this work?  For example, if the keyword is "a'rmy" (showing stress on A) will the search for "army" locate the keyword if I use the above described approach?

[div align=\"right\"][{POST_SNAPBACK}][/a][/div]

Just several comments to my earlier notes.

Emphasis Tag.   Emphasis {em} tag works.  The problem of it not showing was somehow connected to the size of the font I used.  I decreased the size of the font and the emphasis started to show.

Showing Word Accent  I am still struggling with this.  The proper way would be to use "combining diacritical marks" from Unicode standard (see [a href=\"http://www.unicode.org/charts/PDF/U0300.pdf)]http://www.unicode.org/charts/PDF/U0300.pdf)[/url].  0301 looks like the one I would like to use (however, 02c8 or 02b9 could potentially be also used, I guess).  However, it turns out that a lot of programs cannot combine these marks with other glyphs.  Microsoft Word kind of works but still it is a problem.  I have not tested whether Zbedic supports this because I have hard time creating a test dictionary with these marks.  In the perfect world, these combining diacritical marks would combine with a preceding glyph to form one character for displaying purposes.  I would then only list the main character in the char-precedence while putting the diacritical mark 0301 into ignore list.  This way I could see the stress of all words (not just in transcription) but the word stress would be ingnored for searching purposes.  Does this make sense?  Anyone has any ideas or suggestions?

{ph} tag? I have just noticed a {ph} tag.  I don't see a description of this tag other than that it is somehow used for "description" field.  What is it supposed to do?  What are the suggestions for its use?  If nobody knows for sure, I will try it and report back.

One other question.  Will the old linux bedic program (qbedic?) work with the files created by mkbedic (by ignoring the new features) or will it fail to open the new dictionaries?  Again, I will try to test this.  I feat that it will fail and we will have to wait for a zbedic port to linux.

RAFM, do you have plans for zbedic to go in parallel with TEI XML (making it easy to convert between the two)?  Is TEI XML something worth learning?  

I presume that there could be new tags in the future.  Is it possible for Zbedic to ignore all tags that it does not yet know (for example, {bla}, {/bla})?  This way I could put some placeholding tags into my dictionaries for the future and then one day hope that the new tags will be implemented.  In the meantime, the dictionaries could be used as is.  Mostly, these tags would deal with grammatical categories or wordforms, etc.


Otherwise, I am moving along and getting better with using zbedic format.
« Last Edit: December 29, 2005, 12:39:05 pm by kurochka »
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #19 on: December 29, 2005, 12:43:30 pm »
A couple of suggestions:

1. Internal Comments.  I think it would be great to have a tag to comment out text in the dic file.  This way we could put comments or yet unimplemented text in the dictionaries without it showing.

2. Another tag could be for comments that could be displayed or hidden depending on users' choice (similar to how pronounciation tag works).

These are just ideas.
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

rafm

  • Full Member
  • ***
  • Posts: 145
    • View Profile
Various Questions About Zbedic
« Reply #20 on: January 04, 2006, 03:32:48 pm »
Quote
Rafm,

is that possible to create an english-japanese-english dictionary to work on the zbedict? What would be the issue with caracter encoding? From what I understand, ZBEDic accepts only UTF-8 coding, and it seems that lot of Japanese is coded with EUC-JP. What do you think?

Thanks Kurochka.

In the meantime I have been investigating about the EPWING format, which seem dedicated to Japanese. But I come back to Zbedict anyway.

I have no linux machine now, do I need one to run mkbedic, or can I do it on the Z? What Linux distribution should I chose?

As for the japanese, I have seen there is a Japanese- english dictionary available for the Bedic. I am just downloading it now, will install it soon to see how it works.

The japanese english dictionary for the ZBEDic works all right, display is nice. Do you know how is the japanese encoded there?

Ludo
[div align=\"right\"][a href=\"index.php?act=findpost&pid=108170\"][{POST_SNAPBACK}][/a][/div]


* You need either Linux (gcc 3.x) or cygwin on Windows to compile mkbedic. It is also possible to cross-compile mkbedic for zaurus. I will consider including mkbedic for zaurus in the future release of zbedic.

* All bedic dictionaries must be in utf-8. You can use "iconv" under Linux to convert from whatever encoding to utf-8. I use emacs to edit utf-8 text files.
SL-C1000 w/ Cacko ROM 1.23

rafm

  • Full Member
  • ***
  • Posts: 145
    • View Profile
Various Questions About Zbedic
« Reply #21 on: January 04, 2006, 03:51:36 pm »
Quote
Just several comments to my earlier notes.
Emphasis Tag.   Emphasis {em} tag works.  The problem of it not showing was somehow connected to the size of the font I used.  I decreased the size of the font and the emphasis started to show.
[div align=\"right\"][{POST_SNAPBACK}][/a][/div]

Some fonts or font sizes may be missing an italic, which is used for the emphasis.

Quote
Showing Word Accent  I am still struggling with this.  The proper way would be to use "combining diacritical marks" from Unicode standard (see [a href=\"http://www.unicode.org/charts/PDF/U0300.pdf)]http://www.unicode.org/charts/PDF/U0300.pdf)[/url].  0301 looks like the one I would like to use (however, 02c8 or 02b9 could potentially be also used, I guess).  However, it turns out that a lot of programs cannot combine these marks with other glyphs.  Microsoft Word kind of works but still it is a problem.  I have not tested whether Zbedic supports this because I have hard time creating a test dictionary with these marks.  In the perfect world, these combining diacritical marks would combine with a preceding glyph to form one character for displaying purposes.  I would then only list the main character in the char-precedence while putting the diacritical mark 0301 into ignore list.  This way I could see the stress of all words (not just in transcription) but the word stress would be ingnored for searching purposes.  Does this make sense?  Anyone has any ideas or suggestions?

As far as I know combining diacritical marks does not work under QTopia.

Quote
{ph} tag? I have just noticed a {ph} tag.  I don't see a description of this tag other than that it is somehow used for "description" field.  What is it supposed to do?  What are the suggestions for its use?  If nobody knows for sure, I will try it and report back.

This tag does not  / should not work.  Once I decided that this tag was a bad idea and I removed it. If there is a trace of this tag somewhere in the documentation, I should remove it.

Quote
One other question.  Will the old linux bedic program (qbedic?) work with the files created by mkbedic (by ignoring the new features) or will it fail to open the new dictionaries?  Again, I will try to test this.  I feat that it will fail and we will have to wait for a zbedic port to linux.

If you don't use any new features and your dictionaries are <2GB, files generated with mkbedic should work with qbedic. But I haven't checked this.

Quote
RAFM, do you have plans for zbedic to go in parallel with TEI XML (making it easy to convert between the two)?  Is TEI XML something worth learning? 

Freedict project has a script for converting TEI XML -> bedic.  TEI XML could be very useful if there were more software supporting it.  

Quote
I presume that there could be new tags in the future.  Is it possible for Zbedic to ignore all tags that it does not yet know (for example, {bla}, {/bla})?  This way I could put some placeholding tags into my dictionaries for the future and then one day hope that the new tags will be implemented.  In the meantime, the dictionaries could be used as is.  Mostly, these tags would deal with grammatical categories or wordforms, etc.

Currently zbedic will "ignore" unknown tags by displaying them as they are.
SL-C1000 w/ Cacko ROM 1.23

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #22 on: January 11, 2006, 04:31:11 pm »
Quote
This tag ({ph}) does not  / should not work.  Once I decided that this tag was a bad idea and I removed it. If there is a trace of this tag somewhere in the documentation, I should remove it.

Reference to this tag is in one of the examples in the bedic-format.txt file.

Quote
Currently zbedic will "ignore" unknown tags by displaying them as they are.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=109541\"][{POST_SNAPBACK}][/a][/div]
Understood.  But this would look ugly.  Is it hard to implement hiding unrecognized tags without taking any action?

By the way, I have ended up using non-combining accent grave (can't remember the Unicode code) as the word emphasis.  Now, I can even have word accent for the entry words.  I have put accent grave in the ignore-chars list.

By the way, I wonder how wikipedia for zbedic works.  Especially, I am interested in inserting pictures in the body of translation field.  How can it be done (html?)?  WHere should the pictures be?  Will mkbedic compile this type of dictionaries?

Thanks.
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

rafm

  • Full Member
  • ***
  • Posts: 145
    • View Profile
Various Questions About Zbedic
« Reply #23 on: January 15, 2006, 01:26:27 pm »
Quote
Quote
Currently zbedic will "ignore" unknown tags by displaying them as they are.
[div align=\"right\"][{POST_SNAPBACK}][/a][/div]
Understood.  But this would look ugly.  Is it hard to implement hiding unrecognized tags without taking any action?

Is there  a point of keeping in a dictionary file the information that is never displayed? You can easily  remove unwanted tags with awk/perl script from a source file.

Another important thing about zbedic parser: if parsing fails, zbedic will display the entry without any parsing.

Quote
By the way, I have ended up using non-combining accent grave (can't remember the Unicode code) as the word emphasis.  Now, I can even have word accent for the entry words.  I have put accent grave in the ignore-chars list.

If you give more details how to get those accented characters working under Qtopia, more people could benefit from this.  I could put it somewhere in the documentation.

Quote
By the way, I wonder how wikipedia for zbedic works.  Especially, I am interested in inserting pictures in the body of translation field.  How can it be done (html?)?  WHere should the pictures be?  Will mkbedic compile this type of dictionaries?
[div align=\"right\"][a href=\"index.php?act=findpost&pid=110545\"][{POST_SNAPBACK}][/a][/div]

Wikipedia is formated as HTML. mkbedic does not check syntax, so there is no problem if entries are html without any proper zbedic syntax. Images should be theoretically possible, but I have not checked where the files should be located.

I though about an extension for an efficient storage of images in zbedic dictionary files. For example, images could be wavelet compressed ([a href=\"http://www.lizardtech.com/download/]dejavu[/url] compression?)  with lower bit-depth as zaurus screen can handle 6 bits per r,g,b anyway.  Question is whether this is worth the effort.
SL-C1000 w/ Cacko ROM 1.23

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #24 on: April 10, 2006, 01:06:59 pm »
Quote
I though about an extension for an efficient storage of images in zbedic dictionary files. For example, images could be wavelet compressed (dejavu compression?)  with lower bit-depth as zaurus screen can handle 6 bits per r,g,b anyway.  Question is whether this is worth the effort.


Having this functionality (having images in the dictionaries themselves) would allow creation of encyclopedias for zbedic.  In addition, images could be used for inserting information in the form not supported by bedic format (tables, etc.).  The question is how hard is it to implement?  Is it worth the effort?

One of these days I will gather my tips about creating bedic dictionaries (e.g., word stress, etc.) and put them in this thread.
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

ShiroiKuma

  • Hero Member
  • *****
  • Posts: 900
    • View Profile
Various Questions About Zbedic
« Reply #25 on: April 10, 2006, 02:20:16 pm »
kurochka, I'm trying to make a simplified dictionary too. However I can't seem to compile the mkbedic.

Could you send me or post your compiled mkbedic binary?
[span style=\'font-size:8pt;line-height:100%\']Das ganze tschechische Volk ist eine Simulantenbande.[/font][/span]
Militäroberarzt Bautze

ludo

  • Jr. Member
  • **
  • Posts: 53
    • View Profile
Various Questions About Zbedic
« Reply #26 on: April 11, 2006, 10:19:13 pm »
Hi

I'm interested also in putting images in a dictionary. That would be a great thing to do. Worth the effort? I don't know how difficult it would be, but worth the result yes for sure!

Ludo
C3000 sharp ROM

kurochka

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Various Questions About Zbedic
« Reply #27 on: April 12, 2006, 05:15:15 pm »
Quote
kurochka, I'm trying to make a simplified dictionary too. However I can't seem to compile the mkbedic.

Could you send me or post your compiled mkbedic binary?
[div align=\"right\"][a href=\"index.php?act=findpost&pid=122544\"][{POST_SNAPBACK}][/a][/div]

Will do.  It may take a while though.  Have you indicated in another thread that you may have Korean dics.  If so could you share them with me?
« Last Edit: April 12, 2006, 05:16:28 pm by kurochka »
SL-C3100 (from PriceJapan.com): modified Sharp Rom (couldn't make Japanese input work in Cacko Rom)

ex-SL-C3000; ex-SL-5600; ex-Simpad

keisangi

  • Newbie
  • *
  • Posts: 3
    • View Profile
Various Questions About Zbedic
« Reply #28 on: August 28, 2006, 02:40:20 pm »
Quote
Quote
kurochka, I'm trying to make a simplified dictionary too. However I can't seem to compile the mkbedic.

Could you send me or post your compiled mkbedic binary?
[div align=\"right\"][{POST_SNAPBACK}][/a][/div]

Will do.  It may take a while though.  Have you indicated in another thread that you may have Korean dics.  If so could you share them with me?
[div align=\"right\"][a href=\"index.php?act=findpost&pid=122920\"][{POST_SNAPBACK}][/a][/div]


i'm interested in the korean dictionary files too..
i searched a lot but couldn't find anything for the zaurus, except a rather limited one,
here: [a href=\"http://dbrechalov.narod.ru/zaurus/koen.dic.dz]http://dbrechalov.narod.ru/zaurus/koen.dic.dz[/url]

can you share the korean dictionary files pretty please

rafm

  • Full Member
  • ***
  • Posts: 145
    • View Profile
Various Questions About Zbedic
« Reply #29 on: August 28, 2006, 02:54:49 pm »
Quote
Having this functionality (having images in the dictionaries themselves) would allow creation of encyclopedias for zbedic.  In addition, images could be used for inserting information in the form not supported by bedic format (tables, etc.).  The question is how hard is it to implement?  Is it worth the effort?
[div align=\"right\"][a href=\"index.php?act=findpost&pid=122536\"][{POST_SNAPBACK}][/a][/div]

Tables can be put in dictionaries using HTML syntax. I works fine unless the table is too large for the small screen.

zbedic can handle images starting from v1.1. Can be both .jpg and .png. To get better compression, one can experiment with reduction of bit-depth and scaling. Linux 'convert' command from ImageMagick can be useful for these operations.
SL-C1000 w/ Cacko ROM 1.23