New XML <-> DTM tool (dtmdup update)

[dhns]

I have updated and extended the GPLed tool "dtmdump" to

a) fix bugs
 make XML output more compliant
c) add a "write" mode to parse XML and update the DTM - DANGEROUS!

Download link for sources (compiled on a Mac with Zaurus-X-gcc) and binary:

http://www.dsitri.de/download/dtmdump-src.tgz

To install:

1. make sure you have Sharp ROM with DTM database!
2. unpack on your host
3. transfer the executable 'dtmdump' to /home/QtPalmtop/bin
4. do chmod +x dtmdump
5. use AT YOUR OWN RISK!

Command description:

dtmdump AddressBook     read address book
dtmdump ToDo                 read ToDo list
dtmdump Calendar           read Calendar
dtmdump Memo               read Notes (from Text Editor)

Files created with these commands can be read back (do to some sort of backup), e.g.:
dtmdump AddressBook >addresses.xml
dtmdump -w AddressBook <addresses.xml

There is a special code <delete/> to delete entries:

e.g.

Code: [Select]

# dtmdump -w Todo

ctl-D
NOTE: We have just completed the code. So, this command may wipe out all your calendars, address book, to do list, mail and notes entries. So, make a backup before experimenting!

Anyway - if you find bugs, please report.

How to compile
You need a Mac and Zaurus-X-gcc or you have to modify the Makefile.

The Makefile assumes that you have the Zaurus online (and enabled for ssh access). It automatically fetches the libzdtm.so to link against and automatically installs the updated binary on the Zaurus. This speeds up development the cycle dramatically. (Edit in Xcode, Start Makefile, test in ssh commandline on Zaurus).

-- hns
ZMacSync project
http://www.dsitri.de/wiki.php?page=ZMacSync

[dhns]

Dear all,
there is a new version for download: http://www.dsitri.de/download/dtmdump-src.tgz

It is a little more robust, fixes a bug with short vs. long integers and is now capable of having several tables within a single XML document. This allows for sort of complete backup:

Code: [Select]

dtmdump -a >Backup.xml
dtmdump -w should save and restore some (not all) tables.

Here some README:

DTMDUMP

This is a tool to read (dump) and write the Sharp Zaurus DTM Database.

Usage:

dtmdump [-w] [-a] table | -f path.box ... [<input.xml]

-w         read XML from stdin (or file input.xml) and update database before dumping
-a         print all data (ToDo, Calendar, Addressbook, Memo)
table      the database table(s) to read
   ToDo
   Calendar
   Addressbook
   Memo
   Category
   Inbox
   Outbox
-f path.box   file path to a .box file
...         you can list more than one table or box file for reading
         at least one specification is required unless -w or -a is present

XML format:

<Table>                  select database table (Table to replaced by keyword)
  <Entry card=123>         select card (Entry and 123 to replaced by keyword and card number - counting from 0)
     <ATTR>value</ATTR>      change value (ATTR and calue to replaced by 4-character keyword and new value)
     <delete/>               delete current card
  </Entry>
</Table>

NOTE: This is alpha-ware. Use a separate tool to backup all your PIM data as you may loose it. We take no responsibility for any data loss.

This software is based on dtmdump written by S.C. Kremer <stefan@kremer.ca> as described at
http://216.239.57.104/search?q=cache:H1aFQ...m_howto+dtmdump or
http://www.zaurususergroup.com/modules.php...=DTM%20database

This software is using the GPL.

-- hns
ZMacSync project
http://www.dsitri.de/wiki.php?page=ZMacSync

[dhns]

And again an update. No change in functionality:

- just renamed to dtm2xml
- packed into an ipk
- assigned version number 1.0

download links now:
* ipk: http://www.dsitri.de/download/dtm2xml-1.0_arm.ipk
* src: http://www.dsitri.de/download/dtm2xml-1.0-src.tgz

Please test and post/send feedbacks.

-- hns

[acpkendo]

Hi hns,

I've been playing around with dtm2xml- thanks for putting work into this program.  Overall it seems to work pretty well, although I do get a lot of errors like the following:

Code: [Select]

CardRead("/home/zaurus/Applications/dtm/SLDATE.box", CARDID=16) failed: Lack of Memory
This is confusing- according to System Info I have 860kb free, and 15544kb cached. . . which I would think is plenty to run this little utility.

On another note, I notice that for ZMacSync you're using the zdbat utility.  Is there any difference in performance?  I've been playing around with the idea of making an app that would collect links of Datebook, Addressbook, ToDo, and Memo entries into "projects". . . this would require a read-only list of all the cards in the DTM database, so I'm wondering if one or the other of these is faster.

Thanks again for your work. . .

[dhns]

although I do get a lot of errors like the following:

Code: [Select]

CardRead("/home/zaurus/Applications/dtm/SLDATE.box", CARDID=16) failed: Lack of MemoryThis is confusing- according to System Info I have 860kb free, and 15544kb cached. . . which I would think is plenty to run this little utility.

It might come from a very long entry in the database. Currently, the internal buffer is 1024 bytes into which records are copied. Or it might be a bug of the 1.0 version. I have just copied a new 1.1 version on the server:

http://www.dsitri.de/download/dtm2xml-1.1_arm.ipk for installing the tool (only)
http://www.dsitri.de/download/dtm2xml-1.1-src.tgz for the full sources

If the message persists, I can try to increase the buffer size.

On another note, I notice that for ZMacSync you're using the zdbat utility.  Is there any difference in performance?  I've been playing around with the idea of making an app that would collect links of Datebook, Addressbook, ToDo, and Memo entries into "projects". . . this would require a read-only list of all the cards in the DTM database, so I'm wondering if one or the other of these is faster.

I have not made a real performance comparison but I think both have approx. the same speed. Reading entries is done in both by the Sharp DTM libraries. And encoding the cards to stdout is just a printf(). If you take the transfer of the created output file into account, XML output might be some % larger than the Comma Separated List of zdbat.
A further difference is that zdbat handles the Category database internally, i.e. by converting an Addressbook record to the proper CATEGORY list. dtm2xml (currently) does not link both tables.
The reason to write our own utility for ZMacSync was that zdbat does not reveal all internal fields (e.g. CTTM, SYID are missing). So I would recommend to base the decision between both tools for your project to base on the data format you want to get.

Hope this helps,
hns

[acpkendo]

It might come from a very long entry in the database. Currently, the internal buffer is 1024 bytes into which records are copied.

This does help, and is probably the cause--I have some very long text notes and calendar entries (sometimes I put directions to a location in the memo field).  I will try out the new v.1.1 and let you know how it goes. . .

[acpkendo]

hns,

I still get the memory errors, although I believe it is on different (i.e. only the very longest) cards.  Is there any way to introduce a user-configurable buffer length?  Or remove the limit entirely (although this could conceivable cause system problems, it doesn't seem likely that someone would store enough data on a DTM card to eat up all the system memory).

[dhns]

Is there any way to introduce a user-configurable buffer length?  Or remove the limit entirely (although this could conceivable cause system problems, it doesn't seem likely that someone would store enough data on a DTM card to eat up all the system memory).

Ok,
I will enlarge the buffer - or even do that dynamically each time this error occurs (internally). I am not a fan of making such things user-configurable.
And I think there is a limit given by the DTM itself - so that it will not eat up all system memory...

[acpkendo]

Using the limit set by DTM (if it exists) seems like an excellent idea--it should eliminate all problems.

Thank you for your work in producing this program?  Are you planning on adding any additional features?  I have a couple things that I would like to see. . . the ability to generate some sort of index (or read the .IDX index files) would be nice.

[dhns]

Finally,
a new release is out. Called 1.2.

It brings some new features:
* buffer size increased to 10k (the Sharp DTM spec is a little unclear if that is enough now. DTM seems to support up to 249k - but only if the database is manipulated before writing long records).
* new 'reset database' command that installs a fresh dtm directory (in case yours got corrupt)
* fixed some bugs with card="0"
* added default values for CTTM, MDTM, SYID so you can add them (but needn't do so)
* works as daemon for ZMacSync

Here the download links:

http://www.dsitri.de/download/dtm2xml-1.2_arm.ipk for installing the tool (only)
http://www.dsitri.de/download/dtm2xml-1.2-src.tgz for the full sources

Using the limit set by DTM (if it exists) seems like an excellent idea--it should eliminate all problems.

Please report if you still have issues with buffer size.

Are you planning on adding any additional features?  I have a couple things that I would like to see. . . the ability to generate some sort of index (or read the .IDX index files) would be nice.

Currently, the focus is to use it as a daemon for the long awaited ZMacSync tool - syncing right away with the MacOS X Addressbook, iCal, Stickies etc. And for this porpose it does not need additional features (it is mostly working besides 3 or 4 blocking bugs) - except improved handling of CATEGORY.

Indexes are apparently managed well enough by the Qt PIM applications so everything is transparent (unless you write or read records while one of the applications is open and that may desynchronize the index and corrupt the database).

But if you want to add features yourself, the source code is available for free (see links above).

Please report bugs and irregularities.

-- hns
http://www.dsitri.de/wiki.php?page=ZMacSync

[dhns]

Just a note to all:

the source and the ipk is now distributed as part of ZMacSync (it is a .tgz so that you can load it on any system) which you can find at

http://www.dsitri.de/wiki.php?page=ZMacSync

-- hns