Help - Search - Members - Calendar
Full Version: Creating Python 2.4 Zaurus Image
OESF Forums > Archived Forums > Distros, Development, and Model Specific Forums > Zaurus - Everything Development > Python
kopsis
I decided the shiny new Cacko 1.22 ROM on my Z needed a shiny new version of Python to go with it, so I'm building up a new Zaurus Python image with Python 2.4. I have the base Python image built and working and it includes all the "standard" Python modules (with the exception of crypt and nis for which I'm still missing libraries) including distutils, encodings, curses, readline, xml, etc.

In addition to the ext2 format, I'll also be releasing this new Python image as a cramfs for folks that want to run off SD (or internal flash) without chewing up so much space. Its not easy for folks to add stuff to the cramfs image on their own, so before I package it up, I'd like to find out if there are any "site-packages" that people think are critical to doing Python on the Z. The obvious ones (to me) are PyQT and pysqlite so I'll be including those. If there's anything else you'd like to see (that won't bloat things up too much), now's the time to ask.
LarsAC
Would the mySQL API for Python fit into this, too?

I have a number of apps at home for which I might want to write a dedicated client for the Z in favor of porting stuff amde for large screens...

Lars
fpp
Does this mean than when using the cramfs version, we would be unable to add packages that install themselves to site-packages in the standard distutil way :
python setup.py --install
... or am I misunderstanding ?
(this is not a problem for me, I'm comfortable with the ext2 image, but I'm not familiar with these fs tricks, so it's just to make sure...)

On a related topic, maybe you've seen this 'Movable Python' project that appeared recently :
http://sourceforge.net/projects/movpy
http://www.voidspace.org.uk/python/movpy

Do you think it might be a useful option for the Z (without the wxPython part obviously), or would it be no smaller than what you've achieved ?
kopsis
QUOTE(LarsAC @ Jan 7 2005, 04:33 AM)
Would the mySQL API for Python fit into this, too?

Perhaps. The source code looks pretty small, I just have to see if I can get it to build.
kopsis
QUOTE(fpp @ Jan 7 2005, 07:00 AM)
Does this mean than when using the cramfs version, we would be unable to add packages that install themselves to site-packages in the standard distutil way


That is correct. A cramfs filesystem is, by nature, "read only". The only way to make changes is mount it, make a complete copy to a writeable filesystem, make the changes there, then use mkcramfs to build a new image from the copy in the writeable filesystem.

I suspect that Python developers will much prefer the ext2 filesystem image, despite the much larger size. With 1GB SD cards going for under $70 these days, 50MB seems a small price to pay smile.gif

QUOTE
On a related topic, maybe you've seen this 'Movable Python' project that appeared recently


I took a look and it's basically trying to create Python filesystem images for Windows platforms. A standard Windows install doesn't have the luxury of things like loop mounting and cramfs drivers so they have to use a different (and more complex) approach. The images they're building are comparable in size to a cramfs image (and are also read-only), so there's little to be gained by going down that path.
clofland
Sounds great! I've been waiting for 2.4 for the Z before I start using it on my Windows box.

As long as it supports sqlite 3, I'll be happy.

I'll probably use the cramfs version until I run into something I can't do, and then I'll decide how badly I want to switch. smile.gif

One thing I do notice is that every time I start up my Z, it says that the Python file system was not cleanly unmounted. It even does that if I manually stop and start it. I wonder if the script isn't quite right? It doesn't seem to cause any problems though, other than forcing a file system check each time.
fpp
Thanks for both clarifications kopsis !

I'll wait till the new 2.4 ext2 image comes out, and probably the first cacko 1.22 hotfix too, then it'll be flashing time :-)
clofland
One quick add: Isn't squashfs "better" than cramfs? My understanding is that Cacko ROM 1.22 uses squashfs instead of cramfs, because it makes a smaller faster image.
kopsis
squashfs is a bit better than cramfs, but that would force users to install drivers (which may not even be available for all ROMs). cramfs will work everywhere, and it's a pretty trivial matter for those who want squashfs to turn the cramfs image into a squashfs image (I'll probably throw a short howto on the website).
gasolin
Dear kopsis:

Very appreciate for your python image release.

As I saw this post, I'd suggest to make "palmdb" module preinstalled to read/write/modify Palm PDB and PRC format databases with python.

This module is maintained by rick_price, one of PDesk(the Palm outliner "progect"'s desktop program) developers, and avaliable at sourcefotge
http://sourceforge.net/projects/pythonpalmdb/


The second is cjkcodecs, which contained in python 2.4. To support Chinese, Japenese, and Korean characters. This can make your MP3player display CJK characters in Tags.

doctest (and unittest(opt.)) is one of essential model, too.
http://aspn.activestate.com/ASPN/Cookbook/...n/Recipe/305292
I think this module is helpful for programers to make unit test while writing documents.

And how about add "twisted" module to ease network-enabled program develop curve?

regards
kopsis
The palmdb module sounds like a great idea! I'll look into that one.

The CJK codecs from the standard 2.4 distribution are definitely included, as are doctest and unittest smile.gif I'll have to take a look at the "twisted" stuff. If it's small enough and builds easily, I'll throw it in.

Database wise, pysqlite for sqlite3 is in there. Hopefully there won't be too many folks still wanting to use sqlite2 database files.

I've also built and tested MySQL-python 1.1.8 for MySQL 3.23.49 (which seems to be the latest Zaurus IPK around). It's not perfect ... the client libs default to the wrong directory for the localhost socket and choke if you force them to read my.cnf. But those problems are easy enough to work around with the right "connect" parameters. I'll be including the MySQL stuff for expert users who need to work with MySQL databases, but you're on your own when it comes to getting MySQL running on the Z. If you don't know why you would want to choose MySQL over sqlite, then you probably shouldn't smile.gif

The cramfs image is currently running 8.3MiB and the ext2 version will weigh in at around 35MiB (25MiB + 10MiB free space). That's with the .pyo files stripped out (I don't think too many people use the optimize switch) but all the .py and .pyc files retained. With a little luck I'll be able to release in a couple weeks.
Bertram Wooster
dear kopsis, thanks for all your work!
do you also plan to release a non-image version, which does not necessarily have to stay on sd/cf but can be installed/copied to harddisk (on a c3000) directly?
kopsis
QUOTE(Bertram Wooster @ Jan 11 2005, 06:55 AM)
do you also plan to release a non-image version, which does not necessarily have to stay on sd/cf but can be installed/copied to harddisk (on a c3000) directly?

For all practical purposes, the HD in the SL-3000 is a CF card, so the ext2 image should work from it. In fact, my understanding is that the main storage partition on the internal HD is formated FAT (which doesn't allow symlinks or file permissions) so you'd have to use the ext2 image rather than installing the files directly.

The only thing that may not work is my init scripts to search for the image. That should be easy enough to fix, but I'd need someone with a 3000 to send me the output from running the mount command (as root and without any arguments).

Now, if someone does want to install Python to an ext2/ext3 formatted filesystem (internal memory or an ext2 formatted CF/SD card), it's pretty easy. Just mount the image, copy everything over to the target directory, and then make the appropriate symlinks. If there's demand, I can write up semi-detailed instructions.
Bertram Wooster
it's definitely worth a try, though i'd be thankful for your "semi-detailed instructions"- i'm not new to computers however to linux and especially unix commands are quite unknown to me.
LarsAC
QUOTE(kopsis @ Jan 11 2005, 12:04 AM)
But those problems are easy enough to work around with the right "connect" parameters. I'll be including the MySQL stuff for expert users who need to work with MySQL databases, but you're on your own when it comes to getting MySQL running on the Z.


Great stuff, I'm really looking forward towards the release.

Regarding MySQL: I'm rather intending to use a database server on the network rather than a MySQL instance running on the Z. Hopefully this is going to work without problems...

Lars
kopsis
Connections to a remote MySQL database should work just fine as long as you specify things like host, port, user, passwd, etc. as parameters in the connect call (and don't rely on picking them up from [client] section of a my.cnf file). I ran the full suite of dbapi20 tests (about 40 unit tests) and had only three minor failures that shouldn't affect usability.

I've looked at the Twisted networking package and though it looks like really good stuff, it's pretty huge and has some major dependancies (like OpenSSL) that I don't want impose on users that don't need it. At some point I may package it up as an add-on, but it doesn't look like it will make it into the base image. I also looked at PyGames (a Python interface to SDL) but it's also too big and has too many dependancies for the base image.

So, barring any other suggestions, I may be able to package all this good stuff up and have a release by the end of the week. Stay tuned smile.gif
LarsAC
BTW, would this only work on Cacko or other ROMs (e.g. for the 5500/5600) as well ?

Lars
kopsis
I'm testing on a C760 running Cacko 1.22 and a 5500 running the the last official Sharp ROM. So pretty much any Zaurus with a ROM that is reasonably close to the standard Sharp ROM should work just fine. I've even modified the mount scripts to theoretically work with the 3000 (though I have no way to test). It will not be compatible with pdaXrom or OpenZaurus, but I believe they both have their own solutions.
slapout
I'm using kopsis's Python 2.3 image on a 5600 with the Watapon rom. I've just started working with python, so I haven't tried all the modules. But the basic stuff seems to work and the little stuff I've done with pyqt seems to work.
vernm
I just got my 5600 last week. It is great, but I have a lot to learn re. Linix. I am most interested in Python; I have been using it for about 4 years. I have been reading about your port of 2.4. Sounds fantastic.

I noted that the memory footprint is an issue. Have you looked into the Python feature which allows imports from a ZIP file. This might be a good way to shrink the size. There is probably some time penalty, but I bet it is not too bad.

Right now I am trying to figure out what I need to do to setup Python on my Zaurus. I would like to put it on my SanDisk 512MB SD card. Is there a good how-to guide for setting things up for Linix newbies.

Since I have a lot of python experience, I would be happy to help out anyone who has a Python question.
kopsis
You can read up on my approach to Python on the Z here. That write up is for the previous (2.3) image, but most of it will apply to the new 2.4 image as well. The only big differences are 2.4 (obviously smile.gif ) and that I'll offer a cramfs image in addition to the ext2 version.

A cramfs image, for those not familiar with them, is a "Compressed RAM FileSystem". It's basically a ramdisk image that is compressed and saved to a file. Since it's already compressed, the trick of importing modules from a ZIP file is unnecessary. Using the cramfs image will be a bit slower than the ext2, but the whole thing will need less than 10MiB of space on your SD or CF card.

The ext2 image will remain avialable for those who are willing to trade space for speed and for developers who want to be able to add packages or otherwise customize the image (the ext2 image is read-write, the cramfs is read-only). You'll be able to fit either onto a 512MB SD card and still have over 450MB of space left smile.gif
Ragnorok
QUOTE(Bertram Wooster @ Jan 11 2005, 08:06 PM)
it's definitely worth a try, though i'd be thankful for your "semi-detailed instructions"- i'm not new to computers however to linux and especially unix commands are quite unknown to me.
*

- Just did this and it's pretty simple. Thanks kopsis, for putting this thing together. Beats installing all of that stuff separately!!

- Naturally you must have an ext2 file system available. My SD is ext2; since it never comes out of Hiroshi I don't care if it's not compatible with another device. I mount it from the Z when I need to move data around.
- I just looked at the /etc/rc.d/init.d/mntpython script to see it wanted to mount it to /opt/QtPalmtop/lib/python2.4. Farther down you'll find an fsck.ext2 command, then after that a mount command to mount file system using a loop. Take note of these commands. The Z has two loop devices already, so unless you're using them for something else, you won't have to wory about the lines that create a bunch of loop devices. If both of your loops are busy you'll have to make another one.
- I created a place on the SD to hold the python directory. Since it's going to be linked to /opt/QtPalmtop/lib, I used /usr/mnt.rom/card/QtPalmtop/lib/python2.4. Anywhere will work just fine ... simply create a directory.
- Just 'cause, I ran the fsck to be sure the python24.ex2 wasn't corrupted, which is what it does each time it mounts.
- Then I created a temporary dir to mount it to, since the final "resting" place has to be the /opt...lib dir above. I used /home/zaurus/mnt, but anything writable that's not the lib dir above should work fine.
- Use the noted loop mount command, substituting the location of python24.ex2 for $IMAGEDIR/file and the temporary mount point for $PYTHONDIR.
- cp -R <mnt point>\python2.4 <dest dir>. For my example, <mnt point> is /home/zaurus/mnt and <dest dir> is /usr/mnt.rom/card/QtPalmtop/lib/python2.4, just to be painfully explicit. (grin)
- The odd thing is this actually puts stuff in <dest dir>\mnt. I could have copied it down one directory, but I was feeling lazy, so I just linked to there.
- cd /opt/QtPalmtop/lib
- ln -s <dest dir>\mnt python2.4
- umount <mnt point>
- rmdir <mnt point> (only if you want to ... I created it just to mount this .ex2 so I deleted it when I was done)
- rm <path to>\python24.ex2. You don't need this any longer because it' on your SD card in <dest dir>\mnt now.
- You can delete mntpython, or just uncomment the exit 0 at the beginning, which is what I did. If you don't prevent this from running you'll get errors next reboot.
- Run the Hello World example from the installation instructions. Also, type "help()" and press Enter. If it loads help, you know you're golden ... if there were errors but it still gets a python command prompt, the Hello World will still work but it won't load help.
- Enjoy your new kopsis-provided python 2.4 installation.

- I've had very good results moving binaries and libraries to wherever and linking them back to where they started. I haven't done this with python yet, but I certainly will be. Just move python-exe from /home/QtPalmtop/bin to /mnt/card/QtPalmtop/bin and "ln -s" to "put it back". Nethack is the only thing I've seen that fails with this treatment, so even if you have to install to "Internal", it doesn't have to stay there. I have tons of free space because I don't let stuff live there forever. If it doesn't work ... no biggie ... just remove the link and move it back! This is Linux, not Winbloze, and it simply won't care.

- The entire reason I did this is to get gnome-sudoku to run. I scoured the web fo Sudoku generators and this is the only thing I found. Tons of *solvers*, but no generators/players for the Z at all. Gnome-sudoku is written in python, and kopsis so very conveniently provided this package for the latest version.
- I'm going to play with it today.

- Hope this helps...
rjohnson1969
QUOTE(kopsis @ Jan 13 2005, 06:40 AM)
You can read up on my approach to Python on the Z here.

KOPSIS,
This is an ingenious way of packaging python. Until I found your site, I was too intimidated to install it. Now, its on my sl-5500 SD and running fine.

I have a question: Does the loop filesystem mounted from SD throw off the SharpROM System Info application that estimates memory remaining? Mine seems to have gone up by about the size of the read only image... or I have some major cleaning up to do...

Thanks,
-- RJ
hughsieman
Hi

I have installed your python image kopsis and have been using it for the last few weeks to learn python/PyQt so thanks very much for that.

Just one thing though. I would like to set up python/PyQt/Qt on one of my Windows machines and I would like to be able to develop on the Windows machine and then run on the Z and vice versa.

I installed Python obviously, then Qt and then PyQt. Then I needed scintilla which I installed and then I needed sip and it was all getting a bit silly. Needless to say I couldn't get it to work. I was trying to get Idle up and running but failed.

I think I had the wrong versions to match the Z anyway as I was installing PyQt4 and I think it should have been version 3.

Anyway, please would someone let me know what to download and install and how to do it to achieve the above.

Many thanks and apologies for the fairly long post.

Also if this would be best in the Desktop thread then please let me know.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2017 Invision Power Services, Inc.