Python on CF/SD

[kopsis]

For the impatient:

http://pyqplayer.sourceforge.net/cgi-bin/b...qPlayerDownload

The story:

For the past couple months I've been working on a nice iPod-like media player front end for the Zaurus. It's getting close to where I can release it (GPL'd open source) but I've been struggling with the question of how to let ordinary Zaurus users run Python apps.

There are Python binaries/libraries available for the Zaurus (Python for arm-linux) but the versions built for Sharp or Sharp-compatible ROMs have to be installed in internal memory (or an ext2/ext3 formatted CF/SD card) to work properly. To make matters worse, for my app you'd need to install a bunch of those packages and it ends up requiring some knowledge of Python and using a lot of internal memory.

So instead I built an almost complete (missing the Tk stuff and some test scripts) Python 2.3 distribution for Sharp and Sharp-compatible ROMs. Then I packaged all that up into a single file that contains an ext2 image of the Python installation tree. Thanks to some scripts and the magic of "loop" mounting, you can toss that image on any FAT formatted CF/SD card (with enough free space to hold a 50 MiB file), install a 1.2 MiB IPK, and you'll now have a relatively complete Python 2.3 distribution on your Zaurus without chewing up the 30+ MiB of internal memory that it would normally take. Note that you can install all of this without ever seeing a command prompt

My media player still isn't quite ready for prime time, but I'm releasing the Python image and IPK now in hopes that folks with some Python knowledge will try it out and report any problems.

This has been tested on an SL-C760 running Cacko 1.21a and an SL-5500 running a standard Sharp 3.13 ROM. It should work on other Sharp-compatible ROMs. It will probably not work on OE or pdaXrom.

I've found that Python with PyQt (included in this image) is by far the easiest way to write Qtopia applications for the Zaurus. And it has the added advantage that you can develop directly on the Zaurus! If you know or want to learn Python, give this image a try and let me know what you think.

[stupkid]

Sounds cool Kopsis.  Have you tried using cramfs/squashfs for your image? It would get the size down by an order of magnitude.

[kopsis]

The problem with cramfs/squashfs is that they're read-only filesystems. For Python you really want to retain the ability to add third-party stuff to the image. Plus, loop mounted ext2 is a bit faster than cramfs/squashfs because you don't have to go through the decompression processing. Not a huge difference, but every little bit helps  When I release PyqPlayer, I may also release a matching cramfs version of the Python image for those who have no intention of actually messing around with Python directly and want to conserve SD/CF space.

[GoLinux]

Kopsis,

I'm definitely interested in giving Python development on the Zaurus a try (got to do something good for the brain on those long coast to coast business flights....).

However, let me state a couple of key points upfront:

1. I'm new to the Zaurus world (got my SL-6000L just few weeks ago)
2. I have no previous knowledge of Python as a language and its development tools.
3. While I consider myself an advanced PC user and I have past experience in programming with Turbo Pascal first and Delphi later, I'm by no mean a monster programmer.

That being said, I'm finding extremely exciting the Zaurus + Linux combo and I'm enjoying immensely the challenge of learning something new.

Before I jump and install your Python distribution, I have few questions:

1. Do you know of any possible incompatibility with the SL-6000L?
2. Given same speed CF and SD, is there any advantage in using one rather than the  other to install "your" Python?
3. Is there and Integrated Development Environment coming with your distribution? As I mentioned I'm used to Delphi, is it a dream to hope for something like that on a Zaurus? I don't mind the command line coding, but it would be nice to be able to put together an application GUI "visually".
4. How fast (or slow....) is to test on a Zaurus an application being developed?

Thanks in advance.

[kopsis]

While I consider myself an advanced PC user and I have past experience in programming with Turbo Pascal first and Delphi later, I'm by no mean a monster programmer.

I think you're gonna like Python. I ignored it for a long time (Python? Isn't that just another Perl wannabe?) and now I regret that I didn't get into it sooner. The language takes a little getting used to, but once you get the hang of it, it's utterly amazing how much you can do with just a little code. Not to mention that there's a high degree of built-in readability and maintainability in the code.

1. Do you know of any possible incompatibility with the SL-6000L?

No ... but then I don't know if it's ever been tried on a 6000. Theoretically it should work. It's unlikely to do any software damage ... it doesn't delete any files or create a big symlink jungle. If there's a problem it should simply not work. Of course a backup before installing is always a good idea for any new software package

2. Given same speed CF and SD, is there any advantage in using one rather than the  other to install "your" Python?

From a performance standpoint you aren't likely to notice any difference. CF may be a little faster but I use SD and it works great. I've heard that the Sharp ROMs can have problems with loop mounted filesystems (that's how my Python image works) on CF cards when the Zaurus suspends. I haven't experienced that but it could be a point in favor of running from SD. My system also doesn't have an easy way to "unmount" the Python image without entering a command from a shell prompt (and you can't eject the CF card with a mounted image until you unmount). Not a huge deal but if you freqently swap out your CF memory card for WiFi or Bluetooth, it will be a little bit of a pain (another plus for SD).

3. Is there and Integrated Development Environment coming with your distribution? As I mentioned I'm used to Delphi, is it a dream to hope for something like that on a Zaurus? I don't mind the command line coding, but it would be nice to be able to put together an application GUI "visually".

Nope, my Python image doesn't come with an IDE. The closest thing I've seen to a Python IDE for the Zaurus is VisiScript. It's really just a decent editor that will let you execute your script from a menu (and capture the text ouput into an output "pane" in the editor). It doesn't provide any GUI tools for UI construction. I'm really not sure how practical such a tool would even be given the Zaurus' small screen.

If you really want a visual GUI builder you may want to look into something like Blackaddr (though I really hate recommending theKompany's products). You could use it to build your UI framework on a PC and then do the back end coding on the Zaurus. You can similarly use Trolltech's Qt Designer (a free version is available with most Linux distros) and a tool called "pyuic" to build the source code framework. Not as easy or automatic as Delphi or VB, but not as bad as it sounds.

Personally, I've found that PyQt makes things simple enough that just coding the UI from scratch really isn't that difficult. And if you do a Qt GUI correctly, you never use absolute positioning for the widgets (which is what you get from some GUI builders). Instead you construct a window by programatically adding widgets to layout managers. The layout managers compute the correct widget placement at run-time based on the size of the window/display. That way your GUI works on a 320x240 SL-5500, a 640x480 SL-6000, or your an even bigger window on your desktop without changing a single line of code. I know some folks don't like layout manager based GUI design (perhaps due to post-traumatic stress induced by past experiences with Java's AWT  ) but I think it's the only sane way to do a portable GUI. I've found that my perfect Zaurus Python IDE is vim, but then I'm a little "old school"

4. How fast (or slow....) is to test on a Zaurus an application being developed?

Surprisingly fast. Start-up time for Python apps on the Zaurus is a little long (10 - 20 sec ... especially if your app imports big modules like PyQt), but once it's running the code runs much faster than I expected. I run a Python based webserver/wiki (MoinMoin) on my Zaurus for note taking and though it's a pretty hefty Python app with a lot of filesystem access, page load times are on the order of only 3 seconds. I'm also working on an mplayer front end that keeps all my music and videos in a SQLite database and it can populate the full song list (about 400 tracks) from the database and display it in under two seconds and do it sorted by album in three seconds. Of course SQLite is doing most of the work, but that's the hallmark of a good Python app - provide an easy to write, easy to use front-end and let highly optimized back-end libraries/apps that do the heavy lifting.

One last bit of advice ... check out the book GUI Programming with Python: QT Edition by Boudewijn Rempt. You can read the whole book online at http://www.opendocs.org/pyqt or snag a hardcopy from your favorite bookstore (ISBN: 0-97003300-4-4).

[aki]

kopsis, this sounds pretty interesting.  I will give it a try on my sl-6000l and report how it goes.

I was planning in trying to get the juke box PyTone running on the Z, perhaps you are familiar with it?  I find it interesting since it has a simple curses based interface and a built-in player based on libmad.  It does require a C extension module, so I am not sure how simple it is to build it on the Z.  
At any rate, does your package include the distutils package?
To build the C extension module, the instructions are to type the following:

Code: [Select]

$ python setup.py build_ext -i
Will this build the module with with your package?

[GoLinux]

Dave (a.k.a. Kopsis...)

thanks a bunch for your precious comments. You are definitely getting me intrigued with your Python build. Between work and family I haven't spent much time programming in the recent years, although I always have the latest Delphi version on one of my computers... The idea of being able to code on the Zaurus is very appealing.

I have already downloaded the Python image and the ipk from your web site, plus a bunch of html docs from Python.org. I guess the bug bit me already...

I'll be glad to be a guinea pig trying it on my SL-6000L, although it seems aki is going to beat me... I just need to clean-up one of my flash cards (got to buy that 1Gb SD one of these days....) and find some quite time after the kids are in bed.

From what you said on the GUI building I believe I can live with the layout managers method. I used to build UI using Turbo Vision with Borland Turbo Pascal, in the good old days of DOS. If you have ever used it, I think you will agree that few other programming tools can be more intimidating............. (at least at first, by the way)

One last question at this time: would you please elaborate on the unmounting procedure of the card containing your image? Does it apply only to a CF or to a SD as well? I occasionally eject my cards to transfer files using a card reader.

I don't mind typing commands in console, I just want to be sure I understand how to deal with this aspect of using your image.

Thanks also for the book suggestion, I'll certainly check it out.

[kopsis]

At any rate, does your package include the distutils package?
To build the C extension module, the instructions are to type the following:

Code: [Select]

$ python setup.py build_ext -i
Will this build the module with with your package?

My Python image does include distutils but it does not include the whole gcc toolchain that you need to build C extension modules.

There are gcc toolchain images similar to my Python image that, when properly installed, will give you the gcc tools you need for distutils to build C extension modules. The dev-img-1.6 toolchain package is the most complete but its installation can be complicated and it creates a jungle of symlinks with no automated uninstall/clean-up script. I prefer the zgcc2  package. It's less complete but works reasonably well for non-Qt apps and its install is much cleaner and easier to undo. You can locate both distributions by searching the ZUG forums.

As for PyTone, it looks like a good project and I came close to trying it out but it just wasn't quite close enough to the kind of UI I was looking for (so I created my own). I'll be curious to hear if you can get it working on the Zaurus.

[kopsis]

One last question at this time: would you please elaborate on the unmounting procedure of the card containing your image? Does it apply only to a CF or to a SD as well? I occasionally eject my cards to transfer files using a card reader.

Quite simple really. Fire up a shell and do the following:

Code: [Select]

$ su
# /etc/rc.d/init.d/mntpython stopTo remount after putting the memory card back in:

Code: [Select]

$ su
# /etc/rc.d/init.d/mntpython startYou do need to unmount the image before ejecting the memory card that it resides on regardless of whether it's on SD or CF.

[aki]

My Python image does include distutils but it does not include the whole gcc toolchain that you need to build C extension modules.

I prefer the zgcc2  package.

As for PyTone, it looks like a good project and I came close to trying it out but it just wasn't quite close enough to the kind of UI I was looking for (so I created my own). I'll be curious to hear if you can get it working on the Zaurus.

I will give it a try with zgcc2, as you suggest.  
Thanks for the help.
Then again, I might save some agravation and wait for your PyqPlayer ipk...    

[slapout]

I wanted to say thanks for taking the time to set this up. I'm been thinking about learning Python, and I think having it on the Z will be real handy.

[shula]

the problem is the GUI.
though i'm a python enthusiastic, i can't easilky recommend python for GUI apps on the Zaurus.  i couldn't yet find a fast and easy gui toolkit to work with.

Thus, i'm falling back to web(!) gui (SPYCE or php maybe.)
the performance at runtime aren't better,
but with web apps mechanism, the resources are released after each "screen" being sent to you.

if someone found an easy and penalty-free gui toolkit that works with python, is easy to write code with (see anygui API for an example of such ease) on Zaurus, i'll kiss his feet (i.e., i'll thank him very much, and/or pay him some $$$)

[kopsis]

I'm not too familiar with anygui. I found its Sourceforge page but it looks like development has ceased. Never the less, it claims to support a PyQt backend so perhaps I'll take a look and see if I can make it work easily with my Python image.

[kopsis]

if someone found an easy and penalty-free gui toolkit that works with python, is easy to write code with (see anygui API for an example of such ease) on Zaurus, i'll kiss his feet (i.e., i'll thank him very much, and/or pay him some $$$)

Get out your wallet  ... Just kidding.

The good new is that I can (with a few modifications) get anygui to run on the Zaurus and do some basic stuff like display windows and buttons. The bad news is that it doesn't play nice with Qtopia so you can't do some fairly important things (for example close the app). While it may be possible to get this usable, it looks like there could be a fair bit of work involved. And given that anygui never even made it to a 0.2 release, I'm concerned that there could be a lot of bugs/problems even with the Qtopia issues corrected. I may tackle it someday, but it won't be any time soon unless there's a lot of demand or a lot of $$$ involved

One thing that I did notice as I was digging into it is that anygui really isn't that much simpler than just plain old PyQt GUI development. It hides a little of the "boilerplate" that PyQt requires, but the fact is that stuff only looks complicated. Once you "steal" the framework for an application from an example or another app, adding the widgets and hooking in the backend code is really pretty easy. Maybe we just need a couple of good PyQt Zaurus examples to help folks get started (the stuff in the PyQt book isn't Zaurus specific and may be a little complex for folks just starting out).

[GoLinux]

going along with Kopsis suggestion about reusing the framework from an example or another application, shouldn't  it be possible to put togehter a "library" of GUI building blocks? I mean, not a visual library, but just the pieces of code required to make a tabbed window, a button, an input box, etc.

Even if it were only a text file with the fragment of code to be just crudely copied and pasted into your Python source, it wouls still be better than starting from scratch.

Once I get a half-decent grip on Python and put my hands on a Python app written specifically for the Z I'm going to play with this idea.