Author Topic: Introducing quickpython (Read 6779 times)

zenyatta · « **on:** May 21, 2004, 12:54:17 pm »

Hi all.

As I\'ve mentioned in another thread, it would be nice if there was a way to speed up Python apps on the Z the way quickexec speeds up C++ apps. I started thinking about today and it took me 41 minutes to put together a very basic working version. I\'ll call it quickpython because it makes sense (and google yielded just 5 hits mostly related to The Quick Python Book).

Be warned: I\'m a complete Python newbie and I just wanted a proof-of-concept, so it\'s a butt-ugly hack. We\'ll get to that later. First you need to copy the following code into a file and call it quickpython.py:

Code: [Select]

#!/usr/bin/env python

from os import remove

from time import sleep

COMMAND_FILE="quickpython.cmd"

while 1:

 try:

  sleep(1)

  commandFile=open(COMMAND_FILE, "r")

  command=commandFile.read().splitlines()[0]

  commandFile.close()

  remove(COMMAND_FILE)

  if command=="exit":

   break

  else:

   execfile(command, {})

 except:

  continue

The above code plays the \"daemon\" and keeps the Python runtime loaded. Now we need a way to send it innocent Python apps to be executed :twisted:. It will be a trivial shell script called qp.sh:

Code: [Select]

echo $1 >> quickpython.cmd
Once you have both files in place in the same directory, open konsole in that directory (if you\'re not already there) and try this:

chmod a+x qp.sh
./quickpython.py &

Now the daemon is running and we can try to feed it a Python script. Let\'s say you have an app you want to speed up and it\'s launched via myapp.py. Simply type

./qp.sh myapp.py

and it should start up. You will notice no change the first time but if you exit your app and do it again there should be a major difference. In the case of my diary app, launch time went from about 10 seconds to under 1 second (on an SL-5500). The daemon takes up about 2MB of RAM and, last but not least, you can shut it down with

./qp.sh exit

As you can see, it is a butt-ugly hack indeed. The most outrageous shortcomings include:

DrWowe · « **Reply #1 on:** May 21, 2004, 01:41:36 pm »

Good idea. Your implentation could have a problem with race condition if two apps are started at the same time. One might clobber the other before the file is read. This could be improved pretty easily using a named pipe or socket.

nilch · « **Reply #2 on:** May 21, 2004, 02:42:08 pm »

zenyatta, I wont pretend to understand Python concepts that well to suggest any improvements to your scripts, but the overall concept and the idea to have a quickscript is very novel and needed.

I love designing Python/PyQT apps, but the only downside seems to me to be too many runtimes to install - python, PyQY plus other modeules for various functions like Python Webtools for network functions, XML modules for XML parsing etc, and that they seem slow to startup. If we could overcome the 2nd problem, that would be a great advantage still.

allbest to your efforts though, and hope Mickeyl and others like him can find some time off from their busy schedules and help in making Python a more viable solution on the Zaurus.
Mickeyl has always been a great help and a compendium on Python problems, so we somehow have to convince him.

zenyatta · « **Reply #3 on:** May 21, 2004, 04:31:24 pm »

Things are getting very interesting very quickly. I\'ve learnt how to run the app in a separate thread but the \"QApplication: There should be max one application object\" warning kept coming up. Turns out it\'s not a namespace issue. Instead, PyQt fails to destroy the QApplication object once the app is finished. That\'s natural since PyQt has no way of knowing the app is done. I need to tell PyQt after the app stops that it should destroy the QApplication. I can access the singleton instance as the qt.qApp variable but I don\'t know how to explicitly garbage-collect it and -crucially- whether it would also destroy the underlying C++ object (is there a way to call a destructor from Python? PyQt docs don\'t mention it.). I definitely cannot just say qt.qApp=None (PyQt docs say it has no effect).

As I can run apps in threads, I tried running several of them at once. Of course, they fight over the QApplication instance and segfault the daemon. This happens regardless of whether I run multiple instances of the same app or different ones. I don\'t quite know how to solve it. I could isolate the apps further by launching complete OS processes, thus defeating the whole purpose of quickpython :? I wonder how quickexec deals with this. Are there multiple QApplication instances? If not, how is the singleton shared?

nilch: I find the number of Python packages quite manageable - and it\'s good for people with resource constraints as they can install just the stuff they need. I have a 512MB CF card so I just installed the whole lot. And as it came in the standard Opie feed (I use OZ) I didn\'t have to hunt for the packages either. And maybe if there are enough quality Python apps it will get into a flash image or two... someday

z.

Mickeyl · « **Reply #4 on:** May 27, 2004, 11:30:42 am »

Okay folks, I have thought about it.
I want us to come up with a solution featuring the following design goals:

0.) Don\'t require application programmers to change anything to play nice with quicklaunching
1.) Be extensible, i.e. speed up loading arbitrary modules
2.) Save memory and processing power, this means saying \"No\" to multithreading
3.) Don\'t violate Qt, i.e. don\'t use more than one QApplication object
4.) Provide feedback and means to switch between PyQt applications using a dedicated applet

The good news is I have a prototype up and running which I will publish soon :-D

The \"bad\" news is I don\'t want to do this alone and I won\'t, so if I don\'t get patches, this project will die as fast as it has been born.

Until I publish the prototype (which is less than one week away) let\'s discuss about the design goals stated.

nilch · « **Reply #5 on:** May 27, 2004, 12:11:57 pm »

Mickeyl, firstly thanks to you for taking a second look at Python on the Zaurus. Musch as I would like to contribute, I dont think I have the understanding of PyQT that much. Hope zenyatta and others chip in . I will do my best testing and questioniong with my meager understanding though.

as for your design goals ....
0) is a nice objective. designers like me or better would want to stay away from needing to do special tweaking to play with the quicklaunch thing.
1) I dont know how you mean to make it extensible, but speed of loading modules would be great - thats the primary shortcoming - spped to launch PyQT and the extra modules.
2) blank on this point
3) Thats a very objective goal - we dont want to break anything with the structure of QT apps. If that requires just having one QApplication at one time - so be it.
4) I didn\'t understand this point very well.

One point I thought about when designing and publishing an app I wrote - most people seem to have objections to installing many different IPK\'s - 1) Python, 2) PyQT, and 3) extra modules plus 4) the Python/PyQT app itself to be able to run a particular application.
Is it possible (without breaking Python or PyQT licenses) to have a common package (IPK) for Python and PyQT and keep the extra modules seperate (since otherwise it would be huge)

Since the newer models of Zaurus have more memory, maybe Python+PyQT could be fit into one package ?

hope the others can suggest and better the goals you set out.

Mickeyl · « **Reply #6 on:** May 27, 2004, 12:53:54 pm »

Ok, to clarify two goals:

2) To speed up loading modules, the quicklauncher preloads these. This is basically the whole idea. However, some people work on PyQt, others work on XMLRPC etc. We would have to come up with a means to only preload the modules which are needed by the user, because preloading all modules available on the system wouldn\'t be sane.

4) The quicklauncher is the one and only python process in the system which has a QApplication object. This means if at all you would get only one icon in the taskbar. Nothing more, \'cause subsequently started (read: quicklaunched) applications don\'t have own application objects. Hence, how to change between a number of concurrently running quicklaunch applications? Therefore I thought we could come with a dedicated python-quicklaunch-applet which gives up a status (e.g. how many apps are running, how many memory is consumed by the python process, how many modules are preloaded, etc.) and lets you switch between those apps.

zenyatta · « **Reply #7 on:** May 28, 2004, 04:27:35 am »

Darn, I thought I\'d create it by myself under the radar and take all the credit :evil: . Anyway, I\'ve been intending to post my own progress but there hasn\'t been much:

- I\'ve re-implemented the daemon as a SocketServer listening on a datagram BSD socket
- the launch script is now in C (found no way to do sockets in the shell)
- the quicklaunched script rteceives its command-line args

I\'m now stuck trying to grok from Opie sources how the hell Opie quicklauncher shares one QApplication. It is my first serious look at C++ after 9 years and it\'s not pleasant at all (either Python or C++ was surely created by super-advanced aliens, I hope it was Python or we\'re all in trouble). I understand the general quicklauncher architecture, plugins and all, but I still don\'t get how there can be 2 quicklauncher apps running at the same time. Especially two instances of the same plugin (since they are stored in a QMap with the name of the executable as key). Any explanation would be very much appreciated (I thought that was plenty clear from my previous post - hint, hint :wink: ).

As for the design goals:

0 - unmodified apps) it\'s a noble but ambitious goal, I don\'t see how it can be done. But Mickey is obviously on top of a higher hill than me so he probably can. I was thinking of having app authors replace app=QPEApplication(sys.argv) with app=quickpython.getQPEApplication(sys.argv), that wouldn\'t be much of a modification. And how do you want to deal with someone calling QApplication.quit()? Is the object usable after that?

1 - preload config) don\'t see a problem with that, you could have a .cfg file listing the modules, or a directory with symlinks to the modules

2 - no multithreading) I don\'t quite understand. You mean different quicklaunched apps would reside in different processes rather than different threads? How will that save resources?

3 - single QApplication) Agreed. I\'m looking forward to seeing your code

4 - quickpython applet) Again, I don\'t understand. Right now, I can launch textedit and todolist simultaneously. They are both quicklauncher apps yet they both get their icons on the taskbar. How is that? Do they have separate QApplications after all? If so, what does the app variable in core/tools/quicklauncher/main.cpp do? I want to scream...

Mickey, please elaborate or post the code asap. Thanks.

z.

fpp · « **Reply #8 on:** May 28, 2004, 04:51:50 am »

Quote

It is my first serious look at C++ after 9 years and it\'s not pleasant at all (either Python or C++ was surely created by super-advanced aliens, I hope it was Python or we\'re all in trouble).

Fantastic little nugget this !

I\'m cutting \'n pasting this little gem for later reuse ;-)

News:

Author Topic: Introducing quickpython (Read 6779 times)