I have the Cacko ROM running with X/Qt & Debian (PocketWorkstation). It about the closest you can get to having both Qtopia and X at the same time. PdaXrom is more "elegant" and has more of the typical applications most people want compiled to run under it, but in theory the possibilities with the Debian implementation are endless.
One issue is that you have to run the filesystem on a SD card if you want wifi at the same time. And, you need swapspace on the SD too, which can wear the SD card out over time.
I have gotten Dillo, Ghostview, Abiword, Firefox and Gnumeric running on it, along the last three programs take a while to load.
Bill