Ahh, that would make sense.
Well, it would make sense were it true. But the debian kernels kicking around for the Z are not full-featured with lots of things you don't need on a Zaurus, but are specifically compiled for the Zaurus. In fact, I seem to recall that at least some of them _were_ Angstrom kernels. All the scripts are set up specifically for the things you would want on your Z as well, so IIRC there were not a bunch of extra services or logging.
Frankly, my experience was that debian was essentially the same speed as Angstrom. Now, maybe I whittled down some memory hogs, such as blacklisting the sound modules since I didn't use them (maybe that's part of it), and used a lightweight window-manager (I used xfce4 and, later, icewm).
The biggest differences were the ease of upgrade and availability of packages, both strongly in favor of debian.