Well, I read your comment and I was all set to come back and say "It's doing what it's supposed to be doing and you can't change it anyways, so don't worry about it," but then I did some reading.
For the Zaurus I can say with relative certainty that it really is freeing cache just fine exactly when it needs to. Linux is much more agressive about disk caching than other operating systems and it can make people used to other operating systems nervous to see so little memory marked as free. So, you really don't need to worry about it. The 2.6 kernel has some nice options for tuning the VM system through /proc/sys/vm/ , but 2.4 doesn't seem to have that kind of stuff exposed. If you're really concerned about it you might be able to do some basic tuning by looking at information on this page
On your FC4 machine it's an entirely different case for two reasons:
1) The 2.6 kernel has all sorts of neat VM tuning parameters
2) The presence of swap
If you want less cache because you want to reduce how often it swaps out you can tune the "swappiness" value by doing:
echo 60 > /proc/sys/vm/swappiness
Instead of 60 put different values between 0 and 100. At 0 the kernel will avoid swapping until it's really necessary. At 100 it will swap whenever it seems like a good idea to do so. Experiment however much you want since your settings will get put back to default (probably 60) when you reboot. The swappiness value is really a personal choice on a desktop. Basically it's a tradeoff between having a nice big disk cache (start open office, close it, open it again and it shows up instantly since it was still in cache) and not having to hit swap too often. If you really just want to see a bigger number next to "free memory" there's another setting to change called min_free_kbytes. It sets how much memory the kernel will keep for itself. This is almost certainly not something you want to change unless you're having problems with things living in kernel space not being able to allocate memory. Like all settings in /proc however it goes away on reboot, so go crazy.
echo 8192 > /proc/sys/vm/min_free_kbytes
Anyways, my take on all this is that any memory that's called "free" might as well be called "wasted." Linux takes the same view on this and thus tries to do something useful (cache) with any memory that would otherwise just sit there unused.