After some hacking in the audio drivers I am now able to fix the speaker phase with a rather nasty but functional kernel hack that simply inverts the polarity of one channel of the audio data before it is sent to the codec. It is rather incredible how much better the sound is with the phase fixed, not only does the sound not originate from behind your eyes anymore, but there sound is much fuller with more low in it.
The problem I now run in to is that the phase is reversed with using headphones, as the same codec path is used for both headphones and speakers, so I will need to find some kind of detect if the headphones are active and only reverse phase if this is not the case.
I might also be able to fix the crackle-when-nothing-is-playing issue while I'm at it by not powering down the I2S path when audio playing is paused, but that might have implications on power usage, of course.
For the brave of heart, here is my android kernel boot image with the phase fixed:
http://zevv.nl/div/linux_boot-phasefix.img. (No, I take no responsibility for any bricking or other harm done to your device!)