I don't see how rebuilding the modules should change anything. the xscale cache errata code is some arm asm bits contained in the startup and sleep/resume. Enabling or disabling this has no effect on the module code.
I'm not a kernel guru, but I'm sure the kernel size changes with the Cache Errata disabled, so some of the symbols the modules might want to call are certainly displaced, thus you have to recompile them, does it make sense ?
I'm pretty sure the System.map changes, so maybe the only thing needed is a valid System.map somewhere (and a depmod or sth like that) ?
Anayway, I recompiled the modules and I don't have the oops when plugin the USB connector, which lead me to think I was right, but this is maybe just a coincidence. I can't confirm anything about wifi as I don't have such cards.