I tried one rather brutal way to stop xdm.
Exit X by pressing [Ctrl] [Alt] [BS]
Then keep pressing [Ctrl] [z] quickly enough. Otherwise xdm will load very soon.
Once you succeed in getting the console back, do this:
/etc/init.d/xdm stop
update-rc.d -f xdm remove
Then you can go about fixing things or setting up X.