What ./configure command-line are you using? I use:
LDFLAGS="-Wl,-rpath-link,/opt/arm/3.3.2-vfp/armv5tel-cacko-linux/X11R6/lib" ./configure --host=armv5tel-cacko-linux --build=i686-linux --x-includes=/opt/arm/3.3.2-vfp/armv5tel-cacko-linux/X11R6/include --x-libraries=/opt/arm/3.3.2-vfp/armv5tel-cacko-linux/X11R6/lib --disable-debug --prefix=/usr
I think the setting of LDFLAGS at the start is what you need.