Yeah the native build environment definitely helps in some situations, but in others it won't do much. For example, trying to make OpenTTD work (a reverse-engineered Transport Tycoon 'clone'), there were some subtleties involved in the ARM architecture related to structs and alignment that caused some problems in what looked to be a perfectly-compiled executable.
As for libraries, they are 'essentially' the same thing as executables, except that they are not DIRECTLY executable. Usually with shared dynamic libraries (.so files), they are dynamically 'inserted' into your actual executable at run-time instead of compile-time, so one would THINK that maybe you wouldn't need cross-compiled versions at compile-time. However, the linker still needs to match up symbols, etc, so you do in fact need them available beforehand.
If you know C at all (even some basic syntax and have a mind that can 'deduce' the function of various commands, etc), and take a little time to study the various documentation for Makefiles and the autotools (autoconf, automake, autoheader, etc), it makes life a lot easier... Just a place to start when/if you're interested
Dave