"semi" Native Compilation With Distcc, a short howto Options

[pgas]

One way to combine some of the benefits of native compilation with some of the benefits of cross compilation is to use distcc. It is very is easy to set up and its usage is transparent.

You need:
a lan
a linux pc, I suppose its IP is 192.168.1.10
a zaurus, I suppose its IP is 192.168.1.20

On the linux pc:
- download the cross sdk
- cd /; tar -jxvf cross-sdk-armv5tel-cacko-linux-3.4.4-2.2.5-softfloat.tar.bz2
- make a directory for the links to the cross gcc, mkdir /home/pgas/distcc
- make some links so that distcc use the cross compilers

CODE

ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-gcc /home/pgas/distcc/gcc
ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-g++ /home/pgas/distcc/g++
ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-c++ /home/pgas/distcc/c++

these steps are required only the first time

- run /opt/cross/arm/3.4.4-xscale-softvfp/runsdk.sh
- make sure the cross compilers are first in the path ie

CODE

export PATH=/home/pgas/distcc:$PATH

- run distccd (I believe that distcc is included in the cross sdk, else install it) like that distccd --daemon -a 192.168.1.20

On the zaurus
- download the native sdk
- install the sdk

CODE

mkdir -p  /opt/native/arm/3.4.4-xscale-softvfp
mount -o loop zgcc-3.4.4.img /opt/native/arm/3.4.4-xscale-softvfp

- create links so that distcc is used instead of the actual gcc

CODE

mkdir /home/root/distcc
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/g++
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/c++

These steps are required only the first time.

- make sure that the links are first in the path

CODE

export PATH=/home/root/distcc:$PATH

- indicate where the server is

CODE

export DISTCC_HOSTS=192.168.1.10

That's it, you can use the native sdk as usual, the only difference is that now the compilations will take place on your linux pc and not onboard.

Compilation will be faster and the limited ram of the zaurus is not a problem anymore.

I write this from memory, so I might have make some mistakes. Please correct me.

[jh]

Thanks for the write-up. I'll give it a shot on my C1k.

jason

[oneguynick]

CODE

ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc

should be

CODE

ln -s /opt/native/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc

Other than that looks great! Going to compile some stuff right now

Nick

[jgrauman]

One way to combine some of the benefits of native compilation with some of the benefits of  cross compilation is to use distcc. It is very is easy to set up and its usage is transparent.

Ahh, glad I found this post, I almost asked you how to do this I just got my Zaurus, and would like to do development like this. I'm considering whether to buy a 10/100 ethernet card, or a 11 megabit wifi card. How much of a difference will this make in compile times? Any ideas? Thanks.

This post has been edited by jgrauman: Nov 5 2005, 08:13 PM

[pgas]

Ahh, glad I found this post, I almost asked you how to do this smile.gif I just got my Zaurus, and would like to do development like this. I'm considering whether to buy a 10/100 ethernet card, or a 11 megabit wifi card. How much of a difference will this make in compile times? Any ideas? Thanks.

I use wfi, I don't know if it makes a diff for compile times.

[sds]

Thanks for the nice write-up. I finally came around to try the cross sdk timestamped 18th of October using jikes as a test example. There were some minor changed paths/names and I am getting some warnings in the make phase -- but I guess nothing serious.

However, I am getting also plenty of critical errors which seem to indicate problem in the (parsing and/or usage of) the math functions of C++ such as:

CODE

/opt/native/arm/3.4.4-xscale-softvfp/lib/gcc/armv5tel-cacko-linux/3.4.4/../../../../include/c++/3.4.4/cmath: In
  function `float std::acos(float)':
/opt/native/arm/3.4.4-xscale-softvfp/lib/gcc/armv5tel-cacko-linux/3.4.4/../../../../include/c++/3.4.4/cmath:103: error: `
  __builtin_acosf' undeclared (first use this function)

Full trace is in the attachment.

As this problem is over my head, can somebody suggest a solution?

Attached File(s)

 make_errors.txt ( 23.89K ) Number of downloads: 56

 

[pgas]

After a litttle googling, it may be related to the gcc version see
here

the second post from the bottom

[jgrauman]

So I finally got around to trying this, and I'm getting the following error when trying to compile qt-x11:

distcc[2432] (dcc_r_bulk_plain) ERROR: ftruncate fd5 failed: Operation not permitted
distcc[2432] (dcc_build_somewhere) Warning: failed to distribute to 192.168.1.1, running locally instead.

Then it crashes after that with
g++: Internal error: Terminated (program cc1plus)

maybe because it ran out of RAM from trying to do it locally.?

Any ideas?

[jgrauman]

So I finally got around to trying this, and I'm getting the following error when trying to compile qt-x11:

distcc[2432] (dcc_r_bulk_plain) ERROR: ftruncate fd5 failed: Operation not permitted
distcc[2432] (dcc_build_somewhere) Warning: failed to distribute to 192.168.1.1, running locally instead.

Then it crashes after that with
g++: Internal error: Terminated (program cc1plus)

maybe because it ran out of RAM from trying to do it locally.?

Any ideas?

Here is the verbose log from distccd up to the error...

distccd[1484] (main) chdir to /tmp
distccd[1484] (dcc_setup_daemon_path) daemon's PATH is /usr/local/sbin:/usr/sbin:/sbin:/root/distcc:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/lib/java/bin:/usr/lib/java/jre/bin
distccd[1484] (dcc_listen_by_addr) listening on 192.168.1.1:3632
distccd[1484] (dcc_defer_accept) TCP_DEFER_ACCEPT turned on
distccd[1484] (dcc_standalone_server) 1 CPU online on this server
distccd[1484] (dcc_standalone_server) allowing up to 3 active jobs
distccd[1484] (dcc_ignore_sighup) ignoring SIGHUP
distccd[1485] (dcc_detach) setsid to session 1485
distccd[1485] (dcc_log_daemon_started) preforking daemon started (2.18.3 i486-slackware-linux-gnu, built Dec 21 2004 16:43:2
distccd[1485] (dcc_preforking_parent) up to 1 children
distccd[1485] (dcc_preforking_parent) up to 2 children
distccd[1485] (dcc_preforking_parent) up to 3 children
distccd[1486] (dcc_check_client) connection from 192.168.1.2:1048
distccd[1486] (dcc_check_address) match client 0x201a8c0, value 0x201a8c0, mask 0xffffffff
distccd[1486] (dcc_r_token_int) got DIST00000001
distccd[1486] (dcc_r_token_int) got ARGC00000006
distccd[1486] (dcc_r_argv) reading 6 arguments from job submission
distccd[1486] (dcc_r_token_int) got ARGV00000003
distccd[1486] (dcc_r_argv) argv[0] = "gcc"
distccd[1486] (dcc_r_token_int) got ARGV00000002
distccd[1486] (dcc_r_argv) argv[1] = "-c"
distccd[1486] (dcc_r_token_int) got ARGV00000013
distccd[1486] (dcc_r_argv) argv[2] = "-fvisibility=hidden"
distccd[1486] (dcc_r_token_int) got ARGV0000000d
distccd[1486] (dcc_r_argv) argv[3] = "fvisibility.c"
distccd[1486] (dcc_r_token_int) got ARGV00000002
distccd[1486] (dcc_r_argv) argv[4] = "-o"
distccd[1486] (dcc_r_token_int) got ARGV0000000d
distccd[1486] (dcc_r_argv) argv[5] = "fvisibility.o"
distccd[1486] (dcc_r_argv) got arguments: gcc -c -fvisibility=hidden fvisibility.c -o fvisibility.o
distccd[1486] (dcc_scan_args) scanning arguments: gcc -c -fvisibility=hidden fvisibility.c -o fvisibility.o
distccd[1486] (dcc_scan_args) found input file "fvisibility.c"
distccd[1486] (dcc_scan_args) found object/output file "fvisibility.o"
distccd[1486] compile from fvisibility.c to fvisibility.o
distccd[1486] (dcc_run_job) output file fvisibility.o
distccd[1486] (dcc_input_tmpnam) input file fvisibility.c
distccd[1486] (dcc_readx) ERROR: unexpected eof on fd5
distccd[1486] (dcc_r_token_int) ERROR: read failed while waiting for token "DOTI"

[pgas]

I' m not sure why distcc fail to distribute, (do you run /opt/cross..../runsdk.sh before launching distccd?)

but the g++: Internal error: Terminated (program cc1plus) is most probably due to the lack of ram when compiling locally.

[jgrauman]

I' m not sure why distcc fail to distribute, (do you run /opt/cross..../runsdk.sh before launching distccd?)

but the g++: Internal error: Terminated (program cc1plus) is most probably due to the lack of ram when compiling locally.

Yep, I did run /opt/cross..../runsdk.sh... Could it have to do with permissions? I'm running distccd --user nobody, it wouldn't run without it. 'nobody' has permission to /tmp, does distccd need any other permissions? Thanks. Other ideas?

Josh

[jgrauman]

A couple other notes that could be of interest. It says to make links on the Zaurus:

ln -s /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc
etc.

but on my zaurus distcc is in /opt/native/arm/3.4.4-xscale-softvfp/bin... so I linked to there. Shouldn't be a problem, right?

Also, it says to make links on the PC:

ln -s /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-gcc /home/pgas/distcc/gcc
etc.

But on my PC, it's at:
/opt/cross/arm/3.4.4-xscale-softvfp/bin/armv5tel-linux-gcc

And I'm running RC12... Anyone get this to work with RC12? Thanks for the help, I'm really trying to get this to work, any help is greatly appreciated.

Josh

[jgrauman]

Sorry for all the messages...

This seems to be more relevant to my problem than the previous trails I was attempting to track down...

Running /opt/cross/arm/3.4.4-xscale-softvfp/bin/armv5tel-cacko-linux-gcc returns:
error while loading shared libraries: libiconv.so.2: cannot open shared object file: No such file or directory (Even after running runsdk.sh).

It looks like my cross sdk is not quite configured correctly. libiconv.so.2 is in /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/lib as expected, and set has X11LIB=/opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/lib,
Any ideas why gcc (and the all the other programs in the cross compiler) can't find the libraries? Thanks again.

Josh

[pgas]

ibiconv.so.2 is in /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/lib as expected

this one is an arm binary used for linking, you need one for x86 to run the compiler.
attach is a x86 bin, put it in /usr/lib (rename and link to libiconv.so.2 etc..)

I can start distccd as root, it just issue a warning if i remember correctly

Attached File(s)

 libiconv.so.2.2.0.txt ( 863.6K ) Number of downloads: 203

 

[jgrauman]

attach is a x86 bin, put it in /usr/lib (rename and link to libiconv.so.2 etc..)

Thanks, but that didn't do it either, same error still. I've also tried compiling other apps (including joe) and same thing. I'm going to try with a different PC to see if that changes anything... Could someone who has this working let me know what OS they are running on their PC and what version of pdaX they are running? Thanks.

Josh