Author Topic: "semi" Native Compilation With Distcc  (Read 34942 times)

pgas

  • Hero Member
  • *****
  • Posts: 1097
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« on: October 17, 2005, 03:20:16 am »
One way to combine some of the benefits of native compilation with some of the benefits of  cross compilation is to use distcc. It is very is easy to set up and its usage is transparent.

You need:
a lan
a linux pc, I suppose its IP is 192.168.1.10
a zaurus, I suppose its IP is 192.168.1.20

On the linux pc:
- download the cross sdk
- cd /; tar -jxvf cross-sdk-armv5tel-cacko-linux-3.4.4-2.2.5-softfloat.tar.bz2
- make a directory for the  links to the cross gcc, mkdir /home/pgas/distcc
- make some links so that distcc use the cross compilers
Code: [Select]
ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-gcc /home/pgas/distcc/gcc
ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-g++ /home/pgas/distcc/g++
ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-c++ /home/pgas/distcc/c++

these steps are required only the first time

- run /opt/cross/arm/3.4.4-xscale-softvfp/runsdk.sh
- make sure the cross compilers are first in the path ie
Code: [Select]
export PATH=/home/pgas/distcc:$PATH- run distccd (I believe that distcc is included in the cross sdk, else install it) like that distccd --daemon -a 192.168.1.20

On the zaurus
- download the native sdk
- install the sdk
Code: [Select]
mkdir -p  /opt/native/arm/3.4.4-xscale-softvfp
 mount -o loop zgcc-3.4.4.img /opt/native/arm/3.4.4-xscale-softvfp
- create links so that distcc is used instead of the actual gcc
Code: [Select]
mkdir /home/root/distcc
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/g++
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/c++


These steps are required only the first time.

- make sure that the links are first in the path
Code: [Select]
export PATH=/home/root/distcc:$PATH- indicate where the server is
Code: [Select]
export DISTCC_HOSTS=192.168.1.10
That's it, you can use the native sdk as usual, the only difference is that now the compilations will take place on your linux pc and not onboard.

Compilation will be faster and the limited ram of the zaurus is not a problem anymore.

I write this from memory, so I might have make some mistakes. Please correct me.
SLC-860 cacko / senao wifi

jh

  • Full Member
  • ***
  • Posts: 115
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #1 on: October 17, 2005, 09:40:30 am »
Thanks for the write-up.  I'll give it a shot on my C1k.

jason

oneguynick

  • Newbie
  • *
  • Posts: 6
    • View Profile
"semi" Native Compilation With Distcc
« Reply #2 on: October 17, 2005, 11:52:06 am »
Code: [Select]
ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc
should be

Code: [Select]
ln -s /opt/native/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc /home/root/distcc/gcc
Other than that looks great! Going to compile some stuff right now

Nick

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #3 on: November 05, 2005, 02:18:05 am »
Quote
One way to combine some of the benefits of native compilation with some of the benefits of  cross compilation is to use distcc. It is very is easy to set up and its usage is transparent.[div align=\"right\"][a href=\"index.php?act=findpost&pid=99670\"][{POST_SNAPBACK}][/a][/div]

Ahh, glad I found this post, I almost asked you how to do this  I just got my Zaurus, and would like to do development like this. I'm considering whether to buy a 10/100 ethernet card, or a 11 megabit wifi card. How much of a difference will this make in compile times? Any ideas? Thanks.
« Last Edit: November 05, 2005, 11:13:55 pm by jgrauman »
C1000

pgas

  • Hero Member
  • *****
  • Posts: 1097
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #4 on: November 05, 2005, 02:35:06 am »
Quote
Ahh, glad I found this post, I almost asked you how to do this smile.gif I just got my Zaurus, and would like to do development like this. I'm considering whether to buy a 10/100 ethernet card, or a 11 megabit wifi card. How much of a difference will this make in compile times? Any ideas? Thanks.

I use wfi, I don't know if it makes a diff for compile times.
SLC-860 cacko / senao wifi

sds

  • Jr. Member
  • **
  • Posts: 81
    • View Profile
"semi" Native Compilation With Distcc
« Reply #5 on: November 16, 2005, 08:54:13 am »
Thanks for the nice write-up. I finally came  around to try the cross sdk timestamped 18th of October using jikes as a test example. There were some minor changed paths/names and I am getting some warnings in the make phase -- but I guess nothing serious.

However, I am getting also plenty of critical errors which seem to indicate problem in the (parsing and/or usage of) the math functions of C++ such as:
Code: [Select]
/opt/native/arm/3.4.4-xscale-softvfp/lib/gcc/armv5tel-cacko-linux/3.4.4/../../../../include/c++/3.4.4/cmath: In
   function `float std::acos(float)':
/opt/native/arm/3.4.4-xscale-softvfp/lib/gcc/armv5tel-cacko-linux/3.4.4/../../../../include/c++/3.4.4/cmath:103: error: `
   __builtin_acosf' undeclared (first use this function)
Full trace is in the attachment.

As this problem is over my head, can somebody suggest a solution?
C3100 pdaXrom

pgas

  • Hero Member
  • *****
  • Posts: 1097
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #6 on: November 16, 2005, 09:09:11 am »
After a litttle googling, it may be related to the gcc version see
 here

the second post from the bottom
SLC-860 cacko / senao wifi

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #7 on: November 24, 2005, 02:27:26 pm »
So I finally got around to trying this, and I'm getting the following error when trying to compile qt-x11:

distcc[2432] (dcc_r_bulk_plain) ERROR: ftruncate fd5 failed: Operation not permitted
distcc[2432] (dcc_build_somewhere) Warning: failed to distribute to 192.168.1.1, running locally instead.

Then it crashes after that with
g++: Internal error: Terminated (program cc1plus)

maybe because it ran out of RAM from trying to do it locally.?

Any ideas?
« Last Edit: November 24, 2005, 02:31:33 pm by jgrauman »
C1000

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #8 on: November 24, 2005, 03:05:22 pm »
Quote
So I finally got around to trying this, and I'm getting the following error when trying to compile qt-x11:

distcc[2432] (dcc_r_bulk_plain) ERROR: ftruncate fd5 failed: Operation not permitted
distcc[2432] (dcc_build_somewhere) Warning: failed to distribute to 192.168.1.1, running locally instead.

Then it crashes after that with
g++: Internal error: Terminated (program cc1plus)

maybe because it ran out of RAM from trying to do it locally.?

Any ideas?
[div align=\"right\"][a href=\"index.php?act=findpost&pid=104841\"][{POST_SNAPBACK}][/a][/div]

Here is the verbose log from distccd up to the error...

distccd[1484] (main) chdir to /tmp
distccd[1484] (dcc_setup_daemon_path) daemon's PATH is /usr/local/sbin:/usr/sbin:/sbin:/root/distcc:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/lib/java/bin:/usr/lib/java/jre/bin
distccd[1484] (dcc_listen_by_addr) listening on 192.168.1.1:3632
distccd[1484] (dcc_defer_accept) TCP_DEFER_ACCEPT turned on
distccd[1484] (dcc_standalone_server) 1 CPU online on this server
distccd[1484] (dcc_standalone_server) allowing up to 3 active jobs
distccd[1484] (dcc_ignore_sighup) ignoring SIGHUP
distccd[1485] (dcc_detach) setsid to session 1485
distccd[1485] (dcc_log_daemon_started) preforking daemon started (2.18.3 i486-slackware-linux-gnu, built Dec 21 2004 16:43:2
distccd[1485] (dcc_preforking_parent) up to 1 children
distccd[1485] (dcc_preforking_parent) up to 2 children
distccd[1485] (dcc_preforking_parent) up to 3 children
distccd[1486] (dcc_check_client) connection from 192.168.1.2:1048
distccd[1486] (dcc_check_address) match client 0x201a8c0, value 0x201a8c0, mask 0xffffffff
distccd[1486] (dcc_r_token_int) got DIST00000001
distccd[1486] (dcc_r_token_int) got ARGC00000006
distccd[1486] (dcc_r_argv) reading 6 arguments from job submission
distccd[1486] (dcc_r_token_int) got ARGV00000003
distccd[1486] (dcc_r_argv) argv[0] = "gcc"
distccd[1486] (dcc_r_token_int) got ARGV00000002
distccd[1486] (dcc_r_argv) argv[1] = "-c"
distccd[1486] (dcc_r_token_int) got ARGV00000013
distccd[1486] (dcc_r_argv) argv[2] = "-fvisibility=hidden"
distccd[1486] (dcc_r_token_int) got ARGV0000000d
distccd[1486] (dcc_r_argv) argv[3] = "fvisibility.c"
distccd[1486] (dcc_r_token_int) got ARGV00000002
distccd[1486] (dcc_r_argv) argv[4] = "-o"
distccd[1486] (dcc_r_token_int) got ARGV0000000d
distccd[1486] (dcc_r_argv) argv[5] = "fvisibility.o"
distccd[1486] (dcc_r_argv) got arguments: gcc -c -fvisibility=hidden fvisibility.c -o fvisibility.o
distccd[1486] (dcc_scan_args) scanning arguments: gcc -c -fvisibility=hidden fvisibility.c -o fvisibility.o
distccd[1486] (dcc_scan_args) found input file "fvisibility.c"
distccd[1486] (dcc_scan_args) found object/output file "fvisibility.o"
distccd[1486] compile from fvisibility.c to fvisibility.o
distccd[1486] (dcc_run_job) output file fvisibility.o
distccd[1486] (dcc_input_tmpnam) input file fvisibility.c
distccd[1486] (dcc_readx) ERROR: unexpected eof on fd5
distccd[1486] (dcc_r_token_int) ERROR: read failed while waiting for token "DOTI"
C1000

pgas

  • Hero Member
  • *****
  • Posts: 1097
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #9 on: November 24, 2005, 03:48:28 pm »
I' m not sure why distcc fail to distribute, (do you run /opt/cross..../runsdk.sh before launching distccd?)

but the g++: Internal error: Terminated (program cc1plus) is most probably due to the lack of ram when compiling locally.
SLC-860 cacko / senao wifi

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #10 on: November 25, 2005, 01:14:37 pm »
Quote
I' m not sure why distcc fail to distribute, (do you run /opt/cross..../runsdk.sh before launching distccd?)

but the g++: Internal error: Terminated (program cc1plus) is most probably due to the lack of ram when compiling locally.
[div align=\"right\"][a href=\"index.php?act=findpost&pid=104857\"][{POST_SNAPBACK}][/a][/div]

Yep, I did run /opt/cross..../runsdk.sh... Could it have to do with permissions? I'm running distccd --user nobody, it wouldn't run without it. 'nobody' has permission to /tmp, does distccd need any other permissions? Thanks. Other ideas?

Josh
C1000

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #11 on: November 25, 2005, 11:54:24 pm »
A couple other notes that could be of interest. It says to make links on the Zaurus:

ln -s  /opt/native/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/distcc  /home/root/distcc/gcc  
etc.

but on my zaurus distcc is in /opt/native/arm/3.4.4-xscale-softvfp/bin... so I linked to there. Shouldn't be a problem, right?

Also, it says to make links on the PC:

ln -s  /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/bin/armv5tel-linux-gcc  /home/pgas/distcc/gcc
etc.

But on my PC, it's at:
/opt/cross/arm/3.4.4-xscale-softvfp/bin/armv5tel-linux-gcc

And I'm running RC12... Anyone get this to work with RC12? Thanks for the help, I'm really trying to get this to work, any help is greatly appreciated.

Josh
C1000

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #12 on: November 26, 2005, 12:45:02 am »
Sorry for all the messages...

This seems to be more relevant to my problem than the previous trails I was attempting to track down...

Running /opt/cross/arm/3.4.4-xscale-softvfp/bin/armv5tel-cacko-linux-gcc returns:
error while loading shared libraries: libiconv.so.2: cannot open shared object file: No such file or directory  (Even after running runsdk.sh).

It looks like my cross sdk is not quite configured correctly. libiconv.so.2 is in /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/lib as expected, and set has X11LIB=/opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/lib,
Any ideas why gcc (and the all the other programs in the cross compiler) can't find the libraries? Thanks again.

Josh
C1000

pgas

  • Hero Member
  • *****
  • Posts: 1097
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #13 on: November 27, 2005, 02:39:51 am »
Quote
ibiconv.so.2 is in /opt/cross/arm/3.4.4-xscale-softvfp/armv5tel-cacko-linux/lib as expected

this one is an arm binary used for linking, you need one for x86 to run the compiler.
attach is a x86 bin, put it in /usr/lib (rename and link to libiconv.so.2 etc..)

I can start distccd as root, it just issue a warning if i remember correctly
« Last Edit: November 27, 2005, 02:45:42 am by pgas »
SLC-860 cacko / senao wifi

jgrauman

  • Jr. Member
  • **
  • Posts: 59
    • View Profile
    • http://
"semi" Native Compilation With Distcc
« Reply #14 on: November 29, 2005, 03:55:34 pm »
Quote
attach is a x86 bin, put it in /usr/lib (rename and link to libiconv.so.2 etc..)[div align=\"right\"][a href=\"index.php?act=findpost&pid=105091\"][{POST_SNAPBACK}][/a][/div]

Thanks, but that didn't do it either, same error still. I've also tried compiling other apps (including joe) and same thing. I'm going to try with a different PC to see if that changes anything... Could someone who has this working let me know what OS they are running on their PC and what version of pdaX they are running? Thanks.

Josh
C1000