Zaurus Cluster How To With Distcc, Was: Arm Clusters To Speed Up Native Builds? Options

[Capn_Fish]

I can't remember if I've asked this before, but here goes: It it possible/feasable to USB network all of my ARM devices, running a common OS (Ubuntu/Debian, likely) and have a mini-cluster going to speed up native builds? It would seem that with 2 Zaurii and an N810 (and hopefully a Pandora soon), I could get things built 2-3x faster, which would be nice.

I suspect something with distcc could be done, but my impression is that that ignores the "host" CPU (eg, if I started the build on the N810, only the two Zaurii would actually do work), making it less than optimal. Is that correct?

Anyway, thoughts, how-tos, etc.?

EDIT: It looks completely possible. I guess this thread will likely turn into a how-to once I get around to it.

EDIT 2: Changed the thread title, see the next post for the tutorial.

This post has been edited by Capn_Fish: Jun 10 2009, 04:54 AM

[Capn_Fish]

OK, it turns out this is really easy. I'm going to do this from the prospective of two Zaurii running Zubuntu, so adapt it however necessary. This is a rough draft, so please don't expect everything to be absolutely correct the first time around.

Note that you COULD do this over traditional networks, but that the latency would slow things down to some extent (and it's pretty awesome to have a mini USB network cluster going).

A generic example can be found here.

0. Get USB networking between all of the involved devices going (not going in-depth on this, look it up):

Sample /etc/network/interfaces:

CODE

iface usb0 inet static
address 192.168.2.4
netmask 255.255.255.0
gateway 192.168.2.1

Then run:

CODE

ifup usb0

If all went well, you should be networked.

1. Install the necessary tools (both machines):

CODE

sudo apt-get install build-essential automake autoconf python2.5-dev

2. Download the latest distcc source (3.1 as of this writing, both machines unless you want to compile and transfer):

CODE

wget http://distcc.googlecode.com/files/distcc-3.1.tar.gz

3. Build distcc (both machines, once again, unless you want to compile and transfer):

CODE

tar -xf distcc-3.1.tar.gz
cd distcc-3.1
./configure --prefix=/usr --without-get --without-gnome --disable-Werror
make
sudo make install

4. Add a "distcc" user (both machines):

CODE

useradd distcc

5. Start the server (both machines, I believe):

CODE

distccd --daemon --allow "127.0.0.1 xxx.xxx.xxx.xxx"

6. On the machine "doing" the build, export the hosts:

CODE

export DISTCC_POTENTIAL_HOSTS='127.0.0.1 xxx.xxx.xxx.xxx'

7. Run the build ("host" machine only, NUMBER_OF_THREADS should probably be the number of machines available + 1, IIRC):

CODE

cd /to/source/dir/
pump make -jNUMBER_OF_THREADS CC=distcc

8. Check to make sure everything is working (on "host" machine only):

CODE

distccmon-text

If you see something like:

CODE

object_abc.o [0]: localhost
object_xyz.o [1]: 192.168.2.3

Sample speedups:
Building distcc using distcc (N810 and SL-C3100, using "pump make -j3 cc=distcc"):4 minutes, 49 seconds
Building distcc using only the N810: 4 minutes, 52 seconds

That's not really the speedup I was hoping for, but I'll try to build some other things and see how it pans out.

Anyway, I hope somebody finds this useful.

[RX Shorty]

Thanks, I will get my third arm device soon.
So this is something nice to try indeed