Posted inInformation Technology / Raspberry Pi

KDE Neon – Distcc and Qt

seasoned_geek by seasoned_geek
distcc using other machine image

One of the tools which was wildly touted years ago was distcc. This is a distributed compilation system which can be brutal to set up, but can also dramatically reduce compilation times for big jobs. It has fallen out of favor in recent years because most developers end up getting a quad or more core machine with a modern enough CPU to have all kinds of virtualization and hyper threading. These machines also tend to have many more Gig of RAM than they really need so, if Linux really is good about adjusting its disk cache memory usage, in theory you won’t see much boost. At least that is the argument I keep hearing and usually I hear that argument from people using laptops for development.

Flaws in the Thinking

So, please allow me to point out some flaws in that (mostly Millineals) thinking:

  • Laptops, unless their battery life is measured in mere minutes, _always_ have underpowered components. Yes, you may have lots of RAM, a great sounding graphics chipset, etc., but the hardware children will have opted for the lowest power consuming version of each. Even your USB ports will operate at both lower power and slower speed because the overall design goal was to make the batter last as long as possible. Be honest. When you are thinking about a new laptop and see “atrocious battery life” in the reviews, you click to the next one don’t you?
  • A sucky network isn’t going to make anything run faster. Most shops which complained profusely about distcc not returning much bang for the buck typically have a horrible network where people groan any time they have to transfer even a print job on it.
  • Both make and moc have gotten much better when it comes to working with distcc. One of the big drawbacks of building really complex Qt GUI applications with distcc used to be moc didn’t distribute well. I don’t notice the particulars, but I don’t notice a problem anymore.

Distcc Experiments

Any C++ Qt application with a sufficient number of source files can benefit from using distcc. Assuming your network isn’t a three legged dog running in deep snow, that is. As to hand tuning the disk cache and conducting other experiments with it, I don’t bother. You can read about a few experiments here. That mystical “sufficient number of files” threshold is much lower than you think. In order to verify this I need to install the distcc monitor.

distcc monitor install image
distcc monitor install

This is a little graphical tool which lets you see how your build is using the farm. After that I needed to install distcc itself. For some reasons the software application tool doesn’t included that, but you can find it with synaptic.

distcc-pump

distcc synaptic install image

I also installed distcc-pump. There are pluses and minuses here. The default configuration of distcc does not work with pump. There has been a bug with posts and reports dating back to 2007 and probably beyond where you end up with a rash of “connection refused” errors trying to let distcc dynamically find and use distcc servers on your network. I forgot about this and spent some head slamming time figuring it out.

The Next Step

Go that far on each machine which is to be part of your distcc compilation farm then open up a terminal and type the following:

distcc --show-hosts

On my main desktop it returned:

192.168.1.132:3632/24

192.168.1.105:3632/32

Now you need to know just who those machines are.

roland@roland-HP-Compaq-8100-Elite-SFF-PC:~$ nslookup 192.168.1.132
Server: 127.0.1.1
Address: 127.0.1.1#53

132.1.168.192.in-addr.arpa name = roland-desktop.

roland@roland-HP-Compaq-8100-Elite-SFF-PC:~$ nslookup 192.168.1.105
Server: 127.0.1.1
Address: 127.0.1.1#53

105.1.168.192.in-addr.arpa name = roland-HP-Compaq-8100-Elite-SFF-PC.

You need to tweak distcc just a touch

sudo nano /etc/default/distcc

STARTDISTCC="true"
ALLOWEDNETS="192.168.1.0/24 127.0.0.1"
LISTENER=""
ZEROCONF="false"

The above lines need to be in your distcc file. Of course you need to change 192.168.1.0/24 to be whatever your network is. Another major issue is that LISTENER needs to be blank. It just doesn’t seem to work any other way.

As I said, I had quite a bit of head slamming time trying to track down the “connection refused” issue so I made a few changes which may not be required. I’m going to list them here and we will experiment more during another post when I try to distcc from my Raspberry Pi.

$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 roland-HP-Compaq-8100-Elite-SFF-PC
192.168.1.132 roland-desktop

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

 

cat .distcc/hosts
localhost
roland-desktop

 

$ cat /etc/distcc/hosts
# As described in the distcc manpage, this file can be used for a global
# list of available distcc hosts.
#
# The list from this file will only be used, if neither the
# environment variable DISTCC_HOSTS, nor the file $HOME/.distcc/hosts
# contains a valid list of hosts.
#
# Add a list of hostnames in one line, seperated by spaces, here.
localhost
roland-desktop

Make options

Once all of that was done I was able to tweak the xpnsqt2 make parameters as follows:

distcc make options
distcc make options

Basically I added

-j40 CC=distcc CXX=distcc

You will find posts on the Internet telling you to add

QMAKE_CC = distcc
QMAKE_CXX = distcc

to your .pro file. While it is true this will cause qmake to generate

CC = distcc
CXX = distcc

it is also true that people tend to forget those are in there then post a project on SourceForge or their own Web site many people cannot build. The reason many wish to add it directly into the .pro file is so distcc gets used by the next user.

Those additional make options get saved in .pro.user not in the .pro file. If you are building on multiple machines or have multiple developers all using a common build environment it makes sense to put those values in the original .pro file and to look up how to force in the -j40 option as well. If you are working on something which will be released as an OpenSource project of some kind, best not to make those mods.

While I have not tried it I have seen posts stating you can define the CC and CXX environment variables to be distcc gcc and distcc g++ respectively. For someone working on many projects this is definitely the way to go, assuming it works.

Once all of that was done I fired up QtCreator, cleaned the project, then kicked off a build

distcc using other machine image
distcc using other machine

 

The machine I’m using as the build server is horribly named “roland-desktop.” It is an AMD 6-core machine with 20Gb of RAM and an SSD. I should also mention it is running BOINC while idle. The machine which is actually on my desktop is that Compaq-8100-Elite blah blah blah machine. It is a quad-core I7 having 16Gb of RAM and an SSD.

As the monitor clearly shows, even though the build server is weighted down by BOINC, a project this small benefited from distcc.

Related posts:

Where Did My QDebug Output Go?

MOC Parse Error at “std”

So You Can’t Get Your Models to Work with QML?

QtCreator – No qmlScene installed

CopperSpice Experiments

Roland Hughes started his IT career in the early 1980s. He quickly became a consultant and president of Logikal Solutions, a software consulting firm specializing in OpenVMS application and C++/Qt touchscreen/embedded Linux development. Early in his career he became involved in what is now called cross platform development. Given the dearth of useful books on the subject he ventured into the world of professional author in 1995 writing the first of the "Zinc It!" book series for John Gordon Burke Publisher, Inc.

A decade later he released a massive (nearly 800 pages) tome "The Minimum You Need to Know to Be an OpenVMS Application Developer" which tried to encapsulate the essential skills gained over what was nearly a 20 year career at that point. From there "The Minimum You Need to Know" book series was born.

Three years later he wrote his first novel "Infinite Exposure" which got much notice from people involved in the banking and financial security worlds. Some of the attacks predicted in that book have since come to pass. While it was not originally intended to be a trilogy, it became the first book of "The Earth That Was" trilogy:
Infinite Exposure
Lesedi - The Greatest Lie Ever Told
John Smith - Last Known Survivor of the Microsoft Wars

When he is not consulting Roland Hughes posts about technology and sometimes politics on his blog. He also has regularly scheduled Sunday posts appearing on the Interesting Authors blog.