Daemons and friendly Ninjas

Image of Beastie + NinjaThere’s quite a lot of software that uses CMake as a (meta-)buildsystem. A quick count in the FreeBSD ports tree shows me 1110 ports (over a thousand) that use it. CMake generates buildsystem files which then direct the actual build — it doesn’t do building itself.

There are multiple buildsystem-backends available: in regular usage, CMake generates Makefiles (and does a reasonable job of producing Makefiles that work for GNU Make and for BSD Make). But it can generate Ninja, or Visual Studio, and other buildsystem files. It’s quite flexible in this regard.

Recently, the KDE-FreeBSD team has been working on Qt WebEngine, which is horrible. It contains a complete Chromium and who knows what else. Rebuilding it takes forever.

But Tobias (KDE-FreeBSD) and Koos (GNOME-FreeBSD) noticed that building things with the Ninja backend was considerably faster for some packages (e.g. Qt WebEngine, and Evolution data-thingy). Tobias wanted to try to extend the build-time improvements to all of the CMake-based ports in FreeBSD, and over the past few days, this has been a succes.

Ports builds using CMake now default to using Ninja as buildsystem-backend.

Here’s a bitty table of build-times. These are one-off build times, so hardly scientifically accurate — but suggestive of a slight improvement in build time.

Name Size GMake Ninja
liblxt 50kB 0:32 0:31
llvm38 1655kB * 19:43
musescore 47590kB 4:00 3:54
webkit2-gtk3 14652kB 44:29 37:40

Or here’s a much more thorough table of results from tcberner@, who did 5 builds of each with and without ninja. I’ve cut out the raw data, here are just the average-of-five results, showing usually a slight improvement in build time with Ninja.

Name av make av ninj Delta D/Awo
compiler-rt 00:08 00:07 -00:01 -14%
openjpeg 00:06 00:07 +00:01 +17%
marble 01:57 01:43 -00:14 -11%
uhd 01:49 01:34 -00:15 -13%
opencacscade 04:08 03:23 -00:45 -18%
avidemux 03:01 02:49 -00:12 – 6%
kdevelop 01:43 01:33 -00:10 – 9%
ring-libclient 00:58 00:53 -00:05 – 8%

Not everything builds properly with Ninja. This is usually due to missing dependencies that CMake does not discover; this shows up when foo depends on bar but no rule is generated for it. Depending on build order and speed, bar may be there already by the time foo gets around to being built. Doxygen showed this, where builds on 1 CPU core were all fine, but 8 cores would blow up occasionally.

In many cases, we’ve gone and fixed the missing implicit dependencies in ports and upstreams. But some things are intractable, or just really need GNU Make. For this, the FreeBSD ports infrastructure now has a knob attached to CMake for switching a port build to GNU Make.

  • Normal: USES=cmake
  • Out-of-source: USES=cmake:outsource
  • GNU Make: USES=cmake:noninja gmake
  • OoS, GMake: USES=cmake:outsource,noninja gmake
  • Bad: USES=cmake gmake

For the majority of users, this has no effect, but for our package-building clusters, and for KDE-FreeBSD developers who build a lot of CMake-buildsystem software in a day it may add up to an extra coffee break. So I’ll raise a shot of espresso to friendship between daemons and ninjas.

(Image credits: Beastie by Marshall Kirk McKusick on FreeBSD.org, Ninja by irkeninvaderkit on deviantart)

This entry was posted in FreeBSD, KDE. Bookmark the permalink.

3 Responses to Daemons and friendly Ninjas

  1. Sandro says:

    Sounds very interessing – faster builds. But can you please add some links, where I can see how I can actually switch the backend for cmake/qmake? You only added the samples, how to disable the feature 🙂

    • adridg says:

      Well, for raw cmake it’s “cmake -G Ninja” to generate ninja instead of makefiles; so what we did for our ports-building is add “-G Ninja” to the cmake command-line when building ports which have USES=cmake.

      The documentation in CMake itself is rather sparse: https://cmake.org/cmake/help/v3.8/generator/Ninja.html

      • Sandro says:

        Thanks for the explanation. But how can I change cmake within qmake?

        Background is that I want to make qtwebengine build with ninja. because it uses qmake the cmake is hidden through this.