Daemons and friendly Ninjas

Image of Beastie + NinjaThere’s quite a lot of software that uses CMake as a (meta-)buildsystem. A quick count in the FreeBSD ports tree shows me 1110 ports (over a thousand) that use it. CMake generates buildsystem files which then direct the actual build — it doesn’t do building itself.

There are multiple buildsystem-backends available: in regular usage, CMake generates Makefiles (and does a reasonable job of producing Makefiles that work for GNU Make and for BSD Make). But it can generate Ninja, or Visual Studio, and other buildsystem files. It’s quite flexible in this regard.

Recently, the KDE-FreeBSD team has been working on Qt WebEngine, which is horrible. It contains a complete Chromium and who knows what else. Rebuilding it takes forever.

But Tobias (KDE-FreeBSD) and Koos (GNOME-FreeBSD) noticed that building things with the Ninja backend was considerably faster for some packages (e.g. Qt WebEngine, and Evolution data-thingy). Tobias wanted to try to extend the build-time improvements to all of the CMake-based ports in FreeBSD, and over the past few days, this has been a succes.

Ports builds using CMake now default to using Ninja as buildsystem-backend.

Here’s a bitty table of build-times. These are one-off build times, so hardly scientifically accurate — but suggestive of a slight improvement in build time.

Name Size GMake Ninja
liblxt 50kB 0:32 0:31
llvm38 1655kB * 19:43
musescore 47590kB 4:00 3:54
webkit2-gtk3 14652kB 44:29 37:40

Or here’s a much more thorough table of results from tcberner@, who did 5 builds of each with and without ninja. I’ve cut out the raw data, here are just the average-of-five results, showing usually a slight improvement in build time with Ninja.

Name av make av ninj Delta D/Awo
compiler-rt 00:08 00:07 -00:01 -14%
openjpeg 00:06 00:07 +00:01 +17%
marble 01:57 01:43 -00:14 -11%
uhd 01:49 01:34 -00:15 -13%
opencacscade 04:08 03:23 -00:45 -18%
avidemux 03:01 02:49 -00:12 – 6%
kdevelop 01:43 01:33 -00:10 – 9%
ring-libclient 00:58 00:53 -00:05 – 8%

Not everything builds properly with Ninja. This is usually due to missing dependencies that CMake does not discover; this shows up when foo depends on bar but no rule is generated for it. Depending on build order and speed, bar may be there already by the time foo gets around to being built. Doxygen showed this, where builds on 1 CPU core were all fine, but 8 cores would blow up occasionally.

In many cases, we’ve gone and fixed the missing implicit dependencies in ports and upstreams. But some things are intractable, or just really need GNU Make. For this, the FreeBSD ports infrastructure now has a knob attached to CMake for switching a port build to GNU Make.

  • Normal: USES=cmake
  • Out-of-source: USES=cmake:outsource
  • GNU Make: USES=cmake:noninja gmake
  • OoS, GMake: USES=cmake:outsource,noninja gmake
  • Bad: USES=cmake gmake

For the majority of users, this has no effect, but for our package-building clusters, and for KDE-FreeBSD developers who build a lot of CMake-buildsystem software in a day it may add up to an extra coffee break. So I’ll raise a shot of espresso to friendship between daemons and ninjas.

(Image credits: Beastie by Marshall Kirk McKusick on FreeBSD.org, Ninja by irkeninvaderkit on deviantart)

Calamares Testing

Photo of DesktopMy project for Blue Systems is maintaining Calamares, the distro-independent installer framework. Not surprisingly, working on it means installing lots of Linux distro’s. Here’s my physical-hardware testing setup, which is two identical older HP desktop machines and a stack of physical DVDs. Very old-school. Often I use Virtual Box, but sometimes the hum of a DVD is just what I need to calm down. There’s a KDE Neon, a Manjaro and a Netrunner DVD there, but the machine labeled Ubuntu is running Kannolo and sporting an openSUSE Geeko.

I’m all for eclecticism.

So far, I’ve found one new bug in Calamares, and fixed a handfull of them. I’m thankful to Teo, the previous Calamares maintainer, for providing helpful historical information, and to the downstream users (e.g. the distros) for being cheerful in explaining their needs.

Installing a bunch of different modern Linuxen is kind of neat; the variations in KDE Plasma Desktop configuration and branding are wild. Nearly all of them have trouble being usable on small screen sizes (e.g. the 800×600 that Virtual Box starts with — this has since been fixed). They all seem to install Virtual Box guest additions and can handle resizes immediately, so it’s not a huge issue, but just annoying. I’ve only broken one of my Linux installs so far (running an update, which then crashed kscreenlocker, and now it just comes up a black screen). I’ve got a KDE Neon dev/unstable as my main development VM set up, with KDevelop and the whole shizzle .. it’s very nice inside my KDE 4 desktop on FreeBSD.

I’ve got two favorite features, so far, in Linux live CDs and in KDE Plasma installations: ejecting the live CD on shutdown (Neon does this) and skipping the confirmation screen + 30 second timeout when clicking logout or shutdown (Netrunner does this).

So, time to hunker down with the list of issues, and in the meantime: keep on installin’.

KDE FreeBSD updates mid-june

I really wanted to say that Krita was the first KDE Frameworks 5-based application available in the official FreeBSD ports tree (so that pkg install krita just works), but it turns out that labplot has been KF5-based for a month or more and no-one noticed.

Nonetheless, three cheers for Krita and Calligra, which have been updated to the latest, modernest versions.

This is the start of a whole flood of updates, although Plasma is still going to take a while. Several things have come together to make this update possible:

  • FreeBSD KDE CI is running, so we can keep a close(r) eye on upstream. (This is what runs in KDE’s CI system, for FreeBSD)
  • KDE FreeBSD CI is running, so we can keep a close eye on ourselves. (This is what runs in FreeBSD’s CI systems, for KDE)
  • Both KDE and FreeBSD have Phabricator review systems, one for changes to downstream packaging, one for changes to upstream code.
  • KDE FreeBSD ports development is happening in a GitHub fork of the official FreeBSD ports tree.

With good CI and a good review process, we’ve been much happier getting packaging-fixes upstream than ever before. The CI catches unpleasant changes (hey, k3b has turned red, what’s up? Patches forthcoming ..) before they are released. The packaging CI is good for keeping track of where we are in packaging things ourselves. Since there’s a fair amount of package-shuffling going on, that’s important to have in hand. Finally the move to a git clone of the official ports tree makes it much easier to do small topic branches (e.g. updating Frameworks), test, and merge than we ever could with the SVN-based tree.

There’s a handful of smaller updates in-flight, alongside the Great Big Plasma5 branch, which is now shrinking as parts of it start to show up in the official ports already.

Mixed KDE Applications on FreeBSD

The KDE Frameworks have been available in FreeBSD for a while now, but we haven’t seen much movement on the desktop environment or the applications front. KDE4 is still the latest you can get from ports. The plasma5/ branch in the KDE FreeBSD ports repository contains all the applications, and KDE Plasma 5 Desktop, and is very up-to-date with KDE releases — but it’s not in official ports, and that makes it a little more difficult than it needs to be to install the latest KDE  Software on FreeBSD.

The KDE-FreeBSD team is starting to migrate individual applications to recently-released KF5 versions. That sometimes means letting go of some features: Plasma 5 integration isn’t going to happen until we have Plasma 5 in the official ports tree. But KDevelop 5.1 is a valuable upgrade over 4.7, and the IDE suffers very little (except if you wanted the embedded konsole part).

Here’s a screenshot of my KDE4 desktop running on FreeBSD, with the first few KF5 applications alongside.

Those are Krita and Kexi and Calligra Words an KDevelop from KDE Applications 17.04.2, and a KDE4-based JuK.  I haven’t tested this much: they run, and I can type words into the windows and KDevelop can compile something. Krita is out of my league.

So for applications, we’re coming close to a situation where they can be updated to recent releases without breaking everything at once (except for a fistful of “core” applications — we might  keep a konsole-kde4 and a kate-kde4 if there’s both interest and effort put into maintaining them).

For users of the official FreeBSD ports, expect updates to trickle out for a little while yet, while we sort out things like UPDATING and finally decide on what-goes-where in the shiny new (um, I guess we still need a new branding package) KF5-based world.

Generating FreeBSD packages with CPack

For software that uses CMake as (meta) buildsystem, and then gets packaged by distro’s — at least on FreeBSD — there’s something weird going on: the meta-buildsystem knows exactly what files are generated and where they get installed, but then knowledge about those files gets re-created outside of the meta-buildsystem in the ports tree, and that re-created information is used to do the actual packaging. To me, it feels like a duplication of effort, since CPack can be used to (re)use the information straight from the meta-buildsystem.

Back in february I blogged about starting on a CPack generator for FreeBSD packages.

Since then, with some comments and hints from Baptiste Daroussin (FreeBSD pkg(8) maintainer) on the FreeBSD side, and a really slick and constructive review process on Kitware’s gitlab, mostly by Brad King, the FreeBSD generator is being upstreamed, and it may be in CMake 3.10, as near as I can tell. There is still some wrestling to be done because there are multiple sources of libpkg.

This isn’t directly intended to break into the FreeBSD ports system — if only because there’s various staging and QA steps in a ports build that CMake just doesn’t know about. But it does produce packages that can be directly installed. That’s a big help for stuff not-yet-in-ports but which might be distributed and tested already. That, in turn, is something I need for some of my other (non-KDE) projects which are inching towards release and could be subsequently packaged but need testing now.

Anyway, big thanks to the constructive reviewers, and y’all can probably look forward to new package generators in CMake 3.10.

One of the new things CMake is picking up — entirely independently — in 3.9 is a new parameter to the project() command. You can check out CMake 3.9-RC2 now. The project() command now has an optional parameter DESCRIPTION, where you can put the human-readable project description. The FreeBSD package generator will use that description if it is given (although the Debian project description takes precedence, and there’s a CPACK_FREEBSD_PACKAGE_DESCRIPTION if you need to set a FreeBSD-specific description which takes precedence over all).

This new parameter is an opportunity for other CPack generators, too: there’s a Deb and RPM generator, and it struck me while writing the FreeBSD package generator that there’s quite a bit of packaging information that is invariant under the actual package format: homepage of the upstream project, for instance; possibly the package description; the software license the upstream software is published under. The different Free Software distro package generators don’t share that information (much — the FreeBSD package generator falls back to Debian and RPM settings if the FreeBSD-specific settings are not given).

So I think there’s a longer-term project that could be undertaken: massaging software meta-data in CMake and sharing that between package generators. That doesn’t just make sense for the Free Software OS generators: Wix and NSIS could (re)use the same information, although I don’t know offhand if there’s license information fields in those packages (the Wix and NSIS stuff I’ve done is for propritary software that comes with a separate EULA, and doesn’t use CMake anyway). And while we’re at it, that makes project metadata more accessible: suppose you could end up with something like this:

  DESCRIPTION "Terminal emulator using KDE Frameworks 5, which integrates fully with the Plasma5 desktop."
  LICENSE spdx:gplv3+
  WWW https://konsole.kde.org/

Right now you can find those bits of metadata (usually) in the README and the LICENSE file, possibly in the appinfo file, but I think it would be (even more) useful to have that information available in the (meta) buildsystem. And I think that anything that nudges developers towards better metadata maintainence and use of standard-ish notations is a good thing.

Migrating Myself

After all that FreeBSD shuffling, I’m going to move myself, too. For the past six years I’ve been commuting by train to Utrecht, where I worked at the Dutch Association of Audiological Centres on patient systems, financing, some data analysis and all the other stuff that happens in a small organization with a broad remit (also “help! the website is down!”). Starting today I’ll be moving less far, and commuting to my office upstairs instead.

As of today, I’m an independent contractor; the domain cnossos.nl which I registered ages ago when I last worked with Sebas is now in use.

As of today, I’m working with Blue Systems on the Calamares Installer Framework. Not a KDE project, but an independent, Qt-based, Free Software project used by Linux distro’s like Manjaro and Netrunner (which in turn ship KDE Plasma Desktop images). As a FreeBSD person, it feels a little weird to be writing Linux installers, but I have a secret hobby project for Calamares, too: adding ZFS install support and maybe even getting it to work as a FreeBSD installer. But that’s hobby, outside of the regular maintainence work I’ll be doing.

As of today, I’m working with the ARPA2 project on LDAP tooling and the IdentityHub project. That’s a lot of bit-banging, sometimes in C, which feels a little primitive when coming from a C++ background (I should look at Go sometime).

As of today, I’m still working with the Dutch Association for the Evaluation of Hearing Aids, but only as their occasional website-and-database-futzing dude.

As of today, I’m a full time freelance Free Software contributor. I’ve got a mantra: “if I use it, I’ll aggressively improve it”. Which means there’s plenty of patches coming down the line, for Charm time tracker for one thing.