There will be a BSD-centric talk at Akademy / QtCon this september. Yay!

Since it's rainy in Randa today, I'm going to write a little about all the bits-and-pieces of FreeBSD packaging that I worked on. The *NIX-packaging BoF has given me lots of ideas, for extracting metadata and for automating bits of packaging, but nothing has crystallized around that yet.

I've given a little thought to the current hotness, flakpak and snappy and appstream, wondering what parts of that could work on FreeBSD and how. I have a sketch of how we could leverage ZFS snapshots and poudriere and jails to quickly define an OS layer, a platform layer, and an application layer. Using unionfs mounts these can be combined into a single filesystem, allowing quick application testing. I'm not sure about how to wrangle permissions for such a scheme, though.

But maybe these schemes aren't all that necessary on FreeBSD (as a distro) when all is functioning smoothly (that is a major caveat). Since I was asked a few times here in Randa how FreeBSD packaging works and what the process is, I'm writing it down here.

Normally speaking, FreeBSD has two major releases that are supported. Those are 9-STABLE and 10-STABLE, at the moment. A next major release is being prepared. That's 11-CURRENT (not yet STABLE). Those are the operating system base systems, which in FreeBSD is a great deal more expansive than in the Linux world. There's a kernel and a compiler included, and a good chunk of userland. Together, that forms the FreeBSD distribution. Compatibility of a distribution is guaranteed for its lifetime, which I believe is generally two years. 9-STABLE is almost done, 11-CURRENT is almost ready.

The base FreeBSD distribution can vary a great deal from one major version to another: 9-STABLE uses gcc, 10-STABLE uses clang (as does 11-, but that has a much newer clang). So while the base is stable and compatible, moving between base OSses can be a challenge.

On top of the base OS, we have ports. These are all the userland programs, libraries, whatnots that have been ported to FreeBSD. "Ported," here, means that someone has created a Makefile for it that runs in the ports infrastructure. Packages are created from the ports; because of the differences in the base OS, packages are built for each supported release (and for each supported architecture) automatically by the FreeBSD builders.

What this effectively means is that FreeBSD could be considered a rolling-release distro (when looking at it from a Linux-distro point of view). The packages are continuously updated, based on the latest ports-Makefiles.

When things are moving smoothly, ports Makefiles get updated on time, packages are built, and a user can pkg upgrade the system regularly. With ZFS snapshots a rollback after a bad upgrade can be very simple (but pkg(8) itself doesn't have any mechanism for this).

Area51 ports process But how to ports get updated? There's ports committers (people) who do this. The KDE-FreeBSD project has its own repository, area51, where the K-F team assembles ports. The area51 repository is open to more (or rather, other) people than ports committers. In area51, the trunk is the last staging area before ports are sent "downstream" to the official ports tree. In other words, trunk is the "next new thing". For instance, when new versions of software are released that the KDE-FreeBSD maintains / packages in FreeBSD are released, that is often updated in area51 first, tested with our own package builders, and then "flushed" downstream to official ports. So there's a little shared staging area for our ports.

Area51 also has branches for "next big new thing" work. That's for large-scale reorganization efforts (which are sometimes necessary when we discover that things are not as co-installable as we thought, or when the names of ports and packages turn out to be poorly chosen). That's for updates that affect a lot of ports at once, like Qt updates, or new KDE Applications updates.

The diagram (that's my epic Kolourpaint skills there, with a crystal icon-set person-icon, and fonts from Blambot) shows roughly how this works. There are two SVN repositories, with branches visible in area51. FreeBSD committers move stuff from area51 to official ports; in between the other 22000 official ports update independently.

We're just coming off of a long period of time in which we were bottlenecked in getting stuff out of area51 -- i.e. unable to flush downstream. That's an unpleasant situation to be in, because things get clogged up.

Users who were prepared to work with area51 ports could always use the area51 repository and keep up-to-date, but people using official ports or packages were lagging behind quite seriously. We (KDE-FreeBSD) don't really have the capacity to serve up packages built from area51.

The situation is improving. Thanks to Randa, I've had time to work with Tobias Berner, Ralf Nolden, and others to wrestle a bunch of updates through to the ports system and to merge a lot of outstanding work to trunk, ready for the next iteration of flushing trunk downstream.

So here's what we've achieved this week:

  • konsole patch updated and merged to ports; it's still languishing in a reviewboard item on kde.org though. This improves support for home-dir handling and %-expansion which relies on system services.
  • QtCreator and qbs (Qt Build System) flushed downstream.
  • Last iteration of bump-KDE4-to-last-pre-Applications release prepared for downstream.
  • Qt 5.6 merged into area51 trunk.

What this effectively means is that KDE4 is about to be bumped (a little bit: there needs to be another round of reorganization before the changed form of releases under KDE Applications are supported) and that Qt 5.6 is being prepared for inclusion in official FreeBSD packages Real Soon Now.

Of course, this still lags behind on several fronts:

  • Qt 5.7 was released today,
  • We have not yet wrestled webengine into a releasable state,
  • KDE Frameworks 5 are not yet available,
  • KDE Plasma 5 is not yet available,
  • KDE Applications based on frameworks are not yet available.

These are all things that are being worked on, though! There is already a qt-5.7 branch which is close to being mergeable (but we really want webengine in shape for that). The plasma5 branch of area51 has been tracking Frameworks and Plasma and Applications closely for months, so that's basically poised to swoop in as soon as we have some other bits flushed out.

So it's been a good and productive week of getting those little annoyances worked out, and soon (tm) I expect we can have FreeBSD back on track with the latest and greatest of KDE software. We even have a Krita port prepared!