Farewell to the Mountains

So the last train has left Randa, and we can look back at a — judging from the bugs that were fixed and the ideas that were traded and the code that was written — successful sprint. The last two days were characterized by the authentic Randa Internet experience, in which a mountain goat eats 18% of the packets, but that doesn’t stop KDE developers from writing code and sharing AppImages.

Peter Simon from AppImage stopped by to give a talk and lend a hand in building AppImages for those that wanted them. Talking with him has clarified a few ideas in my mind about how I’d do something similar in FreeBSD. It’s also clarified the rationale behind these bundling (upstream-) application formats: a <foo> application image is for when you need the latest version of a specific application for a specific purpose, or when you’re testing an application; in both cases you want nothing else to change. These are not my own words: they have been expressed several times on lwn.net recently, and Boud puts it eloquently on his own.

Given that, we can conclude that a given user will have only a few to a dozen <foo> application images in use.

With those assumptions, certain objections to the whole format go away. Not all of them, but enough to make me wonder if something like that could be done for FreeBSD, and if it would make testing easier.

Saturday I played the cook; I have tremendous respect for Hadrien who ran the show the rest of the week by himself (it is a full-time job). Fortunately I had great help in the kitchen: Mario’s parents who helped clean and drove me to the grocery store. Ayrton and Lays who convinced me to make brigadeiro. Grace and Scarlett who minded pans and cut potatoes. Alexander and Pinak who also stirred and scoured. If I’ve forgotten anyone, my apologies: it’s your help that made the community kitchen a success, too.

The following things were developed for saturday dinner: schwytserbrigadeiro, with 85% cocoa Swiss chocolate, doppelgeschnitzelteknoblauchbraten because we had leftover schnitzels and 20 cloves of garlic, and a potato-salami-raclette melt which just melted away in front of the hungry developers.

We also all got Cholera, but that is a local specialty.

Anyway, this puts me in mind that we could possibly do a cooking thing in Berlin at Akademy, if we put our minds (hearts? tongues?) to it.

Here’s a gigantic thank-you to Mario Fux and Randa Meetings e.V. for putting this week together, special thanks to Scarlett and Grace for being there at the right moment, and to all the other participants for making it a great and productive week.

FreeBSD in Randa

There will be a BSD-centric talk at Akademy / QtCon this september. Yay!

Since it’s rainy in Randa today, I’m going to write a little about all the bits-and-pieces of FreeBSD packaging that I worked on. The *NIX-packaging BoF has given me lots of ideas, for extracting metadata and for automating bits of packaging, but nothing has crystallized around that yet.

I’ve given a little thought to the current hotness, flakpak and snappy and appstream, wondering what parts of that could work on FreeBSD and how. I have a sketch of how we could leverage ZFS snapshots and poudriere and jails to quickly define an OS layer, a platform layer, and an application layer. Using unionfs mounts these can be combined into a single filesystem, allowing quick application testing. I’m not sure about how to wrangle permissions for such a scheme, though.

But maybe these schemes aren’t all that necessary on FreeBSD (as a distro) when all is functioning smoothly (that is a major caveat). Since I was asked a few times here in Randa how FreeBSD packaging works and what the process is, I’m writing it down here.

Normally speaking, FreeBSD has two major releases that are supported. Those are 9-STABLE and 10-STABLE, at the moment. A next major release is being prepared. That’s 11-CURRENT (not yet STABLE). Those are the operating system base systems, which in FreeBSD is a great deal more expansive than in the Linux world. There’s a kernel and a compiler included, and a good chunk of userland. Together, that forms the FreeBSD distribution. Compatibility of a distribution is guaranteed for its lifetime, which I believe is generally two years. 9-STABLE is almost done, 11-CURRENT is almost ready.

The base FreeBSD distribution can vary a great deal from one major version to another: 9-STABLE uses gcc, 10-STABLE uses clang (as does 11-, but that has a much newer clang). So while the base is stable and compatible, moving between base OSses can be a challenge.

On top of the base OS, we have ports. These are all the userland programs, libraries, whatnots that have been ported to FreeBSD. “Ported,” here, means that someone has created a Makefile for it that runs in the ports infrastructure. Packages are created from the ports; because of the differences in the base OS, packages are built for each supported release (and for each supported architecture) automatically by the FreeBSD builders.

What this effectively means is that FreeBSD could be considered a rolling-release distro (when looking at it from a Linux-distro point of view). The packages are continuously updated, based on the latest ports-Makefiles.

When things are moving smoothly, ports Makefiles get updated on time, packages are built, and a user can pkg upgrade the system regularly. With ZFS snapshots a rollback after a bad upgrade can be very simple (but pkg(8) itself doesn’t have any mechanism for this).

Area51 ports processBut how to ports get updated? There’s ports committers (people) who do this. The KDE-FreeBSD project has its own repository, area51, where the K-F team assembles ports. The area51 repository is open to more (or rather, other) people than ports committers. In area51, the trunk is the last staging area before ports are sent “downstream” to the official ports tree. In other words, trunk is the “next new thing”. For instance, when new versions of software are released that the KDE-FreeBSD maintains / packages in FreeBSD are released, that is often updated in area51 first, tested with our own package builders, and then “flushed” downstream to official ports. So there’s a little shared staging area for our ports.

Area51 also has branches for “next big new thing” work. That’s for large-scale reorganization efforts (which are sometimes necessary when we discover that things are not as co-installable as we thought, or when the names of ports and packages turn out to be poorly chosen). That’s for updates that affect a lot of ports at once, like Qt updates, or new KDE Applications updates.

The diagram (that’s my epic Kolourpaint skills there, with a crystal icon-set person-icon, and fonts from Blambot) shows roughly how this works. There are two SVN repositories, with branches visible in area51. FreeBSD committers move stuff from area51 to official ports; in between the other 22000 official ports update independently.

We’re just coming off of a long period of time in which we were bottlenecked in getting stuff out of area51 — i.e. unable to flush downstream. That’s an unpleasant situation to be in, because things get clogged up.

Users who were prepared to work with area51 ports could always use the area51 repository and keep up-to-date, but people using official ports or packages were lagging behind quite seriously. We (KDE-FreeBSD) don’t really have the capacity to serve up packages built from area51.

The situation is improving. Thanks to Randa, I’ve had time to work with Tobias Berner, Ralf Nolden, and others to wrestle a bunch of updates through to the ports system and to merge a lot of outstanding work to trunk, ready for the next iteration of flushing trunk downstream.

So here’s what we’ve achieved this week:

  • konsole patch updated and merged to ports; it’s still languishing in a reviewboard item on kde.org though. This improves support for home-dir handling and %-expansion which relies on system services.
  • QtCreator and qbs (Qt Build System) flushed downstream.
  • Last iteration of bump-KDE4-to-last-pre-Applications release prepared for downstream.
  • Qt 5.6 merged into area51 trunk.

What this effectively means is that KDE4 is about to be bumped (a little bit: there needs to be another round of reorganization before the changed form of releases under KDE Applications are supported) and that Qt 5.6 is being prepared for inclusion in official FreeBSD packages Real Soon Now.

Of course, this still lags behind on several fronts:

  • Qt 5.7 was released today,
  • We have not yet wrestled webengine into a releasable state,
  • KDE Frameworks 5 are not yet available,
  • KDE Plasma 5 is not yet available,
  • KDE Applications based on frameworks are not yet available.

These are all things that are being worked on, though! There is already a qt-5.7 branch which is close to being mergeable (but we really want webengine in shape for that). The plasma5 branch of area51 has been tracking Frameworks and Plasma and Applications closely for months, so that’s basically poised to swoop in as soon as we have some other bits flushed out.

So it’s been a good and productive week of getting those little annoyances worked out, and soon ™ I expect we can have FreeBSD back on track with the latest and greatest of KDE software. We even have a Krita port prepared!

Randa in General

One of the nice things about the Randa meetings is — for me at least — is the return to an almost student-like work scheme. Sleep, eat, code, repeat. The hacking room downstairs (there’s two more upstairs) has 16 people in it, working on all corners of KDE. People move around to form pairs as needed. The Davids are in high demand. Alexander jumps at all kinds of problems, from Python to I18n. Tomaz is doing neat things with Subsurface and explaining things to others. Just sitting here means catching up with KDE technologies and hearing tips.

The consequence of these long hacking days is that a lot of little stuff gets fixed as well. Stuff that get shunted aside at the end of a regular working day, but that fills up little gaps in the 16 hour working day that most of the Randa attendees have here. A common turn of phrase seems to be “I have this little annoyance ..” followed by some nearby participant saying “Yeah, I just fixed that.”

Keeping up 16 hour working days for a week requires a lot of organization. And the wonderful thing is is that that organization is there, and that you can roll out of bed (watch out if you’re in the upper bunk) and breakfast is ready. All the day-to-day necessities are already taken care of, and we can get to work. Catering to these day-to-day needs costs money, and you can support that (and therefore support the work that gets done) by making a donation to the Randa fundraiser.

Randa Fundraiser

A lot of the pictures you’ll see that come out of Randa are the “pretty pictures“, of the times between hacking sessions (which Lays Rodrigues describes above the pretty pictures). From here I can see 14 people behind laptops, water bottles, coffee cups, various candy wrappers, Umbrello, 3D-printing, flatpak, KMyMoney. There’s a lot of work getting done, but it’s not very photogenic.

Zanshin 0.4 on FreeBSD

Zanshin 0.4 was released a little over a month ago. Today I got around to fixing up the packaging (see, this is why tooling and automation is important) for Zanshin on FreeBSD in the area51 repository. One stumbling block was a sudden requirement for C++14 while not checking if the compiler supports it. The base system clang on FreeBSD 10 doesn’t, and that led to weird and hard-to-diagnose build failures. Consistent failures, but it took a long time — and Tobias’s help — to find out that --std=c++14 was causing an error in FindLibIcal and that CMake wasn’t logging the failing compile command or the failing source code.

Anyway. It’s sorted out now, and Zanshin can be had through the unofficial ports repository. There’s no ETA for landing it (and the rest of KDE Frameworks 5) in official ports and packages, though.

Reconciling Packaging Data (2)

So previously I looked at the dependency data available in KDE repositories, and how that translates to information in the FreeBSD packaging system (i.e. the ports Makefiles). It seems likely that judicious use of list_dependencies can help fill in the dependency information when depending on KDE and Qt components.

“Judicious” here means something like “after the Randa *NIX-deployment meeting tomorrow”, since over lunch I learned from upstream (Scarlett) that the inconsistencies in the dependency data I noticed earlier are going to be tackled.

So it’s good to be here in Randa at the cross-platform meeting, also for the *NIX packagers.

But let’s look at two other bits of information in a FreeBSD port: the COMMENT string in the Makefile and the pkg-descr file.

KDE Frameworks all have a metainfo.yaml file, so there’s 81 of those sitting on y hard drive right now. Each of those files has a description line that could be used to fill the COMMENT. Since this is the upstream short description of the software product, that seems like a good pithy thing to use instead of coming up with our own.

The pkg-descr is a larger file, with multiple lines of text allowed describing the package. One special line in the file starts with WWW:, followed by a space, followed by the URL of the package’s homepage. That is information that isn’t available in the metadata in Frameworks. It could be available in the AppStream data, except that none of the Frameworks seem to have that, while AppStream does support generic components.

So my plan is slowly congealing, like lime-Jell-O pudding, into the following:

  • Get kde-build-metadata, on the assumption that it contains a list of all the frameworks (in particular all those frameworks released in a given iteration on KDE Frameworks)
  • For each Framework listed there:
    • grab the released tarball, and extract the description from the metainfo.yaml.
    • if there’s AppData somewhere, extract the description from it (that’s probably XSLT).
    • get dependency information from the build-metadata.
    • read the existing port Makefile to get FreeBSD-metadata.
    • write a new pkg-descr and Makefile.
    • make distinfo and run some FreeBSD-preparatory scripts.

See, with one bit of tooling, we can lighten our maintainence burden and make our package descriptions more consistent with upstream. It’s a win-win!

Reconciling Packaging Data

At the Randa sprint 2016, we’re talking about KDE om all the platforms. And I’m here to represent the traditional packaging scheme. FreeBSD ports are created by writing a Makefile which contains some settings and then invokes some magic. The Porter’s Handbook describes how.

But here in Randa there are efforts on doing Continuous Integration not only on Linux but also on Windows, and there’s talk of automation of package building (packages of what? for which devices? we don’t know yet). So I thought I’d take a look at a particular port Makefile and try to figure out what parts of the information in that Makefile come from — or could be automatically sourced from — somewhere else.

Here’s the port Makefile for devel/kf5-bookmarks (that’s the internal name for the port). I’ve numbered the lines, and will be commenting inline.

     1  # Created by: Tobias Berner
     2  # $FreeBSD: branches/plasma5/KDE/devel/kf5-kbookmarks/Makefile 12693 2016-06-13 18:31:53Z adridg $

These first lines are comments, with an SVN tag that’s been expanded. Seems that I was the last person to touch this port Makefile. This is boilerplate.

     4  PORTNAME=       kbookmarks
     6  CATEGORIES=     devel kde # kde-frameworks

Here’s a really important item: the name of the port. This is the only identifier for the port that we can find inside the Makefile. The categories are a way of organizing the ports; in this case, devel must be listed because the port is devel/kf5-kbookmarks, and kde is because it’s from us. So line 4 is important, and 5 and 6 are boilerplate.

     8  MAINTAINER=     kde@FreeBSD.org
     9  COMMENT=        KF5 library for bookmarks and the XBEL format

Line 8 is boilerplate again, and then line 9! That’s something that could be clipped straight from the metainfo.yaml file. After all, there’s a description: line in there with pretty much the same text. So, knowing that the port is kbookmarks, if we know what repository to get stuff from for kbookmarks, we could get the metainfo and extract the information for the COMMENT.

    13  USES=           cmake:outsource compiler:c++11-lib kde:5 tar:xz
    14  USE_QT5=        buildtools_build core dbus gui linguisttools \
    15                  qmake_build widgets xml

This is dependency and build information. We use various packages for building and set options on those packages: usually the KDE tarballs are xz-compressed (not always: that’s one of the unpleasant surprises packagers have to deal with; at least BSD-tar has a ‘z’ flag that’s smart); we’re doing an out-of-source cmake build. That kind of stuff.

The Qt dependencies are fairly fine-grained. This is a consequence of the BSD run-time linker, which doesn’t like transitively linked shared libraries. In other words, linking an application to QtWidgets (and QtWidgets links to QtCore) doesn’t link the application to QtCore. If the application also uses QtCore symbols, then it needs to explicitly link to QtCore.

I’ll write more about dependencies below.

    16  USE_KDE=        codecs config configwidgets coreaddons ecm \
    17                  iconthemes widgetsaddons xmlgui
    18  KDE_DIST=       frameworks

Here, KDEDIST distinguishes between KDE Frameworks, KDE Plasma and KDE Applications, since they are delivered in slightly different ways. For instance, the Frameworks come with translations, while Applications don’t. So that variable controls some settings elsewhere. USE_KDE is the list of dependencies within KDE that this port has. This list uses the same names as you’d find in PORTNAME; a later KDE Framework or Application that needs bookmarks would list USE_KDE=bookmarks. More about dependencies below.

    21  OPTIONS_SUB=    YES

Let’s call this boilerplate. Frameworks have translations, and we have an option, NLS, for building and installing translations (or not). So Frameworks all have this. Applications don’t, because they have translations shipped separately.

    23  .include <bsd.port.mk>

Line 23 is where the magic happens.

So to sum up:

  • Lines 4, 6 and 18 establish the identity of this port: it’s devel/ in the filesystem, the resulting package is called kbookmarks and it’s a KDE Framework.
  • Lines 13-17 are build and runtime dependencies.
  • The rest is boilerplate, though some of it depends on KDE_DIST being equal to “frameworks”.

Back to dependencies. There are (at least) two ways that we can find dependenciesy information about a KDE Framework. One is to try to build it and to see where CMake complains. The other is to look at the kde-build-metadata repository. The kde-build-metadata repository contains a file where KDE documents the dependencies within the collection of KDE source repositories. So knowing that I’m working on the port for framework kbookmarks, I can query in-KDE dependencies with the tool list_dependencies.

Wouldn’t it be great if CMake’s find_package() also wrote a (machine-readable) record of what package was looked-for and whether it was optional or required? (And, while I’m dreaming, whether it was for build- or runtime use). That way you could get a straight-from-the-source list of dependencies.

Lacking that, list_dependencies uses a dependency file maintained by the KDE community. And using it, I find some surprising differences between the dependency information in there, what CMakeLists.txt says, and what our port Makefile says.

First off, the CMakeLists.txt for kbookmarks says:

find_package(Qt5 ${REQUIRED_QT_VERSION} NO_MODULE REQUIRED Widgets Xml DBus)

find_package(KF5Config ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5CoreAddons ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5Codecs ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5ConfigWidgets ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5IconThemes ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5WidgetsAddons ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5XmlGui ${KF5_DEP_VERSION} REQUIRED)

Maybe there’s more find_package calls, I didn’t look everywhere. But this is a clear statement of dependencies. Matching these lists against our ports Makefile shows:

  • Each direct dependency on another framework is reflected in USE_KDE’s list of frameworks. That’s good!
  • Each Qt component explicitly looked-for can also be found in USE_QT5. That’s good!
  • But there are other entries in USE_QT5. That’s because kbookmarks links (privately) against Qt5DBus. Possibly we could drop core from USE_QT5, and there’s three build-tools listed (I’m not sure linguisttools is a runtime requirement). So this is almost good: I get the feeling there’s a missing find_package, and our list is a little on the bulky side.

Turning to the information in kde-build-metadata, I can ask for direct dependencies, or transitive dependencies. Here’s the direct list:


This is roughly the same list as spotted in the CMakeLists.txt and in the port Makefile, but there’s a catch (or two):

  • The Qt (sub)module information is lost. This is documented at the top of the dependency-information file, so it is not surprising, but it is a little annoying.
  • The frameworks KF5Config and KF5Codecs have vanished, in spite of being explicitly searched for in CMakeLists.txt. There’s an explanation for this, because frameworks/kxmlgui depends on them (and lots else like kitemviews). So it looks like transitive dependencies have been removed here (this is not systematically done: kcmutils depends both on kxmlgui and kitemviews).

So, at least from my point of view, the dependency information could use some refinement (and/or expansion). But that’s why I’m in Randa! So I can sit down with the people most-responsible and figure out what we need, and what can be provided. You can support bringing together the right people in Randa though this year’s fundraiser.

GPG key transition

Some time ago I blogged about resuming the use of my GPG key (with short-ID FEA2A3FE, which I always thought was a neat ID). On that post someone commented that the key way weak and didn’t follow current best practices (no wonder, the key was created in 2002). A recent LWN article on key-ID collisions made me realise it was time to take some action. So here’s a statement, signed with my old and new key. I am aware of the irony of serving those up over HTTP.

The GPG key for Adriaan de Groot is now

sec 4096R/0x7FEA3DA6169C77D6 2016-06-11 [expires: 2018-06-11]
Key fingerprint = 00AC D15E 25A7 9FEE 028B 0EE5 7FEA 3DA6 169C 77D6

it used to be

sec 1024D/0x76ACEE01FEA2A3FE 2002-06-18
Key fingerprint = 934E 31AA 80A7 723F 54F9 50ED 76AC EE01 FEA2 A3FE

I was kind of attached to my short key-ID, so I’m glad to see FEA3 return in my new long key-ID. The new key has been sent to the sks-keyservers pool and should be available where ever needed. While I don’t expect to use much GPG, those things that need signing will be signed with the new key.

I’m going to Randa!

Like Christoph, I’m going to Randa! It’s a long train ride, but that means I can get some hacking done on the way.

Randa Fundraiser Image

You can support the Randa meetings! Click on the image for fundraiser information. The Randa meetings are one of the biggest sprints in KDE. Each year a tightly focused group gets together to work on KDE technology for one goal. This year the goal is KDE technology on every device.

While most of the participants seem to be going to the meeting for the purpose of getting more KDE applications on Windows, MacOS or Android — indeed platforms where our technology can make a difference for developers and where our applications can make a difference for Freedom — I’m going with a slightly different purpose. I’m there for our traditional niche platforms: the BSD’s. But also for packaging in a traditional sense, and for building our software effectively and efficiently.

There’s a lot of infrastructure in the KDE software (git) repositories, information about dependencies and build orders and where to find sources and stuff like that. For the traditional packagers, though, packaging is largely an artisanal process: discover what software is released this time, under what names and in which directory; figure out what new dependencies there are by trying to compile the new stuff in an environment that worked for the previous release; adjust sources for invalid assumptions. Lather, rinse, repeat.

Speaking of repeat, Scarlett has been working on reproducible builds of KDE software for Debian. FreeBSD is part of the same organization, so I hope to learn a lot from Scarlett towards that goal.

If containerized apps are really a thing, I’d hope to be able to build FreeBSD(-ish) containers from the same metadata as other containers are built from.

As preparation, I tried to build KDE (as in, the software stack needed for Zanshin) from source on a Linux distro. I failed. To me, that suggests that we shouldn’t forget the devices running Linux, either, and the packagers that work there.

Posted in KDE