Akademy: the technical bits

Agh, when I wrote about the social bits of Akademy in Berlin, I intended to follow it up closely with a post on the more technical things I did at the conference. Sadly, that escaped my grasp and much has slipped my mind, too. There was GPG keysigning, and I finally wrestled gpg into shape on FreeBSD at home to be able to use my new keys (when it works, it’s fine, when it doesn’t, the error messages are maddeningly meaningless).

There was a great KMail / KDEPIM BoF, which combined a run-through of the options in KMail — led by Dan — to tally which ones made sense in the modern world. It led to two lists, one of “does this work?” and another of “get rid of it.” There was also a regular stream of “can it do that? neat!” KMail is huge, and it has a lot of really neat features that don’t get a lot of promotion. And during the session David piped up with “yup, killed that one” (which was the intended-as-a-demonstration-plugin never-meant-for-release ROT-13 encryption plugin).

There was more technical stuff, packaging, trying to work on CI, but it’s a little hazy by now. I can still recommend the wrap-up page, which also has the videos of each afternoon’s summary session.

Posted in KDE | Comments Off on Akademy: the technical bits

Akademy: the social bits

So far at Akademy and QtCon, I’ve been quiet on the blogging front. That’s mostly because it’s been really really busy from morning to night with technical and social things. Quick recap of the social bits:

  1. Jonathan rented a non-party flat, and having seven (eight?) KDE peeps in a flat is pretty cool. Nicer than hostels and hotels, at any rate.
  2. I’ve written one bedtime story (about Sally the Stegosaurus), but wasn’t really satisfied about the plot.
  3. Helio wants cookies and/or brigadeiro, and I have not yet gotten around to making any.
  4. Scarlett, Hanna and Rohan make the best grocery commando’s a guy could wish for. The resulting rice and veg-curry was most excellent — just a little too hot for Austria. For a lark we also made chocolate-covered pickled piri-piri peppers, but those were a little disappointing.
  5. Chairing sessions means you sometimes hear talks you wouldn’t choose normally, and they can be awesome too. I learned a lot about medical devices, multithreading, PR, KDE Neon, and GIS widgets that I otherwise would have missed.
  6. “Frozen” breakfast cereal with sweetened condensed milk and Nutella is the closest thing you can make to Calvin’s “Chocolate Frosted Sugar Bombs”
Posted in Bla Bla, KDE | 2 Comments

Going to Akademy

I’m going to Akademy! Akademy 2016, as part of QtCon, that is. I missed last year in A Coruña because it conflicted with my family summer vacation, but this year is just fine (although if I was a university student I’d be annoyed that Akademy was smack-dab in the middle of the first week of classes — you can’t please everyone).

Two purely social things I will be doing are baking cookies and telling stories about dinosaurs. I have a nice long train ride to Berlin to think of those stories. But, as those of you who have been following my BSD posts know, the dinosaurs are not so backwards anymore. Qt 5.6 is doing an exp-run on FreeBSD, so it will be in the tree Real Soon Now ™, and the Frameworks are lined up, etc. etc. For folks following the plasma5 branch in area51 this is all old hat; that tends to follow the release of new KDE software — be it Frameworks, or Plasma, or Applications, or KDevelop — by a few days. The exciting thing is having this all in the official ports tree, which means that it becomes more accessible to downstreams as well.

Er .. yeah, dinosaurs. Technically, I’m looking forward to talking about Qt on BSD and KDE Plasma desktop and other technologies on BSD, and about the long-term effects of this year’s Randa meeting. I have it on good authority that KDE Emerge^WRunda^W KDE Cauldron is being investigated for the BSDs as well.

Posted in Bla Bla, KDE | 1 Comment

Time flies for FBSD updates, too

So it’s been some time since I last wrote something about the KDE-FreeBSD ports. Time flies when you’re having fun (in my case, that means biccling around Zeeland and camping). In the meantime, Ralf has been hammering on the various Qt ports updates that need to be done — official ports are still Qt 5.5.1, but we have 5.6 done and 5.7 on the way. Tobias has been watching over the plasma5 branch of the unofficial ports, so that it’s up-to-date with KDE Frameworks, Plasma and Applications.

The older KDE stuff — that is, KDE4, which is still the current official release for the desktop on FreeBSD — is also maintained, although (obviously) not much changes there. We did run into a neat C++-exceptions bug recently, which was kind of hard to trigger: running k3b (or ksoundconverter and some other similar ones) with a CD that is unknown to MusicBrainz would crash k3b. There’s not that many commercially-available CDs unknown to that database, so I initially brushed off the bug report as Works For Me.

Good thing Methyl Ethel is still relatively obscure (but a nice band to listen to). They trigger the crash, too.

At issue is the visibility of C++ exception symbols; when -fvisibility=hidden is used and C++ exceptions are thrown out of a shared object, then users of that shared object may not catch the exceptions. Making exception classes explicitly visible fixes the problem — which in our case was because libkcddb catches exceptions from MusicBrainz, except when -fvisibility=hidden is used in linking and executable that uses libkcddb and libmusicbrainz5, in which case the exception isn’t caught and ends up crashing the application.

https://gcc.gnu.org/wiki/Visibility has been very useful in sorting this bug out. After Tobias identified the exceptions as the root cause, I played with code across the shared-object-dependency chain until I came up with a patch which I’ve proposed upstream (e.g. libmusicbrainz5) and in the packaging for FreeBSD.

Even older software sometimes shows interesting bugs. And time flies when you’re having fun, but also when tracking down unhandled exceptions.

Posted in FreeBSD, KDE | Comments Off on Time flies for FBSD updates, too

Farewell to the Mountains

So the last train has left Randa, and we can look back at a — judging from the bugs that were fixed and the ideas that were traded and the code that was written — successful sprint. The last two days were characterized by the authentic Randa Internet experience, in which a mountain goat eats 18% of the packets, but that doesn’t stop KDE developers from writing code and sharing AppImages.

Peter Simon from AppImage stopped by to give a talk and lend a hand in building AppImages for those that wanted them. Talking with him has clarified a few ideas in my mind about how I’d do something similar in FreeBSD. It’s also clarified the rationale behind these bundling (upstream-) application formats: a <foo> application image is for when you need the latest version of a specific application for a specific purpose, or when you’re testing an application; in both cases you want nothing else to change. These are not my own words: they have been expressed several times on lwn.net recently, and Boud puts it eloquently on his own.

Given that, we can conclude that a given user will have only a few to a dozen <foo> application images in use.

With those assumptions, certain objections to the whole format go away. Not all of them, but enough to make me wonder if something like that could be done for FreeBSD, and if it would make testing easier.

Saturday I played the cook; I have tremendous respect for Hadrien who ran the show the rest of the week by himself (it is a full-time job). Fortunately I had great help in the kitchen: Mario’s parents who helped clean and drove me to the grocery store. Ayrton and Lays who convinced me to make brigadeiro. Grace and Scarlett who minded pans and cut potatoes. Alexander and Pinak who also stirred and scoured. If I’ve forgotten anyone, my apologies: it’s your help that made the community kitchen a success, too.

The following things were developed for saturday dinner: schwytserbrigadeiro, with 85% cocoa Swiss chocolate, doppelgeschnitzelteknoblauchbraten because we had leftover schnitzels and 20 cloves of garlic, and a potato-salami-raclette melt which just melted away in front of the hungry developers.

We also all got Cholera, but that is a local specialty.

Anyway, this puts me in mind that we could possibly do a cooking thing in Berlin at Akademy, if we put our minds (hearts? tongues?) to it.

Here’s a gigantic thank-you to Mario Fux and Randa Meetings e.V. for putting this week together, special thanks to Scarlett and Grace for being there at the right moment, and to all the other participants for making it a great and productive week.

Posted in Bla Bla, KDE | 1 Comment

FreeBSD in Randa

There will be a BSD-centric talk at Akademy / QtCon this september. Yay!

Since it’s rainy in Randa today, I’m going to write a little about all the bits-and-pieces of FreeBSD packaging that I worked on. The *NIX-packaging BoF has given me lots of ideas, for extracting metadata and for automating bits of packaging, but nothing has crystallized around that yet.

I’ve given a little thought to the current hotness, flakpak and snappy and appstream, wondering what parts of that could work on FreeBSD and how. I have a sketch of how we could leverage ZFS snapshots and poudriere and jails to quickly define an OS layer, a platform layer, and an application layer. Using unionfs mounts these can be combined into a single filesystem, allowing quick application testing. I’m not sure about how to wrangle permissions for such a scheme, though.

But maybe these schemes aren’t all that necessary on FreeBSD (as a distro) when all is functioning smoothly (that is a major caveat). Since I was asked a few times here in Randa how FreeBSD packaging works and what the process is, I’m writing it down here.

Normally speaking, FreeBSD has two major releases that are supported. Those are 9-STABLE and 10-STABLE, at the moment. A next major release is being prepared. That’s 11-CURRENT (not yet STABLE). Those are the operating system base systems, which in FreeBSD is a great deal more expansive than in the Linux world. There’s a kernel and a compiler included, and a good chunk of userland. Together, that forms the FreeBSD distribution. Compatibility of a distribution is guaranteed for its lifetime, which I believe is generally two years. 9-STABLE is almost done, 11-CURRENT is almost ready.

The base FreeBSD distribution can vary a great deal from one major version to another: 9-STABLE uses gcc, 10-STABLE uses clang (as does 11-, but that has a much newer clang). So while the base is stable and compatible, moving between base OSses can be a challenge.

On top of the base OS, we have ports. These are all the userland programs, libraries, whatnots that have been ported to FreeBSD. “Ported,” here, means that someone has created a Makefile for it that runs in the ports infrastructure. Packages are created from the ports; because of the differences in the base OS, packages are built for each supported release (and for each supported architecture) automatically by the FreeBSD builders.

What this effectively means is that FreeBSD could be considered a rolling-release distro (when looking at it from a Linux-distro point of view). The packages are continuously updated, based on the latest ports-Makefiles.

When things are moving smoothly, ports Makefiles get updated on time, packages are built, and a user can pkg upgrade the system regularly. With ZFS snapshots a rollback after a bad upgrade can be very simple (but pkg(8) itself doesn’t have any mechanism for this).

Area51 ports processBut how to ports get updated? There’s ports committers (people) who do this. The KDE-FreeBSD project has its own repository, area51, where the K-F team assembles ports. The area51 repository is open to more (or rather, other) people than ports committers. In area51, the trunk is the last staging area before ports are sent “downstream” to the official ports tree. In other words, trunk is the “next new thing”. For instance, when new versions of software are released that the KDE-FreeBSD maintains / packages in FreeBSD are released, that is often updated in area51 first, tested with our own package builders, and then “flushed” downstream to official ports. So there’s a little shared staging area for our ports.

Area51 also has branches for “next big new thing” work. That’s for large-scale reorganization efforts (which are sometimes necessary when we discover that things are not as co-installable as we thought, or when the names of ports and packages turn out to be poorly chosen). That’s for updates that affect a lot of ports at once, like Qt updates, or new KDE Applications updates.

The diagram (that’s my epic Kolourpaint skills there, with a crystal icon-set person-icon, and fonts from Blambot) shows roughly how this works. There are two SVN repositories, with branches visible in area51. FreeBSD committers move stuff from area51 to official ports; in between the other 22000 official ports update independently.

We’re just coming off of a long period of time in which we were bottlenecked in getting stuff out of area51 — i.e. unable to flush downstream. That’s an unpleasant situation to be in, because things get clogged up.

Users who were prepared to work with area51 ports could always use the area51 repository and keep up-to-date, but people using official ports or packages were lagging behind quite seriously. We (KDE-FreeBSD) don’t really have the capacity to serve up packages built from area51.

The situation is improving. Thanks to Randa, I’ve had time to work with Tobias Berner, Ralf Nolden, and others to wrestle a bunch of updates through to the ports system and to merge a lot of outstanding work to trunk, ready for the next iteration of flushing trunk downstream.

So here’s what we’ve achieved this week:

  • konsole patch updated and merged to ports; it’s still languishing in a reviewboard item on kde.org though. This improves support for home-dir handling and %-expansion which relies on system services.
  • QtCreator and qbs (Qt Build System) flushed downstream.
  • Last iteration of bump-KDE4-to-last-pre-Applications release prepared for downstream.
  • Qt 5.6 merged into area51 trunk.

What this effectively means is that KDE4 is about to be bumped (a little bit: there needs to be another round of reorganization before the changed form of releases under KDE Applications are supported) and that Qt 5.6 is being prepared for inclusion in official FreeBSD packages Real Soon Now.

Of course, this still lags behind on several fronts:

  • Qt 5.7 was released today,
  • We have not yet wrestled webengine into a releasable state,
  • KDE Frameworks 5 are not yet available,
  • KDE Plasma 5 is not yet available,
  • KDE Applications based on frameworks are not yet available.

These are all things that are being worked on, though! There is already a qt-5.7 branch which is close to being mergeable (but we really want webengine in shape for that). The plasma5 branch of area51 has been tracking Frameworks and Plasma and Applications closely for months, so that’s basically poised to swoop in as soon as we have some other bits flushed out.

So it’s been a good and productive week of getting those little annoyances worked out, and soon ™ I expect we can have FreeBSD back on track with the latest and greatest of KDE software. We even have a Krita port prepared!

Posted in Bla Bla | Comments Off on FreeBSD in Randa

Randa in General

One of the nice things about the Randa meetings is — for me at least — is the return to an almost student-like work scheme. Sleep, eat, code, repeat. The hacking room downstairs (there’s two more upstairs) has 16 people in it, working on all corners of KDE. People move around to form pairs as needed. The Davids are in high demand. Alexander jumps at all kinds of problems, from Python to I18n. Tomaz is doing neat things with Subsurface and explaining things to others. Just sitting here means catching up with KDE technologies and hearing tips.

The consequence of these long hacking days is that a lot of little stuff gets fixed as well. Stuff that get shunted aside at the end of a regular working day, but that fills up little gaps in the 16 hour working day that most of the Randa attendees have here. A common turn of phrase seems to be “I have this little annoyance ..” followed by some nearby participant saying “Yeah, I just fixed that.”

Keeping up 16 hour working days for a week requires a lot of organization. And the wonderful thing is is that that organization is there, and that you can roll out of bed (watch out if you’re in the upper bunk) and breakfast is ready. All the day-to-day necessities are already taken care of, and we can get to work. Catering to these day-to-day needs costs money, and you can support that (and therefore support the work that gets done) by making a donation to the Randa fundraiser.

Randa Fundraiser

A lot of the pictures you’ll see that come out of Randa are the “pretty pictures“, of the times between hacking sessions (which Lays Rodrigues describes above the pretty pictures). From here I can see 14 people behind laptops, water bottles, coffee cups, various candy wrappers, Umbrello, 3D-printing, flatpak, KMyMoney. There’s a lot of work getting done, but it’s not very photogenic.

Posted in Bla Bla, KDE | Comments Off on Randa in General

Zanshin 0.4 on FreeBSD

Zanshin 0.4 was released a little over a month ago. Today I got around to fixing up the packaging (see, this is why tooling and automation is important) for Zanshin on FreeBSD in the area51 repository. One stumbling block was a sudden requirement for C++14 while not checking if the compiler supports it. The base system clang on FreeBSD 10 doesn’t, and that led to weird and hard-to-diagnose build failures. Consistent failures, but it took a long time — and Tobias’s help — to find out that --std=c++14 was causing an error in FindLibIcal and that CMake wasn’t logging the failing compile command or the failing source code.

Anyway. It’s sorted out now, and Zanshin can be had through the unofficial ports repository. There’s no ETA for landing it (and the rest of KDE Frameworks 5) in official ports and packages, though.

Posted in FreeBSD, KDE | 1 Comment

Reconciling Packaging Data (2)

So previously I looked at the dependency data available in KDE repositories, and how that translates to information in the FreeBSD packaging system (i.e. the ports Makefiles). It seems likely that judicious use of list_dependencies can help fill in the dependency information when depending on KDE and Qt components.

“Judicious” here means something like “after the Randa *NIX-deployment meeting tomorrow”, since over lunch I learned from upstream (Scarlett) that the inconsistencies in the dependency data I noticed earlier are going to be tackled.

So it’s good to be here in Randa at the cross-platform meeting, also for the *NIX packagers.

But let’s look at two other bits of information in a FreeBSD port: the COMMENT string in the Makefile and the pkg-descr file.

KDE Frameworks all have a metainfo.yaml file, so there’s 81 of those sitting on y hard drive right now. Each of those files has a description line that could be used to fill the COMMENT. Since this is the upstream short description of the software product, that seems like a good pithy thing to use instead of coming up with our own.

The pkg-descr is a larger file, with multiple lines of text allowed describing the package. One special line in the file starts with WWW:, followed by a space, followed by the URL of the package’s homepage. That is information that isn’t available in the metadata in Frameworks. It could be available in the AppStream data, except that none of the Frameworks seem to have that, while AppStream does support generic components.

So my plan is slowly congealing, like lime-Jell-O pudding, into the following:

  • Get kde-build-metadata, on the assumption that it contains a list of all the frameworks (in particular all those frameworks released in a given iteration on KDE Frameworks)
  • For each Framework listed there:
    • grab the released tarball, and extract the description from the metainfo.yaml.
    • if there’s AppData somewhere, extract the description from it (that’s probably XSLT).
    • get dependency information from the build-metadata.
    • read the existing port Makefile to get FreeBSD-metadata.
    • write a new pkg-descr and Makefile.
    • make distinfo and run some FreeBSD-preparatory scripts.

See, with one bit of tooling, we can lighten our maintainence burden and make our package descriptions more consistent with upstream. It’s a win-win!

Posted in FreeBSD, KDE | 7 Comments

Reconciling Packaging Data

At the Randa sprint 2016, we’re talking about KDE om all the platforms. And I’m here to represent the traditional packaging scheme. FreeBSD ports are created by writing a Makefile which contains some settings and then invokes some magic. The Porter’s Handbook describes how.

But here in Randa there are efforts on doing Continuous Integration not only on Linux but also on Windows, and there’s talk of automation of package building (packages of what? for which devices? we don’t know yet). So I thought I’d take a look at a particular port Makefile and try to figure out what parts of the information in that Makefile come from — or could be automatically sourced from — somewhere else.

Here’s the port Makefile for devel/kf5-bookmarks (that’s the internal name for the port). I’ve numbered the lines, and will be commenting inline.

     1  # Created by: Tobias Berner
     2  # $FreeBSD: branches/plasma5/KDE/devel/kf5-kbookmarks/Makefile 12693 2016-06-13 18:31:53Z adridg $

These first lines are comments, with an SVN tag that’s been expanded. Seems that I was the last person to touch this port Makefile. This is boilerplate.

     4  PORTNAME=       kbookmarks
     6  CATEGORIES=     devel kde # kde-frameworks

Here’s a really important item: the name of the port. This is the only identifier for the port that we can find inside the Makefile. The categories are a way of organizing the ports; in this case, devel must be listed because the port is devel/kf5-kbookmarks, and kde is because it’s from us. So line 4 is important, and 5 and 6 are boilerplate.

     8  MAINTAINER=     kde@FreeBSD.org
     9  COMMENT=        KF5 library for bookmarks and the XBEL format

Line 8 is boilerplate again, and then line 9! That’s something that could be clipped straight from the metainfo.yaml file. After all, there’s a description: line in there with pretty much the same text. So, knowing that the port is kbookmarks, if we know what repository to get stuff from for kbookmarks, we could get the metainfo and extract the information for the COMMENT.

    13  USES=           cmake:outsource compiler:c++11-lib kde:5 tar:xz
    14  USE_QT5=        buildtools_build core dbus gui linguisttools \
    15                  qmake_build widgets xml

This is dependency and build information. We use various packages for building and set options on those packages: usually the KDE tarballs are xz-compressed (not always: that’s one of the unpleasant surprises packagers have to deal with; at least BSD-tar has a ‘z’ flag that’s smart); we’re doing an out-of-source cmake build. That kind of stuff.

The Qt dependencies are fairly fine-grained. This is a consequence of the BSD run-time linker, which doesn’t like transitively linked shared libraries. In other words, linking an application to QtWidgets (and QtWidgets links to QtCore) doesn’t link the application to QtCore. If the application also uses QtCore symbols, then it needs to explicitly link to QtCore.

I’ll write more about dependencies below.

    16  USE_KDE=        codecs config configwidgets coreaddons ecm \
    17                  iconthemes widgetsaddons xmlgui
    18  KDE_DIST=       frameworks

Here, KDEDIST distinguishes between KDE Frameworks, KDE Plasma and KDE Applications, since they are delivered in slightly different ways. For instance, the Frameworks come with translations, while Applications don’t. So that variable controls some settings elsewhere. USE_KDE is the list of dependencies within KDE that this port has. This list uses the same names as you’d find in PORTNAME; a later KDE Framework or Application that needs bookmarks would list USE_KDE=bookmarks. More about dependencies below.

    21  OPTIONS_SUB=    YES

Let’s call this boilerplate. Frameworks have translations, and we have an option, NLS, for building and installing translations (or not). So Frameworks all have this. Applications don’t, because they have translations shipped separately.

    23  .include <bsd.port.mk>

Line 23 is where the magic happens.

So to sum up:

  • Lines 4, 6 and 18 establish the identity of this port: it’s devel/ in the filesystem, the resulting package is called kbookmarks and it’s a KDE Framework.
  • Lines 13-17 are build and runtime dependencies.
  • The rest is boilerplate, though some of it depends on KDE_DIST being equal to “frameworks”.

Back to dependencies. There are (at least) two ways that we can find dependenciesy information about a KDE Framework. One is to try to build it and to see where CMake complains. The other is to look at the kde-build-metadata repository. The kde-build-metadata repository contains a file where KDE documents the dependencies within the collection of KDE source repositories. So knowing that I’m working on the port for framework kbookmarks, I can query in-KDE dependencies with the tool list_dependencies.

Wouldn’t it be great if CMake’s find_package() also wrote a (machine-readable) record of what package was looked-for and whether it was optional or required? (And, while I’m dreaming, whether it was for build- or runtime use). That way you could get a straight-from-the-source list of dependencies.

Lacking that, list_dependencies uses a dependency file maintained by the KDE community. And using it, I find some surprising differences between the dependency information in there, what CMakeLists.txt says, and what our port Makefile says.

First off, the CMakeLists.txt for kbookmarks says:

find_package(Qt5 ${REQUIRED_QT_VERSION} NO_MODULE REQUIRED Widgets Xml DBus)

find_package(KF5Config ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5CoreAddons ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5Codecs ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5ConfigWidgets ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5IconThemes ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5WidgetsAddons ${KF5_DEP_VERSION} REQUIRED)
find_package(KF5XmlGui ${KF5_DEP_VERSION} REQUIRED)

Maybe there’s more find_package calls, I didn’t look everywhere. But this is a clear statement of dependencies. Matching these lists against our ports Makefile shows:

  • Each direct dependency on another framework is reflected in USE_KDE’s list of frameworks. That’s good!
  • Each Qt component explicitly looked-for can also be found in USE_QT5. That’s good!
  • But there are other entries in USE_QT5. That’s because kbookmarks links (privately) against Qt5DBus. Possibly we could drop core from USE_QT5, and there’s three build-tools listed (I’m not sure linguisttools is a runtime requirement). So this is almost good: I get the feeling there’s a missing find_package, and our list is a little on the bulky side.

Turning to the information in kde-build-metadata, I can ask for direct dependencies, or transitive dependencies. Here’s the direct list:


This is roughly the same list as spotted in the CMakeLists.txt and in the port Makefile, but there’s a catch (or two):

  • The Qt (sub)module information is lost. This is documented at the top of the dependency-information file, so it is not surprising, but it is a little annoying.
  • The frameworks KF5Config and KF5Codecs have vanished, in spite of being explicitly searched for in CMakeLists.txt. There’s an explanation for this, because frameworks/kxmlgui depends on them (and lots else like kitemviews). So it looks like transitive dependencies have been removed here (this is not systematically done: kcmutils depends both on kxmlgui and kitemviews).

So, at least from my point of view, the dependency information could use some refinement (and/or expansion). But that’s why I’m in Randa! So I can sit down with the people most-responsible and figure out what we need, and what can be provided. You can support bringing together the right people in Randa though this year’s fundraiser.

Posted in FreeBSD, KDE | 1 Comment