Everything old is new again

Just because KDE4-era software has been deprecated by the KDE-FreeBSD team in the official ports-repository, doesn’t mean we don’t care for it while we still need to. KDE4 was released on January 11th, 2008 — I still have the T-shirt — which was a very different C++ world than what we now live in. Much of the code pre-dates the availability of C++11 — certainly the availability of compilers with C++11 support. The language has changed a great deal in those ten years since the original release.

The platforms we run KDE code on have, too — FreeBSD 12 is a long way from the FreeBSD 6 or 7 that were current at release (although at the time, I was more into OpenSolaris). In particular, since then the FreeBSD world has switched over to Clang, and FreeBSD current is experimenting with Clang 7. So we’re seeing KDE4-era code being built, and running, on FreeBSD 12 with Clang 7. That’s a platform with a very different idea of what constitutes correct code, than what the code was originally written for. (Not quite as big a difference as Helio’s KDE1 efforts, though)

So, while we’re counting down to removing KDE4 from the FreeBSD ports tree, we’re also going through and fixing it to work with Clang 7, which defaults to a newer C++ standard and which is quite picky about some things. Some time in the distant past, when pointers were integers and NULL was zero, there was some confusion about booleans. So there’s lots of code that does list.contains(element) > 0 .. this must have been a trick before booleans were a supported type in all our compilers. In any case it breaks with Clang 7, since contains() returns a QBool which converts to a nullptr (when false) which isn’t comparable to the integer 0. Suffice to say I’ve spent more time reading KDE4-era code this month, than in the past two years.

However, work is proceeding apace, so if you really really want to, you can still get your old-school kicks on a new platform. Because we care about packaging things right, even when we want to get rid of it.

.. in with the New

It’s been a long time since I wrote anything about the state of current KDE software in FreeBSD. So, without further ado:

  • Qt is now at 5.10.1; work on 5.11 is proceeding. Right now, WebEngine is still at 5.9.5, and that’s remaining so for 5.11 unless we have a sudden influx of free time to work on that monster.
  • KDE Frameworks are at 5.49, which is the august 2018 release; 5.50 hasn’t come out yet but we’re ready for them when they do.
  • KDE Applications are at 18.08.0, which again is the august 2018 release; 18.08.1 was just released today.
  • KDE Plasma is at 5.12.5, so we’re one whole minor release behind (5.13.5 came out on tuesday). This is actually blocked by libinput, which is lagging in FreeBSD.
  • KDevelop version 5.2.3 is one patch level behind

So except for the Qt version, we’re keeping up reasonably well with the modern stuff. And we’ve finally joined most of the Linux distributions in deprecating KDE4 software. For KDE4-using ports that are not “ours”, we’re encouraging other ports maintainers to update them (e.g. to KF5-enabled versions) or follow in deprecating the software.

Out with the Old ..

KDE4 ports will be removed from FreeBSD ports on December 31st, 2018

The KDE-FreeBSD team met at Akademy this year, and while hacking on some other stuff, we also got around to deciding what to do with the KDE4 ports. We have wanted to get rid of them for some time, and there is an increasing pressure of maintainence on them: code written in 2003 doesn’t play well with the C++ of 2018 (in particular clang keeps getting more picky, which is good).

As for KDE4 itself: there haven’t been any upstream KDE4 releases since Applications 17.08.3, and Qt4 upon which it depends is EOL since 2015. The latest KDE Plasma desktop has been available in the official ports tree for over four months (and has been in use by users of Area51 for much much longer).

So, given that there is a viable upgrade path (although, truth be told, you’ll probably have to re-configure KMail and get used to Falkon), we’ve decided to put a four month deprecation period on all the KDE4 ports. They will be removed at the end of this year, which will free up some maintainence time for chasing the steady stream of updates from the KDE community.

My First Clang Bug

Part of the role of being a packager is compiling lots (and lots) of packages. That means compiling lots of code from interesting places and in a variety of styles. In my opinion, being a good packager also means providing feedback to upstream when things are bad. That means filing upstream bugs when possible, and upstreaming patches.

One of the “exciting” moments in packaging is when tools change. So each and every major CMake update is an exercise in recompiling 2400 or more packages and adjusting bits and pieces. When a software project was last released in 2013, adjusting it to modern tools can become quite a chore (e.g. Squid Report Generator). CMake is excellent for maintaining backwards compatibility, generally accomodating old software with new policies. The most recent 3.12 release candidate had three issues filed from the FreeBSD side, all from fallout with older software.  I consider the hours put into good bug reports, part of being a good citizen of the Free Software world.

My most interesting bug this week, though, came from one line of code somewhere in Kleopatra:

Q_UNUSED(gpgagent_data);

That one line triggered a really peculiar link error in KDE’s FreeBSD CI system. Yup .. telling the compiler something is unused made it fall over. Commenting out that line got rid of the link error, but introduced a warning about an unused function. Working with KDE-PIM’s Volker Krause, we whittled the problem down to a six-line example program — two lines if you don’t care much for coding style. I’m glad, at that point, that I could throw it over the hedge to the LLVM team with some explanatory text. Watching the process on their side reminds me ever-so-strongly of how things work in KDE (or FreeBSD for that matter): Bugzilla, Phabricator, and git combine to be an effective workflow for developers (perhaps less so for end-users).

Today I got a note saying that the issue had been resolved. So brief a time for a bug. Live fast. Get squashed young.

(wanted) Poudriere Workflow Support

One of the premiere tools in FreeBSD CI work is poudriere. It’s a collection of shell scripts that leverage FreeBSD jails (chroot on steroids; build-in containers) and ZFS to build ports in a clean environment. You can also “cross” build for different versions of FreeBSD (e.g. on my 11-STABLE box, I can also build 10-STABLE and 12-CURRENT, although forward-compatibility to 12- can be tricky because of kernel changes). It will even truly cross build, but that’s beastly slow due to QEMU (e.g. my Skylake box can just about keep up with my Pine64). And it supports multiple ports trees, so you can do builds for the official ports tree and for local experiments too.

A typical invocation of poudriere looks like this:

poudriere bulk -j 111amd64 -p github-kde -t -i -C math/freemat

That means “build packages in the 111amd64 jail, from the ports tree called github-kde; test each port, afterwards give an interactive shell, remove and rebuild a clean version of math/freemat“. The -C is generally only used when building something multiple times in the same environment as you refine a port. There’s also a -c “rebuild the whole world” (also known as “damn, I may as well go to bed then”) flag.

Poudriere will grind away at dependencies and everything, and in the end spits out a nicely colored status line; it looks like this (here, I was rebuilding octave in order to test Qt5 compatibility, and most of the dependencies were already done).

Screenshot of Poudriere's console outputDuring the build, poudriere can run hooks in response to various events. Those events include build success, or failure (in various flavors). Using the hooks, it’s easy to move the errors to a separate directory, to end up with the build logs of those things that actually failed. Tobias has done that, and we end up with a directory listing like this:

We call a listing like this fallout. Ports that fail during a poudriere run. Since the poudriere run is often a test-run for the upgrade of an important package (e.g. upgrading CMake is tested by building 2500+ packages), handling the fallout afterwards is important: we need to go through each failed port and figure out why it has failed.

In the screenshot above, Coin was a C++-compatibility issue; so were freeMat and ampas. Apviv is hilariously silly bad C++ code (from 2014, so only in today’s context is it bad). .. and so on, and so on. Most recently, I was working from the top of the list, Tobias from the bottom, fixing actual CMake issues and optionally fixing non-CMake issues (e.g. all the dodgy C++ code). And then it struck me, we need better tool support for our very simple workflow.

Wanted!

What we need is a way to associate two pieces of data with each entry in that directory.

  • Who has “claimed” the entry (file) to look at. This is just to prevent double efforts. It should also be possible to “unclaim” an entry.
  • Tags on an entry, explaining what the problem is (e.g. “derp++”, “CMake”, “Upstream gone”).

Screenshot showing possible design for workflow tool.

Kolourpaint don’t fail me now! Three failed ports, with an image indicating who has claimed them and some tags.


In a way, it’s like an issues tracker, only slightly more free-form. There doesn’t have to be any kind of persistence: the workflow applies to one run of poudriere and the next one is basically independent. It might be nice to have user-selectable nicknames, but the owner could be indicated as an IP address, or a hash, or a color .. perhaps the set of possible tags should persist from one incarnation to another.

Does anyone know of an existing tool that does this?

If all else fails, I may sit down with Cutelyst and see what I can do there (or in Pyramid, or whatever; most of the work is probably in the CSS and Javascript, with only a very small core to serve up the page and handle the AJAX requests).

KDE on FreeBSD – June 2018

The KDE-FreeBSD team (a half-dozen hardy individuals, with varying backgrounds and varying degrees of involvement depending on how employment is doing) has a status message in the #kde-freebsd channel on freenode. Right now it looks like this:

http://FreeBSD.kde.org | Bleeding edge http://FreeBSD.kde.org/area51.php | Released: Qt 5.10.1, KDE SC 4.14.3, KF5 5.46.0, Applications 18.04.1, Plasma-5.12.5, Kdevelop-5.2.1, Digikam-5.9.0

It’s been a while since I wrote about KDE on FreeBSD, what with Calamares and third-party software happening as well. We’re better at keeping the IRC topic up-to-date than a lot of other sources of information (e.g. the FreeBSD quarterly reports, or the f.k.o website, which I’ll just dash off and update after writing this).

In no particular order:

  • Qt 5.10 is here, in a FrankenEngine incarnation: we still use WebEnging from Qt 5.9 because — like I’ve said before — WebEngine is such a gigantic pain in the butt to update with all the necessary patches to get it to compile.
  • Our collection of downstream patches to Qt 5.10 is growing, slowly. None of them are upstreamable (e.g. libressl support) though.
  • KDE Frameworks releases are generally pushed to ports within a week or two of release. Actually, now that there is a bigger stack of KDE software in FreeBSD ports the updates take longer because we have to do exp-runs.
  • Similarly, Applications and Plasma releases are reasonably up-to-date. We dodged a bullet by not jumping on Plasma 5.13 right away, I see. Tobias is the person doing almost all of the drudge-work of these updates, he deserves a pint of something in Vienna this summer.
  • The freebsd.kde.org website has been slightly updated; it was terribly out-of-date.

So we’re mostly-up-to-date, and mostly all packaged up and ready to go. Much of my day is spent in VMs packaged by other people, but it’s good to have a full KDE developer environment outside of them as well. (PS. Gotta hand it to Tomasz for the amazing application for downloading and displaying a flamingo .. niche usecases FTW)

CMake 3.12 Update on FreeBSD

CMake 3.12 has reached rc1. That means we’re testing the update on FreeBSD, and building lots and lots of packages. And, as I’ve written previously, every CMake update triggers a bunch of interesting software findings.

As a motto, I’ve got “use it, aggressively improve it” on my website (you can hire me for odd CMake and C++ jobs, too). So hitting compile issues makes me turn to fixing software outside of KDE.

  • Spring is a 3D RTS engine, with only a minor CMakeLists fix — CMake 3.12 is strict about file(GLOB) and the FOLLOW_SYMLINKS keyword, which is documented only for file(GLOB_RECURSE). Since CMake 3.5, probably much earlier, that keyword has been ignored, and now it’s an error (this is considered a regression).
  • Coin3D is a 3D toolkit, which in the version currently available on FreeBSD, doesn’t even use CMake. It hit a bunch of Clang6 compatibility issues, and after some investigation it turns out they had all been fixed already in later releases; I put in a little time to improve FreeBSD compatibility for the next release.

What I found interesting in those two was once again the variety in CMake styles — “Modern CMake Style” still needs to catch on in many places, and the wider ecosystem. Mantis bug-tracker! Mercurial! I remember being a big Mercurial fan years ago when doing KDE-Solaris and complaining how obtuse git is. (It’s still obtuse, but I’m used to it now).

There’s another four dozen ports that have fallout from this update; amusingly Kitware’s VTK 5 and VTK 6 are among them — although a first glance tells me that’s C++ problems and not CMake problems, actually. (Generally, using NULL in places where you want to write 0; older macro definitions of NULL would re-write to something that could successfully be cast to 0, but clang6 in C++17 mode, the default, uses nullptr which doesn’t cast).

Other People’s Work

Most of my writing on this blog is about FreeBSD, KDE, or Calamares. So it gives a bit of a one-sided view of what I do. There’s lots of pictures of rhubarb crumble, for instance, that never see the bloggy light-of-day. But I can build more than just software! Two months ago an unusually heavy storm blew down part of the fence in my back yard, which wasn’t really good for the privacy of that yard.

Photos of fence

I could pretend this fits into the KDE privacy goals, but really I just want to show off that yes, I can dig post holes, cut lumber, hammer and fasten. While doing so I also found tomatoes, cilantro and dill growing in the yard like weeds, so that’s a bonus.

Software thingies:

Since Daniel Nicoletti keeps writing about Cutelyst, I took a stab at a FreeBSD port, since web-frameworks should be plenty portable. Well, except for the logic failure that UNIX AND NOT APPLE means LINUX. After a half-hour or so of trying to get FreeBSD’s libepoll-shim to be used, I noticed that the shim API is incomplete, so I just punted LINUX out of there. After a minor code update to deal with implicit includes, I’ve got a port file for it that needs some polishing for porter’s-handbook compliance. Expect Cutelyst in the FreeBSD ports tree within a few days.

Atelier and AtCore are active, so here’s a blue-blobs picture of what’s going on (since january). Plenty of commits from the core developer, and some incidental contributions. Lays is less visible in the contribution blobs recently, but that’s probably because of fun Free Software events.

Screenshot of activity-blobs

So that’s what everyone else has done (well, some of everyone else; I’ll leave broader coverage to Nate). Next week, come back for bicycle repairs and Calamares releases.

Welcome x11/kde5 to the FreeBSD Ports Tree

Desktop wallpaper with Konqui and FreeBSD LogoThere is no KDE5. There are KDE Frameworks 5 (releasing monthly, now reaching version 5.45) and KDE Plasma Desktop 5 (releasing quarterly, I think, now 5.12) and KDE Applications (releasing semi-anually, called 18.04).

For the FreeBSD ports tree, there is a x11/kde5. It is a metaport, which means it collects other ports together; in this case, x11/kf5-frameworks (metaport for all the frameworks), x11/plasma5-plasma-desktop and a fistful of KDE Applications metaports (e.g. the metaport for KDE games, and the metaport for KDE graphics applications, and the metaport for what-we-consider-essential KDE applications like konsole, konqueror, dolphin, and okular). So, from a bare FreeBSD installation, installing x11/xorg, x11/sddm, and x11/kde5 should get you close to a working modern KDE Desktop experience. Throw in www/falkon and devel/kdevelop for a developer workstation, or graphics/krita for an artists workstation, and you’ve got a daily driver.

I’m not really happy with naming the port this way, but it is just an identifier that needs to distinguish it from other bits and pieces on the system. And our naming was historically such a mess that it’s taken a long time to sort out; there are still some odd corners like x11/kde-baseapps.

KDE4-style exit buttonWhat this means is that KDE4 users and modern KDE Desktop users can now be separated out effectively with the KDE packages on FreeBSD. Since there’s no more upstream releases of KDE4-era code and Qt4 was end-of-lifed long ago, I can see us going the same route as Ubuntu (and Debian, and basically everyone else) soon-ish (in FreeBSD time) and handing the whole unmaintained stack to people with an active interest in maintaining it themselves, or dropping it.

Photo of FOSDEM booth with Tobias

The elusive Tobias at FOSDEM

Special and exceptional thanks needs to go to Tobias Berner for pushing the last bits to the official ports tree and for catherding this process over the course of several years (of course, you could run a modern KDE Desktop from Area51 since 2016 or so, but not from the official ports tree).

So, what’s next? Well, in no particular order:

  • Qt 5.10 in the official tree. We’ve pushed it quickly to the KDE-FreeBSD CI systems so that git master can continue to build, but it needs to go to the official ports tree too. Main issue is dealing with WebEngine (no surprise there), so we’re looking at Qt 5.10 and the unmodified WebEngine from 5.9 — a FrankenEngine, which, for all its frightening and unnatural connotations, is probably the right name for it anyway.
  • Improving overall system integration and dealing with papercuts.
  • Chasing our CMake and KDE bug lists.
  • Bringing Wayland to fruition on FreeBSD. This in cooperation with the Mesa and GNOME teams.
  • Fixing ports and things all over the tree as we bump into things (I’ve spent some time with FreeRDP recently, and should say thanks to Kyle Evans for taking my throw-them-at-the-wall patches and making them stick).

That’ll keep us busy through 2018.

CMake 3.11 in FreeBSD

The latest release of CMake has landed in FreeBSD. Prior to release we had good contact with KitWare via the bug tracker so there were few surprises left in the actual release. There were still a few last-minute fixes left, in KDE applications no less. Here is a brief summary of changes we made:

  • FindQt doesn’t match the way FreeBSD ports are built and installed, so we defer to QMake rather than looking for directories,
  • FindOpenMP gets a tweak because it won’t find the system pthreads library when gcc (for Fortran) is used,
  • FindBLAS gets a larger tweak because BLAS may need to link to libgcc_s, and which gcc_s that is needs to be figured out via ldd(1).

Some older patches have gone away because upstream has picked them up. Tweaks downstream, in package-building-terms, of cmake that were necessary:

  • check_include_files respects the required libraries, which can be a surprise when the required libraries have been found but not fully plugged into the build (e.g. missing -L flags).
  • the order of includes in automoc sources has changed, which reveals places where a C++ header file doesn’t actually include all of the headers it needs to fully define the types it uses; previously the include order might implicitly include them and the issue is papered over.

CPack now fully supports producing FreeBSD packages from a build / install tree by default, so for non-ports software which uses CMake, cpack -G FREEBSD does the right thing. Previously, this was a non-default tweak to CPack as built in FreeBSD ports.
Edit: as of CMake 3.11.0_1, CPack no longer supports producing FreeBSD packages. There were some unexplored corners of the build process that cause build failures when the FreeBSD pkg(8) support is enabled. So it’s off again until we shine some more light into those corners.

.. and a PS., CMake 3.11.1 has just been released, which reverts the change to check_include_files which I’d been working around.