Trying a new Calamares workflow

I just released Calamares 3.2.1, a new-features release, and I’m trying something new for the next release. This is vaguely inspired by Paul Adams’s talk about “The Art of the Pull Request”. Immediately after the release, I’ve added several branches, one for each set of features that I want to implement for the next release. The graph displayed by QGit looks more like an octopus than ever:

Master has three immediate branches

Each branch starts with an empty commit that just says what I indend to accomplish in the branch. That way, intentions are documented inside the code repository — portably (that is, not tied up with a Pull Request or other coding-platform feature). I can of course delete the branches later if something happens to make the branches obsolete or useless. But from here, there’s a pretty clean picture of how the next release (3.2.2) is going to work:

  • Bugfixes go to master,
  • Planned feature commits go to the relevant branch,
  • Unplanned features go to new topic branches off of master.

When a planned feature is done, I will merge that branch back to master, delete the branch, and call it a day. When all of them have been merged that way, the planned features for the release are done — and it should be ready to release.

Slightly sneaky, I can even claim “FIXES #” in the initial empty commit in the branch, and assuming I’m honest with myself about the intentions of, and effects of, the branch, when I merge to master the corresponding issues in the issue tracker will be closed automatically.

Mostly I’m curious whether working this way will help me keep the total cycle time low for the next release, while also informing others where works is “needed right now”. And if it works, then in future the graph of a Calamares release will look more like an octopus, tying its feet together. Seems like a good thing for an installer framework named after squid.

The World and Calamares

This last week, my desk mostly looked like this:

Photo of desk with atlas and monitors

WQHD and even more HD on paper.

There was a bug report for Calamares that Reykjavik was misplaced. And once I started looking at that, it was pointed out that Johannesburg was misplaced as well, and then I did some serious clicking around and found that most of the southern hemisphere was subtly off in Calamares.

Why?

The timezone-selection widget in Calamares was borrowed from some distro installer a long time ago, and at some point the images were resized, and some math corrected to map image points to latitude and longitude on the map. The resizing introduced some aliasing artifacts, and then the math moved locations far north — where a typical map projection is “stretched out” to the wrong spot. Some time ago I fixed up things above 65 degrees north or so. Reykjavik is at 64-and-a-bit north, so wasn’t handled then. And it was a bodge anyway.

Screenshot of timezone selection widget

Your Reykjavik is in another timezone

So this week I sat down with an atlas (because that provides a much nicer overview than any online map, and has latitude and longitude lines) and a list of places to test. The selection works better now, much better far south, but it’s not right yet. Basically, I’m going to have to re-draw this stuff, so I’m learning a little qgis to try to get this done (and otherwise I’ll ask a friend in Nijmegen who does GIS stuff professionally).

The atlas nearest to my desktop was Velhagen & Klasings Kleiner Handatlas, printed in Leipzig, Germany, in 1922. I didn’t feel like walking downstairs and grabbing a newer atlas. I figured the continents haven’t moved that much in the past 96 years.

What I hadn’t counted on was how much the world has changed. When I realised that, I kept the atlas open as an education.

It’s not really visible in the photograph above, but in 1922 Africa was almost entirely under the yoke of colonialism. What wasn’t “owned” by Europe, was blank. Germany had recently lost its colonies. Long-range air travel didn’t exist (although the Atlantic had been crossed by a non-stop flight by then). The number of telegraph cables across the Pacific ocean could be counted with a two-bit integer.

Photo of pages from atlas

Colonies and Steamship Routes

Remember that: the world wasn’t always as it is today, and needn’t be the way it is today, tomorrow. Strive to make it better.

Clicking around, I selected Harare, the capital of Zimbabwe. It’s been Harare since my earliest memories (insofar as I thought about cities in Africa as a child). Knowing the latitude and longitude I could look it up in my atlas (to check that its visual place on the map matches where it should be). Unfortunately, there’s only Salisbury in South Rhodesia there. Lagos, now 20 million people, is a small dot at six degrees north. All the struggles for independence are yet-to-come on this map. Almost everywhere I looked, a world that is very different.

So the thing I set out to fix in Calamares has been fixed, but the atlas is still on my desk, as a reminder to me of things that have changed, and things that haven’t.

Calamares on Krypton

Calamares is a Linux system installer (and some day, a FreeBSD system installer, but that is a long way off) which is distro- and desktop-independent. OpenSUSE Krypton is a live CD and installer for the latest-and-greatest .. but it already has an installer, so why try Calamares on it?

Well, sometimes it’s just to show that a derivative could be made (there is one, called GeckoLinux), or to experiment with tools and configurations.

Calamares has a script called deploycala.py, which like every gaping huge security hole is expected to be downloaded from the Calamares site, then run. It is recommended to only use this in a VM, with a live CD / ISO image running. What the script does is install a basic dev environment for Calamares, install up-to-date dependencies, and then it builds and installs Calamares. That then gives you a way to experiment, installing with Calamares from an already-set-up live CD.

The deploy script supports many different package managers and host systems, so it’s just a matter of running python3 deploycala.py -n to get started (and then wait for a while as packages are installed, Calamares is cloned, and then built). Calamares builds with no issues on Krypton (at least today, when I tried it).

Screrenshot of Calamares in Krypton

Calamares in Krypton (Qt 5.11)

Screenshot of Calamares in Manjaro

Calamares in Manjaro (Qt 5.10)

Having built Calamares, there’s a few bits I notice:

  • Esperanto isn’t supported in Qt applications (neither in Krypton, nor in Manjaro, nor in anything else I tested); QLocale has a constructor that takes an enum value specifying the language, but for a bunch of languages in that enum, it then creates a “C” locale. This is documented with the wriggly description “… if found in the database …”, but is rather unsatisfying.
  • Manjaro (Qt 5.10) does a better job of displaying Indic scripts than OpenSUSE Krypton (Qt 5.11), although this might be an artifact of installing updated packages into the live system.
  • The keyboard-layout picker displays no keycaps. It also doesn’t provide any useful debugging output. This is probably a combination of missing packages in the live system, and Calamares not providing enough useful feedback when the live image isn’t quite right. The latter, I can fix.

So, by briefly switching distro’s today, I’ve found one bug of my own, and one configuration thing for myself to document. And then from this not-using-Calamares distro, I can move on to another one.

Calamares and Google Summer of Code

This year Calamares is participating in Google’s Summer of Code. While Calamares doesn’t live under the KDE umbrella — for political reasons, basically, to emplasize that it is a desktop-agnostic system installer — it has a great deal of KDE DNA inside. The maintainers (that’s Teo, then myself) have been KDE people, some of the technology is definitely KDE (KPMCore in particular). So we’re happy to be participating under the KDE umbrella in a mixed KPMCore / Calamares role.

Andrius (KPMCore) and myself (Calamares) will be working with our student Caio Jord√£o de Lima Carvalho on a list of advanced partitioning scheme improvements. With this, we’ll make KPMCore a more powerful library for dealing with complicated partitioning schemes, and Calamares a more powerful installer. The “advanced” schemes here are all based around LVM (Linux Volume Manager) which is something I’ve been dreading having to deal with — so I’m really happy Caio will.

He’s pushed stuff to KPMCore before, and his first pull request to Calamares arrived this week, so I feel we’re off to a good start. Look for more on the Calamares site.

Calamares GeoIP

Calamares is a distribution-independent (Linux) system installer. Outside of the “big five” distro’s, many smaller “boutique” distro’s need an installer, and Calamares is a highly configurable one that they can use. There’s a few dozen distro’s that I know of that use it (although I’ve only actually installed maybe six of them).

One optional feature — optional at the discretion of the distro which is deploying Calamares, that is — is the use of GeoIP location to find the time zone of the device being installed. This is used to automatically set the clock, for instance. If it’s not switched on, or there’s no network, Calamares defaults to a location defined by the distro — generally New York, although there’s an Austria-based distro that I think defaults to UTC.

For years, the default and suggested GeoIP provider has been freegeoip.net. That service is shutting down — well, being upgraded to a nicer API, at a different location, and with different access limitations. Older ISO images that have Calamares configured to use GeoIP with that service will cease to get usable GeoIP data on july 1st, 2018.

I don’t actually know which distro’s have GeoIP enabled, nor what provider they use.

However, the fact that this provider is shutting down prompted me to go over the existing code and make it more flexible and more general (prodding from Gabriel and Phil and Harald helped here as well). So Calamares is now ready to use more, and different, providers of GeoIP data. The providers can’t agree on using “time_zone” or “timezone” or “TimeZone” as the attribute (JSON) or element (XML) name where the data lives, so there’s a little bit more code and configuration needed. But you (as a distro) can now shop around and pick a GeoIP provider that respects your privacy.

A Day on Krypton

It’s a bird! It’s a plane! No, it’s a shiny stable-yet-bleeding-edge KDE Plasma distro!

Since Calamares has to run all over the place, and is used in derivatives of all of the “Big Five” Linux distributions, I regularly switch distro’s as a development platform. Also because I inevitably blow up the VM while running Calamares, or because an update renders the system useless. At FOSDEM I had the pleasure of chatting with the folks from the SUSE stand about OpenQA and OBS.

(Note, when I originally wrote this I was going to just fiddle around a bit and then return to my Manjaro dev VM; instead it’s turned into a week and Krypton is likely to stay lodged on my VMs and spare machines for the foreseeable future.)

Last week I spent the day with openSUSE Krypton, which is a almost-bleeding-edge KDE Plasma desktop (today’s version has Plasma 5.11.5) on top of openSUSE’s rolling-release, Tumbleweed. Most of my Linux systems (e.g. the kids gaming boxes) run openSUSE of some sort, as did all my work systems at my previous job, but I have not yet used it as a development platform for Calamares. Here’s some usage notes.

Day 1 First day with a distro is usually roughly the same: install it, copy some stuff over, install tools, checkout and build Calamares. With Krypton, it’s no different.

  1. Installation looks a little wonky here and there. The installer could use a careful go-over by a designer to smooth out lines, reduce drawing glitches, etc. It may have been an artifact of installing in an 800×600 VirtualBox window, but it didn’t seem very polished, even if the installer procedure was.
  2. Install basic development tools: zypper in git cmake make gcc gcc-c++. Huh, kdevelop is already installed, that’s a good sign (except it seems like it’s broken, and can’t find the plugin KDevWelcomePage, but see below). Shame Linux systems are otherwise so poorly prepared for being development systems.
  3. Run deploycala.py on the installed system (there’s big fat warnings saying never to do that, but I’m the developer and this is a fresh VM, so nyah nyah). Fall over backwards when it turns out that apt-get exists on this system (and invokes zypper via aptitude) so that the deploy script thinks it’s on Debian and is going to do all of the wrong things. Debug the script. Figure out dependency names (e.g. it’s gcc-c++ on openSUSE, g++ on Debian and just gcc on Arch).
  4. Find there’s no PythonQt packaged; while this is a strictly optional dependency, I would like to find a distro that actually ships something usable for PythonQt (seems Arch does, and KaOS).
  5. Build Calamares.
  6. Profit!

So where does that last, profit, step come in? Well, openSUSE has Secure Boot support, while distro’s using Calamares generally don’t — for the simple reason that Calamares doesn’t support it yet. So I’ll be peeking at what, and how, openSUSE does it and massaging that into Calamares.

Day 2 Ran an update, hoping that KDevelop would be fixed by now. That’s a nice thing about rolling- and bleeding-edge distro’s, stuff gets fixed and/or broken on a daily basis. With Krypton, the underlying rolling base is touted as stable while the KDE bits are bleeding-edge. It wasn’t, but a quick question in the right IRC channel (#opensuse-kde for Krypton) got me sorted and a fix scheduled for the next build. Well done, Kryptonites.

Spent the day hacking on Calamares, mostly fiddling with other bits-and-pieces rather than doing what I intended to do, which was examine secure boot.

Day 3 Still stable. Today’s bleeding-edge update is 112MB, as KDE Plasma is updated to 5.12. I decide to do some ARM development today as well. This is obviously not ideal, since I’m then cross-compiling to aarch64 in a Linux VM running on FreeBSD, but hey. After installing cross-aarch64-gcc7 and adjusting some build instructions that assume Debian naming (e.g. CROSS_COMPILE=aarch64-suse-linux- instead of CROSS_COMPILE=aarch64-linux-gnu-), spent a thoroughly frustrating morning building U-Boot and watching it panic. That’s the downside to using very new hardware which isn’t supported by anything yet except the OEM’s binary-blob package.

Day 4 (after the weekend) A total of 733 package updates today, 810MB to download. They’re not kidding about bleeding-edge and up-to-date. In the meantime I’ve learned that my deploycala script could be much simplified by using the package-manager. Since Calamares is packaged for openSUSE, I could have done zypper mr --enable repo-source ; zypper source-install -d calamares to get the build dependencies for it.

Anyway, after a week I’ve I have not yet broken the system, it’s fast and up-to-date. I’ll be keeping this one around. (And if I was looking for something between Krypton and Leap, I’d probably go for GeckoLinux, which uses Calamares — a bit of dogfooding, as it were).

Switching Distro’s

Obviously I still use FreeBSD on the desktop; with the packages from area51 I have a full and modern KDE Plasma environment. We (as in, the KDE-FreeBSD team) are still wrestling with getting the full Plasma 5 into the official ports tree (stalled, as so often it has been, on concerns of backwards compatibility), but things like CMake 3.10.1 and Qt 5.9 are sliding into place. Slowly, like brontosauruses driving a ’57 Cadillac.

In the meantime, I do most of my Calamares development work — it is a Linux installer, after all — in VMs with some Linux distro installed. Invariably — and especially when working on tools that do the most terrible things to the disks attached to a system — I totally break the system, the VM no longer starts at all, and my development environment is interrupted for a bit.

That’s always a good moment to switch distro’s .. since I’m going to spend an hour or so re-invigorating the VM anyway, reminding myself that this time I’ll make a clone and keep snapshots and whatnot, I may as well see what others use Calamares for. I’ve left a dusty and broken KDE Neon and a Manjaro behind me, and today I’m starting on Kaos.

For the very reason that Calamares can break stuff, and because new Calamares versions need to be tested on (older) live ISO images, there’s a deploycala Python3 script that helps install all the (development) dependencies needed. This is a bit of a drag, since it’s tied to a whole bunch of distro’s specific package names, but it gets the job done. I update it whenever I hit a new distro.

Kaos Linux Logo

Kaos Linux Logo

Kaos is a KDE-leading-edge distro, and daaannnggg is it ever slick in the first 10 minutes of use. The customization of Calamares is some of the nicest I’ve seen. The release-notes page could use some work (on my part) .. if only because I could read that stuff during unsquashfs, instead of before it. First-time startup with Kaptan is a hardly-intrusive way of configuring a bunch of separate things that would require a careful trip through systemsettings.

Look for the next few Calamares releases (in particular 3.2) to emerge from the Kaos, then.

More Calamares Releases

Another month passed, just like that. I spent last week holed up with some KDE people in the hills, watching the snow come down. While they did impressive things to the KDE codebase, I hacked on Calamares. Since my last post on the topic, I’ve been running a roughly every-other-week release schedule for the Calamares 3.1-stable branch. We’ve just reached 3.1.10. The reason for these stable releases is small bugfixes, minor polishing, and occasional non-interfering features.

Each release is announced on the Calamares site, and can be found on the Calamares GitHub page.

Calamares isn’t a KDE project, and aims to support whatever Linux distro wants to use it, and to configure the stuff that is needed for that distro. But when feature requests show up for KDE integration, there’s no special reason for me to reject them — as long as things can remain modular, the SKIP_MODULES mechanism in Calamares can avoid unwanted KDE Frameworks dependencies.

One new module in development is called “Plasma Look-and-Feel” (plasmalnf) and the purpose there is to configure the look-and-feel of Plasma. No surprise there, but ther point is that this can be done during installation, before Plasma is started by the (new) user on the target system. So kind of like the Look-and-Feel KCM, but entirely outside of the target environment. The UI is more primitive, more limited, but that’s the use-case that was asked for. I’d like to thank the Plasma developers for helping out with tooling and guiding me through some of the Plasma code, and deployers of Calamares (that is, distro’s) can look forward to this feature in the 3.2 series.

Speaking of Calamares 3.2, I’m intending to put out an RC fairly soon, with the features as developed so far. There will probably be a few RCs each of which integrates a new feature branch (e.g. a giant requirements-checking rewrite) with 3.2.0 showing up for real in the new year.

Calamares releases

It’s been a quiet month for me for blogging, but one filled with unexpected and weird and not-really-bloggable things. There was a trip to Berlin, where I had the pleasure of meeing up with a bunch of KDE people whom I hadn’t seen for over a month. Long time. There was also an accident with maple syrup, I’m sure.

Anyway, there have been four (!) Calamares releases since I last wrote about it on august 23rd (two months ago). These mix various bugfixes with various regression-fixes. This illustrates that I need a better set of acceptance tests before releasing — which take a surprisingly long time to set up, since the regressions are things like “Installing Chakra with network packages fails”, not simple stuff that is OS-, hardware- and installation-independent.

The latest release fixes that Chakra (or, rather, Calamares netinstall) problem. I’m happy to report that there’s a few new Linuxen evaluating Calamares as a system installer.

Plans for the near future include a 3.2 with some new features and much nicer support from new KPMCore releases, and ongoing care for the 3.1.x series. When and how, is in the hands of the vagaries of inspiration and long sessions with pen and paper figuring out just how things should work.

Randa Approaches

Later this week, I’m leaving for Zurich, and from there I’ll take the train up to Randa (up, in the sense that I live at sea level, and Randa is the length of one million micro-SD cards laid end-to-end higher).

In Randa, I’ll be working as a KDE developer, and as a Calamares developer, and learning about accessibility tooling. There’s about 60 hacking hours in that week. I’ll also be working as the cook, for one day. There’s about 12¬†cooking hours in a day, since feeding 30 people takes a lot of vegetable-chopping, bread-slicing, and dish-washing.

That is something special about Randa, I think: the feeling of much closer “living together”, and the way the attendees work together to create an optimal hacking environment. And cooking (along with the hacking) is my way of supporting the Randa meeting.

You can support the Randa meeting, too. That doesn’t support me; it supports other attendees who need to make long trips, it defrays the costs associated with infrastructure, it brings networking to town for a week. Support the Randa meetings for this year’s theme, or for the idea of a focussed retreat for hacking.

There’s a dot story about plans for the meeting. There will be summaries as well, and blog-roundups. But blog-roundups are tricky, because of the kind of things we (attendees) tend to write about. When blogging about the Randa meeting, I’ll probably blog more about food and hikes than about the hacking. The non-hacking bits make for better stories, even if the point of being there is the hacking. The results of the long coding sessions — privilege-separating Calamares, double-checking accessibility of KDE on FreeBSD — will show up later, in a future Calamares release or KDE-FreeBSD update. That’s the long-term payoff.