Calamares is a distro- and desktop-agnostic installer for Linux distributions. Dozens of distributions, medium and small, use Calamares every day for getting the distro from an ISO image onto the computers of their users. There have been three Calamares releases in the past seven days: release, hotfix and another hotfix. That’s embarrassing, and annoying for the distributions that pick up the latest release on their rolling ISO images. There is one specific cause of this gaggle of releases: Python

Calamares is built to be extensible. There are about 60 modules in the core distribution, another dozen in an -extensions repository, and distributions have their own collections of modules as well. Some modules are written in C++, some are written in Python.

The Python modules are there because it’s easy to write some Python, it’s a nice language, and the requirements for many modules are quite straightforward: read a file, write a file, run some command in the target system. Distributions can also easily contribute to the codebase, because there’s more Python programmers than C++ programmers out there (at least in the making-a-distro scene).

Calamares Releases

Calamares releases are largely automated: when I’ve merged the last bits-and-pieces, I check that the whole thing compiles, that the tests pass, and then it’s time for tagging. There is a shell script called RELEASE.sh that does the work.

Up through the release of Calamares 3.2.44.1 the tests on the Python modules were limited to “does it load”. That catches egregious errors that would cause a SyntaxError exception, at least.

After that it’s up to the downstream consumers – Linux distributions – to test that the modules work. Oftentimes the modules contain special cases for Linux distributions – Arch does one thing, Debian another, Fedora a third and openSUSE a fourth. So hitting code-paths really does depend on doing an installation of a given distribution.

How Adriaan Causes Problems

Recent issues are on me: when looking at code, I tend to refactor as well. Shuffle some duplicated code into a translation-unit-local (e.g. static) free function. Rename variables for better readability. Add a method working on a reference for code sharing.

You can tell by the way I word this that I’m used to C++ and the support that the compiler and IDE give me.

My Python refactorings suffer from “oh, there’s a self. left in there” or incomplete renamings or module-globals (I’m telling you, the Python code is rarely pretty) that sneak in somewhere. That kind of error does not get picked up by a simple syntax check.

Better Tooling

Anke Boersma, the driving force behind KDE showcase-distro KaOS, pointed me (on Matrix) at pylint. This is a tool that does more than simple syntax checking. Much more. So much more that it took some effort to pare it down to “just tell me where the variables are undefined”.

The straightforward version of pylint issomething like

pylint src/modules/networkcfg/main.py

This checks a single module (conventionally, in Calamares, these all have a main.py and no other files). Unfortunately this simple use barfs all over because the module assumes an API that is provided to Python by Calamares, but which isn’t visible anywhere else.

I ended up writing a quick-and-dirty “stub” for the API: it mocks all of the functions that the modules seem to use, although I have not mechanically checked that it is exactly the same as the API provided programmatically; definitions like

def warning(_): pass

stand in for the API method that allows Python modules to print a warning via the same channels that Calamares itself uses.

This commit adds tests – running pylint, using the stub-implementation of the API, for each Python module that Calamares ships. That shook out a number of existing problems, highlighted the issues I knew about in recent releases, demonstrated some undiscovered problems on rare code-paths, and generally has made things better.

The tooling is part of Calamares’s CMake modules, which means that extensions modules that use the framework get the improved testing “for free”.

Takeaway

I should have done this ages ago. It took half a day to sort out and implement nicely and would have saved us – me and collective downstreams – much more time than that over the past year. More tests are (nearly universally) good.