Users of Akonadi (databases/akonadi) on FreeBSD who chase the main ports / packages tree may have noticed that Akonadi started crashing on startup. I know I did, because I couldn’t read mail anymore. There’s a fix, it is in ports, and this post describes what I did about it.

The Problem

Assertion failed: (param->buffer_length != 0), 
  function setup_one_fetch_function, 
  file mysql-5.7.34/libmysql/libmysql.c, line 4112.

Akonadi (the server) crashes on startup in a MySQL function; if you restart it a couple of times quickly-enough, it stops auto-restarting and the sad-faced DrKonqi goes away from the task bar.

I wrote about the problem previously with the somewhat derpy suggestion “use an older libmysql”. That’s a workaround, but any accidental upgrade (like I did saturday morning) will pull in the latest MySQL 5.7.34 libraries and start the problem all over again. (It actually took me until saturday afternoon or so to realise that I had this problem and that I had already described it – derp on me).

Software Stack

The behavior in libmysql changed between 5.7.33 and 5.7.34. I fetched the source tarballs (5.7.34 isn’t available from Oracle’s downloads site, but FreeBSD packaging snags it from somewhere; 5.7.33 is from Oracle) and compared libmysql.c from the two releases: a DBUG_ASSERT() has turned into an assert(), which in the way FreeBSD builds these things is still “live” and triggers a crash when it fails.

MySQL source code change
MySQL source code change

I’m going to assume that there’s a good reason for that change, and a good reason that FreeBSD builds the library with asserts-enabled. So I need to move up in the stack to find a good place to attack the problem.

By running akonadiserver directly in gdb I could get a decent backtrace. To get to that point I needed to rebuild MySQL and parts of Qt with debugging enabled. That’s easy enough with a local package repository and poudriere, which I had anyway. The backtrace looks like this (edited for readability):

#4  setup_one_fetch_function from lib/mysql/libmysqlclient.so.20
#5  mysql_stmt_bind_result from lib/mysql/libmysqlclient.so.20
#6  QMYSQLResult::exec from lib/qt5/plugins/sqldrivers/libqsqlmysql.so
#7  QSqlQuery::exec from lib/qt5/libQt5Sql.so.5
#8  Akonadi::Server::QueryBuilder::exec at querybuilder.cpp:418
#9  Akonadi::Server::PartHelper::remove at parthelper.cpp:108

Reading up through the stacktrace,

  • (#9, #8) Akonadi builds an SQL query using the Qt SQL module,
  • (#7) Qt SQL hands off the query to a backend plugin, the Qt MySQL plugin,
  • (#6) MySQL plugin calls the MySQL C API to build the query for MySQL itself,
  • (#5, #4) MySQL C API says “no”.

I bunged a bunch of printf-debugging (ok, qCDebug() streaming-debugging) into Akonadi, Qt SQL and the Qt MySQL plugin looking for something special. There’s something special about this query (reformatted for readability):

SELECT PartTable.id, PartTable.pimItemId, PartTable.partTypeId, 
       PartTable.data, PartTable.datasize, PartTable.version, 
       PartTable.storage 
FROM PartTable 
WHERE ( PartTable.partTypeId = :0 AND storage = :1 AND data IS NOT NULL )

That none of the queries prior to it have. It may be that this is the first query that has a blob attached to it. In any case, by the time we end up in the MySQL plugin, there is fieldInfo describing the blob field, and that has a max_length. There is a comment in the source saying that max_length is known later. Right now, though, it’s 0.

The 0 ends up in the buffer length for the bound parameter, and then that ends up in libmysqlclient, where it triggers the assert.

Patching Qt

I decided to patch the issue as close to the changed API as I could. Since I don’t have ownership of the MySQL libraries, but I do control Qt ports (this is a FreeBSD administrative thing, really) – along with my assumptions about MySQL – this meant patching in the Qt MySQL plugin. Looking at the various exec() functions, I could see bindBlobs() being called (that’s internal to the Qt MySQL plugin) just before the call to mysql_stmt_bind_result() that triggers the crash.

Here’s the code that is setting up the parameter binding that will be passed along to the MySQL library:

bind->buffer_length = fieldInfo->max_length;
delete[] static_cast<char*>(bind->buffer);
bind->buffer = new char[fieldInfo->max_length];

When max_length is 0, this is going to trigger the assert. It’s also going to do a 0-length allocation, but that is legal.

I briefly experimented with setting the length to 1 when it was 0, but that feels wrong and also didn’t get me a working software stack. Instead I added a check for a non-zero field size when doing the bind:

if (qIsBlob(inBinds[i].buffer_type) && meta && 
    fieldInfo && fieldInfo->max_length) {

That last check is new; this means that 0-size blobs don’t bind and don’t set an out-field buffer – but since that would have been a 0-sized allocation (that’s not nullptr, new T[0] is supposed to return a distinct non-null, but non-dereferenceable, pointer) that seems reasonable to me.

With this change, Akonadi starts again and I can read my mail again.

Next Steps

I’ve pushed this patch to Qt packaging on FreeBSD. I’ve been using it for a few days via the now-works-again Akonadi. What I don’t know is whether this has an effect anywhere else in the stack: since I can read mail, at least I can also read complaints if something goes wrong.

If there is a next step, it is to push this to the Qt5 patch collection maintained by the KDE community. I suspect, though, that I don’t have the energy to chase this upstream and that the use of MySQL 5.7 is rare enough that non-FreeBSD consumers of the MySQL plugin aren’t going to notice the problem anyway.