Personal clones on KDE infrastructure

I’m doing a little work on Tupi — the 2D animation application that joined the KDE community some months back — so that it builds on FreeBSD (the C++ code is wonderful, but the build system is qonf, which is not).
This has led me to the maze of git documentation on KDE’s infrastructure, and I’m taking notes so I don’t forget what I did. It’s also part of one of the things-to-do-at-Akademy on my list: talk to the techbase people to find out what the status and intentions are.
For the purpose of futzing with the build system, I’m using a personal clone of the repository. This is so that whatever weird-ass things I’m doing, they don’t pollute the upstream repository; eventually I hope to code-dump a CMake-based buildsystem into upstream. One that can live alongside the existing build system for platforms other than FreeBSD — although I think that in the long run having the same build system and dependency-finding-modules as all the rest of the KDE software would be good.
Basic git configuration is covered on techbase, which describes the mandatory (username) and optional (URL rewriting, colors, templates and exclusions) configurations for using git with the KDE infrastructure.
The KDE sysadmin git FAQ explains how to create a personal clone of a project repository. I’ve created a personal clone of tupi, and cloned that one locally. Then I work on that (in a project Neon VM and a FreeBSD jail).
The only thing is, my personal clone doesn’t get updates from the upstream (here, that means the original project repository) unless I pull them in myself. This is where additional remote repositories come from. In my clone, by default there’s two origin remotes (one for pull from anongit, one for push over ssh, as documented in the git configuration page).

[adridg@beastie]$ git remote -v
origin  git.kde.org:clones/tupi/adridg/tupi-cmake (fetch)
origin  git.kde.org:clones/tupi/adridg/tupi-cmake (push)

So what I need to do locally is pull in commits from the upstream, merge all of them with my local clone, and then push (which goes to my personal clone on KDE infrastructure). The git-scm book has clear instructions; for me it means adding an upstream repository:

[adridg@beastie]$ git remote add upstream kde:tupi

And then to keep up-to-date with the upstream commits and store them in my own personal copy, I need to do these four steps (this assumes that I don’t do anything in the master branch myself):

[adridg@beastie]$ git fetch upstream
[adridg@beastie]$ git checkout master
[adridg@beastie]$ git merge upstream/master
[adridg@beastie]$ git push

There’s probably a faster / better way of doing this, but at least I know I can keep up with the upstream. Now I can keep rebasing my work inside my local clone onto the latest upstream like this, until such a time as I’m ready to push my CMake branch to my personal clone on the KDE infrastructure. Once that’s done, I can ask the maintainer to pull it into upstream.

Posted in KDE

Prep for Akademy

In preparation for Akademy I wanted to swap out the drive from my laptop — which is full of work-work things — and drop in a new one with stuff I actually want to have with me at Akademy, like git clones of various repositories. I spent a few hours wrestling with my Lenovo x121e (AMD) laptop and FreeBSD, which taught me the following:

  1. You can update the BIOS from a USB stick using only Linux tools, and
  2. FreeBSD does not like it when the SATA controller is in compatibility mode, and either hangs or fails to find the hard drive at all; in AHCI mode things are fine, but
  3. Even the updated BIOS cannot boot from GPT partitions, so I had to be careful during installation to manually do an MBR / fdisk-based installation (this seems to preclude ZFS as well), and then
  4. Wireless isn’t automatically detected (but the WAN modem is), and suspend-resume doesn’t resume.

This makes for  less-than-stellar performance for a conference laptop; I’ll fiddle with it a little before departing for Berlin in two weeks time (isn’t Akademy in Brno? Yes, it is, but the most effective train journey takes me to Berlin first to catch up with the trainful of KDE people at 12:46 from HBf), so I may end up being a FreeBSD person sporting an OpenSUSE laptop.

For development purposes — sort of as a quick counterpart to the FreeBSD VM where I’m doing qt5-based things for KDE applications — I installed a project Neon VM. This way, too, I can check that I’m not breaking anything on non-FreeBSD systems. What I’m seeing on the desktop in that VM is not very encouraging to me, though. As used as I am to the current KDE software on OpenSUSE or FreeBSD (4.12 or whatever), the newer software feels weird and arbitrarily changed and oddly slow. That last bit might be due to VirtualBox, I don’t really know. I’ll have to attend some of the VDG or HCI topics to get a better feeling for the (visual) changes already made.

My Akademy schedule

I for one applaud the decision to put Akademy-badges on the KDE community wiki.

I was afraid I was going to have to apply my awesome Kolourpaint skills again. Albert’s reminder has caused me to figure out my reasons for attending Akademy this year. So my agenda is four-or-fivefold:
  1. attend the AGM of KDE e.V.
  2. talk to Paul Adams about measuring community health. Technical aspects of measurement aside, I have some serious methodological misgivings about what he’s measuring and how he’s presenting it. This will require several beer mats of exposition.
  3. see if there’s any other FreeBSD users about. Or, for that matter, OpenSolaris people or anything else non-Linux.
  4. talk to Lydia and Valorie and other folks with knowledge of techbase and community to see if I can contribute there. There’s a lot of stuff that is undergoing a complete rewrite — but has been undergoing that for a long time.
  5. hear from the FrogLogic folks how their test-tools have evolved. It was a long time ago that we had some ideas of doing KDE HIG checks on the EBN.

There is of course also an implied 6(a) meet new people in the KDE community and 6(b) drink beer with them.

SVN-to-git bridge (for practice)

Some time ago, I wrote that I needed to be less of a scaredy-cat about git (in particular so as to get back into KDE development, and the rat’s-nest of git repositories there was scaring me off — in that sense I’m a data point in what Paul Adams is writing about). The best way of learning is by doing, so I looked for something to do with git that would basically force me to use it regularly.
That really means “find a way to use git at work-work”, since most of my development hours happen there now (largely administrative number crunching in Python).
A bit of background: at work-work we have a central SVN repository. It has a non-standard naming scheme: trunk is called development, and branches are in the releases subdirectory. The only branches are for actual releases. For various reasons we are also using SVN 1.5, which means that we don’t have any of the more-modern merge and branch features that SVN has grown. So feature work by the developers happens in trunk directly, not in feature branches, and we end up with some pretty confusing history of interleaved commits.
I have 38 minutes on the train between Arnhem and Utrecht that I could use effectively for development of small things: typo-fixes, message improvements, adding unittests, that kind of thing. But I don’t want to end up with one big set of changes for all the little things I do on the train; I want sensible commits of one logical change after another.
So basically I want simple feature branches and offline commits. It needs to linearize history and integrate with a weird SVN setup. Previously I used Mercurial and its hgsubversion extension to get this effect; now I wanted to do the same with git, for learning purposes (and then I can futz with the KDE repositories again).
Most of what I eventually built to give me a nice git-based workflow that meshes with the central SVN repository is based on a series of blog posts
from TF Nicolaisen. They were really useful.
Anyway, my workflow now looks like this:
  • Pick a ticket N from our bug tracker (it’s TRAC, with some customized statistics modules I wrote and a handful of third-party TRAC-hacks for planning purposes),
  • Update my git repo from SVN with git pull,
  • Start a git branch for my work on the ticket with git checkout -b ticket-N,
  • Do my thing, with as many commits and experimental branches as needed, and then clean up (remove debug-commits, maybe merge some small steps), with git rebase -i,
  • Rebase onto the updated upstream SVN with git svn rebase,
  • Push the whole thing into SVN with git svn dcommit,
  • Drop the branch, since it’s in SVN now; the detached commits will get garbage collected eventually.
That’s a cromulent git workflow, and except for excessive rebasing and the push to SVN at the end, usable for regular git work as well.
The setup I’ve ended up with is illustrated here; on the server side, there are three repositories: first is the central SVN repository. This is the official and canonical source and other developers commit to SVN normally. Then there’s the git-fetch repository, which pulls revisions from SVN and puts them into git commits. This one has only the SVN commits in it. Third is a bare git repository, which is where I pull from and where the fetcher repository pushes things. This bare repository also has a few things that are not from SVN commits — I push branches here if I want to share them over git with other machines I work on, and there are a few tags in it marking some events in the history of the repository.
Client-side, there’s whatever clones I make of the bare repository.
Getting this set up was a matter of configuring and cloning the right bits; this I mostly did by following the steps described by TF Nicolaisen, except that I needed to get the authors map just right ahead of time, and I’m only git-bridging one single branch from SVN (namely, development, which is our trunk), and I’m not interested in ancient history (which is gnarly), but only fairly-recent commits from SVN. So here’s what I did:
  1. Figure out what part of the repository is interesting; for me, that was the development/ branch in SVN, from revision 28754 onwards.
  2. Figure out who is committing to the repository. SVN has the usernames, while git needs to have a name and email address. This requires a map of SVN authors to git authors. It’s also pretty much essential that the author in the mapping file matches the author and SVN username you configure in client-side git clones, or you’ll get a multitude of branches, all twisty and all very much alike. After some history examination (in SVN) and discussion on useful git author names with the other developers that might use git, we ended up with a file like this:

    adriaan = Adriaan <ade@example.com>
    bassie = Bassie <bas@example.com>

    I ended up committing this file into SVN so that it would be available — and could be updated — for general use. It lives indevelopment/.git-authors, ie. in the root of what I’m going to follow with git.

  3. Do the initial clone of the repository. Unlike the recipe here, I have an authors file (a copy exported from SVN, because it’s got to be there before the clone runs), and I don’t use the standard layout. I did this on the server, so that the SVN repository is local. The repository lives at/home/svn/project, and the fetching repo will be git-project-fetch

    cd /home/svn
    git svn clone -A author-map -r 28754:HEAD file:///home/svn/project/development git-project-fetch

  4. Set up the bare repo, which will be git-project:

    cd /home/svn
    git init –bare git-project

  5. Configure the fetching repo to push changes:

    cd /home/svn/git-project-fetch
    git remote add origin ../git-project

  6. Then modify .git/config so it reads as follows (again, this is all according to TF Nicolaisen’s recipe, only with a restricted SVN tree, a starting revision, and an author map):

    [remote “origin”]
      url = ../websites.git/
      fetch = +refs/remotes/*:refs/remotes/origin/*
      push = refs/remotes/*:refs/heads/*

    Because there’s only one branch here (namely development), there’s no need to configure which branch needs to be checked out by default.

  7. Add a post-commit hook to update the git repositories:

    if /usr/bin/lockfile -2 -r1 /tmp/project-gitsvn ; then
      ( cd /home/svn/project-fetch && /usr/bin/git svn fetch && /usr/bin/git push origin )
      rm -f /tmp/project-gitsvn
    fi

At this point, every SVN commit gets pulled into the fetcher-repo by the git-svn code and then pushed into the bare repository as if it’s a normal git repo. The server side is done. One the client side, I set up ssh access to the SVN (and now git, too) server. Then getting a correctly configured client-side clone is as follows:
  1. Use an ssh URL for access to both the SVN and the git repositories

    git clone -o mirror ssh://project.example.com/home/svn/git-project
    cd git-project/
    git checkout -t mirror/git-svn
    git svn init –prefix=mirror/ ssh://project.example.com/home/svn/project
    git svn dcommit

    That last dcommit — since the repo has just been cloned — just updates all the revision numbers. It’s not really necessary, since it will happen with the first real SVN commit from git anyway.

  2. Client side, we also need to set up the git author and the SVN authors files, so that they match with what happens on the server side. Failing to configure these consistently will cause lots of extra commits to show up in your local clone. Note I’m configuring this in the repo-local configuration, not globally, so it doesn’t interfere with the recommended KDE git setup. It uses the authors file I previously checked into SVN at the root of the development branch — now in the root of the git repo:

    cd git-project/
    git config user.name “Adriaan”
    git config user.email “ade@example.com”
    git config svn.authorsfile .git-authors

And with that, I’ve got a usable git clone that I can use for feature branches on the train, and that can easily push back to SVN. After using this for a few months I’ve finally gotten comfortable enough with git — feature branches, and sometimes futzing aroud to massage history into a usable form, and dealing with the rest of the git tools — to touch KDE git repositories again.