This post documents how to convert a directory full of project release ZIP files into a git history with better diffs and commit dates. It was an itch I needed to scratch and there is no reason to keep the hacky tooling to myself.

The NX-KS project

I'm an active user of the NX-KS2 mod on my 2014 Samsung NX500 camera. The mod implements additional functionality and offers an UI overlay to access it. Under the hood, it's shell scripts and small UI helpers. The original developer KinoSeed vanished around 2017, and all official download locations are gone.

I've been carrying a few minor modifications in my $HOME and on my cameras since installing the mod, and I always wanted to properly track them in git. However, there was no original repository, and just using the last release 2.88 as the initial commit felt wrong to me.

Luckily, somebody collected the known releases and even added a README.html with a change log of sorts.

I want to convert that set of files into a git history now.

Preparing the inputs

I've downloaded all the ZIP files from NX-KS-mod/ and NX-KS-mod/archive/ into one common directory. To make the file names consistent, I had to rename the v2 files:

rename s/NX-KS2-/NX-KS-2./ nx-ks-mod-input/NX-KS2*

The change log in README.html contains the commit id of the ZIP file in the upstream project (yes, don't ask me!), the version string, a changelog, and a date.

I manually converted the README into tab-separated values as follows:

The full metadata table was saved to nx-ks-mod-changes.tsv.

The changes in 2.88 and 2.07 were undocumented, so I left them as "TODO" for the first iteration, checked the git diff after it, and improved the changelog on the second conversion re-run.

The 2.00m-repack version was based on the ZIP file name, I decided to keep it as-is in the commit history, but to manually change the git tag to v2.00m.

Converting the README

The project is providing a NX-KS_readme.odt (renamed to NX-KS2_readme.odt in the 2.x versions), which contains a description and sometimes can indicate changes. However, ODT is not a good format to git-diff (even though there are workarounds). As a hack, I'm using Pandoc to create a markdown approximation of the original document:

[ -f "NX-KS_readme.odt" ] && pandoc NX-KS_readme.odt -o NX-KS_readme.md
[ -f "NX-KS2_readme.odt" ] && pandoc NX-KS2_readme.odt -o NX-KS_readme.md

Unfortunately, the markup in the ODT is not using proper styles, so the result is far from perfect.

Creating the repository

We need to create an empty git repository first:

mkdir nx-ks-mod
cd nx-ks-mod
git init .

Committing the ZIP files

Then, for each ZIP file we have to perform the following steps:

1. Remove all previous content (in case a file was deleted):

    git rm -r .
   

2. Unpack the ZIP file (we are using find because we do not have a full file-name in our table):

     fn=$(find ../nx-ks-mod-input/ -name *$version.zip)
     unzip "$fn"
   

3. Pandoc the README (see above)

4. Commit everything, back-dating to $date and cheating the file name into the committer id (it could be added into the commit message as well), then tagging it with the version from the table:

     git add .
     git commit --date="$date 00:00:00" \
                --author "$(basename $fn) <$email>" \
                -m "$changes"
     git tag "v$version"
   

The final script

There is one more thing. The order in the README was newest-first, but for a proper commit history, we need to reverse the order, which we do using tac.

The complete result is this:

#!/bin/bash

mkdir nx-ks-mod && (cd nx-ks-mod ; git init .)

tac nx-ks-mod-changes.tsv | \
while IFS=$'\t' read version changes date ; do
    echo "-- $date -- $version -- $changes --"
    echo ""
    fn=$(find nx-ks-mod-input/ -name *$version.zip)
    cd nx-ks-mod
    git rm -r .
    unzip "../$fn"
    [ -f "NX-KS_readme.odt" ] && pandoc NX-KS_readme.odt -o NX-KS_readme.md
    [ -f "NX-KS2_readme.odt" ] && pandoc NX-KS2_readme.odt -o NX-KS_readme.md
    git add .
    git commit --date="$date 00:00:00" --author "$(basename $fn) <georg@op-co.de>" -m "$changes"
    git tag "v$version"
    cd ..
    echo
done

Not perfect, but good enough to create a commit history, as can be seen on github/nx-ks-mod.

Discuss on Mastodon

As a long-time Mutt user I always looked with envy at you Thunderbird and Kmail and what-not fans, as you could spawn new windows for reading and writing e-mails with a mere click (or sometimes a double-click).

It was just too bothersome to have $EDITOR block my inbox until I finish writing or give up and postpone the mail, losing track. As I am using Screen to run Mutt anyway, it seemed like a logical step to make it spawn new screen windows for writing mails and opening other mailboxes. The problem of multiplexing Mutt seems to have appeared to other people before as well, however the solution is not quite easy to spot.

The Idea Behind Mutt Multiplexing

The suggestion for people like me is: replace editor with a command line that spawns a new terminal and opens up mutt -H on a copy of the temp file. The original mutt will then see that the message was not changed and drop it silently, while the new instance allows editing. Fortunately, you can call screen from inside screen, thus creating a new window, so project mutt-screen was born! Unfortunately, not everything was as easy as expected.

The Gory Details, version 1

So, to make the whole thing better manageable, lets split that editor setting line into two parts: a script that is called in the new screen window and what needs to be done in the original mutt instance.

Lets call the script ~/bin/mutt-compose.sh:

#!/bin/sh
mutt -H "$1"
rm "$1"

Now, lets make a copy of the mail draft (%s), and run the composer in a new screen window:

# edit messages in a new screen window
set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"
unset wait_key

Great! We did it! Oh wait, no! It's a fork bomb! The new composer of course evaluates editor and launches... another copy of itself.

Edit the editor, version 2

Consequently, we need to override editor for the editing instance. Because we need to prevent the new Mutt from inserting a second .sig and more headers, lets create a second config file. Then we can call mutt -P ~/.mutt_compose -H "$1" from the script, with ~/.mutt_compose containing:

# read main config
source ~/.muttrc

# remove hooks, headers and sig, they are already in the draft
unhook send-hook
unset signature
unmy_hdr *

# call the right editor immediately
set autoedit=yes
set editor="vim +'set tw=72' +/^$/+1"

But now its going to work, isn't it? Not quite - something happened to postponed messages! Why are there now two edit windows? It seems, Mutt has a different code-path for messages which have been edited already. Here, the first instance ignores that the message was not edited and falls through to the compose window, while the second instance runs the editor (and permanently deletes the message if you do not explicitly save it, oops!).

Forking the editor - or not? Version 3

Our forking-out of the editor causes problems in three cases, as it seems: <recall-message>, <edit> and <resend-message>/<forward-message>. The former can be worked around by using a custom macro:

# override the <recall-message> hotkey
macro index,pager R "<shell-escape>screen -t postponed mutt -F ~/.mutt/compose -p<enter>"
# prevent recall on <mail>
set recall=no

This comes at the additional discomfort of not being asked if you want to resume writing your last mail when pressing m. That might be possible to achieve with some workaround - feel free to leave a comment if you find one.

For <edit>, the only viable workaround seems to be resetting editor to its former state, editing the message in-place and setting editor back to the screen call. Inconvenient, blocking your main Mutt, but possible.

# bonus points for outsourcing the two different editor settings into their own
# source'able one-liners...
macro index,pager e '<enter-command>set editor="vim +set\ tw=72 +/^$/+1"<enter><edit><enter-command>set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"<enter>'

With <resend-message>/<forward-message>, there is no workaround. I tried pushing keys into the buffer to quit the compose window, changing relevant settings, making blood sacrifices to various gods, all without success. So far I have to live with manually quitting the first composer and editing the message in the second one. Yikes.

The resulting config, version 4 (final)

When we put everything together, the following settings files should be in place.

~/.muttrc

# do not forget your own settings...

# edit messages in a new screen window
set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"
unset wait_key

# override the <recall-message> hotkey
macro index,pager R "<shell-escape>screen -t postponed mutt -F ~/.mutt/compose -p<enter>"
# prevent recall on <mail>
set recall=no

# set the editor for for editing messages in-place
macro index,pager e '<enter-command>set editor="vim +set\ tw=72 +/^$/+1"<enter><edit><enter-command>set editor="cp %s %s.2; screen -t composer ~/bin/mutt-compose.sh %s.2"<enter>'

# open mailbox listing in a new window
macro index,pager y "<shell-escape>screen -t mboxes mutt -y<enter>"

~/bin/mutt-compose.sh

#!/bin/sh
# set the screen window title to the message receiver
awk -F 'To: ' '/^To:/ { print "\033k" $2 "\033\\" }' "$1"
mutt -F ~/.mutt_compose -H "$1"
rm "$1"

~/.mutt_compose

# read main config
source ~/.muttrc

# remove hooks, headers and sig, they are already in the draft
unhook send-hook
unset signature
unmy_hdr *

# call the right editor immediately
set autoedit=yes
set editor="vim +'set tw=72' +/^$/+1"

The following .screenrc.mutt file can be used to spawn a screen with your inbox in it and a status bar showing the outsourced instances:

~/.screenrc.mutt

hardstatus alwayslastline "%-Lw%{=b BW} %50>%n%f* %t %{-}%+Lw%< "
sorendition = bG
vbell off

# start mutt directly in a window
screen -t INBOX mutt

Conclusion

This solution is not perfect. In certain corner cases (<resend-message>) it just does not behave. In other situations (caching the PGP passphrase does not work) it has less comfort. Nevertheless, it has increased my Mutt productivity and decreased my frustration a lot. Now all is missing is clean support for multiple profiles (read: without having many scripts bound to clumsy keyboard shortcuts) and proper in-mutt mechanism for message tags.

cgit, a gitweb replacement?

Recently I was made aware of the existence of cgit, a git web frontend written in C. It looked promising, so I tried it out.

It seems to perform better than gitweb, which I kind of expected from the fact that it is a native binary, as opposed to a perl script forking git for the actual work. Also, there is a comparison performed by the author of cgit.

However, I was convinced by the clean, short, nice looking URLs provided by cgit, which allow to checkout files from your default HEAD without nasty hash tags:

/cgit/libmpq.git/tree/Makefile.am

versus the rather ugly gitweb URL:

/git?p=libmpq.git;a=blob;f=Makefile.am;
    h=1f70fdb06cc13ae0338c380139c352383aea7700;hb=HEAD

I think in the good old new days of Web 0.2, this is called RESTful behaviour, which also coincidentally contains the substring "STFU".

Syntax highlighting with cgit

Anyway, once I was at playing with cgit, I wanted to add some syntax highlighting, too. To enable it, cgit already comes with a filter script (you see the elegant URL syntax at work again?). This script uses the highlight command, but it does not support all the file formats of highlight, and might be missing some elegance.

I made an attempt at supporting all the files I have in the different git repositories, which you can find here:

#!/bin/sh
# store filename and extension in local vars
BASENAME="$1"
EXTENSION="${BASENAME##*.}"

# map Makefile and Makefile.* to .mk
[ "${BASENAME%%.*}" == "Makefile" ] && EXTENSION=mk

exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null

This variant extracts the file extension of the file to be displayed, contains detection of Makefile.* files and spares one fork by running exec highlight. If the syntax is not known, --force makes the command just pass through the input.

Do not forget to add the CSS required by highlight, as documented in the original syntax-highlight.sh.

What is still missing?

Cgit comes with cache support to further reduce the CPU load. However, I yet have to figure out how to make the caching time long enough to make a difference on this low-traffic server without taking forever to update the cache content whenever I push changes.

A cosmetic issue remains with directories in the tree view. They should get a more decent mark-up for people to notice them. Update: this is easy with a small css tweak (example).

And, last but not least, I am missing a feature to limit the blob-size for generating HTML. This is especially important for binary objects which are served as a hex-dump, generating gazillions of bytes for something you are not going to read anyway. Update: If you want something done, send a patch upstream! (example) :-)

Update 3: All three patches have found their way into the cgit master repository, thanks very much, Lars!

Am I actually making a blog? I don't even want to know!

Setting up the CSS for iki is not quite a nice & easy job, but at least it is not the usual Web 0.2 bloat.

Update: ikiwiki has the nasty misfeature of overwriting local.css whenever you regenerate the wiki without --refresh. Because I do not like it and I did not want to dig too deep, I added the following to my .htaccess:

Redirect /blog/local.css /ikiwiki.css

Up-Update: The obvious solution is to put local.css into the source directory instead of the destination directory. Thanks to Spida for this hint.