A simple example of git bisect run with ruby

git bisect run is my favourite git command, and I'll find any excuse to use it.

If you're not familiar with this tool, the git manpage succinctly describes it as:

git-bisect - Find by binary search the change that introduced a bug

And Bisect run as:

Bisect run
    If you have a script that can tell if the current source code is good or
    bad, you can bisect by issuing the command:

        $ git bisect run my_script arguments

    Note that the script (my_script in the above example) should exit with
    code 0 if the current source code is good, and exit with a code between
    1 and 127 (inclusive), except 125, if the current source code is bad.

An example

One of the applications at Zendesk uses a very handy gem called Phony that has the noble and non-trivial aim of formatting any phone number in the world, here's an example:

# United States
> puts Phony.format('14155556666')
+1 415 555 6666
# United Kingdom
> puts Phony.format('441914980000')
+44 191 498 0000
# France
> puts Phony.format('33123456789')
+33 1 23 45 67 89

It's really cool.

Recently I updated our application to use the latest version of Phony and some tests started to fail, specifically the ones around emergency numbers. After a bit of investigation in the rails console I discovered that the following command ran successfully before the version bump and failed afterwards.

# Using v1.9.0
> Phony.normalize("999")
=> "999"

# Using v2.10.0
> Phony.normalize("999")
Phony::NormalizationError: Phony could not normalize the given number. Is it a phone number?
    from .../phony-2.10.0/lib/phony.rb:102:in `rescue in normalize!'
    from .../phony-2.10.0/lib/phony.rb:100:in `normalize!'
    from .../phony-2.10.0/lib/phony.rb:97:in `normalize'

I cloned Phony locally and tried the same thing within the gem. I was able to write a one-line script that I could run from my bash prompt:

# Using v1.9.0
$ phony ((v1.9.0))> ruby -e 'require "./lib/phony"; Phony.normalize("999")'
$ phony ((v1.9.0))> echo $?

# Using v2.10.0
$ phony (master)> ruby -e 'require "./lib/phony"; Phony.normalize("999")'
./phony/lib/phony.rb:102:in `rescue in normalize!': Phony could not normalize the given number. Is it a phone number? (Phony::NormalizationError)
    from ./lib/phony.rb:100:in `normalize!'
    from ./lib/phony.rb:97:in `normalize'
    from -e:1:in `<main>'
$ phony (master)> echo $?

As you can see from the output of echo $?, the script had an exit code of 0 in version 1.9.0, while in version 2.10.0 the script had an exit code of 1.

So now I knew of one 'good' commit and one 'bad' commit, and I had a script that I could run which exited with 0 on a good commit and 1 on a bad commit.

Next I wanted to find out at which point Phony started treating the number '999' differently.

The basic process for starting a bisect goes like this:

$ phony (master)> git bisect start
$ phony (master|BISECTING)> git bisect bad
$ phony (master|BISECTING)> git bisect good v1.9.0

Bisecting: 221 revisions left to test after this (roughly 8 steps)
[164dfc36f83b5d0c5b3c11073395a3324622449f] Add Rubinius performance tool.

$ phony ((164dfc3...)|BISECTING)> 

At this point if I hadn't had a script that I could run I could have checked this version of the code manually and then marked it as good or bad myself:

$ phony ((164dfc3...)|BISECTING)> git bisect [good/bad]

But fortunately I had a script! So I ran:

$ phony ((164dfc3...)|BISECTING)> git bisect run ruby -e 'require "./lib/phony"; Phony.normalize("999")'

At this point git took over and jumped through commits, executing this script until it found the first commit that caused this script to fail. Here's a video of the process:

The final message that git output was:

9b4234a5024780f3b781b9a68e9c12104dea9c94 is the first bad commit
commit 9b4234a5024780f3b781b9a68e9c12104dea9c94
Author: Florian R. Hanke <florian.hanke@gmail.com>
Date:   Sat Sep 13 14:23:18 2014 +0200

    Add reserved keyword to Phony DSL.

:100644 100644 dbbb2f6a5a2db77645c622256d42dac45f187a3e 0ddb203544988f2c5f76e53a17b30788716abaeb M  history.textile
:040000 040000 7aee4152514e4aa8540bac2b5aab02de07fbc4a7 26952552d99a3e3387afb88ed84cab47eabf1a0a M  lib
:100644 100644 84cd8783223cbac50acdceec0a2d9790f40cf4ae c36a4c9f458a8516fadd7ae2f4a50e8bed9f0557 M  phony.gemspec
bisect run success

So now I knew the first commit that caused Phony.normalize('999') to behave differently!

Next steps

The commit in question implied that the change was intentional, and that Phony would no longer allow any operations on the number '999'.

I didn't want to make any assumptions, so I created an issue on the Phony repo and the author, Florian explained that this really was the intended behaviour, and that according to E.164, '999' is not considered an international number (and technically neither is '911', but this didn't cause any tests to fail because +91 is the country code for India, and +911 is currently considered to be a valid number).

Thanks to the feedback from Florian I changed our application to handle all emergency numbers in a different way.

I hope this was useful! If you have any feedback send me an email at paul at paulboxley dot com or tweet me at @baxt3r.