In the last article we explored the problem of reproducibility and the Gitian build environment Bitcoin Core uses to achieve it. We encoutered the gitian.sigs git repository, where anyone can upload signed claims about the correct hashs for different releases of the Bitcoin software. In this post you'll learn to verify your own Bitcoin software downloads against gitian.sigs and learn to submit your own claims. This is a nice way to get involved in Bitcoin Core development. You help increase overall confidence that the software is being built correctly, and familiarity with the command line is the only prerequisite knowledge.

Verify Bitcoin Software

As was the case in the SHA256SUMS.asc post, let's focus on a specific build of Bitcoin Core: bitcoin-0.20.1-aarch64-linux-gnu.tar.gz. Instead of comparing the hash of this file against SHA256SUMS.asc, let's instead compare against the .assert files submitted by everyone who executed the 0.20.1 Gitian build targeting Linux. Instead of trusting a website, we trust an army of paranoid independent volunteers. This is the way!

The Gitian build project requires a certain folder structure, let's make a new folder to contain everything. If you follow along to the end, you'll make a pull request to gitian.sigs. For this reason, you'll also want to fork gitian.sigs.

Click here for instructions on how to fork a GitHub repo if you've never done it before.

$ mkdir gitian-building
$ cd gitian-building
$ git clone <your gitian.sigs fork>

Now let's download bitcoin-0.20.1-aarch64-linux-gnu.tar.gz from bitcoincore.org and hash it:

$ curl -O https://bitcoincore.org/bin/bitcoin-core-0.20.1/bitcoin-0.20.1-aarch64-linux-gnu.tar.gz
$ sha256sum bitcoin-0.20.1-aarch64-linux-gnu.tar.gz
60c93e3462c303eb080be7cf623f1a7684b37fd47a018ad3848bc23e13c84e1c  bitcoin-0.20.1-aarch64-linux-gnu.tar.gz

Next we need to choose one of the Gitian signers for the 0.20.1 release. I'll choose luke-jr because his GPG key is all over the place.

Here's his .assert file for the 0.20.1 release. Note that line 4 contains exactly the same hash we produced above. To convince ourselves Luke authored this proof, we need to PGP-verify this .assert file:

# Download Luke's PGP pubkey
$ curl -O https://bitcoin.org/luke-jr.asc

# Import into gpg
$ gpg --import luke-jr.asc

# Signature verification
$ gpg --verify  gitian.sigs/0.20.1-linux/luke-jr/bitcoin-core-linux-0.20-build.assert.sig
gpg: assuming signed data in 'gitian.sigs/0.20.1-linux/luke-jr/bitcoin-core-linux-0.20-build.assert'
gpg: Signature made Sun 02 Aug 2020 11:02:03 AM CDT
gpg:                using RSA key E463A93F5F3117EEDE6C7316BD02942421F4889F
gpg: Good signature from "Luke Dashjr <luke@dashjr.org>" [unknown]
gpg:                 aka "Luke Dashjr <luke-jr@dashjr.org>" [unknown]
gpg:                 aka "Luke Dashjr <luke-jr+git@utopios.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: E463 A93F 5F31 17EE DE6C  7316 BD02 9424 21F4 889F

The signature is good, but once again we see a warning that Luke's key isn't "trusted" because we haven't marked this pubkey with a high degree of trust.

Before we try to verify the signatures of other Gitian builders, let's compare Luke's .assert with Jon Atack's. We'll use the Unix diff tool, which generates a report of differences between text files:

$ diff gitian.sigs/0.20.1-linux/luke-jr/bitcoin-core-linux-0.20-build.assert \
       gitian.sigs/0.20.1-linux/jonatack/bitcoin-core-linux-0.20-build.assert

Here's what I get. Lines starting with < lines mean luke-jr has that line but jonatack doesn't, and > means the opposite. The string bitcoin doesn't appear in the diff, but it is present in all filenames in out_manifest and in_manifest sections of each .assert file. This means there is no disagreement about inputs to the build -- source code or build instructions contained in Gitian descriptor -- or releases output by the build. All differences are in the base_system section which contains hashes of dependencies. Apparently this isn't a problem, but it is strange Gitian fails to exactly pin dependencies ...

At this point we could continue doing everything by hand and go searching for Jon Atack's PGP key. But doing so for every builder would be tedious. Luckily, regular Gitian builders add their PGP key fingerprints directly to the Bitcoin Core git repo here. Let's follow those instructions to import the PGP public keys for regular Gitian builders:

$ git clone git@github.com:bitcoin/bitcoin.git
$ pushd bitcoin/contrib/gitian-keys
$ while read fingerprint keyholder_name; do gpg --keyserver hkp://subset.pool.sks-keyservers.net --recv-keys ${fingerprint}; done < ./keys.txt
$ popd

pushd / popd are commands to enter and exit a directory back to where you were before. After this we should have PGP keys for many of the Gitian builders. For instance, we can now verify Jon Atack's .assert.sig files from earlier:

$ gpg --verify gitian.sigs/0.20.1-linux/jonatack/bitcoin-core-linux-0.20-build.assert.sig
...
gpg: Good signature from "Jon Atack <jon@atack.com>" [unknown]
...

Gitian's gverify script we encountered last time can automate this signature verification for us. Recall that gverify requires a path to a "Gitian descriptor" file. This is located in the bitcoin repo at the git tag v0.20.1:

$ git clone https://github.com/devrandom/gitian-builder.git
$ cd bitcoin
$ git checkout v0.20.1
$ cd ../gitian-builder
$ bin/gverify --destination ../gitian.sigs/ --release 0.20.1-linux ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
$ cd ..

For all signers contained in the keys.txt file from earlier you should see <username>: OK. For signers who weren't in that file, you should see:

gpg: Can't check signature: No public key
<username>: BAD SIGNATURE

Let's look at the gverify script to get a better idea what it's doing:

• Iterates over folder in "signer directory"
• PGP-verifies each .assert.sig file
• Checks if there are any differences in the out_manifest, release or name fields of the .assert file and prints MISMATCH if there are and OK if there aren't.
• In summary, it checks that important parts of .assert files are identical and verifies signature in .assert.sig.

Become A Gitian Builder

You, too, can become a Gitian builder. There are a few quality guides available so I won't write a new one here:

• The official bitcoin-core/docs repo has a guide using VirtualBox. It's labor intensive. I tried the Debian guide and it didn't work for me until I switched from Debian 8 to Debian 10.
• fanquake has a bare-bones guide using Docker, which is less work.
• Jon Atack maintains an expanded version of fanquake's guide which I found easiest to follow.

I encourage you to try Jon's guide, now!

Commentary on Jon's Guide

So far with Gitian we've just been building and verifying for Linux to keep things simple. But Bitcoin Core also has MacOS and Windows releases. Both of these platforms have "code signing" systems that require developers to complete a non-PGP proprietary code signing process so that these operating systems will trust their apps. You've likely seen what happens when developers screw this up:

Individual Bitcoin Core developers control the keys for MacOS and Windows code signing. When a new release is tagged in git, they wait for a few people to upload matching builds to gitian.sigs. Then they create "detached signatures" like so which Gitian builders can utilize to produce final MacOS and Windows "code signed" releases. For each platform you end up with 5 builds: Linux, unsigned MacOS, code-signed MacOS, unsigned Windows, code-signed Windows. I believe the "unsigned MacOS/Windows" builds are inputs into the "detached signatures" process. Jon Attack's guide builds the unsigned releases here and the code-signed releases here

The MacOS Gitian build requires you extract files from SDKs published by Apple. Jon's guide downloads these extracted files (4th snippet here). The official guide shows you how to extract these yourself, which is tedious but increases the independence of your Gitian build.

Bitcoin Core has a gitian-build.py script which can build, sign and make commits to gitian.sigs for each of the 5 release targets. You need to copy it to your base folder containing the other repositories and run it so:

$ cp bitcoin/contrib/gitian-build.py .
$ ./gitian-build.py -b <username> <version, e.g. 0.20.1>

This script reduces the typing / copypasta required by Jon's guide, but when it fails you need to run the whole thing over again. Not sure if it helps or hurts.

Lastly, only make PRs to gitian.sigs if you have a PGP key you plan to take care of. Provide a way for other developers to download your PGP public key in your PR. For example, upload it to a key server and provide the 40-character fingerprint in your PR.

Conclusion

Now you have the tools to achieve a much higher degree of confidence in the Bitcoin software you use, and help others increase their confidence by contributing to gitian.sigs.

One last question: What are you trusting now? Where might the NSA hide their precious chain split bug? Stop and think about this for a moment.

...

Gitian uses Ubuntu to run the actual builds and downloads all dependencies from Ubuntu's package manager. We are trusting Ubuntu and its package ecosystem.

Due to concerns like this, Bitcoin Core is transitioning to a system called Guix which promises to deterministically build not just Bitcoin Core but the entire toolchain and environment used to build Bitcoin Core.

We'll discuss Guix next time.