To build Bitcoin Core on Unix systems (Linux or MacOS) the docs tell you to do the following. Let's try to figure out what these actually do:

./autogen.sh
./configure
make
make install # optional

In order the steps seems to do the following:

• Execute a shell script
• Execute a shell script
• Build executables using Make
• Optionally, install executables using Make

So far so good. After visiting the Bitcoin Core GitHub repository, we find autogen.sh. It seems to do some setup and then executes autoreconf --install --force --warnings=all. Before investigating this command further, let's check out the next script script we execute, configure.

Hmmmm ... this script isn't in the Bitcoin Core source code. In fact, it's in the .gitignore file. This usually means it's a file the user is responsible for generating and different users on different systems might generate different files -- so it can't be checked into the source code. Where does this file come from? Note there is a similarly-named configure.ac script. Perhaps it has something to do with the autoreconf we saw above? This is mystery #1.

The third step is make. If you've never used make before, check out this 2 minute tutorial. Briefly, make is a system for building code. You write a Makefile containing shell commands necessary and configuration variables to builds your code. It locates all the dependencies on your system, compile your code, and output a final executable. For example, here's the Makefile for the Linux project. Once again, the Bitcoin Core git repository doesn't have a Makefile, but there is a similarly-named Makefile.am file. This is mystery #2.

🕵️ Mystery #1: Case of the Missing configure

Let's run the first command:

$ ./autogen.sh
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'build-aux/m4'.
libtoolize: copying file 'build-aux/m4/libtool.m4'
libtoolize: copying file 'build-aux/m4/ltoptions.m4'
libtoolize: copying file 'build-aux/m4/ltsugar.m4'
libtoolize: copying file 'build-aux/m4/ltversion.m4'
libtoolize: copying file 'build-aux/m4/lt~obsolete.m4'
configure.ac:45: installing 'build-aux/compile'
configure.ac:28: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'build-aux/m4'.
libtoolize: copying file 'build-aux/m4/libtool.m4'
libtoolize: copying file 'build-aux/m4/ltoptions.m4'
libtoolize: copying file 'build-aux/m4/ltsugar.m4'
libtoolize: copying file 'build-aux/m4/ltversion.m4'
libtoolize: copying file 'build-aux/m4/lt~obsolete.m4'
configure.ac:15: installing 'build-aux/compile'
configure.ac:9: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'build-aux/m4'.
libtoolize: copying file 'build-aux/m4/libtool.m4'
libtoolize: copying file 'build-aux/m4/ltoptions.m4'
libtoolize: copying file 'build-aux/m4/ltsugar.m4'
libtoolize: copying file 'build-aux/m4/ltversion.m4'
libtoolize: copying file 'build-aux/m4/lt~obsolete.m4'
configure.ac:93: installing 'build-aux/compile'
configure.ac:47: installing 'build-aux/missing'
Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
Makefile.am:139: ... '.INTERMEDIATE' previously defined here
src/Makefile.am: installing 'build-aux/depcomp'

If this output looks foreign to you, that makes two of us! But we might recognize a few things: configure.ac and Makefile.am. As a comment to our blameless beloved Core developers, it would be nice if this script told us what it did! Let's inspect our filesystem to find out for ourselves (ls -1 shows one listing per line):

$ ls -1
CONTRIBUTING.md
COPYING
INSTALL.md
Makefile
Makefile.am
Makefile.in               <- Hey, you weren't here before ...
README.md
REVIEWERS
SECURITY.md
aclocal.m4
autogen.sh
autom4te.cache
build-aux
build_msvc
ci
config.log
config.status
configure                 <- FOUND YOU!
configure.ac
contrib
depends
doc
libbitcoinconsensus.pc
libbitcoinconsensus.pc.in
libtool
share
src
test

Two observations: configure is generated by autoreconf and so is Makefile.in. Onwards!

🕵️ Mystery #2: Case of the Missing Makefile

Let's run the second command:

$ ./configure
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking ...
...

The script is inspecting our to make sure all dependencies needed to build Bitcoin Core are available. Some dependencies are optional. If QT is missing then it will won't build the GUI, but it will print a warning and proceed building everything else.

configure: WARNING: Qt5Core >= 5.5.1 not found; bitcoin-qt frontend will not be built

Other dependencies are required. One my Ubuntu system I got this error:

configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)

I'm missing development library for the ancient Berkeley DB version used by the Bitcoin Core wallet. I can pass a --without-bdb to disable wallet support, --with-incompatible-bdb to use the wallet in a way where the wallet.dat file might not work on other systems, or run ./contrib/install_db4.sh to install the proper version of BDB.

Consult the build-x.md files in Bitcoin Core's documentation folder for more information about installing dependencies for your system.

Autotools

Meet GNU Autotools, the suite of build system tools behind all this madness. The Wikipedia page explains what's happening here. Some highlights:

It can be difficult to make a software program portable: the C compiler differs from system to system; certain library functions are missing on some systems; header files may have different names. One way to handle this is to write conditional code, with code blocks selected by means of preprocessor directives (#ifdef); but because of the wide variety of build environments this approach quickly becomes unmanageable. Autotools is designed to address this problem more manageably.

The GNU Build System makes it possible to build many programs using a two-step process: configure followed by make.

Autoconf generates a configure script based on the contents of a configure.ac file which characterizes a particular body of source code. The configure script, when run, scans the build environment and generates a subordinate config.status script which, in turn, converts other input files and most commonly Makefile.in into output files (Makefile) which are appropriate for that build environment. Finally the make program uses Makefile to generate executable programs from source code.

It's all clear now. Bitcoin Core prioritizes portability -- ensuring code runs on many computer architectures and oeprating systems. Ability to run Bitcoin Core on a Raspberry Pi keeps the cost of running a fully validating Bitcoin node far below that of gold. Running Bitcoin Core on computers made pre-2009 offers some assurances against state-level backdoors. Bitcoin Core uses Autotools to help enable these and other usecases.

There's only one thing left to explain. The wiki mentions autoconf, but autogen.sh uses a similarly-named autoreconf. Check out this StackOverflow answer to understand the difference:

autoconf generates the configure script from various input files, some of which are created using other tools like aclocal, automake, etc.

autoreconf is a helper that knows how to call all these tools in the right order

You'll usually just call autoreconf yourself and let it deal with all the lower level tools ....

I was curious what differences there would actually be if I ran autogen.sh on two different computer. Here's a diff with Mac configure script on the left and a Ubuntu one on the right. They're slightly different, but mysteriously much of the difference seems to be whitespace related. Odd! If you have multiple computers you might try generating and comparing configure or Makefile scripts.

We can also see what changes when we pass different flags to configure.

$ ./configure --without-bdb
...
$ cp Makefile without.txt
$ ./configure
...
$ diff without Makefile
305c305
< BDB_LIBS =
---
> BDB_LIBS = -ldb_cxx

-ldb_cxx stands for libdb-cxx, which is the C++ library for Berkeley DB we discussed earlier. This makes sense -- if we remove bdb wallet support then the BDB library code is removed as a dependency. If we disable the wallet entirely with the --disable-wallet flag, it also removes the dependencies for SQLite, which is the successor to BDB.

$ ./configure --disable-wallet
...
$ cp Makefile without.txt
$ ./configure
...
$ cp Makefile with.txt
$ diff without with
305c305
< BDB_LIBS =
---
> BDB_LIBS = -ldb_cxx
505c505
< SQLITE_LIBS =
---
> SQLITE_LIBS = -lsqlite3

Make

Now that we have a Makefile generated, let's use it to build Bitcoin core:

$ make
...

This will take a while. After it's done you should be able to execute the generated bitcoind executable:

$ ./src/bitcoind
2020-12-18T19:18:58Z Bitcoin Core version v21.99.0-1811e488d (release build)
...

Talk to it with the generated bitcoin-cli executable:

$ ./src/bitcoin-cli getblockchaininfo
{
    "chain": "main",
    "blocks": 0,
    ...
}

Don't want to type out relative paths to these files every time you check your wallet balance? That's what make install is for. All this does is copies bitcoind, bitcoin-cli and other files to standard locations where your shell should look for programs. First, kill the ./src/bitcoind process above with control-c (only one bitcoind can run at-a-time). Then:

// note that bitcoind and bitcoin-cli not installed
$ bitcoind
bitcoind: command not found
$ bitconi-cli
bitcoin-cli: command not found

$ make install
...

$ bitcoind
...

// in another terminal
$ bitcoin-cli
{
  ...
}

And now let's return the git repo to it's original state, deleting the configure script, the Makefile, and everything else that was generated along the way:

$ make clean