DuckDB has an extensive benchmark suite. When making changes that have potential performance implications, it is important to run these benchmarks to detect potential performance regressions.

Getting Started

To build the benchmark suite, run the following command in the DuckDB repository:

BUILD_BENCHMARK=1 BUILD_TPCH=1 make

Listing Benchmarks

To list all available benchmarks, run:

build/release/benchmark/benchmark_runner --list

Running Benchmarks

Running a Single Benchmark

To run a single benchmark, issue the following command:

build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark

The output will be printed to stdout in CSV format, in the following format:

name	run	timing
benchmark/micro/nulls/no_nulls_addition.benchmark	1	0.121234
benchmark/micro/nulls/no_nulls_addition.benchmark	2	0.121702
benchmark/micro/nulls/no_nulls_addition.benchmark	3	0.122948
benchmark/micro/nulls/no_nulls_addition.benchmark	4	0.122534
benchmark/micro/nulls/no_nulls_addition.benchmark	5	0.124102

You can also specify an output file using the --out flag. This will write only the timings (delimited by newlines) to that file.

build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark --out=timings.out

The output will contain the following:

0.182472
0.185027
0.184163
0.185281
0.182948

Running Multiple Benchmark Using a Regular Expression

You can also use a regular expression to specify which benchmarks to run. Be careful of shell expansion of certain regex characters (e.g., * will likely be expanded by your shell, hence this requires proper quoting or escaping).

build/release/benchmark/benchmark_runner "benchmark/micro/nulls/.*"

Running All Benchmarks

Not specifying any argument will run all benchmarks.

build/release/benchmark/benchmark_runner

Other Options

The --info flag gives you some other information about the benchmark.

build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark --info

display_name:NULL Addition (no nulls)
group:micro
subgroup:nulls

The --query flag will print the query that is run by the benchmark.

SELECT min(i + 1) FROM integers;

The --profile flag will output a query tree.