Benchmarking Rust Code with #[bench] and Criterion

Benchmarking is a critical step in software development when it comes to optimizing and ensuring your Rust code runs efficiently. Rust provides a built-in library path #[bench] for benchmarking, although it is more idiomatic and thorough to use the Criterion library for more comprehensive analysis.

In this article, we will explore both methods of performing benchmarks in Rust, with an emphasis on best practices and detailed explanations of the output.

Using #[bench] for Benchmarking

In Rust, the #[bench] attribute is similar to the #[test] attribute but is used for creating benchmark tests. Note that as of writing, you need to enable a special compiler flag to use the standard benchmarking library.

Setup

First, ensure you are using the nightly toolchain as the #[bench] feature is only available there:

rustup default nightly

Next, create a new Rust project if you haven't already:

cargo new rust_benchmark_example

Edit your Cargo.toml file to enable benchmarks:

[[bench]]
name = "my_benchmark"
path = "benches/my_benchmark.rs"

Create a new file my_benchmark.rs within the benches directory and add the following content:

#![feature(test)]
extern crate test;

#[bench]
fn my_benchmark(b: &mut test::Bencher) {
    b.iter(|| {
        // code to benchmark
    });
}

Run your benchmarks using Cargo with the following command:

cargo bench

Benchmarking with Criterion

Criterion offers a more flexible and powerful framework compared to #[bench]. With Criterion, you can handle complex scenarios such as parameterized benchmarks and custom data collection.

Setup

Add Criterion as a dependency in your Cargo.toml file:

[dependencies]
criterion = "0.3"

Create a new file criterion_bench.rs in the benches directory:

use criterion::{black_box, criterion_group, criterion_main, Criterion};

pub fn criterion_benchmark(c: &mut Criterion) {
    c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
}

fn fibonacci(n: u64) -> u64 {
    match n {
        0 => 1,
        1 => 1,
        n => fibonacci(n - 1) + fibonacci(n - 2),
    }
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

Note the use of black_box, which prevents compiler optimizations that would otherwise skip calculations resulting in skewed benchmarks.

Run your Criterion benchmarks with the same command:

cargo bench

Understanding Output

The output from running your benchmarks will provide insights into the performance of your code. With #[bench], you will see execution time metrics. Criterion will offer more detailed statistical analysis, including multiple runs, averages, and estimates of statistical confidence.

Leverage these tools wisely to identify bottlenecks and optimize performance-critical parts of your Rust application. Happy coding and optimizing!