thodg/libgit2/tests/benchmarks/README.md
-
Show log
Commit
-
Hash : 93037bab
Author :
Date : 2022-02-22T22:00:37
tests: add benchmark tests Add a benchmark test suite that wraps hyperfine and is suitable for producing data about test runs of a CLI or A/B testing CLIs.
libgit2 benchmarks
This folder contains the individual benchmark tests for libgit2, meant for understanding the performance characteristics of libgit2, comparing your development code to the existing libgit2 code, or comparing libgit2 to the git reference implementation.
Running benchmark tests
Benchmark tests can be run in several different ways: running all benchmarks, running one (or more) suite of benchmarks, or running a single individual benchmark. You can target either an individual version of a CLI, or you can A/B test a baseline CLI against a test CLI.
Specifying the command-line interface to test
By default, the git in your path is benchmarked. Use the
-c (or --cli) option to specify the command-line interface
to test.
Example: libgit2_bench --cli git2_cli will run the tests against
git2_cli.
Running tests to compare two different implementations
You can compare a baseline command-line interface against a test
command-line interface using the -b (or–baseline-cli) option. Example:libgit2bench –baseline-cli git –cli git2_cliwill run the tests against bothgitandgit2_cli. ### Running individual benchmark tests Similar to how a test suite or individual test is specified in [clar](https://github.com/clar-test/clar), the-s(or–suite) option may be used to specify the suite or individual test to run. Like clar, the suite and test name are separated by::, and like clar, this is a prefix match. Examples: *libgit2_bench -shash_objectwill run the tests in thehash_objectsuite. *libgit2_bench -shash_object::random_1kbwill run thehash_object::random_1kbtest. *libgit2_bench -shash_object::randomwill run all the tests that begin withhash_object::random. ## Writing benchmark tests Benchmark tests are meant to be easy to write. Each individual benchmark is a shell script that allows it to do set up (eg, creating or cloning a repository, creating temporary files, etc), then running benchmarks, then teardown. Thebenchmark_helpers.shscript provides many helpful utility functions to allow for cross-platform benchmarking, as well as a wrapper forhyperfinethat is suited to testing libgit2. Note that the helper script must be included first, at the beginning of the benchmark test. ### Benchmark example This simplistic example compares the speed of running thegit helpcommand in the baseline CLI to the test CLI. ```bash #!/bin/bash -e # include the benchmark library . "$(dirname "$0")/benchmark_helpers.sh" # run the "help" command; this will benchmarkgit2_cli helpgitbench help ``` ### Naming The filename of the benchmark itself is important. A benchmark's filename should be the name of the benchmark suite, followed by two underscores, followed by the name of the benchmark. For example,hash-object__random_1kbis therandom_1kbtest in thehash-objectsuite. ### Options Thegitbenchfunction accepts several options. *–sandbox <path>The name of a test resource (in thetests/resources` directory).
This will be copied as-is to the sandbox location before test
execution. This is copied _before the prepare script is run.
This option may be specified multiple times.
--repository <path>
The name of a test resource repository (in the tests/resources
directory). This repository will be copied into a sandbox location
before test execution, and your test will run in this directory.
This is copied before the prepare script is run.
--prepare <script>
A script to run before each invocation of the test is run. This can
set up data for the test that will not be timed. This script is run
in bash on all platforms.
Several helper functions are available within the context of a prepare
script:
flush_disk_cache
Calling this will flush the disk cache before each test run.
This should probably be run at the end of the prepare script.
create_random_file <path> [<size>]
Calling this will populate a file at the given path with size
bytes of random data.
create_text_file <path> [<size>]
Calling this will populate a file at the given path with size
bytes of predictable text, with the platform line endings. This
is preferred over random data as it’s reproducible.
--warmup <n>
Specifies that the test should run n times before actually measuring
the timing; useful for “warming up” a cache.
Source
-
tests/benchmarks/README.md
-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121
# libgit2 benchmarks This folder contains the individual benchmark tests for libgit2, meant for understanding the performance characteristics of libgit2, comparing your development code to the existing libgit2 code, or comparing libgit2 to the git reference implementation. ## Running benchmark tests Benchmark tests can be run in several different ways: running all benchmarks, running one (or more) suite of benchmarks, or running a single individual benchmark. You can target either an individual version of a CLI, or you can A/B test a baseline CLI against a test CLI. ### Specifying the command-line interface to test By default, the `git` in your path is benchmarked. Use the `-c` (or `--cli`) option to specify the command-line interface to test. Example: `libgit2_bench --cli git2_cli` will run the tests against `git2_cli`. ### Running tests to compare two different implementations You can compare a baseline command-line interface against a test command-line interface using the `-b (or `--baseline-cli`) option. Example: `libgit2_bench --baseline-cli git --cli git2_cli` will run the tests against both `git` and `git2_cli`. ### Running individual benchmark tests Similar to how a test suite or individual test is specified in [clar](https://github.com/clar-test/clar), the `-s` (or `--suite`) option may be used to specify the suite or individual test to run. Like clar, the suite and test name are separated by `::`, and like clar, this is a prefix match. Examples: * `libgit2_bench -shash_object` will run the tests in the `hash_object` suite. * `libgit2_bench -shash_object::random_1kb` will run the `hash_object::random_1kb` test. * `libgit2_bench -shash_object::random` will run all the tests that begin with `hash_object::random`. ## Writing benchmark tests Benchmark tests are meant to be easy to write. Each individual benchmark is a shell script that allows it to do set up (eg, creating or cloning a repository, creating temporary files, etc), then running benchmarks, then teardown. The `benchmark_helpers.sh` script provides many helpful utility functions to allow for cross-platform benchmarking, as well as a wrapper for `hyperfine` that is suited to testing libgit2. Note that the helper script must be included first, at the beginning of the benchmark test. ### Benchmark example This simplistic example compares the speed of running the `git help` command in the baseline CLI to the test CLI. ```bash #!/bin/bash -e # include the benchmark library . "$(dirname "$0")/benchmark_helpers.sh" # run the "help" command; this will benchmark `git2_cli help` gitbench help ``` ### Naming The filename of the benchmark itself is important. A benchmark's filename should be the name of the benchmark suite, followed by two underscores, followed by the name of the benchmark. For example, `hash-object__random_1kb` is the `random_1kb` test in the `hash-object` suite. ### Options The `gitbench` function accepts several options. * `--sandbox <path>` The name of a test resource (in the `tests/resources` directory). This will be copied as-is to the sandbox location before test execution. This is copied _before_ the `prepare` script is run. This option may be specified multiple times. * `--repository <path>` The name of a test resource repository (in the `tests/resources` directory). This repository will be copied into a sandbox location before test execution, and your test will run in this directory. This is copied _before_ the `prepare` script is run. * `--prepare <script>` A script to run before each invocation of the test is run. This can set up data for the test that will _not_ be timed. This script is run in bash on all platforms. Several helper functions are available within the context of a prepare script: * `flush_disk_cache` Calling this will flush the disk cache before each test run. This should probably be run at the end of the `prepare` script. * `create_random_file <path> [<size>]` Calling this will populate a file at the given `path` with `size` bytes of random data. * `create_text_file <path> [<size>]` Calling this will populate a file at the given `path` with `size` bytes of predictable text, with the platform line endings. This is preferred over random data as it's reproducible. * `--warmup <n>` Specifies that the test should run `n` times before actually measuring the timing; useful for "warming up" a cache.