tests/generate.py


Log

Author Commit Date CI Message
Patrick Steinhardt 394951ad 2019-06-07T14:11:29 tests: allow for simple data-driven tests Right now, we're not able to use data-driven tests at all. E.g. given a set of tests which we'd like to repeat with different test data, one has to hand-write any outer loop that iterates over the test data and then call each of the test functions. Next to being bothersome, this also has the downside that error reporting is quite lacking, as one never knows which test data actually produced failures. So let's implement the ability to re-run complete test modules with changing test data. To retain backwards compatibility and enable trivial addition of new runs with changed test data, we simply extend clar's generate.py. Instead of scanning for a single `_initialize` function, each test module may now implement multiple `_initialize_foo` functions. The generated test harness will then run all test functions for each of the provided initializer functions, making it possible to provide different test data inside of each of the initializer functions. Example: ``` void test_git_example__initialize_with_nonbare_repo(void) { g_repo = cl_git_sandbox_init("testrepo"); } void test_git_example__initialize_with_bare_repo(void) { g_repo = cl_git_sandbox_init("testrepo.git"); } void test_git_example__cleanup(void) { cl_git_sandbox_cleanup(); } void test_git_example__test1(void) { test1(); } void test_git_example__test2(void) { test2(); } ``` Executing this test module will cause the following output: ``` $ ./libgit2_clar -sgit::example git::example (with nonbare repo).. git::example (with bare repo).. ```
Patrick Steinhardt 583e4141 2017-08-30T14:35:57 tests: deterministically generate test suite definitions The script "generate.py" is used to parse all test source files for unit tests. These are then written into a "clar.suite" file, which can be included by the main test executable to make available all test suites and unit tests. Our current algorithm simply collects all test suites inside of a dict, iterates through its items and dumps them in a special format into the file. As the order is not guaranteed to be deterministic for Python dictionaries, this may result in arbitrarily ordered C structs. This obviously defeats the purpose of reproducible builds, where the same input should always result in the exact same output. Fix this issue by sorting the test suites by name previous to dumping them as structs. This enables reproducible builds for the libgit2_clar file.
Patrick Steinhardt 8d22bcea 2017-05-10T12:21:53 generate.py: generate clar cache in binary directory The source directory should usually not be touched when using out-of-tree builds. But next to the previously fixed "clar.suite" file, we are also writing the ".clarcache" into the project's source tree, breaking the builds. Fix this by also honoring the output directory for the ".clarcache" file.
Patrick Steinhardt 9e240bd2 2017-05-10T12:02:03 generate.py: enable overriding path of generated clar.suite The generate.py script will currently always write the generated clar.suite file into the scanned directory, breaking out-of-tree builds with read-only source directories. Fix this issue by adding another option to allow overriding the output path of the generated file.
Patrick Steinhardt 0a513a94 2017-05-10T11:46:00 generate.py: disallow generating test suites for multiple paths Our generate.py script is used to extract and write test suite declarations into the clar.suite file. As is, the script accepts multiple directories on the command line and will generate this file for each of these directories. The generate.py script will always write the clar.suite file into the directory which is about to be scanned. This actually breaks out-of-tree builds with libgit2, as the file will be generated in the source tree instead of in the build tree. This is noticed especially in the case where the source tree is mounted read-only, rendering us unable to build unit tests. Due to us accepting multiple paths which are to be scanned, it is not trivial to fix though. The first solution which comes into mind would be to re-create the directory hierarchy at a given output path or use unique names for the clar.suite files, but this is rather cumbersome and magical. The second and cleaner solution would be to fold all directories into a single clar.suite file, but this would probably break some use-cases. Seeing that we do not ever pass multiple directories to generate.py, we will now simply retire support for this. This allows us to later on introduce an additional option to specify the path where the clar.suite file will be generated at, defaulting to "clar.suite" inside of the scanned directory.
Linquize 3a1eb9e5 2014-12-07T22:09:00 Update clar to e3985dd
Ben Straub 17820381 2013-11-14T14:05:52 Rename tests-clar to tests