kc3-lang/angle/src/tests/test_utils/runner

Diff

  • Show log

    Commit

  • Hash : e495e7fd
    Author : Jamie Madill
    Date : 2022-10-14T13:52:47 

    Redo perf and trace test parameters.

- Moves common argument parsing code into test_util.
- Changes the perf test arg parsing to use the common functions.
- Adds new --use-angle and --use-gl parameters to the trace tests.
- Also adds new --offscreen and --vsync parameters to the traces.
- Removes the now unneeded --enable-all-trace-tests argument.
- Both --arg=value and --arg value work in test suites now.

Now, instead of using --enable-all-trace-tests you can specify the
backend with --use-angle=swiftshader, --offscreen, or combinations
of those parameters. The test names are the same as they were
before, but only the configured tests will run in a session. We
could opt to simplify the test names in later CLs if we want to
simplify the test running. Ideally we'd keep the perf reporting
the same because then we'd keep the time series the same on the
test infra.

This also allows us to split up the trace tests into separate
targets on the bots, which will better allow us to control the
workloads and sampling of the tests. For example:

- angle_perftests becomes
  - angle_perftests (microbenchmarks)
  - angle_trace_perf_vulkan_tests (traces with vulkan back-end)
  - angle_trace_perf_native_tests (traces with system GL)

Bug: angleproject:7755
Change-Id: I537168f3a6de96425dfda05ed98220eff9b19b76
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3956937
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>

  • Properties

  • Git HTTP https://git.kmx.io/kc3-lang/angle.git
    Git SSH git@git.kmx.io:kc3-lang/angle.git
    Public access ? public
    Description

    A conformant OpenGL ES implementation for Windows, Mac, Linux, iOS and Android.

    Homepage

    Github
    Users
    		 thodg_m kc3_lang_org thodg_w www_kmx_io thodg thodg_l
    Tags

  • README.md

  • ANGLE Test Harness

    The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It features:

    • splitting a test set into shards
    • catching and reporting crashes and timeouts
    • outputting to the Chromium JSON test results format
    • multi-process execution

    Command-Line Arguments

    The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the following additional command-line arguments:

    • --batch-size limits the number of tests to run in each batch
    • --batch-timeout limits the amount of time spent in each batch
    • --bot-mode enables multi-process execution and test batching
    • --debug-test-groups dumps the test config categories when using bot-mode
    • --filter-file allows passing a larger gtest_filter via a file
    • --histogram-json-file outputs a formatted JSON file for perf dashboards
    • --max-processes limits the number of simuntaneous processes
    • --results-directory specifies a directory to write test results to
    • --results-file specifies a location for the JSON test result output
    • --shard-count and --shard-index control the test sharding
    • --test-timeout limits the amount of time spent in each test
    • --flaky-retries allows for tests to fail a fixed number of times and still pass
    • --disable-crash-handler forces off OS-level crash handling
    • --isolated-outdir specifies a test artifacts directory
    • --max-failures specifies a count of failures after which the harness early exits.

    --isolated-script-test-output and --isolated-script-perf-test-output mirror --results-file and --histogram-json-file respectively.

    As well as the custom command-line arguments we support a few standard GoogleTest arguments:

    • gtest_filter works as it normally does with GoogleTest
    • gtest_also_run_disabled_tests works as it normally does as well

    Other GoogleTest arguments are not supported although they may work.

    Implementation Notes

    • The test harness only requires angle_common and angle_util.
    • It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms.
    • It uses rapidjson to read and write JSON files.
    • Test execution is not currently deterministic in multi-process mode.

    Normal Mode vs Bot Mode

    The test runner has two main modes of operation: normal and bot mode.

    During normal mode:

    • Tests are executed single-process and single-thread.
    • The test runner executes the GoogleTest Run function.
    • We use a TestEventListener to record test results for our output JSON file.
    • A watchdog thread will force a fast exit if no test results get recorded after a timeout.
    • Crashes are handled via ANGLE’s test crash handling code.

    During bot mode:

    • Tests are run in multiple processes depending on the system processor count.
    • A server process records the child processes’ stdout and stderr.
    • The server terminates a child process if there’s no progress after a timeout.
    • The server sorts work into batches according to the back-end configuration.
    • This prevents driver errors from using multiple back-ends in the same process.
    • Batches are striped to help split up slow groups of tests.
    • The server passes test batches to child processes via a gtest_filter file.
    • Bot mode does not work on Android or Fuchsia.

    See the source code for more details: TestSuite.h and TestSuite.cpp.

    Potential Areas of Improvement

    • Deterministic test execution.
    • Using sockets to communicate with test children. Similar to dEQP’s test harness.
    • Closer integration with ANGLE’s test expectations and system config libraries.
    • Supporting a GoogleTest-free integration.

kmx.io     mail     discord kmxgit v0.4.0