kc3-lang/angle/src/tests/test_utils/runner

Diff

  • Show log

    Commit

  • Hash : 5407aaa0
    Author : Jamie Madill
    Date : 2019-12-16T15:50:12 

    Re-land "Add new test runner harness." (#2)

Re-land #2 changes:

 * export labels are fixed for the CFI build
 * crash test disabled because of flakiness and issues with asan

Re-land changes:

 * Unit test is suppressed in ASAN
 * --deqp-case is fixed
 * Debug layer errors should correctly work with failure expectations

Original message:

The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:

 * splitting a test set into shards
 * catching and reporting crashes and timeouts
 * outputting to the Chromium JSON test results format
 * multi-process execution

Unit tests are added in test_utils_unittest.cpp.

Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>

  • Properties

  • Git HTTP https://git.kmx.io/kc3-lang/angle.git
    Git SSH git@git.kmx.io:kc3-lang/angle.git
    Public access ? public
    Description

    A conformant OpenGL ES implementation for Windows, Mac, Linux, iOS and Android.

    Homepage

    Github
    Users
    		 thodg_m kc3_lang_org thodg_w www_kmx_io thodg thodg_l
    Tags

  • README.md

  • ANGLE Test Harness

    The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It features:

    • splitting a test set into shards
    • catching and reporting crashes and timeouts
    • outputting to the Chromium JSON test results format
    • multi-process execution

    Command-Line Arguments

    The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the following additional command-line arguments:

    • --shard-count and --shard-index control the test sharding
    • --bot-mode enables multi-process execution and test batching
    • --batch-size limits the number of tests to run in each batch
    • --batch-timeout limits the amount of time spent in each batch
    • --max-processes limits the number of simuntaneous processes
    • --test-timeout limits the amount of time spent in each test
    • --results-file specifies a location for the JSON test result output
    • --results-directory specifies a directory to write test results to
    • --filter-file allows passing a larget gtest_filter via a file

    Implementation Notes

    • The test harness only requires angle_common and angle_util.
    • It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms.
    • It uses rapidjson to read and write JSON files.
    • Timeouts are detected via a watchdog thread.
    • Crashes are handled via ANGLE’s test crash handling code.
    • Currently it does not entirely support Android or Fuchsia.
    • Test execution is not currently deterministic in multi-process mode.
    • We capture stdout to output test failure reasons.

    See the source code for more details: TestSuite.h and TestSuite.cpp.

    Potential Areas of Improvement

    • Deterministic test execution.
    • Using sockets to communicate with test children. Similar to dEQP’s test harness.
    • Closer integration with ANGLE’s test expectations and system config libraries.
    • Supporting a GoogleTest-free integration.

kmx.io     mail     discord kmxgit v0.4.0