|
b4fb7cc9
|
2020-10-15T08:58:06
|
|
Test Runner: Print reproduction steps on a batch failure.
This prints out a large gtest_filter of the tests in a batch. Makes
reproducing a particular failure much easier.
In the future we could potentially look at using an encoding to
reproduce a specific batch: e.g. a flag that acceps a batch / shard
number and runs those tests based on using the same internal counts.
Bug: angleproject:5172
Change-Id: Iefad8f5ca23761417ad1ceb11fd664c9cf23c8be
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2475455
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
ad80b184
|
2020-10-13T17:15:59
|
|
Test Runner: List tests explicitly without GTest.
This dramatically speeds up Android shards that use batches of
hudreds of tests. Because ANGLE already has the test list, it no
longer needs to call GTest with an enourmous gtest_filter.
Bug: angleproject:5164
Change-Id: I28fee2f36c50006f2a35a4dcd90f44f8ebe4f78c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2468464
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
336202d7
|
2020-10-09T16:43:32
|
|
Test Runner: Pass batch ID to child processes.
This will let both the ANGLETest and dEQP test runners handle child
processes correctly. The existing ANGLETest runner will reuse displays,
and the dEQP test runner will write to a batch-unique filename.
This solves the problem of the multi-process --bot-mode writing to the
same file from multiple children simultaneously.
Long-term it will be good to include the dEQP QPA files into the test
artifacts.
Also disables flushing after every log write when running as a child
process. This hugely improves performance on machines with no SSD.
Test: https://chromium-swarm.appspot.com/task?id=4f2b87c4f8234910
Bug: angleproject:5157
Change-Id: I226d24adf55e0f8b98e5a8e7947862e6906b4c40
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464424
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Ian Elliott <ianelliott@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d3aa1e9c
|
2020-10-03T01:31:33
|
|
Test Runner: Pass dEQP args to child processes.
This fixes the sharding to use the specified back-end. With
the bug ANGLE would start the platform default back-end instead.
Bug: angleproject:3162
Change-Id: Ib2453dd3c58ea40fb36619301865ae0818038d15
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447043
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
acbe145c
|
2020-07-02T14:59:55
|
|
Test Runner: Batch tests by config.
This CL sorts tests into subprocesses organized by a single config.
It will allow for the test runner to use a single EGL display per
process in most cases. It doesn't implement the display sharing right
now, that'll be in a forthcoming CL. It assumes test configs are split
between / and (optional ending) __ characters.
Bug: angleproject:3162
Change-Id: I28e8b4687640da6e501f08b8b9347185c99ce30f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2280403
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fe902f4b
|
2020-06-19T14:28:36
|
|
Test Runner: Stub histogram writer.
The Android tests require a valid output for the perf test
results. The smallest valid output is an empty array.
Bug: angleproject:3162
Change-Id: I1bf7602b02358d634347d587b406c0c2d7c838a2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2254088
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|