|
e5c5efc0
|
2021-02-05T02:39:55
|
|
Revert "Test Runner: Pick single/multi-process mode automatically."
This reverts commit 573d7f34a462fa5b2ca767dd84773ae6fa460d8a.
Reason for revert: Breaking ANGLE roll.
No-try: true
Original change's description:
> Test Runner: Pick single/multi-process mode automatically.
>
> This enables multi-process when running with more than one test
> config. This saves developers from remembering to add the arg.
> It can also simplify our test configurations by allowing us to omit
> the default arguments. On Android by default we stay with single
> process. It also changes the --bot-mode flag into --single-process
> and --multi-process which override the default behaviour.
>
> Bug: angleproject:5585
> Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=kbr@chromium.org,syoussefi@chromium.org,jonahr@google.com,jmadill@chromium.org,jonahr@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: angleproject:5585
Change-Id: I11ee1034c301aa246c4b03e29ff9b56235a1005c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2676247
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
573d7f34
|
2021-02-01T10:36:56
|
|
Test Runner: Pick single/multi-process mode automatically.
This enables multi-process when running with more than one test
config. This saves developers from remembering to add the arg.
It can also simplify our test configurations by allowing us to omit
the default arguments. On Android by default we stay with single
process. It also changes the --bot-mode flag into --single-process
and --multi-process which override the default behaviour.
Bug: angleproject:5585
Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
85efb9d5
|
2021-01-26T14:39:19
|
|
Log dEQP QPA files as test artifacts.
This adds artifact output to the test runner. We add a fake test
at the start of a test run that owns the artifacts.
Bug: angleproject:5236
Change-Id: Ice8001bf1f2aafbd8123fee76e0e7fcc3e5a8a0c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2657535
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Mohan Maiya <m.maiya@samsung.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
e4b4ff7a
|
2021-01-12T13:48:20
|
|
Test Runner: Add a slow tests list.
Each slow test in the list will use a 3x longer timeout. The list is
implemented using the same filter wildcard we use in the test
expectations logic.
We can test this out using a slow D3D11 varying test.
Bug: angleproject:5076
Bug: angleproject:5496
Change-Id: I31cf45e6ee8a8bbd6e460d675ff8a0cf5f19a504
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2625172
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
3e5b6f81
|
2020-10-31T12:33:28
|
|
Enable writing histogram-set-json-format.
This uses the protobuf histogram functionality to build up a histogram
set. The test suites then output the histograms to JSON. This is only
implemented for angle_perftests.
Bug: angleproject:5161
Change-Id: Ia5a7868e8d8dcf4f13d83115ae622828c63ef0d9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2482295
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
c06a424f
|
2020-11-02T21:13:39
|
|
Test Runner: Robustly handle unexpected crashes.
This adds the ability for the test runner to parse unexpected crashes
from the stdout. It also processes the stdout to determine which tests
failed. Tests that run after the crash are re-tried in a follow-up
child process.
Will allow for the test runner to handle very crashy test suites and
also processes crashes from win-asan and other configs where it is
harder to intercept crashes.
Bug: angleproject:5251
Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
59fbb989
|
2020-11-02T13:27:52
|
|
Test Runner: Add ability to retry flaky tests.
Bug: angleproject:5273
Change-Id: Ie89559bb0897a04213981aa8fe4e2f2bfe78959a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513287
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
5c4c37dc
|
2020-10-25T15:55:45
|
|
Capture/Replay: Test improvements.
Streamlines some arguments.
Use a common subdirectory in out/.
Show process name when logging.
Clean up environment variable handling.
Bug: angleproject:5247
Change-Id: I85eb5b8194a64f90691c4d03db3ee65c5e9b2313
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497560
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
31868665
|
2020-10-23T10:54:02
|
|
Test Runner: Only run watchdog in child processes.
For single test execution we don't need to be running the watchdog
thread. This only serves to cause timeouts in slow running tests.
Adds a new method to keep the --batch-id argument in the list of
parameters so other functions can process them.
Bug: angleproject:5124
Bug: angleproject:5218
Change-Id: I958230149b7d79830802cd55f67f0190de607200
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2495282
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
26cd1cc6
|
2020-10-18T11:44:27
|
|
Perf Tests: Add stdout printing arg to bot mode.
This can be useful for debugging and also for perf tests. The perf
tests output gets scraped from stdout.
Bug: angleproject:5124
Change-Id: Id5005df3f2b66be9497a9ba422abd47b95a1a330
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2486097
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
b4fb7cc9
|
2020-10-15T08:58:06
|
|
Test Runner: Print reproduction steps on a batch failure.
This prints out a large gtest_filter of the tests in a batch. Makes
reproducing a particular failure much easier.
In the future we could potentially look at using an encoding to
reproduce a specific batch: e.g. a flag that acceps a batch / shard
number and runs those tests based on using the same internal counts.
Bug: angleproject:5172
Change-Id: Iefad8f5ca23761417ad1ceb11fd664c9cf23c8be
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2475455
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
ad80b184
|
2020-10-13T17:15:59
|
|
Test Runner: List tests explicitly without GTest.
This dramatically speeds up Android shards that use batches of
hudreds of tests. Because ANGLE already has the test list, it no
longer needs to call GTest with an enourmous gtest_filter.
Bug: angleproject:5164
Change-Id: I28fee2f36c50006f2a35a4dcd90f44f8ebe4f78c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2468464
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
336202d7
|
2020-10-09T16:43:32
|
|
Test Runner: Pass batch ID to child processes.
This will let both the ANGLETest and dEQP test runners handle child
processes correctly. The existing ANGLETest runner will reuse displays,
and the dEQP test runner will write to a batch-unique filename.
This solves the problem of the multi-process --bot-mode writing to the
same file from multiple children simultaneously.
Long-term it will be good to include the dEQP QPA files into the test
artifacts.
Also disables flushing after every log write when running as a child
process. This hugely improves performance on machines with no SSD.
Test: https://chromium-swarm.appspot.com/task?id=4f2b87c4f8234910
Bug: angleproject:5157
Change-Id: I226d24adf55e0f8b98e5a8e7947862e6906b4c40
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464424
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Ian Elliott <ianelliott@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d3aa1e9c
|
2020-10-03T01:31:33
|
|
Test Runner: Pass dEQP args to child processes.
This fixes the sharding to use the specified back-end. With
the bug ANGLE would start the platform default back-end instead.
Bug: angleproject:3162
Change-Id: Ib2453dd3c58ea40fb36619301865ae0818038d15
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447043
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
acbe145c
|
2020-07-02T14:59:55
|
|
Test Runner: Batch tests by config.
This CL sorts tests into subprocesses organized by a single config.
It will allow for the test runner to use a single EGL display per
process in most cases. It doesn't implement the display sharing right
now, that'll be in a forthcoming CL. It assumes test configs are split
between / and (optional ending) __ characters.
Bug: angleproject:3162
Change-Id: I28e8b4687640da6e501f08b8b9347185c99ce30f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2280403
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fe902f4b
|
2020-06-19T14:28:36
|
|
Test Runner: Stub histogram writer.
The Android tests require a valid output for the perf test
results. The smallest valid output is an empty array.
Bug: angleproject:3162
Change-Id: I1bf7602b02358d634347d587b406c0c2d7c838a2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2254088
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|