|
1d019f49
|
2021-05-13T10:33:06
|
|
Test Runner: Fixes to skipped tests.
Uses a different token for skipped and not run tests in the json. The
token isn't standard but should only be visible when batches report
to the parent process that a test wasn't run. It shouldn't be visible
to the scripts that consume test results on the bots.
Also fixes the expected result for skipped tests to also be "SKIP".
Bug: angleproject:5951
Change-Id: I0bd7d13b2b25b583f9c42718786cb588bacd4bc6
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2894524
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
9a219a14
|
2021-05-12T09:52:16
|
|
Test Runner: Handle skipped gtests.
GTEST_SKIP() allows us to skip tests using the GoogleTest APIs.
This CL updates our test runner code to handle these skips.
Bug: angleproject:5951
Change-Id: Idef5802c7c81bd92226f5a93281a1cea0f1b24ee
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2892272
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e5c5efc0
|
2021-02-05T02:39:55
|
|
Revert "Test Runner: Pick single/multi-process mode automatically."
This reverts commit 573d7f34a462fa5b2ca767dd84773ae6fa460d8a.
Reason for revert: Breaking ANGLE roll.
No-try: true
Original change's description:
> Test Runner: Pick single/multi-process mode automatically.
>
> This enables multi-process when running with more than one test
> config. This saves developers from remembering to add the arg.
> It can also simplify our test configurations by allowing us to omit
> the default arguments. On Android by default we stay with single
> process. It also changes the --bot-mode flag into --single-process
> and --multi-process which override the default behaviour.
>
> Bug: angleproject:5585
> Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=kbr@chromium.org,syoussefi@chromium.org,jonahr@google.com,jmadill@chromium.org,jonahr@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: angleproject:5585
Change-Id: I11ee1034c301aa246c4b03e29ff9b56235a1005c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2676247
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
573d7f34
|
2021-02-01T10:36:56
|
|
Test Runner: Pick single/multi-process mode automatically.
This enables multi-process when running with more than one test
config. This saves developers from remembering to add the arg.
It can also simplify our test configurations by allowing us to omit
the default arguments. On Android by default we stay with single
process. It also changes the --bot-mode flag into --single-process
and --multi-process which override the default behaviour.
Bug: angleproject:5585
Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
c06a424f
|
2020-11-02T21:13:39
|
|
Test Runner: Robustly handle unexpected crashes.
This adds the ability for the test runner to parse unexpected crashes
from the stdout. It also processes the stdout to determine which tests
failed. Tests that run after the crash are re-tried in a follow-up
child process.
Will allow for the test runner to handle very crashy test suites and
also processes crashes from win-asan and other configs where it is
harder to intercept crashes.
Bug: angleproject:5251
Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
59fbb989
|
2020-11-02T13:27:52
|
|
Test Runner: Add ability to retry flaky tests.
Bug: angleproject:5273
Change-Id: Ie89559bb0897a04213981aa8fe4e2f2bfe78959a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513287
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
47e66f22
|
2020-10-28T14:24:37
|
|
Revert "Re-enable crash testing in angle_unittests."
This reverts commit 21add1240950c2e2a6ec997f9a128005d84266b6.
Reason for revert: Fails on Windows 7 x64 Chromium config.
https://ci.chromium.org/p/chromium/builders/ci/Win%207%20Tests%20x64%20%281%29/73194
Original change's description:
> Re-enable crash testing in angle_unittests.
>
> This is an experimental test to see if this works now.
> Disabled on Win/ASAN because of conflicts with the
> crash callback.
>
> Bug: angleproject:5239
> Change-Id: I280b18ab8846de79b561f26716df9ba374cfd9a5
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2496040
> Reviewed-by: Jamie Madill <jmadill@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,jmadill@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: angleproject:5239
Change-Id: I83c150c2c2090b7d1aeb78e53e96115a167a7491
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2505479
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
21add124
|
2020-10-25T15:55:45
|
|
Re-enable crash testing in angle_unittests.
This is an experimental test to see if this works now.
Disabled on Win/ASAN because of conflicts with the
crash callback.
Bug: angleproject:5239
Change-Id: I280b18ab8846de79b561f26716df9ba374cfd9a5
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2496040
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
4968245c
|
2020-10-25T12:58:15
|
|
Adjust TestSuite test timeout values.
This should prevent flakiness in this test.
Bug: angleproject:5245
Change-Id: I841de2cee093ea04b35763314f0a4f5497bcf778
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497223
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
1fd3e5d8
|
2020-10-01T10:00:18
|
|
Test Runner: Fix race in watchdog timeouts.
This fixes a TSAN warning that popped up with the standalone test
runner. The watchdog timer actually was never started so timeouts
were not working as intended. It also switches the timeout mode
to call _Exit which skips all the atexit handlers and avoids some
races on teardown. This change also speeds up the TestSuiteTest.
Also a small fix to GetTempDir that was including an extra path
separator on Windows.
Bug: angleproject:3162
Bug: angleproject:5117
Change-Id: I0e7880a08b61bbb6e30c65665d5c0acec2d78db2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2442381
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|