|
05b6740c
|
2020-08-09T12:23:42
|
|
TestRunner: Fix test result handling.
The bots expect us to explicitly label "is_unexpected" for all
unexpected results (e.g. failures).
Also they construct a test filter from our test names so our result
names have to match the tests exactly. That means removing prefixes
like we have for the test suite name.
Bug: angleproject:3162
Change-Id: I1a02ff8e26545b10a8829a3ee47be91f7459aa65
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2345028
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
debe4683
|
2020-07-15T14:36:33
|
|
Test Runner: Accept Chromium args.
This change lets the test runner accept the same isolated script
parameter syntax as Chromium. It keeps the old parameter names so
the tests accept both. This will let us more easily integrate with
the Chromium Android test_runner.py script.
Also update the README which was missing a few flags.
Bug: angleproject:3162
Change-Id: Ic22b289a095418ffdaa19a04caa1b45c6c657872
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2300530
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
acbe145c
|
2020-07-02T14:59:55
|
|
Test Runner: Batch tests by config.
This CL sorts tests into subprocesses organized by a single config.
It will allow for the test runner to use a single EGL display per
process in most cases. It doesn't implement the display sharing right
now, that'll be in a forthcoming CL. It assumes test configs are split
between / and (optional ending) __ characters.
Bug: angleproject:3162
Change-Id: I28e8b4687640da6e501f08b8b9347185c99ce30f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2280403
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fe902f4b
|
2020-06-19T14:28:36
|
|
Test Runner: Stub histogram writer.
The Android tests require a valid output for the perf test
results. The smallest valid output is an empty array.
Bug: angleproject:3162
Change-Id: I1bf7602b02358d634347d587b406c0c2d7c838a2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2254088
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
e9a0cb87
|
2020-05-18T17:45:52
|
|
Fix assertion when running tests without suffix on Windows
On Windows, we may omit suffix ".exe" when running an executable.
However, this will trigger an assertion failure in current code and this
CL is to fix this issue.
Bug: angleproject:4640
Change-Id: I7edfdc0b4a7c590c874817530b19a812018b9288
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2206425
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
|
390be9bb
|
2020-03-01T08:47:37
|
|
Test Runner: Use system path for gtest-internal.
This ensures we use the correct internal path in Chromium builds as
well.
Bug: angleproject:3162
Change-Id: I036dcbd8802f67162e5f5eccb6158910f0a5efcc
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2082134
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
593706bc
|
2020-02-10T16:46:52
|
|
Test Runner: Increase default timeouts.
In particular the Debug timeout was not generous enough. Increase the
test timeout to 20 seconds in Release and 60 seconds in Debug. Increase
the batch timeout to 240 seconds in Release and 600 seconds in Debug.
Bug: angleproject:3162
Change-Id: I0bd0ea18cb5cdab005143b8deed9500d9d2e8ea1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2047415
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
ee07cb31
|
2020-01-29T18:02:50
|
|
Test Harness: Allow filter with sharding params.
Some tweaks to when we call gtest initialization allows this to work.
Bug: angleproject:3162
Change-Id: I75159108d3f746f666e57c1a6495a8c8616ef1cf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2024335
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
238adb7a
|
2020-01-13T17:21:53
|
|
Fix test harness running disabled tests.
Adds special handling for the DISABLED_ GTest test name. Fixes the test
for the test suite harness itself when run with the new "--bot-mode"
flag.
Bug: angleproject:3162
Change-Id: Idf34098d2d6bcb78263773be7b01c55df9ec624b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1998660
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|