|
72d2bd0c
|
2021-06-21T09:59:21
|
|
Allow capturing process stdout and stderr interleaved
The test utils are enhanced to allow redirecting stderr to stdout. This
is in preparation for a change that makes the test runner capture stderr
together with stdout. Currently, on failure logs originating from
UNIMPLEMENTED() and other such macros are not captured.
Bug: angleproject:6077
Change-Id: I7a3c6c4732a59dac3ff0cc20a7835d5ed6f0f22e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2976183
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
c06a424f
|
2020-11-02T21:13:39
|
|
Test Runner: Robustly handle unexpected crashes.
This adds the ability for the test runner to parse unexpected crashes
from the stdout. It also processes the stdout to determine which tests
failed. Tests that run after the crash are re-tried in a follow-up
child process.
Will allow for the test runner to handle very crashy test suites and
also processes crashes from win-asan and other configs where it is
harder to intercept crashes.
Bug: angleproject:5251
Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
95a736bd
|
2019-10-09T10:26:25
|
|
Reland "Add more test_utils functions."
This is a reland of 5fcfcea4a9379633a83a67fc1d94938cb31f2a9c
Re-land uses static linking with angle_util. The root cause
of the CFI error wasn't solved. Static linking works around
the problem by not using any export rules.
Second re-land fixes missing imports for libEGL and libGLESv2
static varieties.
Original change's description:
> Add more test_utils functions.
>
> Includes methods for creating temporary files, deleting files, and
> reading files into a string. Also renames GetPathSeparator to mention
> it's only used for environment variables. Includes a new virtual type
> angle::Process that will be used to implement cross-platform async
> Process launching for tests. Also includes a way to specify a custom
> crash handler callback.
>
> Also adds a few unit tests for the new functionality. They are disabled
> on Android because the functions are not needed by the new test runner.
>
> Bug: angleproject:3162
> Change-Id: I3e2c2e9837608884c98379fa0f78c9ffbe158d73
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1821940
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Bug: chromium:1015810
Bug: angleproject:3162
Change-Id: I2a18b819b0f91df610ad12ffedea2b38349fe7cf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1879859
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
f939cb63
|
2019-10-20T21:44:36
|
|
Revert "Reland "Add more test_utils functions.""
This reverts commit 7923e234bc458eda5936b8cb4f09fa19e28c0452.
Reason for revert: Fails compilation on rollers:
[2097/6048] SOLINK ./lib_angle_unittests__library.cr.so
FAILED: lib_angle_unittests__library.cr.so lib_angle_unittests__library.cr.so.TOC lib.unstripped/lib_angle_unittests__library.cr.so
python "../../build/toolchain/gcc_solink_wrapper.py" --readelf="../../third_party/android_ndk/toolch...(too long)
ld.lld: error: undefined symbol: glCreateShader
>>> referenced by shader_utils.cpp
>>> angle_util_static/shader_utils.o:(CompileShader(unsigned int, char const*)) in archive obj/third_party/angle/libangle_util_static.a
ld.lld: error: undefined symbol: glShaderSource
Original change's description:
> Reland "Add more test_utils functions."
>
> This is a reland of 5fcfcea4a9379633a83a67fc1d94938cb31f2a9c
>
> Re-land uses static linking with angle_util. The root cause
> of the CFI error wasn't solved. Static linking works around
> the problem by not using any export rules.
>
> Original change's description:
> > Add more test_utils functions.
> >
> > Includes methods for creating temporary files, deleting files, and
> > reading files into a string. Also renames GetPathSeparator to mention
> > it's only used for environment variables. Includes a new virtual type
> > angle::Process that will be used to implement cross-platform async
> > Process launching for tests. Also includes a way to specify a custom
> > crash handler callback.
> >
> > Also adds a few unit tests for the new functionality. They are disabled
> > on Android because the functions are not needed by the new test runner.
> >
> > Bug: angleproject:3162
> > Change-Id: I3e2c2e9837608884c98379fa0f78c9ffbe158d73
> > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1821940
> > Commit-Queue: Jamie Madill <jmadill@chromium.org>
> > Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
>
> Bug: chromium:1015810
> Bug: angleproject:3162
> Change-Id: I6a2c1e7b585a13ca846759f32da0777c00d7f7e6
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1869541
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I975b2214411906cb981bffa04fa50e0a65ff8b4e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: chromium:1015810, angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1870811
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
7923e234
|
2019-10-09T10:26:25
|
|
Reland "Add more test_utils functions."
This is a reland of 5fcfcea4a9379633a83a67fc1d94938cb31f2a9c
Re-land uses static linking with angle_util. The root cause
of the CFI error wasn't solved. Static linking works around
the problem by not using any export rules.
Original change's description:
> Add more test_utils functions.
>
> Includes methods for creating temporary files, deleting files, and
> reading files into a string. Also renames GetPathSeparator to mention
> it's only used for environment variables. Includes a new virtual type
> angle::Process that will be used to implement cross-platform async
> Process launching for tests. Also includes a way to specify a custom
> crash handler callback.
>
> Also adds a few unit tests for the new functionality. They are disabled
> on Android because the functions are not needed by the new test runner.
>
> Bug: angleproject:3162
> Change-Id: I3e2c2e9837608884c98379fa0f78c9ffbe158d73
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1821940
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Bug: chromium:1015810
Bug: angleproject:3162
Change-Id: I6a2c1e7b585a13ca846759f32da0777c00d7f7e6
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1869541
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
0cb09633
|
2019-10-18T15:50:10
|
|
Revert "Add more test_utils functions."
This reverts commit 5fcfcea4a9379633a83a67fc1d94938cb31f2a9c.
Reason for revert:
1 Test Suite(s) failed. angle_unittests failed because of:
TestUtils.RunApp on Linux CFI
https://ci.chromium.org/p/chromium/builders/ci/Linux%20CFI
Original change's description:
> Add more test_utils functions.
>
> Includes methods for creating temporary files, deleting files, and
> reading files into a string. Also renames GetPathSeparator to mention
> it's only used for environment variables. Includes a new virtual type
> angle::Process that will be used to implement cross-platform async
> Process launching for tests. Also includes a way to specify a custom
> crash handler callback.
>
> Also adds a few unit tests for the new functionality. They are disabled
> on Android because the functions are not needed by the new test runner.
>
> Bug: angleproject:3162
> Change-Id: I3e2c2e9837608884c98379fa0f78c9ffbe158d73
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1821940
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I1441bfbae31712f72b4aebeeea9cd711c3975a5d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1869254
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
5fcfcea4
|
2019-10-09T10:26:25
|
|
Add more test_utils functions.
Includes methods for creating temporary files, deleting files, and
reading files into a string. Also renames GetPathSeparator to mention
it's only used for environment variables. Includes a new virtual type
angle::Process that will be used to implement cross-platform async
Process launching for tests. Also includes a way to specify a custom
crash handler callback.
Also adds a few unit tests for the new functionality. They are disabled
on Android because the functions are not needed by the new test runner.
Bug: angleproject:3162
Change-Id: I3e2c2e9837608884c98379fa0f78c9ffbe158d73
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1821940
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|