src/tests/test_utils/runner/TestSuite_unittest.cpp


Log

Author Commit Date CI Message
Tom Sepez 25390156 2025-08-21T00:13:19 Suppress unsafe buffers on a file-by-file basis in src/ [1 of N] In this CL, we suppress many files but stop short of actually enabling the warning by not removing the line from the unsafe_buffers_paths.txt file. That will happen in a follow-on CL, along with resolving any stragglers missed here. This is mostly a manual change so as to familiarize myself with the kinds of issues faced by the Angle codebase when applying buffer safety warnings. -- Re-generate affected hashes. -- Clang-format applied to all changed files. -- Add a few missing .reserve() calls to vectors as noticed. -- Fix some mismatches between file names and header comments. -- Be more consistent with header comment format (blank lines and trailing //-only lines when a filename comment adjoins license boilerplate). Bug: b/436880895 Change-Id: I3bde5cc2059acbe8345057289214f1a26f1c34aa Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6869022 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Geoff Lang 87ed2c9d 2022-08-11T11:19:17 TestSuiteTest: Generate more unique result file names. Speculative fix for flaky runs of TestSuiteTest. Flakes appear to read the results of other tests suggesting rand is not unique enough to protect against tests generating the same results file. Bug: chromium:1352134 Change-Id: Icadf64c557e23e6d87fd18aa60aa48373ecec9fd Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3827144 Commit-Queue: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Eddie Hatfield 916a35dd 2022-06-28T14:40:53 Merge getTempPath and CreateTemporaryFile Also includes moving these two (and some helpers) to common/system_utils.h Bug: angleproject:5095 Change-Id: I7d2fcbd27a67dc13215265ac5b5e8391b15a980b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3733519 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Eddie Hatfield <eddiehatfield@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Yuly Novikov 7a9856f6 2022-04-05T19:25:26 Report durations of all flaky retries in JSON test output Bug: angleproject:7184 Change-Id: I19295daed7b13919a2b528a43ebe8d9cbac9d0a1 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3572713 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 666fcf38 2021-10-08T12:19:15 Rename functions that overlap with Windows APIs. Bug: angleproject:6283 Change-Id: Ifcd9ea9e3bf729fd2066178eb9429050b2f10518 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3212894 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Zequan Wu 178b8481 2021-09-03T23:30:52 Revert "Skip validation of stderr in TestSuiteTest.RunFlakyTests" This reverts commit 89dbbb2ac6875c92c99feb24353641142bbd49b1. Reason for revert: The test failure was gone in latest clang roll. Original change's description: > Skip validation of stderr in TestSuiteTest.RunFlakyTests > > There is a bug (crbug.com/1234124) causing the profile runtime to print > errors about failing to write the profraw file. Don't validate stderr in > these builds until that is fixed. > > Bug: chromium:1234124 > Change-Id: Ie1e3a3eaf7dc0e2c20064822f774cc85fe79d3bc > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3076479 > Reviewed-by: Jamie Madill <jmadill@chromium.org> > Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> > Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org> Bug: chromium:1234124 Change-Id: If33e92b533c99677d2b8f55de618ecf15e3f19d2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3141973 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Reid Kleckner 89dbbb2a 2021-08-05T13:14:12 Skip validation of stderr in TestSuiteTest.RunFlakyTests There is a bug (crbug.com/1234124) causing the profile runtime to print errors about failing to write the profraw file. Don't validate stderr in these builds until that is fixed. Bug: chromium:1234124 Change-Id: Ie1e3a3eaf7dc0e2c20064822f774cc85fe79d3bc Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3076479 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Shahbaz Youssefi 22ae4ce0 2021-07-28T16:06:49 Revert "Disable RunAppAsync and RunAppAsyncRedirectStderrToStdout" This reverts commit f4f866ce289423d691a0edc0d44d13f4de16f897. Reason for revert: These tests should not be disabled. The culprit is fixed per crbug.com/1233361 Original change's description: > Disable RunAppAsync and RunAppAsyncRedirectStderrToStdout > > The tests are blocking the CQ due to flakiness. > > Bug: chromium:1233361 > Change-Id: Ib0283e201c271379364a177bea130d63322eee53 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3058233 > Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> > Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org> Bug: chromium:1233361 Change-Id: I24682f74d790d40f58a9526aaaaa842c3df23ec0 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3058502 Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Shahbaz Youssefi 8593e0a2 2021-07-28T13:52:06 TestSuite tests: Don't validate stderr until fixed Bug: chromium:1233361 Change-Id: Ibcd66520d7b842211eafb26f9188298c281cbe64 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3058503 Reviewed-by: Tim Van Patten <timvp@google.com> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Shahbaz Youssefi 72d2bd0c 2021-06-21T09:59:21 Allow capturing process stdout and stderr interleaved The test utils are enhanced to allow redirecting stderr to stdout. This is in preparation for a change that makes the test runner capture stderr together with stdout. Currently, on failure logs originating from UNIMPLEMENTED() and other such macros are not captured. Bug: angleproject:6077 Change-Id: I7a3c6c4732a59dac3ff0cc20a7835d5ed6f0f22e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2976183 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Jamie Madill 1d019f49 2021-05-13T10:33:06 Test Runner: Fixes to skipped tests. Uses a different token for skipped and not run tests in the json. The token isn't standard but should only be visible when batches report to the parent process that a test wasn't run. It shouldn't be visible to the scripts that consume test results on the bots. Also fixes the expected result for skipped tests to also be "SKIP". Bug: angleproject:5951 Change-Id: I0bd7d13b2b25b583f9c42718786cb588bacd4bc6 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2894524 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 9a219a14 2021-05-12T09:52:16 Test Runner: Handle skipped gtests. GTEST_SKIP() allows us to skip tests using the GoogleTest APIs. This CL updates our test runner code to handle these skips. Bug: angleproject:5951 Change-Id: Idef5802c7c81bd92226f5a93281a1cea0f1b24ee Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2892272 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill e5c5efc0 2021-02-05T02:39:55 Revert "Test Runner: Pick single/multi-process mode automatically." This reverts commit 573d7f34a462fa5b2ca767dd84773ae6fa460d8a. Reason for revert: Breaking ANGLE roll. No-try: true Original change's description: > Test Runner: Pick single/multi-process mode automatically. > > This enables multi-process when running with more than one test > config. This saves developers from remembering to add the arg. > It can also simplify our test configurations by allowing us to omit > the default arguments. On Android by default we stay with single > process. It also changes the --bot-mode flag into --single-process > and --multi-process which override the default behaviour. > > Bug: angleproject:5585 > Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246 > Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> > Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> TBR=kbr@chromium.org,syoussefi@chromium.org,jonahr@google.com,jmadill@chromium.org,jonahr@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: angleproject:5585 Change-Id: I11ee1034c301aa246c4b03e29ff9b56235a1005c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2676247 Reviewed-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 573d7f34 2021-02-01T10:36:56 Test Runner: Pick single/multi-process mode automatically. This enables multi-process when running with more than one test config. This saves developers from remembering to add the arg. It can also simplify our test configurations by allowing us to omit the default arguments. On Android by default we stay with single process. It also changes the --bot-mode flag into --single-process and --multi-process which override the default behaviour. Bug: angleproject:5585 Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill c06a424f 2020-11-02T21:13:39 Test Runner: Robustly handle unexpected crashes. This adds the ability for the test runner to parse unexpected crashes from the stdout. It also processes the stdout to determine which tests failed. Tests that run after the crash are re-tried in a follow-up child process. Will allow for the test runner to handle very crashy test suites and also processes crashes from win-asan and other configs where it is harder to intercept crashes. Bug: angleproject:5251 Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com>
Jamie Madill 59fbb989 2020-11-02T13:27:52 Test Runner: Add ability to retry flaky tests. Bug: angleproject:5273 Change-Id: Ie89559bb0897a04213981aa8fe4e2f2bfe78959a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513287 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com>
Jamie Madill 47e66f22 2020-10-28T14:24:37 Revert "Re-enable crash testing in angle_unittests." This reverts commit 21add1240950c2e2a6ec997f9a128005d84266b6. Reason for revert: Fails on Windows 7 x64 Chromium config. https://ci.chromium.org/p/chromium/builders/ci/Win%207%20Tests%20x64%20%281%29/73194 Original change's description: > Re-enable crash testing in angle_unittests. > > This is an experimental test to see if this works now. > Disabled on Win/ASAN because of conflicts with the > crash callback. > > Bug: angleproject:5239 > Change-Id: I280b18ab8846de79b561f26716df9ba374cfd9a5 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2496040 > Reviewed-by: Jamie Madill <jmadill@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> TBR=ynovikov@chromium.org,jmadill@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: angleproject:5239 Change-Id: I83c150c2c2090b7d1aeb78e53e96115a167a7491 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2505479 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 21add124 2020-10-25T15:55:45 Re-enable crash testing in angle_unittests. This is an experimental test to see if this works now. Disabled on Win/ASAN because of conflicts with the crash callback. Bug: angleproject:5239 Change-Id: I280b18ab8846de79b561f26716df9ba374cfd9a5 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2496040 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 4968245c 2020-10-25T12:58:15 Adjust TestSuite test timeout values. This should prevent flakiness in this test. Bug: angleproject:5245 Change-Id: I841de2cee093ea04b35763314f0a4f5497bcf778 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497223 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 1fd3e5d8 2020-10-01T10:00:18 Test Runner: Fix race in watchdog timeouts. This fixes a TSAN warning that popped up with the standalone test runner. The watchdog timer actually was never started so timeouts were not working as intended. It also switches the timeout mode to call _Exit which skips all the atexit handlers and avoids some races on teardown. This change also speeds up the TestSuiteTest. Also a small fix to GetTempDir that was including an extra path separator on Windows. Bug: angleproject:3162 Bug: angleproject:5117 Change-Id: I0e7880a08b61bbb6e30c65665d5c0acec2d78db2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2442381 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 5407aaa0 2019-12-16T15:50:12 Re-land "Add new test runner harness." (#2) Re-land #2 changes: * export labels are fixed for the CFI build * crash test disabled because of flakiness and issues with asan Re-land changes: * Unit test is suppressed in ASAN * --deqp-case is fixed * Debug layer errors should correctly work with failure expectations Original message: The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It supports: * splitting a test set into shards * catching and reporting crashes and timeouts * outputting to the Chromium JSON test results format * multi-process execution Unit tests are added in test_utils_unittest.cpp. Bug: angleproject:3162 Bug: chromium:1030192 Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill b92ec244 2019-12-06T15:08:54 Revert "Re-land "Add new test runner harness."" This reverts commit e20560faf1de86c01198143ef7733a12a098a90b. Reason for revert: Now fails on Linux CFI and also is a bit flaky. Original change's description: > Re-land "Add new test runner harness." > > Re-land changes: > > * Unit test is suppressed in ASAN > * --deqp-case is fixed > * Debug layer errors should correctly work with failure expectations > > Original message: > > The ANGLE test harness is a harness around GoogleTest that provides > functionality similar to the Chromium test harness. It supports: > > * splitting a test set into shards > * catching and reporting crashes and timeouts > * outputting to the Chromium JSON test results format > * multi-process execution > > Unit tests are added in test_utils_unittest.cpp. > > Bug: angleproject:3162 > Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240 > Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> > Reviewed-by: Yuly Novikov <ynovikov@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: angleproject:3162 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill e20560fa 2019-12-04T13:18:36 Re-land "Add new test runner harness." Re-land changes: * Unit test is suppressed in ASAN * --deqp-case is fixed * Debug layer errors should correctly work with failure expectations Original message: The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It supports: * splitting a test set into shards * catching and reporting crashes and timeouts * outputting to the Chromium JSON test results format * multi-process execution Unit tests are added in test_utils_unittest.cpp. Bug: angleproject:3162 Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240 Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 8f31872c 2019-12-03T18:34:52 Revert "Add new test runner harness." This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d. Reason for revert: Has a bug with the ASan build and also has a few bugs with ANGLE standalone test expectations an filter. Bug: chromium:1030192 Bug: angleproject:4193 Original change's description: > Add new test runner harness. > > The ANGLE test harness is a harness around GoogleTest that provides > functionality similar to the Chromium test harness. It supports: > > * splitting a test set into shards > * catching and reporting crashes and timeouts > * outputting to the Chromium JSON test results format > * multi-process execution > > Unit tests are added in test_utils_unittest.cpp. > > Bug: angleproject:3162 > Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016 > Commit-Queue: Jamie Madill <jmadill@chromium.org> > Reviewed-by: Yuly Novikov <ynovikov@chromium.org> TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: angleproject:3162 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill fb40d231 2019-12-02T16:39:18 Add new test runner harness. The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It supports: * splitting a test set into shards * catching and reporting crashes and timeouts * outputting to the Chromium JSON test results format * multi-process execution Unit tests are added in test_utils_unittest.cpp. Bug: angleproject:3162 Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>