kc3-lang/angle/src/tests/test_utils/runner/TestSuite.cpp

src/tests/test_utils/runner/TestSuite.cpp

Log

Author Commit Date CI Message
Tom Sepez 25390156 2025-08-21T00:13:19 Suppress unsafe buffers on a file-by-file basis in src/ [1 of N] In this CL, we suppress many files but stop short of actually enabling the warning by not removing the line from the unsafe_buffers_paths.txt file. That will happen in a follow-on CL, along with resolving any stragglers missed here. This is mostly a manual change so as to familiarize myself with the kinds of issues faced by the Angle codebase when applying buffer safety warnings. -- Re-generate affected hashes. -- Clang-format applied to all changed files. -- Add a few missing .reserve() calls to vectors as noticed. -- Fix some mismatches between file names and header comments. -- Be more consistent with header comment format (blank lines and trailing //-only lines when a filename comment adjoins license boilerplate). Bug: b/436880895 Change-Id: I3bde5cc2059acbe8345057289214f1a26f1c34aa Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6869022 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Tim Van Patten 899f3505 2025-08-07T09:59:19 TestSuite: Android: Call onCrashOrTimeout() on test crash Currently, when a test crashes on Android we don't get any results for tests that ran or for the crashing test itself. This makes it difficult for developers to debug the issue, since they don't know which test crashed. Add signal handlers which call onCrashOrTimeout() when a test crashes. Typically, crashes occur within the individual tests rather than in the test runner (TestSuite), so it should be "safe" to attempt to dump the results up to and including the crash to aid with debugging. This is still risky since things are in a bad state, but we expect (essentially) all crashes to be in the individual tests, ANGLE, or the underlying driver code, rather than in the TestSuite instance, so writing the results JSON file should have a good chance of succeeding. Worst case, we just crash again and get nothing (like we have before this change). Bug: b/279980674 Change-Id: Ifd5f3b97d5af731586e7bbf765bb60b3cb4f503b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6827986 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Tim Van Patten <timvp@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Solti Ho <solti@google.com>
Gyuyoung Kim e15d293c 2025-06-16T16:24:09 tvOS: Disable child process creation code tvOS does not support multi-process applications. As a result, the code that creates a new process causes build failures when building ANGLE unit tests on tvOS. This CL disables that code to prevent the build from breaking. Bug: b/424256388 Change-Id: Ib638d3f75b2839e1a75efc571645921f0d336e55 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6640919 Commit-Queue: Yuxin Hu <yuxinhu@google.com> Reviewed-by: Yuxin Hu <yuxinhu@google.com> Reviewed-by: Solti Ho <solti@google.com>
Roman Lavrov 2fcd88e1 2025-04-08T09:30:59 Tests: order bot mode batch configs alphabetically Currently ordered by hashmap key ordering which shuffles the order even with --max-processes=1. Adding a sort to make this deterministic. Bug: angleproject:408276172 Change-Id: Ice96e350ba4247a1f40870e275d04d2f654bc4f5 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6438024 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Mark Lobodzinski d50cd35e 2025-02-20T08:29:07 Tests: Add expectations file for angle_trace_tests Skipping trace tests was handled in the TracePerfTest source and is now specified in angle_trace_tests_expectations.txt. Bug: b/398048048 Change-Id: Idbf5e95e74699374eb46330dc3880a0c2dfc9bc1 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6302912 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Roman Lavrov 729b55ba 2025-03-04T10:45:34 Tests: Fix --gtest_repeat reporting Pass on intermediate Fail When running with --gtest_repeat=N, the same test is executed multiple times and the harness was overwriting a previous Fail status with a subsequent Pass. Also made Android logging more clear. Bug: None Change-Id: Ib1d71ca89e9e9b2af8f6a0df2c9e280b86143265 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6320657 Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Austin Annestrand 957bafb5 2025-01-30T11:49:41 Infra: Print stdout for TIMEOUT cases Bug: angleproject:392929827 Change-Id: I47625474e4ce9253a4cf275ea027290a6a2fbc33 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/6207671 Commit-Queue: Austin Annestrand <a.annestrand@samsung.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Roman Lavrov df90bbc5 2024-07-12T18:04:53 Refactoring: move angle::HashMap and HashSet to own header Underlying abseil includes pull in a large set of headers Bug: angleproject:42266508 Change-Id: Icee47143a8a59bb0795a054b67c0aa4ddcfca4d4 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5704137 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Shahbaz Youssefi d193d51b 2024-06-17T22:46:08 Replace issue ids post migration to new issue tracker This change replaces anglebug.com/NNNN links. Bug: None Change-Id: I8ac3aec8d2a8a844b3d7b99fc0a6b2be8da31761 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5637912 Reviewed-by: Geoff Lang <geofflang@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Roman Lavrov cbdbe3bd 2024-01-26T10:05:41 deqp: Unused test expectations do not cause failure, only log deqp is the only suite that appears to be handling unused expectations this way. This causes an inconvenience where a test cannot be pre-emptively added to the expectations file. For example, a VK-GL-CTS roll might be adding a new test that is failing in some configurations. The expectation cannot be added to the expectations file pre-emptively because of the failure due to the unused expectation, but that is the more convenient way to solve this problem than patching the roll manually. Bug: angleproject:8507 Change-Id: If2ee65d9119043c42e8ca0705a1cc8c40827fae3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5238992 Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com> Auto-Submit: Roman Lavrov <romanl@google.com> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Daniel Cheng bc7af34c 2023-12-06T00:15:33 Use googletest flag macros to access googletest flags. The implementation details of flags can change; fixing this proactively makes it easier to roll googletest. Bug: chromium:1409870 Change-Id: I5efa2487d9e38e122ca20f201748faf8e8050347 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5092928 Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Auto-Submit: Daniel Cheng <dcheng@chromium.org>
Igor Nazarov c38f3374 2023-09-15T19:22:44 Fix "win-asan-test" random failures and timeouts While exact reason for the failures is unknown, it seems like disabling Vulkan Loader DLL unloading fixes the problem. More details in the issue 8347. This CL also sets the variable on the Linux platform, because it does not hurt and will probably help with similar issues. Bug: angleproject:8347 Change-Id: I398d55a7c13f1ce6790988983fdf8fee7cc5776c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4868220 Commit-Queue: Igor Nazarov <i.nazarov@samsung.com> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Jamie Madill 968c429e 2023-01-04T14:18:11 Tests: Use ISOLATED_OUTDIR for test artifact directory. This should fix the missing CAS outputs on the bots. Bug: angleproject:7926 Change-Id: Id09e5ccc19a454e2c0bf63076788187f3c7ef24f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4135158 Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Auto-Submit: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Roman Lavrov 056f8044 2022-10-11T11:58:20 Perf tests: write metrics to file and read from merge script. The idea behind is to save metrics in a structured output with full context so that we don't have to parse stdout and reconstruct that context (e.g. which test was running when the output was printed etc). Note: we already have a "histogram" file (addHistogramSample) written but it is structured using some UUID-based ids making its digestion more complicated (https://anglebug.com/7299#c3). This CL just writes events as one-json-per-line simplifying downstream handling and debugging. Just a sanity check / logging in the merge script for now. Bug: angleproject:7299 Change-Id: I56dada643eceef180ce9bb1aa9ae6a641ea41e4c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3945112 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Jamie Madill e495e7fd 2022-10-14T13:52:47 Redo perf and trace test parameters. - Moves common argument parsing code into test_util. - Changes the perf test arg parsing to use the common functions. - Adds new --use-angle and --use-gl parameters to the trace tests. - Also adds new --offscreen and --vsync parameters to the traces. - Removes the now unneeded --enable-all-trace-tests argument. - Both --arg=value and --arg value work in test suites now. Now, instead of using --enable-all-trace-tests you can specify the backend with --use-angle=swiftshader, --offscreen, or combinations of those parameters. The test names are the same as they were before, but only the configured tests will run in a session. We could opt to simplify the test names in later CLs if we want to simplify the test running. Ideally we'd keep the perf reporting the same because then we'd keep the time series the same on the test infra. This also allows us to split up the trace tests into separate targets on the bots, which will better allow us to control the workloads and sampling of the tests. For example: - angle_perftests becomes - angle_perftests (microbenchmarks) - angle_trace_perf_vulkan_tests (traces with vulkan back-end) - angle_trace_perf_native_tests (traces with system GL) Bug: angleproject:7755 Change-Id: I537168f3a6de96425dfda05ed98220eff9b19b76 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3956937 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Roman Lavrov 6f037273 2022-09-28T18:46:13 Make last lines of batch stdout more visible in logs. So that it looks like this: https://gist.github.com/romanl-g/dd0d2f7e0eddf16fd802bed97789024c Bug: angleproject:7708 Change-Id: Ic92a9de6cba24637119ae9041e9126364d414948 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3923459 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Auto-Submit: Roman Lavrov <romanl@google.com>
Roman Lavrov 3d0faa02 2022-08-09T16:48:06 Remove workaround affecting --list-tests --gtest_filter=X. One of the multiple workarounds added in https://crrev.com/c/3062240. It makes `--list-tests --gtest_filter=X` behave as if it was just `--list-tests`. Not sure why this was necessary, bots appear to work without it?.. Could be something that changed since that CL? Bug: angleproject:7508 Change-Id: I9988add40ddf3168e00cea2739e5a9209f633161 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3821374 Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Jamie Madill 08df7ce4 2022-07-28T10:59:30 Test Runner: Disable --bot-mode on Fuchsia. This flag is a bit hard to get going because of the differences in the Fuchsia system APIs. Disable it explicitly for now to avoid issues when we switch bot configs. Bug: angleproject:7312 Change-Id: I47cd038b198f265a6b60893faf6f50f02be49fda Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3792163 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Roman Lavrov 2b43d630 2022-07-22T14:19:05 Remove unused (and incorrect on Android) mTestSuiteName. On Android it ends up being just: _ Bug: angleproject:7299 Change-Id: I3dabcfcbacc3f43338b6fa91ebd9917e43f783ce Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3780587 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Eddie Hatfield 916a35dd 2022-06-28T14:40:53 Merge getTempPath and CreateTemporaryFile Also includes moving these two (and some helpers) to common/system_utils.h Bug: angleproject:5095 Change-Id: I7d2fcbd27a67dc13215265ac5b5e8391b15a980b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3733519 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Eddie Hatfield <eddiehatfield@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Jamie Madill ca96cba9 2022-06-16T11:33:14 Make ReadEntireFileToString return a std::string. Note: this makes the method unavailable for export from angle_util, which is probably why it was designed the way it was in the first place. However, we import the test utils source file as a static lib into each test executable and test shared module, so it works. Bug: angleproject:7404 Change-Id: Ia957268882c2b8529643660d7d4f34d142c0dc43 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3708602 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Roman Lavrov <romanl@google.com>
Yuly Novikov 5f1d0742 2022-06-17T13:11:00 Increase batch timeout with sanitizer MemoryBarrierBufferOnlyTest got slow recently Bug: angleproject:7439 Change-Id: I6f24fa9abb9dcc98ebb8c964860a292e89ec1e19 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3711745 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Auto-Submit: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 81585fe2 2022-06-08T14:37:38 Use consistent test names with dEQP. This removes the GTest/dEQP test name distinction. Using the "RegisterTests" method in GTest we can use the full dEQP test names. The one limitation is that "-" is reserved in GoogleTest for exclusion filters, so we need to do some extra work to sanitize our filters before we pass them to GoogleTest. Makes maintaining test expectations easier going forward. Bug: angleproject:7390 Change-Id: I23893c425ddc934c11fe539b9173ec06c58abcc2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3695518 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Roman Lavrov b9bdc53b 2022-06-07T14:50:46 Make TestEventListener and inner class of TestSuite. Avoids having to pass dependencies to the class as it's automatically a friend; external linkage sadly but such is life. Bug: angleproject:7299 Change-Id: I5726dcab5a1a98b93c0713073cd02d6eb3a8e216 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3693826 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Rohit Rao 4b911686 2022-06-06T22:25:37 Converts from sprintf() to snprintf(). sprintf() is deprecated in Xcode 14, so update to safer equivalents in order to keep the compiler happy on iOS and macOS. Bug: chromium:1331345 Change-Id: Id5348088bf69cbd360d9251e6323596cb710666d Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3690747 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Peter Kasting 54a6b76f 2022-05-19T07:36:54 Roll third_party/rapidjson/src/ 7484e06c5..781a4e667 (156 commits) Bug: chromium:1284275 Change-Id: I5351f6b38ad56da0a5ccae82a1b340dfe2a3c588 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3646003 Auto-Submit: Peter Kasting <pkasting@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Peter Kasting <pkasting@chromium.org>
Jamie Madill 580f90c7 2022-05-12T09:53:17 Test Runner: Accept Chromium bot mode flag. This is for compatiblity with Chrome's windowed test launcher. Bug: angleproject:5417 Change-Id: I3a366b2b8aa895ab996fbcab68a777514038a206 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3644854 Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Auto-Submit: Jamie Madill <jmadill@chromium.org>
Yuly Novikov 7a9856f6 2022-04-05T19:25:26 Report durations of all flaky retries in JSON test output Bug: angleproject:7184 Change-Id: I19295daed7b13919a2b528a43ebe8d9cbac9d0a1 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3572713 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 9b7a77b0 2022-02-11T11:34:19 Double batch timeout for sanitizer tests. It seems the batch was timing out in ASAN builds. Bug: angleproject:6965 Change-Id: I059e138e7fc43928fb25f905bf85fbb34becd500 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3456662 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill ef3fffd6 2022-01-19T16:00:42 Fix running tests with empty test set. This was causing a crash when using an overly strict filter. Bug: angleproject:5417 Change-Id: I38bae1a155a6b1047bae0856348de67f9cda21eb Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3402939 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Yuly Novikov f071aaaf 2021-07-22T21:21:35 Use first test name instead of shard index for TestArtifacts Shard index is not available on Android if run_gtest_angle_test.py is not used, since host doesn't pass it to device by default. We'd like to remove run_gtest_angle_test.py usage, since custom scripts are not supported on iOS, so this CL is a pre-requisite to that. We still need to distinguish between dEQP test artifacts when we merge test results of different shards, so we'll be using the name of the first test in a batch instead. However, now, on Android, instead of a single TestArtifactsFakeTest entry per shard, we will have multiple entries, an entry for each GTest batch of 256 tests. On other platforms we will still have one entry per shard, since batching is done by TestSuite and not GTests there, just with a different name: Old: "tests": { "TestArtifactsFakeTest-Shard00": { "actual": "PASS", "artifacts": { "TestResults-Batch001.qpa": [ "TestResults-Batch001.qpa" ], New: "tests": { "TestArtifactsFakeTest-GLES3/info_renderer": { "actual": "PASS", "artifacts": { "TestResults-Batch001.qpa": [ "TestResults-Batch001.qpa" ], Bug: angleproject:5417 Change-Id: I649036c151e078c862e380ccab347d6c59acc929 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3046764 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill fdadc420 2021-09-17T14:39:36 Fix non-bot mode sharding on Android and disable elsewhere. There is no need to apply sharding internally on Android, since it is done on the host, except when listing tests. And on other platforms sharding in non-bot mode is not useful. Patch authored by Yuly Novikov (ynovikov@chromium.org) Bug: angleproject:5417 Change-Id: Iddc0e1a38fb514617dce527acc99d0dce85ca177 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3399250 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 14f55a7f 2021-11-02T10:16:58 infra: Enable TSAN and UBSAN tests. These tests are not yet enabled on the CQ. Bug: angleproject:5795 Change-Id: I3fc58ab800ebbc0840da8221b03b4ab145d28028 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3258003 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Lingfeng Yang e9f9fa17 2021-11-09T18:13:15 Add cpu time measurement to perf tests Bug: angleproject:6667 Change-Id: I16570a7b51d363dd1c0f35789b985520cb76d488 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270601 Commit-Queue: Lingfeng Yang <lfy@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Charlie Lao <cclao@google.com>
Jamie Madill 3ce1abd8 2021-11-16T14:38:15 Test Runner: Print output snippet in test timeout. This will be helpful in diagnosing test failures. Bug: angleproject:6694 Change-Id: I0ea491fe8ab48dfd017ce40150f606c571f3fa6c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3285805 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jonah Ryan-Davis f422f21a 2021-11-09T18:42:15 Reland "Metal: Reintroduce GPU power preference selection code." This reverts commit 67a8cf07a740c5ce3aafd2ad7fddd370451b3525. Reason for revert: Landed Chromium-side dependency: https://chromium-review.googlesource.com/c/chromium/src/+/3271170 Original change's description: > Revert "Metal: Reintroduce GPU power preference selection code." > > This reverts commit 017161701b7dbf70a13f3c180a39e1fa45c27d9f. > > Reason for revert: Blocking roller, please re-land with the fix. > > Original change's description: > > Metal: Reintroduce GPU power preference selection code. > > > > This CL re-introduces the GPU power preference code to > > the metal backend. It also reworks EGLDisplay caching > > in the frontend to cache based on the native display > > as well as the power preference attribute. > > A new extension, EGL_ANGLE_display_power_preference is > > added based on EGL_ANGLE_power_preference. This extension > > is a client extension that allows selection of GPU on > > display creation, similar to how GPUs are selected on > > context creation in EGL_ANGLE_power_preference. > > This CL adds EGLDisplayPowerPreferenceTest and enables it on > > the metal backend. > > > > Bug: angleproject:6143 > > Change-Id: I0a081dcd2e3f18ab365fdd3498ddcb6e2ba35212 > > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3231986 > > Reviewed-by: Kenneth Russell <kbr@chromium.org> > > Reviewed-by: Jamie Madill <jmadill@chromium.org> > > Reviewed-by: Gregg Tavares <gman@chromium.org> > > Commit-Queue: Jonah Ryan-Davis <jonahr@google.com> > > TBR=kbr@chromium.org,gman@chromium.org,jonahr@google.com,jmadill@chromium.org,angle-scoped@luci-project-accounts.iam.gserviceaccount.com > > Change-Id: I4f775bf7139253a87b033a30e0da2100b3c1bb02 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: angleproject:6143 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270749 > Reviewed-by: Jamie Madill <jmadill@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> # Not skipping CQ checks because this is a reland. Bug: angleproject:6143 Change-Id: Id9b0a5cbb76e4dea9e2f2da2b1c47a0587dfdaf5 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270970 Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill 67a8cf07 2021-11-09T15:10:50 Revert "Metal: Reintroduce GPU power preference selection code." This reverts commit 017161701b7dbf70a13f3c180a39e1fa45c27d9f. Reason for revert: Blocking roller, please re-land with the fix. Original change's description: > Metal: Reintroduce GPU power preference selection code. > > This CL re-introduces the GPU power preference code to > the metal backend. It also reworks EGLDisplay caching > in the frontend to cache based on the native display > as well as the power preference attribute. > A new extension, EGL_ANGLE_display_power_preference is > added based on EGL_ANGLE_power_preference. This extension > is a client extension that allows selection of GPU on > display creation, similar to how GPUs are selected on > context creation in EGL_ANGLE_power_preference. > This CL adds EGLDisplayPowerPreferenceTest and enables it on > the metal backend. > > Bug: angleproject:6143 > Change-Id: I0a081dcd2e3f18ab365fdd3498ddcb6e2ba35212 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3231986 > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Reviewed-by: Jamie Madill <jmadill@chromium.org> > Reviewed-by: Gregg Tavares <gman@chromium.org> > Commit-Queue: Jonah Ryan-Davis <jonahr@google.com> TBR=kbr@chromium.org,gman@chromium.org,jonahr@google.com,jmadill@chromium.org,angle-scoped@luci-project-accounts.iam.gserviceaccount.com Change-Id: I4f775bf7139253a87b033a30e0da2100b3c1bb02 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: angleproject:6143 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270749 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jonah Ryan-Davis 01716170 2021-10-14T13:24:41 Metal: Reintroduce GPU power preference selection code. This CL re-introduces the GPU power preference code to the metal backend. It also reworks EGLDisplay caching in the frontend to cache based on the native display as well as the power preference attribute. A new extension, EGL_ANGLE_display_power_preference is added based on EGL_ANGLE_power_preference. This extension is a client extension that allows selection of GPU on display creation, similar to how GPUs are selected on context creation in EGL_ANGLE_power_preference. This CL adds EGLDisplayPowerPreferenceTest and enables it on the metal backend. Bug: angleproject:6143 Change-Id: I0a081dcd2e3f18ab365fdd3498ddcb6e2ba35212 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3231986 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Gregg Tavares <gman@chromium.org> Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill 666fcf38 2021-10-08T12:19:15 Rename functions that overlap with Windows APIs. Bug: angleproject:6283 Change-Id: Ifcd9ea9e3bf729fd2066178eb9429050b2f10518 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3212894 Reviewed-by: Bruce Dawson <brucedawson@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
angle-autoroll 66c89b0f 2021-10-06T16:28:42 Fix and disable MSVC warnings Needed because some warnings are no longer disabled after http://crrev.com/c/3189512. Also includes https://github.com/KhronosGroup/OpenCL-Headers/pull/179, needed after clang upgrade to llvmorg-14-init-5410-gd0473681 Bug: chromium:1257173 Change-Id: I4f844aa972362c488cb6d37244439e2126f2c1c3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3210629 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 0004ecc2 2021-09-22T13:39:55 Write GL info to test artifact in end2end_tests. These will show up as glinfo.json in the "cas outputs" of the shard that runs these tests. We currently have information for Pixel 4, NVIDIA and Intel Win/Linux, and SwiftShader Windows. GLES 1.x extensions are treated separately. Some WebGL/Mac-only/D3D-only extensions show as unsupported anywhere. In order to capture artifacts on Android, we use a render test output parameter. Bug: angleproject:6379 Change-Id: I560b219b5a586cdbb68d272090f38eedc76aba20 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3176780 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Ian Elliott <ianelliott@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill e1bc8de3 2021-09-08T07:59:37 Change links from 'master' to 'main' branch. Bug: chromium:1226949 Change-Id: Ie7b28b2fa094cf0c0a407968d681e580519cbb88 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3148210 Reviewed-by: Geoff Lang <geofflang@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill f65d6acc 2021-08-18T12:20:40 Test Runner: Determine slow tests from test expectations. This updates the test harness to handle slow test based on the TIMEOUT test expectation. It removes the "registerSlowTests" API in favor of using the test expectation files. Also updates some of the timeout handling for various angle tests, including on SwiftShader for multithreading tests. Increases the timeout for tests in a batch to attempt to fix the problem of the first test being much slower. Bug: angleproject:6261 Change-Id: I7427344da5a0c1ea26d8c2d6eb0e9be2557b56e6 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3104007 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill d25c9d7f 2021-07-07T12:15:12 Add perf tests runner script. This script will be responsible for running the perf tests multiple times to try and stabilize measurements. We'll use it on the bots instead of just running the perf tests directly. Because the script invokes the binary multiple times, this slows down execution. Most significantly on Android, where we now need to use 20 shards, up from 6. Also marks one test as flaky on OpenGL. Bug: angleproject:6090 Change-Id: I5280035cb0bdb290a68dc6961a384eaf4b40dd4b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3011422 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com>
Jamie Madill 26329cb9 2021-07-30T09:55:24 Test Runner: Allow listing tests on Android. Because of quirks with the Chromium-provided Android test runner, we need to use a few tricks to get the test list output. We add placeholder output for a single test to trick the test runner into thinking it ran the tests successfully. We also add an end marker for the tests list so we can parse the list from the more spammy Android stdout log. Will enable a Python wrapper script to list and run performance tests outside of the GTest harness. Bug: angleproject:6090 Change-Id: I810d4722e9a6efa03ef94208e516401890af1c37 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3062240 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com>
Kenneth Russell e74638fe 2021-07-28T19:03:43 Disable Metal shader cache again in angle_end2end_tests. This hooking was removed in https://chromium-review.googlesource.com/3010486 and sped up local test runs, but seems to have reintroduced flakiness on the Mac AMD Retina bots. Bug: chromium:1233864 Bug: angleproject:5505 Change-Id: I46d95545fbbedc0e881e3a2832a5ac5d5384f405 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3057823 Commit-Queue: ccameron <ccameron@chromium.org> Reviewed-by: ccameron <ccameron@chromium.org>
Kenneth Russell b3cbfd55 2021-07-07T12:31:57 Fix multiple end2end crashes in direct-to-MSL compiler. This set of changes: - Adds a single place (CompilerMtl::useDirectToMSLCompiler) where the direct-to-MSL compiler can be enabled, still at compile time rather than run time. It is still disabled by default. - Initializes MTLWaitableCompileEventImpl::mShader, without which all shader compiles crash. - Fixes a mismatch of the ANGLERasterizerDisabled variable name in the generated MSL. - Removes the transform feedback bindings as an argument to the vertex shader's main0(). Transform feedback support needs to be reimplemented in this compiler backend. - Added an option to the DriverUniform class to emit as either an interface block or a struct. The direct-to-MSL backend assumes it is a struct. - Disable variable reference validation when referencing either sample mask or rasterizer discard functionality in the shader. - Disable struct usage validation when inserting references to ANGLE_TextureEnv. This occurs during pipeline rewriting and it's infeasible to disable it at that point, so disable it during pipeline rewriting - which means it's disabled for all shaders. - For angle_end2end_tests, disable the file API hooking which disables the Metal shader cache. This speeds up the tests significantly - by roughly a factor of 5. With these changes, several hundred angle_end2end_tests run without any assertion failures, though still with many test failures. The next crash is in: GLSLTest.NamelessScopedStructs/ES2_Metal and will likely require a larger bug fix, or more changes to be upstreamed from WebKit's repository. Bug: angleproject:5505 Change-Id: Ia1ea9a13867f00035d4aeccf907fd032255588e3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3010486 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill 15623ada 2021-06-23T08:46:21 Test Runner: Omit results files when too many failures. This will cause the test shards to be re-tried due to an invalid result. The goal is to retry flaky configs where whole shards fail. Bug: angleproject:6075 Change-Id: I76d0f08da29ffc4bcad7f282f7be38618c384bce Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2982500 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Shahbaz Youssefi 09bd5578 2021-06-21T23:47:15 Test runner: Capture test stderr With this change, stderr is also output when a test fails, including logs such as those originating from UNIMPLEMENTED(). Bug: angleproject:6077 Change-Id: Ice1e1f9aff26acfb630aef99865503643f0a9658 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2976184 Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Shahbaz Youssefi 72d2bd0c 2021-06-21T09:59:21 Allow capturing process stdout and stderr interleaved The test utils are enhanced to allow redirecting stderr to stdout. This is in preparation for a change that makes the test runner capture stderr together with stdout. Currently, on failure logs originating from UNIMPLEMENTED() and other such macros are not captured. Bug: angleproject:6077 Change-Id: I7a3c6c4732a59dac3ff0cc20a7835d5ed6f0f22e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2976183 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Jamie Madill 4194d400 2021-06-21T13:08:56 Test Runner: Add maximum failure count. This will make the test runner early exit after a certain count of failures. This will prevent the bots from running tens of thousands of tests when the whole test run is corrupt. Bug: angleproject:6075 Change-Id: I8f5c19ea694a5ad3a5d5df1429d2ba4b1fe868e3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2976179 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill dd686e48 2021-05-11T19:08:01 Test Runner: Add test expectations parser. Moves the test expectations from dEQP into the test runner. Also updates angle_end2end_tests to take an expectations file. Includes some very simple angle_end2end_tests expectations. Note that the expectations in the file are less expressive than the skips we use in the cpp. Bug: angleproject:5951 Change-Id: Ib92235575bc3ea5f3a977ce416b0e78fe806e39b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2892274 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill 1d019f49 2021-05-13T10:33:06 Test Runner: Fixes to skipped tests. Uses a different token for skipped and not run tests in the json. The token isn't standard but should only be visible when batches report to the parent process that a test wasn't run. It shouldn't be visible to the scripts that consume test results on the bots. Also fixes the expected result for skipped tests to also be "SKIP". Bug: angleproject:5951 Change-Id: I0bd7d13b2b25b583f9c42718786cb588bacd4bc6 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2894524 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 9a219a14 2021-05-12T09:52:16 Test Runner: Handle skipped gtests. GTEST_SKIP() allows us to skip tests using the GoogleTest APIs. This CL updates our test runner code to handle these skips. Bug: angleproject:5951 Change-Id: Idef5802c7c81bd92226f5a93281a1cea0f1b24ee Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2892272 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill e5c5efc0 2021-02-05T02:39:55 Revert "Test Runner: Pick single/multi-process mode automatically." This reverts commit 573d7f34a462fa5b2ca767dd84773ae6fa460d8a. Reason for revert: Breaking ANGLE roll. No-try: true Original change's description: > Test Runner: Pick single/multi-process mode automatically. > > This enables multi-process when running with more than one test > config. This saves developers from remembering to add the arg. > It can also simplify our test configurations by allowing us to omit > the default arguments. On Android by default we stay with single > process. It also changes the --bot-mode flag into --single-process > and --multi-process which override the default behaviour. > > Bug: angleproject:5585 > Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246 > Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> > Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> TBR=kbr@chromium.org,syoussefi@chromium.org,jonahr@google.com,jmadill@chromium.org,jonahr@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: angleproject:5585 Change-Id: I11ee1034c301aa246c4b03e29ff9b56235a1005c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2676247 Reviewed-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 573d7f34 2021-02-01T10:36:56 Test Runner: Pick single/multi-process mode automatically. This enables multi-process when running with more than one test config. This saves developers from remembering to add the arg. It can also simplify our test configurations by allowing us to omit the default arguments. On Android by default we stay with single process. It also changes the --bot-mode flag into --single-process and --multi-process which override the default behaviour. Bug: angleproject:5585 Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 85efb9d5 2021-01-26T14:39:19 Log dEQP QPA files as test artifacts. This adds artifact output to the test runner. We add a fake test at the start of a test run that owns the artifacts. Bug: angleproject:5236 Change-Id: Ice8001bf1f2aafbd8123fee76e0e7fcc3e5a8a0c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2657535 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Mohan Maiya <m.maiya@samsung.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill feb470cc 2021-01-19T10:10:18 Test Runner: Prevent race by not resetting timeout. This instead only sets the timeout when tests start to run. That way the timeout should always be at least the default timeout and we won't get into a situation where the watchdog times out the test immediately when the test starts. Bug: angleproject:5562 Change-Id: I6b12bb8fe8edcf35f46ba4fb106fdf80ff9402a1 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2637182 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill e4b4ff7a 2021-01-12T13:48:20 Test Runner: Add a slow tests list. Each slow test in the list will use a 3x longer timeout. The list is implemented using the same filter wildcard we use in the test expectations logic. We can test this out using a slow D3D11 varying test. Bug: angleproject:5076 Bug: angleproject:5496 Change-Id: I31cf45e6ee8a8bbd6e460d675ff8a0cf5f19a504 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2625172 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Le Hoang Quyen cb8903b1 2020-12-08T01:08:00 Metal: Ignore OS's internal shader cache when testing. Internal shader cache caused timeout in some dEQP tests. Work-around: ignore the cache by hooking fopen function and return null when the cache related files are accessed. Bug: angleproject:5354 Change-Id: I12ca228540925e67454bf24ce1ba83d703882c87 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2580918 Commit-Queue: Le Hoang Quyen <le.hoang.q@gmail.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org>
Tim Van Patten c859c0ac 2020-11-24T17:21:38 Batch replace std::unordered_map with angle::HashMap in src/ There are a few places that will remain std::unordered_map due to build or run-time errors, which will need to be evaluated more closely to determine if they should remain std::unordered_map or if there is another Abseil data structure that would be more efficient while still working correctly. Bug: angleproject:4873 Change-Id: Ib04253e3ad6398e63f4cc2bfe12c0f9e57cb112b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2558873 Commit-Queue: Tim Van Patten <timvp@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Charlie Lao <cclao@google.com>
Jamie Madill 9fe2d211 2020-12-01T13:24:33 Test Runner: Handle flaky failures. When a test mark as flaky fails consistently we need to avoid marking the test as flaky. We treat is as a deterministic failure. This means we output "PASS" as the expected result instead of "PASS FAIL". That change will produce the correct behaviour with the test scripts. Also adds missing test runner output when we retry a flaky test. Bug: chromium:1152515 Change-Id: I9e3cbaf04ea57a198717ee81505b5d63a9183410 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2568231 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 3e5b6f81 2020-10-31T12:33:28 Enable writing histogram-set-json-format. This uses the protobuf histogram functionality to build up a histogram set. The test suites then output the histograms to JSON. This is only implemented for angle_perftests. Bug: angleproject:5161 Change-Id: Ia5a7868e8d8dcf4f13d83115ae622828c63ef0d9 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2482295 Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill c06a424f 2020-11-02T21:13:39 Test Runner: Robustly handle unexpected crashes. This adds the ability for the test runner to parse unexpected crashes from the stdout. It also processes the stdout to determine which tests failed. Tests that run after the crash are re-tried in a follow-up child process. Will allow for the test runner to handle very crashy test suites and also processes crashes from win-asan and other configs where it is harder to intercept crashes. Bug: angleproject:5251 Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com>
Jamie Madill 59fbb989 2020-11-02T13:27:52 Test Runner: Add ability to retry flaky tests. Bug: angleproject:5273 Change-Id: Ie89559bb0897a04213981aa8fe4e2f2bfe78959a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513287 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com>
Jamie Madill 5c4c37dc 2020-10-25T15:55:45 Capture/Replay: Test improvements. Streamlines some arguments. Use a common subdirectory in out/. Show process name when logging. Clean up environment variable handling. Bug: angleproject:5247 Change-Id: I85eb5b8194a64f90691c4d03db3ee65c5e9b2313 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497560 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 31868665 2020-10-23T10:54:02 Test Runner: Only run watchdog in child processes. For single test execution we don't need to be running the watchdog thread. This only serves to cause timeouts in slow running tests. Adds a new method to keep the --batch-id argument in the list of parameters so other functions can process them. Bug: angleproject:5124 Bug: angleproject:5218 Change-Id: I958230149b7d79830802cd55f67f0190de607200 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2495282 Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill cd88bccf 2020-10-21T08:44:32 Test Runner: Command line sharding args override env. Previously ANGLE would error out when both the enviornment and the command line args are specified at the same time. Because the perf bots use both at once we need to handle the conflict in the same way as the prior test runner does. In this case the command line takes precedence over the environment. Bug: angleproject:5124 Change-Id: I1ba765b4e75759922bf9fe2db9f153cfc5995f85 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2489722 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 26cd1cc6 2020-10-18T11:44:27 Perf Tests: Add stdout printing arg to bot mode. This can be useful for debugging and also for perf tests. The perf tests output gets scraped from stdout. Bug: angleproject:5124 Change-Id: Id5005df3f2b66be9497a9ba422abd47b95a1a330 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2486097 Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 556d812a 2020-10-18T11:48:45 Test Runner: Disable watchdog when debugger attached. Bug: angleproject:3162 Change-Id: I3685a4fe068b6e82b8bb04448c212788f563ab8b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2483463 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill b4fb7cc9 2020-10-15T08:58:06 Test Runner: Print reproduction steps on a batch failure. This prints out a large gtest_filter of the tests in a batch. Makes reproducing a particular failure much easier. In the future we could potentially look at using an encoding to reproduce a specific batch: e.g. a flag that acceps a batch / shard number and runs those tests based on using the same internal counts. Bug: angleproject:5172 Change-Id: Iefad8f5ca23761417ad1ceb11fd664c9cf23c8be Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2475455 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 9580f3b4 2020-10-10T10:06:04 Test Runner: Accept GTEST shard environment vars. This makes interoperation with Chromium tests easier. For instance when triggering tests with scripts when using sharding. And also using the run/ wrapper scripts instead of run_gtest_angle_test. Bug: angleproject:5158 Change-Id: Ia5b1f94c02a081caf576615175791fff4e0d3fb3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464030 Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill ad80b184 2020-10-13T17:15:59 Test Runner: List tests explicitly without GTest. This dramatically speeds up Android shards that use batches of hudreds of tests. Because ANGLE already has the test list, it no longer needs to call GTest with an enourmous gtest_filter. Bug: angleproject:5164 Change-Id: I28fee2f36c50006f2a35a4dcd90f44f8ebe4f78c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2468464 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 336202d7 2020-10-09T16:43:32 Test Runner: Pass batch ID to child processes. This will let both the ANGLETest and dEQP test runners handle child processes correctly. The existing ANGLETest runner will reuse displays, and the dEQP test runner will write to a batch-unique filename. This solves the problem of the multi-process --bot-mode writing to the same file from multiple children simultaneously. Long-term it will be good to include the dEQP QPA files into the test artifacts. Also disables flushing after every log write when running as a child process. This hugely improves performance on machines with no SSD. Test: https://chromium-swarm.appspot.com/task?id=4f2b87c4f8234910 Bug: angleproject:5157 Change-Id: I226d24adf55e0f8b98e5a8e7947862e6906b4c40 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464424 Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Ian Elliott <ianelliott@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill d3aa1e9c 2020-10-03T01:31:33 Test Runner: Pass dEQP args to child processes. This fixes the sharding to use the specified back-end. With the bug ANGLE would start the platform default back-end instead. Bug: angleproject:3162 Change-Id: Ib2453dd3c58ea40fb36619301865ae0818038d15 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447043 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill f0d9471c 2020-10-02T23:51:57 Test Runner: More misc improvements. - reduces default batch size from 1000 -> 256 - reduces default max processes to a max of 16 - stripes tests when in bot mode - prints less spam when waiting for child processes The striping and smaller batch size can help with slow tests. The max process limit seemed to prevent errors on Linux. Bug: angleproject:3162 Change-Id: Ibed89ca35db54c62ec8ed63b32bc5aed916faec9 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447037 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 23825b12 2020-10-02T16:32:32 Test Runner: Set interrupted to 'false' normally. The bug here was to output 'interrupted: true' when running in --bot-mode. Instead in normal execution we will output that we were not interrupted. Bug: angleproject:3162 Bug: chromium:1134619 Change-Id: I3842e90466473ff05b0b90bc4660f88606a3c9f6 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2446851 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 1fd3e5d8 2020-10-01T10:00:18 Test Runner: Fix race in watchdog timeouts. This fixes a TSAN warning that popped up with the standalone test runner. The watchdog timer actually was never started so timeouts were not working as intended. It also switches the timeout mode to call _Exit which skips all the atexit handlers and avoids some races on teardown. This change also speeds up the TestSuiteTest. Also a small fix to GetTempDir that was including an extra path separator on Windows. Bug: angleproject:3162 Bug: angleproject:5117 Change-Id: I0e7880a08b61bbb6e30c65665d5c0acec2d78db2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2442381 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 867358b7 2020-09-21T23:40:57 Test Runner: Improve sharding validation. The prior check was incorrectly flagging index=0/count=2 as invalid. It also missed a few cases like a shard count of 0. Bug: angleproject:3162 Change-Id: Ida4c0f454ece15e4304d024d205475acf0f135e6 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2423207 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 5d7de5fd 2020-09-20T13:09:07 Test Runner: Fix default config grouping. Running "angle_unittests" with --bot-mode would produce noisy test groupings for each permutation of parameterized tests. Fix this by only splitting configs that begin with "ES". e.g. ES2_Vulkan, ES3_OpenGL, etc. Bug: angleproject:3162 Change-Id: I0b0ebe1c9cef9cbbce0489a2bc3f62761d43a1c7 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419046 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Geoff Lang <geofflang@chromium.org>
Jamie Madill 763e8b81 2020-09-20T21:59:23 Test Runner: Add more timeout logging. Will help diagnose a timeout failure on Windows Intel. Also reduce the significant digits in time output. Bug: angleproject:3162 Change-Id: I6d6a852f572875f613dec8e4ab6e798dfde0ca9b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2420331 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill dd0a9ec9 2020-09-20T12:50:21 Test Runner: Allow empty sharding parameters. The prior check was incorrectly validating a shard index of zero and a shard count of one as inconsistent. Bug: angleproject:3162 Change-Id: I63184c16e82f08080b275672a43e6dacf291560d Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419045 Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill 76e90947 2020-09-19T10:48:55 Test Runner: Really reuse displays with --bot-mode. The prior design didn't pass the correct flag to the subprocess. Instead add a new --reuse-displays flag that forces display reuse on in ANGLE tests. Pass the new flag to the child processes when running with --bot-mode. Bug: angleproject:3162 Change-Id: I5f62125d83d339b5e8b2506b4ca4656976f7398e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419638 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 3f20c713 2020-09-19T11:29:56 Test Runner: Fix "--bot-mode" error level return value. Previously the runner was returning "1" on success and "0" on failure. Swap this so that it correctly returns "0" on success and "1" on fail. This only affected "--bot-mode" which isn't yet enabled. Bug: angleproject:3162 Change-Id: I1ee942223272d24eff137279d5a0638d3a65694e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419639 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill d6806d73 2020-09-18T17:56:48 Test Runner: Fix sharding and --bot-mode. We were using the incorrect test list to set up initial test expectations. Fix this by using the already generated test set in the test suite. Test: angle_unittests with --bot-mode, sharding, and filter Bug: angleproject:3162 Change-Id: Id108943494130410caab404faad25115792e794d Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419596 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 58463573 2020-09-16T19:30:56 Test Runner: Fix dirty thread teardown. When the standalone tests shut down they would not wait for the watchdog thread to exit cleanly. Fix this by setting a flag that shuts down the watchdog and attempting to wait. This was detected by running angle_end2end_tests with TSAN. Bug: angleproject:3162 Bug: b/168744561 Change-Id: I765ce7a21c3be11703619470a81adb1882b009e4 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2415175 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 23cff4e1 2020-09-16T13:41:49 Test Runner: Handle GTest sharding args. These were messing up the test runner when used in combination with the test runner's specific args. Make it an error if the arguments are used together. Bug: angleproject:3162 Change-Id: I6fa1c0fb346a8c376ca455f7f4bf0492cfc0198a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2414931 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 7fc325cc 2020-09-12T13:48:38 Test Runner: Print total test execution time. Also has a few other tweaks to test spin-up. Bug: angleproject:3162 Change-Id: I8753f1afedbcc5c4b5ae257965ce68dc4a02ae06 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2407836 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 2879eb55 2020-09-12T09:04:16 TestRunner: Fix missing "=" on swarming args. These were missing from debe4683095450940883164ca6. Also fixes a misnamed "perf" parameter. Bug: angleproject:3162 Change-Id: Iad4c4d8d35cc8a08d0e30d50dd9bbf159a4a7ab9 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2407833 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 05b6740c 2020-08-09T12:23:42 TestRunner: Fix test result handling. The bots expect us to explicitly label "is_unexpected" for all unexpected results (e.g. failures). Also they construct a test filter from our test names so our result names have to match the tests exactly. That means removing prefixes like we have for the test suite name. Bug: angleproject:3162 Change-Id: I1a02ff8e26545b10a8829a3ee47be91f7459aa65 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2345028 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill debe4683 2020-07-15T14:36:33 Test Runner: Accept Chromium args. This change lets the test runner accept the same isolated script parameter syntax as Chromium. It keeps the old parameter names so the tests accept both. This will let us more easily integrate with the Chromium Android test_runner.py script. Also update the README which was missing a few flags. Bug: angleproject:3162 Change-Id: Ic22b289a095418ffdaa19a04caa1b45c6c657872 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2300530 Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill acbe145c 2020-07-02T14:59:55 Test Runner: Batch tests by config. This CL sorts tests into subprocesses organized by a single config. It will allow for the test runner to use a single EGL display per process in most cases. It doesn't implement the display sharing right now, that'll be in a forthcoming CL. It assumes test configs are split between / and (optional ending) __ characters. Bug: angleproject:3162 Change-Id: I28e8b4687640da6e501f08b8b9347185c99ce30f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2280403 Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill fe902f4b 2020-06-19T14:28:36 Test Runner: Stub histogram writer. The Android tests require a valid output for the perf test results. The smallest valid output is an empty array. Bug: angleproject:3162 Change-Id: I1bf7602b02358d634347d587b406c0c2d7c838a2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2254088 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Yang Gu e9a0cb87 2020-05-18T17:45:52 Fix assertion when running tests without suffix on Windows On Windows, we may omit suffix ".exe" when running an executable. However, this will trigger an assertion failure in current code and this CL is to fix this issue. Bug: angleproject:4640 Change-Id: I7edfdc0b4a7c590c874817530b19a812018b9288 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2206425 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Jamie Madill 390be9bb 2020-03-01T08:47:37 Test Runner: Use system path for gtest-internal. This ensures we use the correct internal path in Chromium builds as well. Bug: angleproject:3162 Change-Id: I036dcbd8802f67162e5f5eccb6158910f0a5efcc Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2082134 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 593706bc 2020-02-10T16:46:52 Test Runner: Increase default timeouts. In particular the Debug timeout was not generous enough. Increase the test timeout to 20 seconds in Release and 60 seconds in Debug. Increase the batch timeout to 240 seconds in Release and 600 seconds in Debug. Bug: angleproject:3162 Change-Id: I0bd0ea18cb5cdab005143b8deed9500d9d2e8ea1 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2047415 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill ee07cb31 2020-01-29T18:02:50 Test Harness: Allow filter with sharding params. Some tweaks to when we call gtest initialization allows this to work. Bug: angleproject:3162 Change-Id: I75159108d3f746f666e57c1a6495a8c8616ef1cf Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2024335 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill 238adb7a 2020-01-13T17:21:53 Fix test harness running disabled tests. Adds special handling for the DISABLED_ GTest test name. Fixes the test for the test suite harness itself when run with the new "--bot-mode" flag. Bug: angleproject:3162 Change-Id: Idf34098d2d6bcb78263773be7b01c55df9ec624b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1998660 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 5407aaa0 2019-12-16T15:50:12 Re-land "Add new test runner harness." (#2) Re-land #2 changes: * export labels are fixed for the CFI build * crash test disabled because of flakiness and issues with asan Re-land changes: * Unit test is suppressed in ASAN * --deqp-case is fixed * Debug layer errors should correctly work with failure expectations Original message: The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It supports: * splitting a test set into shards * catching and reporting crashes and timeouts * outputting to the Chromium JSON test results format * multi-process execution Unit tests are added in test_utils_unittest.cpp. Bug: angleproject:3162 Bug: chromium:1030192 Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Jamie Madill b92ec244 2019-12-06T15:08:54 Revert "Re-land "Add new test runner harness."" This reverts commit e20560faf1de86c01198143ef7733a12a098a90b. Reason for revert: Now fails on Linux CFI and also is a bit flaky. Original change's description: > Re-land "Add new test runner harness." > > Re-land changes: > > * Unit test is suppressed in ASAN > * --deqp-case is fixed > * Debug layer errors should correctly work with failure expectations > > Original message: > > The ANGLE test harness is a harness around GoogleTest that provides > functionality similar to the Chromium test harness. It supports: > > * splitting a test set into shards > * catching and reporting crashes and timeouts > * outputting to the Chromium JSON test results format > * multi-process execution > > Unit tests are added in test_utils_unittest.cpp. > > Bug: angleproject:3162 > Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240 > Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> > Reviewed-by: Yuly Novikov <ynovikov@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: angleproject:3162 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
kmx.io     mail     discord kmxgit v0.4.0