src/tests/py_utils/angle_test_util.py


Log

Author Commit Date CI Message
angle-autoroll f2ee319d 2024-07-03T07:02:13 Roll Chromium from 51d79108bbb3 to 34f452e68b67 (970 revisions) Due to http://crrev.com/c/5651996: 1. Don't use Xvfb for angle_unittests. 2. Force --use-xvfb in restricted_trace_gold_tests. https://chromium.googlesource.com/chromium/src.git/+log/51d79108bbb3..34f452e68b67 If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/chromium-angle-autoroll Please CC abdolrashidi@google.com,angle-team@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Chromium: https://bugs.chromium.org/p/chromium/issues/entry To file a bug in ANGLE: https://bugs.chromium.org/p/angleproject/issues/entry To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md Changed dependencies * build: https://chromium.googlesource.com/chromium/src/build.git/+log/feff4a53b0..229704c27a * buildtools: https://chromium.googlesource.com/chromium/src/buildtools.git/+log/2984b8fcc4..819c7ae6e8 * testing: https://chromium.googlesource.com/chromium/src/testing/+log/106f24b1e1..bf1591ad6e * third_party/catapult: https://chromium.googlesource.com/catapult.git/+log/36641ab0e3..e190315d54 * third_party/depot_tools: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/6a00e594c9..ca091f0d16 * third_party/flatbuffers/src: https://chromium.googlesource.com/external/github.com/google/flatbuffers.git/+log/6ede1ccc9e..fb9afbafc7 * third_party/libc++/src: https://chromium.googlesource.com/external/github.com/llvm/llvm-project/libcxx.git/+log/144e2174e9..0312683647 * third_party/zlib: https://chromium.googlesource.com/chromium/src/third_party/zlib/+log/8b7eff801b..68e57e619c * tools/android: https://chromium.googlesource.com/chromium/src/tools/android/+log/66f8276b4e..a05e27145f * tools/clang: https://chromium.googlesource.com/chromium/src/tools/clang.git/+log/7e2aebd75d..303336503e * tools/mb: https://chromium.googlesource.com/chromium/src/tools/mb/+log/7c0882a310..04455f259c * tools/perf: https://chromium.googlesource.com/chromium/src/tools/perf/+log/aaadc0d4ed..04ec6d97b6 No update to Clang. Bug: chromium:40257169 Change-Id: Ibe9a6bd3906df8cedb42f56adfe5085046b68bd2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5673851 Reviewed-by: Solti Ho <solti@google.com> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Roman Lavrov 8e744e9f 2023-08-14T14:40:41 Android: support dEQP in android_helper and add bundle script Example: autoninja -C out/Android angle_deqp_gles31_tests && (cd out/Android; ../../src/tests/run_angle_android_test.py --suite=angle_deqp_gles31_tests --gtest_filter='dEQP-GLES31.functional.image_load_store.3d.*') For angle_deqp_bundle.py example usage see comments at the top of the file. Note: this change is not supposed to have any impact on CI. dEQP tests are still ran using Chromium's test runner, not android_helper. Bug: b/293590167 Change-Id: I605102ebea8b135f3239ffac012ce5dc13e84943 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4776109 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Yuxin Hu <yuxinhu@google.com> Auto-Submit: Roman Lavrov <romanl@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Roman Lavrov 3a42d024 2023-06-13T13:37:35 Only import xvfb on Linux. Allows to run certain scripts with python3 (instead of vpython3) on Windows. Same as testing/scripts/common.py: https://crsrc.org/c/testing/scripts/common.py;drc=dfebdf76453bb843c0910d8a3e74468f6ab4a474;l=25 Bug: b/286067106 Change-Id: I57f5478082c2147bb5e9cf98baae762b8b844cec Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4610084 Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Auto-Submit: Roman Lavrov <romanl@google.com> Commit-Queue: Amirali Abdolrashidi <abdolrashidi@google.com>
Jamie Madill 71e89320 2023-01-03T09:10:08 Tests: Support multiple filters with ":". Now we can run multiple traces per invocation, e.g.: "trex_200:among_us" Bug: angleproject:7775 Change-Id: I8e8176d444f20ae97234469ab8c553543b81b331 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4132944 Auto-Submit: Jamie Madill <jmadill@chromium.org> Reviewed-by: Roman Lavrov <romanl@google.com> Commit-Queue: Roman Lavrov <romanl@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Roman Lavrov 30439960 2022-10-24T17:30:18 Unbreak run_angle_android_test. Broken by ListTests removed in https://crrev.com/c/3963651 Move helper to angle_test_util. Bug: angleproject:7755 Change-Id: I1eec442cf3e2bf6c24335d4c294a68e7e868b814 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3975353 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Jamie Madill d6ca85bd 2022-10-18T17:10:41 run_perf_tests.py: Remove unneeded Android args. These only apply to the Chromium Android test runner, which we no longer use for this script. Bug: angleproject:7755 Change-Id: I311ac204e1999a77872a76f6d0b9d149f7ac409e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3963651 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Roman Lavrov <romanl@google.com> Commit-Queue: Roman Lavrov <romanl@google.com> Auto-Submit: Jamie Madill <jmadill@chromium.org>
Jamie Madill 0e0ea026 2022-10-13T18:29:33 Split trace tests out into angle_trace_tests. This makes angle_perftests strictly for microbenchmarks. The trace tests are a better separated target because they are for both gold correctness tests and performance. You can now use the GN arg "build_angle_trace_tests" to control building the trace tests. "build_angle_trace_perf_tests" is still supported for compatibility. Updates the infra scripts to use this new trace tests target. Also updates the build so by default we'll build the trace tests if you check them out, without needing a custom arg. Also updates docs. Bug: angleproject:7755 Change-Id: I2b1ae67799234cc1aefe7cda5f3af945c3f867fd Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3953340 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Roman Lavrov <romanl@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 65d3d0f6 2022-09-19T10:42:43 run_perf_tests: Add option to find test binary in out dir. This allows us to run the perf test runner script from any directory and use that as a basis to find the most recently modified test suite binary. This saves us from chdir()ing into test directory every time we want to run a test suite. Ideally this should move into the test utils folder to be used by the gold tests script as well. Bug: angleproject:7671 Change-Id: Iead98670bb5f7627ae32f7ac41b4ebcee2726841 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3904427 Reviewed-by: Roman Lavrov <romanl@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Roman Lavrov 2deae29e 2022-09-06T16:34:18 Change xvfb screen to 3120x3120. Support upstream added in https://crrev.com/c/3865972 Maximum DrawSurfaceWidth and DrawSurfaceHeight from src/tests/restricted_traces/*/*.json. They're never both at 3120 but due to portrait/landscape orientation it's easies to just bump up both. Bug: angleproject:7620 Change-Id: I4a24eed8898d2cd4f65b7c2dbacb7bd1ffd4ba35 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3877784 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov f30cfd18 2022-08-16T16:50:33 Perf and gold tests detect test SKIPs from json results. Make angle_test_util.RunTestSuite() gets and returns json results (using --isolated-script-test-output). perf tests has an additional special case (filed https://anglebug.com/7578): Previously when metrics were missing it was assumed that this was due to tests being skipped - which seems to currently be the case except for the special case of GLMark2Benchmark* tests. Those produce .fps/.score metrics instead of the metrics expected by this script. This CL keeps those tests running (once) but then marks them as SKIP in test results when no metrics are found (which is the behavior before this CL; it might actually make more sense to count this as PASS because everything matches our expectations). We're still running them once, so if they FAIL we'll get notified. gold tests: Use json results file instead of checking for '[ SKIPPED ]' in stdout. Bug: angleproject:7299 Bug: angleproject:7578 Change-Id: Ia751784ad1aa94dc855c8b58ebfe5ba3e06e462f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3826167 Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov c5efd2b5 2022-08-15T16:20:25 Improve Ctrl-c handling. 1. Restore original handlers after subprocess terminates. Otherwise the installed handlers persist and will swallow sigint outside of subprocess invocations. 2. Raise KeyboardInterrupt when subprocesses terminate to interrupt the script itself. run_perf_tests.py allows for some subprocess failures (--max-errors, with non-zero default). This results in ctrl-c interrupting subprocesses but the script just marks the test as FAIL and keeps running. Bug: angleproject:7299 Change-Id: I2917008baa389491c7591596fc14ee139fa6417e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3831887 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Roman Lavrov 64c652d7 2022-08-12T18:14:39 Use stdout instead of (default) stderr in logging.StreamHandler Currently logging output appears before stdout on bots, makes logs so confusing as subprocess logs get out of order with logs from python. Using StreamHandler(stdout) seems to be the common practice in other python scripts in code search. Note: subprocess's stderr is redirected to stdout: https://crsrc.org/c/third_party/angle/src/tests/py_utils/angle_test_util.py;drc=2de8bb42a9ef93dfafe476e7dc81643285fa7ef1;l=92 Bug: angleproject:7299 Change-Id: I825f16d9c479f33a280d8fdbafb8297cda0c18b8 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3827957 Auto-Submit: Roman Lavrov <romanl@google.com> Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Commit-Queue: Amirali Abdolrashidi <abdolrashidi@google.com>
Roman Lavrov 2de8bb42 2022-08-03T18:15:52 Converge run wrappers into angle_test_util.RunTestSuite Bug: angleproject:7299 Change-Id: Ifa882028a6f49579b4a9c6893053cb3bee8d83fd Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808302 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 19c6600c 2022-08-02T12:45:57 Refactoring: move a few common functions to angle_test_util. Drop ScriptTest, Jamie says no longer needed. Bug: angleproject:7299 Change-Id: I32bdd1d70701f63b578b4f79512868daac3ad520 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3805381 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov ebfb9b9f 2022-04-26T10:37:52 Configure logging in perf and gold tests to include time. Currently no timestamp is logged: [I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd} INFO:root:adb --version ... Before switching to android_helper relative timestamps in seconds were printed, set up by catapult. Timestamps proposed in this CL are more consistent with outer logging but keep prefix short. Example: [I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd} I18:45:14.999999Z adb --version ... Bug: angleproject:6854 Change-Id: I8ef7c9ae44eb5dba564a41456ce9417e3fe0f06c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606913 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>