kc3-lang/angle/src/tests/restricted_traces/restricted_trace_gold_tests.py

src/tests/restricted_traces/restricted_trace_gold_tests.py

Log

Author Commit Date CI Message
Roman Lavrov 37c69bd6 2024-10-07T11:00:14 Tests: gold tests assert that filter matches a test As described on the bug, test retries result in an unexpected value passed to --isolated-script-test-filter, which doesn't match any traces. Assert that there is a match as a (permanent) workaround. Bug: chromium:371977204 Change-Id: I415507ea999005a9ac04489582cf77b7d3e29e2a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5912615 Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 0f4c807a 2024-09-16T12:55:40 Tests: disable batching in gold tests on Android setRequestedOrientation from a previous trace appears to affect pixels. We often see this on CI when a new trace is added which reshuffles traces. For example, in this sequence the first trace sets landscape orientation and the second one goes back to portrait: angle_trace_tests --gtest_filter=TraceTest.dota_underlords:TraceTest.dragon_raja \ --run-to-key-frame --save-screenshots --verbose \ --render-test-output-dir=/tmp/g When the second trace is run independently, the resulting pixels are slightly different. This CL disables batching tests on Android so each trace runs as a new instrumentation/activity. This way the previous trace orientation does not seem to matter. The impact on shard runtime seems minor. Bug: b/367401313 Change-Id: If1c41e45d063d434f92d46be8dd381e645a3ff0a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5866850 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 18e2038e 2023-09-01T12:59:51 Revert "Gold tests: add temporary logging of angledata hashes" This reverts commit 4fffc9938375f2b84632eb661fc175457fb2ee77. Reason for revert: Hashes are ok - http://b/296921272#comment3 Original change's description: > Gold tests: add temporary logging of angledata hashes > > Should allow us to determine whether corruption happens between CAS and > local files on bots or something's wrong with the test harness. > > Bug: angleproject:8307 > Change-Id: I8a599777a38d7a1b5c56143eb8d02829895175a7 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4777836 > Commit-Queue: Yuly Novikov <ynovikov@chromium.org> > Reviewed-by: Yuly Novikov <ynovikov@chromium.org> > Auto-Submit: Roman Lavrov <romanl@google.com> Bug: angleproject:8307 Change-Id: Idae3bf3bac810af63e44c0c7553e0c6e9bb6af5b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4834062 Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Roman Lavrov 2c624a5e 2023-08-17T15:36:37 android_helper cleanup: remove smoke test and timeout This was added to help debug cases of stuck tests on bots, but this doesn't seem to have happened in over a year and the code (and timeout) keeps getting in the way. Bug: angleproject:7242 Change-Id: Ic30d976408bd32a745d830921f6e4d98d17e863c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4789845 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Roman Lavrov 4fffc993 2023-08-15T14:10:58 Gold tests: add temporary logging of angledata hashes Should allow us to determine whether corruption happens between CAS and local files on bots or something's wrong with the test harness. Bug: angleproject:8307 Change-Id: I8a599777a38d7a1b5c56143eb8d02829895175a7 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4777836 Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Auto-Submit: Roman Lavrov <romanl@google.com>
Roman Lavrov 84644dd3 2023-03-03T12:30:35 Improve logging to make batch failures more clear. When logging suddenly interrupts it's hard to tell what's going on (see bug) Bug: angleproject:8060 Change-Id: I83c33818d79e442956045772913c6418ac705466 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4307774 Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Brian Sheedy 9566011b 2023-02-23T15:01:59 Add support for permissive pixel comparison Adds support for running ANGLE pixel tests with a more permissive inexact matching algorithm. This is done by passing in --use-permissive-pixel-comparison=1 to the test runner. This is intended to reduce the amount of manual triage work required on CLs that are expected to have a larger amount of differences, such as SwiftShader rolls. On the bots, this will normally be disabled, but will be enabled if "Use-Permissive-Angle-Pixel-Comparison: True" is present as a CL footer. This footer is not yet included automatically anywhere, so there should be no functional change as a result of this CL yet. Bug: angleproject:7985 Change-Id: Ie815fac42edb2198dd4d115fc50650504df136c0 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4288612 Auto-Submit: Brian Sheedy <bsheedy@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Cody Northrop b0739436 2023-02-13T14:28:17 Traces: Add key frame support Allow specifying a key frame in the trace JSON, i.e: "KeyFrames": [ 20 ], This allows our infra to use a frame other than 1 when taking screenshots for quality comparison. Adds new flag `--run-to-key-frame`, which will stop the trace once key frame has been reached. If no key frame in JSON, frame 1 will be used. Note the name in JSON is plural, but we only support one key frame for now. Multiple key frame support can come in the future. This CL also updates the code to allow ending traces early with `--max-steps-performed` which has been broken since http://crrev/c/4008998 It also removes `--one-frame-only` which is superseded by `--run-to-key-frame`, and can be replicated using `--max-steps-performed 1`. Test: angle_trace_tests --gtest_filter="*tmnt_shredders_revenge*" Bug: angleproject:8035 Bug: b/270426257 Change-Id: Ib02ef60d887ae5efb0288f5a9b8c2914dafc6efc Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4284637 Commit-Queue: Cody Northrop <cnorthrop@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Brian Sheedy 81c7a7d9 2023-02-15T11:33:05 Use per-channel fuzzy threshold Switches the fuzzy threshold used in Gold-backed pixel tests to be per-channel instead of the sum of all channels. This means that a difference of [1, 1, 1, 0] between two pixels will now pass, whereas before that would have required a threshold of 3. This is intended to only ever auto-approve changes that can be attributed to rounding errors. Bug: angleproject:7985 Change-Id: Id057534c3e92ee9a4fbd1545fd1a16b28fea61b1 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4256778 Auto-Submit: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Brian Sheedy 2510afd2 2023-02-02T13:30:26 Enable Skia Gold inexact matching Enables inexact matching for ANGLE tests that use Skia Gold using the fuzzy algorithm. This will allow images to be auto-approved if it appears that the differences are due to rounding. Bug: angleproject:7985 Change-Id: Iee791ff2bd06ce426d27e05cf5a6eb0058e857af Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4219862 Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 71e89320 2023-01-03T09:10:08 Tests: Support multiple filters with ":". Now we can run multiple traces per invocation, e.g.: "trex_200:among_us" Bug: angleproject:7775 Change-Id: I8e8176d444f20ae97234469ab8c553543b81b331 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4132944 Auto-Submit: Jamie Madill <jmadill@chromium.org> Reviewed-by: Roman Lavrov <romanl@google.com> Commit-Queue: Roman Lavrov <romanl@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Roman Lavrov e6491c2f 2022-11-15T19:35:47 Revert "Add temporary logging of gold_session.Initialize stdout." This reverts commit 05d86e06003bc9bcfdf62d769d6c6229e7680f5e. Reason for revert: this helped caught an issue https://anglebug.com/7550#c6, but recent outputs seem normal so no longer needed Original change's description: > Add temporary logging of gold_session.Initialize stdout. > > Normally looks like: > Loaded 1549 known hashes > ... > > Suspecting that in failure cases it's going to result in something > different. > > Bug: angleproject:7550 > Change-Id: I0190d4f679fc4161fc36a3008ba2e7d6d4764277 > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3804752 > Commit-Queue: Roman Lavrov <romanl@google.com> > Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Bug: angleproject:7550 Change-Id: I7c6e1569299ca4c834f29fcd01c3e707e373de16 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4024759 Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Commit-Queue: Brian Sheedy <bsheedy@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com> Auto-Submit: Roman Lavrov <romanl@google.com>
Jamie Madill a41c6173 2022-10-16T08:28:53 Simplify trace test names. Originally we had a lot of conditionally enabled test configs in angle_perftests, that we enabled behind a flag because the total runtime would be too large if we left everything enabled. This CL switches to using a single test set, one for each trace, and using a the same command-line API we use in dEQP and the ANGLE samples to pick a single configuration. This makes it easier to pick exactly what configurations you want from the command line. Because we only run one configuration each time, we don't need to specialize the test name to include the configuration info. We can also simplify TracePerfTest -> TraceTest since these tests are run for both perf and correctness. Old name: TracePerfTest.Run/vulkan_trex_200 New name: TraceTest.trex_200 Example for tests that don't use default arguments: Old args: --enable-all-trace-tests Old name: TracePerfTest.Run/native_offscreen_trex_200 New args: --use-gl=native --offscreen New name: TraceTest.trex_200 Bug: angleproject:7755 Change-Id: Id93cf5725eff2e2c6b3b14804bc2902f38306e7e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3966535 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Roman Lavrov <romanl@google.com>
Jamie Madill e495e7fd 2022-10-14T13:52:47 Redo perf and trace test parameters. - Moves common argument parsing code into test_util. - Changes the perf test arg parsing to use the common functions. - Adds new --use-angle and --use-gl parameters to the trace tests. - Also adds new --offscreen and --vsync parameters to the traces. - Removes the now unneeded --enable-all-trace-tests argument. - Both --arg=value and --arg value work in test suites now. Now, instead of using --enable-all-trace-tests you can specify the backend with --use-angle=swiftshader, --offscreen, or combinations of those parameters. The test names are the same as they were before, but only the configured tests will run in a session. We could opt to simplify the test names in later CLs if we want to simplify the test running. Ideally we'd keep the perf reporting the same because then we'd keep the time series the same on the test infra. This also allows us to split up the trace tests into separate targets on the bots, which will better allow us to control the workloads and sampling of the tests. For example: - angle_perftests becomes - angle_perftests (microbenchmarks) - angle_trace_perf_vulkan_tests (traces with vulkan back-end) - angle_trace_perf_native_tests (traces with system GL) Bug: angleproject:7755 Change-Id: I537168f3a6de96425dfda05ed98220eff9b19b76 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3956937 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Jamie Madill 0e0ea026 2022-10-13T18:29:33 Split trace tests out into angle_trace_tests. This makes angle_perftests strictly for microbenchmarks. The trace tests are a better separated target because they are for both gold correctness tests and performance. You can now use the GN arg "build_angle_trace_tests" to control building the trace tests. "build_angle_trace_perf_tests" is still supported for compatibility. Updates the infra scripts to use this new trace tests target. Also updates the build so by default we'll build the trace tests if you check them out, without needing a custom arg. Also updates docs. Bug: angleproject:7755 Change-Id: I2b1ae67799234cc1aefe7cda5f3af945c3f867fd Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3953340 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Roman Lavrov <romanl@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Roman Lavrov 6eff905c 2022-10-11T10:08:12 Make Android gold tests populate os and os_version correctly. os = Android os_version = {ro.build.fingerprint} These show up in Skia gold. Currently set to Linux info. Bug: None Change-Id: If2ffe5e8a6620fde9bd0a983f9c0d11ffe6a75c4 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3946179 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Roman Lavrov f30cfd18 2022-08-16T16:50:33 Perf and gold tests detect test SKIPs from json results. Make angle_test_util.RunTestSuite() gets and returns json results (using --isolated-script-test-output). perf tests has an additional special case (filed https://anglebug.com/7578): Previously when metrics were missing it was assumed that this was due to tests being skipped - which seems to currently be the case except for the special case of GLMark2Benchmark* tests. Those produce .fps/.score metrics instead of the metrics expected by this script. This CL keeps those tests running (once) but then marks them as SKIP in test results when no metrics are found (which is the behavior before this CL; it might actually make more sense to count this as PASS because everything matches our expectations). We're still running them once, so if they FAIL we'll get notified. gold tests: Use json results file instead of checking for '[ SKIPPED ]' in stdout. Bug: angleproject:7299 Bug: angleproject:7578 Change-Id: Ia751784ad1aa94dc855c8b58ebfe5ba3e06e462f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3826167 Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 2de8bb42 2022-08-03T18:15:52 Converge run wrappers into angle_test_util.RunTestSuite Bug: angleproject:7299 Change-Id: Ifa882028a6f49579b4a9c6893053cb3bee8d83fd Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808302 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 88b05d3c 2022-08-03T15:21:00 Use globals in android_helper to simplify flow. Explicit .Initialize(suite_name) call to detect Android Track suite (apk) that was installed. We use the same package com.android.angle.test in multiple apks (eg system info and perftests) so keeping track of which one is currently installed avoids the need to take care of it manually (with PrepareTestSuite). Bug: angleproject:7299 Change-Id: I4b96f0fb24e38fc42d3814fb0eeda5aae527de5a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808292 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 05d86e06 2022-08-02T18:30:52 Add temporary logging of gold_session.Initialize stdout. Normally looks like: Loaded 1549 known hashes ... Suspecting that in failure cases it's going to result in something different. Bug: angleproject:7550 Change-Id: I0190d4f679fc4161fc36a3008ba2e7d6d4764277 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3804752 Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Roman Lavrov 19c6600c 2022-08-02T12:45:57 Refactoring: move a few common functions to angle_test_util. Drop ScriptTest, Jamie says no longer needed. Bug: angleproject:7299 Change-Id: I32bdd1d70701f63b578b4f79512868daac3ad520 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3805381 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov d6acced1 2022-07-25T18:11:20 Add flag --save-screenshots (implied by --screenshot-dir). --screenshot-dir and --render-test-output-dir behave differently on Android. --screenshot-dir is passed as is and therefore means a device directory, and --render-test-output-dir has special handling in Android wrappers that substitutes this flag with a temporary location on the device then copying files to the local directory set by this flag. Cody pointed out that there are many docs referring to --screenshot-dir so behavior should be preserved. So --save-screenshots is implied when this flag is set. Otherwise, it defaults to false, so --render-test-output-dir just on its own won't save screenshots, only when --save-screenshots is added as well. This also allows to produce other artifacts without necessarily producing screenshots by using --render-test-output-dir without --save-screenshots. Bug: angleproject:7299 Change-Id: Ib1a268ddeb7eba9f120e3f37a429ea0e4bbe2411 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3785296 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov b71c6427 2022-05-31T16:23:51 Add a wrapper script to run traces using android_helper. Also add missing `appops set MANAGE_EXTERNAL_STORAGE` that appears to be necessary on newer builds (catapult does it starting from Android 11). Add a hash check to skip transferring .gz files that are already there. Bug: angleproject:6854 Change-Id: Ib612d1235fe1274b8fea47718af8389e8810d34e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679486 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com> Auto-Submit: Roman Lavrov <romanl@google.com>
Roman Lavrov b03f2789 2022-05-31T20:31:23 Revert "Use a temporary timeout multipler for trace tests." This reverts commit f810e998993290f049bbdad4fae975e4867100ad. Reason for revert: Uploads are now sharded so this shouldn't be needed. Original change's description: > Use a temporary timeout multipler for trace tests. > > The trace tests upload is frequently timing out on our test infra > because of the size of the test suite. Until we can find a better > solution we can increase the timeout non-invasively using an > environment variable. > > Bug: angleproject:6854 > Change-Id: Ifbe43c4d9dc8928cf35df8a26b844f884c90eb9d > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3427560 > Reviewed-by: Cody Northrop <cnorthrop@google.com> > Reviewed-by: Yuly Novikov <ynovikov@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> Bug: angleproject:6854 Change-Id: Iaad7b63073ed900acdc575fbac927a8463b0c530 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679487 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov c9794d6c 2022-04-27T14:20:47 Run a smoke test in each shard and crash on timeouts. The smoke test (words_with_friends_2) typically takes only a few seconds to run. Run it first to check that everything is working as intended. As we know it's a short test, we can set a small timeout (60s) and fail early in case something goes wrong. This will speed up tests failures when something is completely broken. I also changed the main shard run to crash when timeout is detected instead of just dumping debug info as if the tests didn't finish in 10 minutes they're also unlikely to finish in the 20 minutes after which swarming times out the shard. Bug: angleproject:7242 Change-Id: Ia99ca4b2e1676d09b9ff7d67ab116aba94f66512 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3611446 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Roman Lavrov ebfb9b9f 2022-04-26T10:37:52 Configure logging in perf and gold tests to include time. Currently no timestamp is logged: [I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd} INFO:root:adb --version ... Before switching to android_helper relative timestamps in seconds were printed, set up by catapult. Timestamps proposed in this CL are more consistent with outer logging but keep prefix short. Example: [I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd} I18:45:14.999999Z adb --version ... Bug: angleproject:6854 Change-Id: I8ef7c9ae44eb5dba564a41456ce9417e3fe0f06c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606913 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov d1788614 2022-04-22T15:52:16 Use adb instead of catapult to run perf tests on Android. Add support for replacing flags: --isolated-script-test-output --isolated-script-test-perf-output --render-test-output-dir Always set --isolated-script-test-output to check for test failures using num_failures_by_type, interrupted, is_unexpected fields. Bug: angleproject:6854 Change-Id: Ia701f4d4d5d1adc9a24c057bb5c58fd5457cbd65 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3602834 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov e6fc5e62 2022-04-22T12:45:33 Use _FindAdb (cached) directly instead of Adb wrapper. A little simpler, no need to pass around an additional `apk` arg. Also fix missed _GetAdbRoot() call. Bug: angleproject:6854 Change-Id: Id64e5f35fb21a6aef7d99e614dc0eae753c24317 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3600912 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 4ac3ec13 2022-04-22T12:15:51 Switch py_utils import to pathlib. os.stat path before adding just to catch future copy-paste mistakes. Bug: angleproject:6854 Change-Id: If58c46f99aca93e6aeda90a987ceebbbb1b66955 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3600911 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 7c091cb3 2022-04-19T21:12:35 Use adb directly (instead of catapult) in gold tests. Android detected by angle_perftests apk presence relative to pwd. All of the restricted_traces/*/*.json files are copied but those are small. Only the necessary .angledata.gz files are copied. angle_system_info_test is also handled via adb. Bug: angleproject:6854 Change-Id: I7a89ff57fcdd8ce5dc63a5e3a8f5c0132f766894 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3595241 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Jamie Madill 9907f22d 2022-04-19T13:22:26 gold tests: Add command-line arg for gold instance. This will allow other users of the script to override the ANGLE Skia Gold instance with their own. Bug: angleproject:6854 Change-Id: Id6a0966d4db4fd0265d427d684f07dc1e4c610e2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3594619 Auto-Submit: Jamie Madill <jmadill@chromium.org> Reviewed-by: Roman Lavrov <romanl@google.com> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov b5d34da0 2022-04-11T18:17:03 Add src/tests/py_utils/ for utils, angle_path_util for imports. Detect if angle or chromium checkout when adding deps paths, so that random paths from above angle_root don't get added to sys.path when imported from an angle checkout. Bug: angleproject:6854 Change-Id: I4cd3334a2313d921f8651de7056f4f3798b8e072 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3582978 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Roman Lavrov 1a3411c7 2022-04-01T18:42:51 Set SKIPPED status on skipped tests, rely on it in Gold tests. Gold tests check for '[ SKIPPED ] {test_name}' notice in test output instead of assuming that missing screenshot means SKIP. Now missing screenshot raises an exception. Also log the reason why the test was skipped. Example: [ RUN ] TracePerfTest.Run/native_asphalt_8 ../../src/tests/perf_tests/ANGLEPerfTest.cpp:837: Skipped Test skipped due to missing extension: GL_KHR_texture_compression_astc_ldr [ SKIPPED ] TracePerfTest.Run/native_asphalt_8 (182 ms) Bug: angleproject:6854 Change-Id: I2d88e2063a68ae95399a7932700f74032737ec91 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3565561 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Roman Lavrov 08e007dd 2022-03-15T18:23:54 angle_system_info_test passes json via file --render-test-output-dir is supported by Android test runner so the file is first written on the device then copied over to the host. Bug: angleproject:6854 Change-Id: Ia56fd9c36ab68a495459efa3b1014e871a615565 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3526855 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Roman Lavrov <romanl@google.com>
Jamie Madill f810e998 2022-01-31T11:39:37 Use a temporary timeout multipler for trace tests. The trace tests upload is frequently timing out on our test infra because of the size of the test suite. Until we can find a better solution we can increase the timeout non-invasively using an environment variable. Bug: angleproject:6854 Change-Id: Ifbe43c4d9dc8928cf35df8a26b844f884c90eb9d Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3427560 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill e0b4cf6c 2021-11-26T10:23:24 Fix Python3 usage with perf tests/gold tests. Bug: angleproject:6733 Change-Id: I9d1baf78ed182db77c6a6d8e896737413ca417f3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3304012 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 316dcb56 2021-10-06T09:31:49 Add SwANGLE Skia Gold testing. Required updating our system info helper to be able to retrieve SwiftShader device information. Bug: angleproject:6496 Change-Id: Ib38ea4da65d199433e17b87df2630c3fd77cb619 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3208646 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Jamie Madill 0e021199 2021-09-08T12:31:47 Gold Tests: Append "show all results" to triage links. This should allow for easier triage when CL authors run into new images that don't match the goldens. Bug: skia:12428 Change-Id: I60fee615f23f5a1b36ddbe6c6708498fb3def46e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3149571 Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 047e9833 2021-08-30T10:00:51 Traces: Move skia gold Python scripts. Moving this folder up one level allows us to enforce that every subdirectory in the restricted_traces folder is a trace folder. This makes it easier to batch delete or move traces. Bug: angleproject:5133 Change-Id: I5af24ce6c233f50dea2e6ffcaef6d489b03914c8 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3130441 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com>
Jamie Madill 48da1c35 2021-07-16T13:24:34 Vulkan: Prefer the local vulkan loader over the system one. Load the Vulkan loader ourselves and give vkGetInstanceProcAddr to volk. This allows us to always prefer loading from the current module directory instead of using the platform-specific ordering. Refactor angle::Library loading to use ModuleDir instead of ApplicationDir. CL originally authored by Geoff Lang. Bug: chromium:1219969 Change-Id: I21d1926e90fd66e1c23cea7323991ae55f3d22d4 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3035444 Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill d99864c0 2021-06-07T12:48:01 Gold Tests: Fix flaky test handling. Indead of just retrying the batch, retry individual tests. This should correctly handle the Intel flake pattern. Bug: angleproject:5415 Change-Id: I029c2514ff8e7a45184c5c105bf2d0350171da29 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2940049 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 8c51ac2b 2021-06-02T19:50:46 Gold Tests: Fix sharding + batching. Clearing the GTEST args from the env vars will run the proper subset of tests instead of applying sharding arguments twice. This should fix the tests on the bots running a small subset of the tests. Also updates logging control for the script. Bug: angleproject:5966 Change-Id: Ic8a21e6aacfed17eff836d4301a6a50464b2cb92 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2935196 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Yuly Novikov 1429130b 2021-05-28T14:02:58 Gold Tests: Fix batch result reporting Previously artifacts were not reset for a new trace in a batch Bug: angleproject:5966 Change-Id: Ia6ed8c070eaaeca7e6cea887419666f630648df8 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2925218 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Yuly Novikov ce7d80bd 2021-05-27T18:16:07 Gold Tests: Fix batch uploading Uploading to Gold used to stop after the first failure in the batch Bug: angleproject:5966 Change-Id: Iff89c106b18a9aab5a43742a6085c5f9a3c9a188 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2923784 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 163f6ac0 2021-05-26T10:57:18 Gold Tests: Add batching. This should speed up test runs on Android, which are currently much slower than the other platforms. Bug: angleproject:5966 Change-Id: I57890522b64d1d195ea0be56b4f68b6d54192d64 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2919888 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 17c45b65 2021-05-25T10:06:47 infra: Shard restricted_trace_gold_tests. Reduces shard time from 50+ min to <10 min per shard. Bug: angleproject:5966 Change-Id: Ia230d71a66ac9a273a10e0389cc04d8c5737bc4a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2915566 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 3ff08c44 2021-05-14T15:41:50 Gold Tests: Implement flaky retries and sharding. This CL implements two features: first off we can now split the tests into several shards which we can run independently on different machines. Second it implements a simple retry mechanism that will retry a failed test a number of times and count any first passing test as a pass. Bug: angleproject:5875 Bug: angleproject:5966 Change-Id: Ieef4ea0e4eebdb286a32de3b1e3fef4f2eda0600 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2897546 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 31d48df1 2021-04-07T11:15:14 restricted trace tests: Fix script. This script was not parsing the trace list properly and was running no tests. Bug: angleproject:5811 Change-Id: I925bc16ac476823ca2f850d4a9e208b9bb5deb62 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2810805 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Yuly Novikov 4ec7021c 2021-01-27T16:41:45 Report angle_restricted_trace_gold_tests as interrupted on Exception Otherwise step is reported as failed but build as succeeded Bug: chromium:1153299 Change-Id: Id21d7ceafd04553634463be5acb64c070d251abb Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2653859 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill d4439824 2020-11-08T16:17:20 Gold Trace Tests: Fix JSON output with empty test set. Previously we would leave an empty test suite name with no tests run. According to the JSON test results spec this is likely a malformed output and we should instead include just an empty "tests" key. Bad: "tests": { "angle_restricted_trace_gold_tests": {} } Good: "tests": {} Bug: angleproject:5299 Change-Id: I459da593df150d804130f9c63e05d031e77a16a3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2525147 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill c07bd2f2 2020-11-05T08:58:14 Ensure trace test failures are unexpected. This will make the names of the failures show up in Milo. Bug: angleproject:5310 Change-Id: I64e1f5ffff04ec70a44f8f3d0c339da2bca6b216 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2520539 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 6372a56f 2020-11-04T14:07:03 Gold trace tests: Add logging for filter misses. The prior fix still didn't work. Add logging to debug why. Bug: angleproject:5299 Change-Id: I758cfa87066e1cdf4a5d1289c030943e9bded58d Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2519400 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 6ef8f848 2020-11-04T11:10:48 Trace Gold Tests: Fix filter name. Was incorrectly checking the individual test name instead of the full name including the test suite. Bug: angleproject:5299 Change-Id: I60df15bf1a748ce24d54f56b48ffe6eaca2ba54b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2519396 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 302f234d 2020-11-03T14:55:55 Gold Trace Tests: Handle test filter. This will allow the Chromium recipe retry mechanism to work properly. That in turn will let the recipe properly fail tests that pass without the patch and fail with the patch. Bug: angleproject:5299 Change-Id: Ic744e125779c6bc6a091e1ed89ac1be6b97fd942 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2518172 Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill cb1a3ce5 2020-10-29T10:13:40 Skia Gold: Fix Android stdout parsing. A recent change in Chromium broke how we were parsing stdout for capturing system info on Android. Fix the break and also update some helper scripts to work better when run manually. Bug: angleproject:5261 Change-Id: Iecc00bbe540461ab50ea31b5245b8b38a7250956 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2507633 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 11d94d86 2020-09-23T15:16:19 Integrate Skia Gold with restricted trace tests. The implementation is based heavily on the Chromium Skia Gold Python classes in //build/skia_gold_common . Results will be visible here: https://angle-gold.skia.org/ Currently results are not visible until this CL lands. Bug: angleproject:4090 Bug: b/168049670 Change-Id: I7a5d64fe35583ad7ea6360804a0f6b72cd9f6d8b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2430179 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Brian Sheedy <bsheedy@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 962e3473 2020-09-25T00:22:27 Fix return code of restricted trace tests. The prior script was incorrectly comparing a boolean result with a number and producing the opposite result: True would be reported as a failure, and False would be reported as success. Bug: angleproject:4090 Bug: b/168049670 Change-Id: I47ec684a57c455c6a833232d0a49774b265f7b39 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2430667 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 2a09e891 2020-09-22T09:32:32 Isolated script args in restricted trace tests. This will allow us to pass CQ since CQ requires isolated script output args. We can also update this logic later to fail CQ if the golden image check fails. Bug: angleproject:4090 Bug: b/168049670 Change-Id: I81c5571e8742e641ebd8c9cbd8e1c89f583c725c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2423212 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill eed069b3 2020-09-22T08:55:32 Give restricted gold tests executable bit. Bug: angleproject:4090 Bug: b/168049670 Change-Id: Ie3a46d4689e40958460c1f26a228645d84b3305c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2423211 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 5ec560fb 2020-09-17T17:18:25 Add restricted trace golden image script test. The script reads the list of tests from the restricted traces JSON and runs angle_perftests once for each test. Running the tests in a batch seems unstable on some platfroms so running each test separately works around that limitation. It does not yet interact with Skia Gold. The Gold integration will come later once this basic skeleton is running on the bots. Also updates the perf tests to exit more quickly with --one-frame-only. This will speed up the screenshots. Example run with trigger.py: https://chromium-swarm.appspot.com/task?id=4ecb985330a33910 Bug: angleproject:4090 Bug: b/168049670 Test: tested with trigger.py Change-Id: I52820dfba2acfbc5d638673c37297877e8476adb Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2417641 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
kmx.io     mail     discord kmxgit v0.4.0