|
30439960
|
2022-10-24T17:30:18
|
|
Unbreak run_angle_android_test.
Broken by ListTests removed in https://crrev.com/c/3963651
Move helper to angle_test_util.
Bug: angleproject:7755
Change-Id: I1eec442cf3e2bf6c24335d4c294a68e7e868b814
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3975353
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
|
|
a41c6173
|
2022-10-16T08:28:53
|
|
Simplify trace test names.
Originally we had a lot of conditionally enabled test configs
in angle_perftests, that we enabled behind a flag because the
total runtime would be too large if we left everything enabled.
This CL switches to using a single test set, one for each
trace, and using a the same command-line API we use in dEQP
and the ANGLE samples to pick a single configuration. This
makes it easier to pick exactly what configurations you want
from the command line.
Because we only run one configuration each time, we don't need
to specialize the test name to include the configuration info.
We can also simplify TracePerfTest -> TraceTest since these
tests are run for both perf and correctness.
Old name: TracePerfTest.Run/vulkan_trex_200
New name: TraceTest.trex_200
Example for tests that don't use default arguments:
Old args: --enable-all-trace-tests
Old name: TracePerfTest.Run/native_offscreen_trex_200
New args: --use-gl=native --offscreen
New name: TraceTest.trex_200
Bug: angleproject:7755
Change-Id: Id93cf5725eff2e2c6b3b14804bc2902f38306e7e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3966535
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
cc35022f
|
2022-10-21T08:42:31
|
|
Only allow missing output file when listing tests.
Avoids potentially swallowing an error in non-list-test mode
Bug: angleproject:7755
Change-Id: I4e7fba6a5d110d568d71c31585fffdb189ec12a1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3971078
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
873650f4
|
2022-10-19T16:51:53
|
|
run_perf_tests.py: Fix test list unexpected error.
The tests were failing because of a failure to pull the test
json output from the device after --list-tests. This failure
was benign because the test lists still came through to the
test runner and it was able to proceed, but it was causing
an error message.
Also changes the runner to fail early if there's any error
during --list-tests.
Bug: angleproject:7755
Change-Id: I67bceec6b69fd2856136313d5e456e3c1f1caccf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3964753
Commit-Queue: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
d6ca85bd
|
2022-10-18T17:10:41
|
|
run_perf_tests.py: Remove unneeded Android args.
These only apply to the Chromium Android test runner, which we
no longer use for this script.
Bug: angleproject:7755
Change-Id: I311ac204e1999a77872a76f6d0b9d149f7ac409e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3963651
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
|
|
|
0e0ea026
|
2022-10-13T18:29:33
|
|
Split trace tests out into angle_trace_tests.
This makes angle_perftests strictly for microbenchmarks.
The trace tests are a better separated target because they
are for both gold correctness tests and performance.
You can now use the GN arg "build_angle_trace_tests" to control
building the trace tests. "build_angle_trace_perf_tests" is
still supported for compatibility.
Updates the infra scripts to use this new trace tests target.
Also updates the build so by default we'll build the trace
tests if you check them out, without needing a custom arg.
Also updates docs.
Bug: angleproject:7755
Change-Id: I2b1ae67799234cc1aefe7cda5f3af945c3f867fd
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3953340
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
6eff905c
|
2022-10-11T10:08:12
|
|
Make Android gold tests populate os and os_version correctly.
os = Android
os_version = {ro.build.fingerprint}
These show up in Skia gold. Currently set to Linux info.
Bug: None
Change-Id: If2ffe5e8a6620fde9bd0a983f9c0d11ffe6a75c4
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3946179
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
b681e939
|
2022-09-19T13:20:23
|
|
Turn on trace data hash checking automatically.
This should save time uploading trace data already present on the
device.
Bug: angleproject:7671
Change-Id: Ie655cba284b6b3aec2e264cd4d64e4d18ebff8a7
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3905902
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
094b49db
|
2022-09-19T10:13:28
|
|
Android Test Runner: Skip APK install if hash matches.
Bug: angleproject:7670
Change-Id: I984fe9377728e5a575fd04be2f5cb3ed1cdbafb5
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3904425
Reviewed-by: Roman Lavrov <romanl@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
f30cfd18
|
2022-08-16T16:50:33
|
|
Perf and gold tests detect test SKIPs from json results.
Make angle_test_util.RunTestSuite() gets and returns json results
(using --isolated-script-test-output).
perf tests has an additional special case
(filed https://anglebug.com/7578):
Previously when metrics were missing it was assumed that this was due to
tests being skipped - which seems to currently be the case except for
the special case of GLMark2Benchmark* tests. Those produce .fps/.score
metrics instead of the metrics expected by this script. This CL keeps
those tests running (once) but then marks them as SKIP in test results
when no metrics are found (which is the behavior before this CL; it
might actually make more sense to count this as PASS because everything
matches our expectations).
We're still running them once, so if they FAIL we'll get notified.
gold tests:
Use json results file instead of checking for '[ SKIPPED ]' in stdout.
Bug: angleproject:7299
Bug: angleproject:7578
Change-Id: Ia751784ad1aa94dc855c8b58ebfe5ba3e06e462f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3826167
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
2de8bb42
|
2022-08-03T18:15:52
|
|
Converge run wrappers into angle_test_util.RunTestSuite
Bug: angleproject:7299
Change-Id: Ifa882028a6f49579b4a9c6893053cb3bee8d83fd
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808302
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
88b05d3c
|
2022-08-03T15:21:00
|
|
Use globals in android_helper to simplify flow.
Explicit .Initialize(suite_name) call to detect Android
Track suite (apk) that was installed. We use the same package
com.android.angle.test in multiple apks (eg system info and perftests)
so keeping track of which one is currently installed avoids the need to
take care of it manually (with PrepareTestSuite).
Bug: angleproject:7299
Change-Id: I4b96f0fb24e38fc42d3814fb0eeda5aae527de5a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808292
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
b7bd0808
|
2022-06-06T10:54:30
|
|
Support running end2end tests.
There is no batching though so all tests matching the filter will be ran
at once which will block without any output until evertyhing is done
(android runner runs tests in batches of 256 IIRC).
Also make suite arg positional and explicitly list supported tests.
Bug: angleproject:6854
Change-Id: Ibe108f24d6c3ec8b966eda1cf9ae92119a517548
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3691048
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
b71c6427
|
2022-05-31T16:23:51
|
|
Add a wrapper script to run traces using android_helper.
Also add missing `appops set MANAGE_EXTERNAL_STORAGE` that appears to be
necessary on newer builds (catapult does it starting from Android 11).
Add a hash check to skip transferring .gz files that are already there.
Bug: angleproject:6854
Change-Id: Ib612d1235fe1274b8fea47718af8389e8810d34e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679486
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Auto-Submit: Roman Lavrov <romanl@google.com>
|
|
|
c9794d6c
|
2022-04-27T14:20:47
|
|
Run a smoke test in each shard and crash on timeouts.
The smoke test (words_with_friends_2) typically takes only a few seconds
to run. Run it first to check that everything is working as intended. As
we know it's a short test, we can set a small timeout (60s) and fail
early in case something goes wrong. This will speed up tests failures
when something is completely broken.
I also changed the main shard run to crash when timeout is detected
instead of just dumping debug info as if the tests didn't finish in 10
minutes they're also unlikely to finish in the 20 minutes after which
swarming times out the shard.
Bug: angleproject:7242
Change-Id: Ia99ca4b2e1676d09b9ff7d67ab116aba94f66512
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3611446
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
0c59b5d9
|
2022-04-26T19:20:49
|
|
Log debuggerd $PID output when instrumentation gets stuck.
Bug: angleproject:7242
Change-Id: I2b12e94177bce65acf68e4420fd1fcabffdb993e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3607499
Auto-Submit: Roman Lavrov <romanl@google.com>
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Commit-Queue: Yuxin Hu <yuxinhu@google.com>
|
|
|
d0c8363e
|
2022-04-26T11:51:47
|
|
Dump logcat if instrumentation doesn't finish in 10 minutes.
Let's see if this helps us understand stuckness as described in
https://anglebug.com/6854#c46
Normally each instrumentation call takes way less than 10 minutes as we
run tests individually or in small batches. When stuck, runner times out
after 20 minutes.
Bug: angleproject:6854
Change-Id: I20074db0fb3e783555a530b9ecef3765f1911fd9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3605763
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
c0ac374f
|
2022-04-25T14:06:58
|
|
Replace `adb exec-out cat` with pull to temp location.
Flaked here:
https://chromium-swarm.appspot.com/task?id=5a7a6cbff20e8510
File "/b/swarming/w/ir/src/tests/py_utils/android_helper.py", line 86, in _ReadDeviceFile
assert len(out) == expected_size, 'exec-out mismatch'
AssertionError: exec-out mismatch
Bug: angleproject:6854
Change-Id: I846474cb64c436a67bbc559839d41967108be8c1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606328
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
d1788614
|
2022-04-22T15:52:16
|
|
Use adb instead of catapult to run perf tests on Android.
Add support for replacing flags:
--isolated-script-test-output
--isolated-script-test-perf-output
--render-test-output-dir
Always set --isolated-script-test-output to check for test failures
using num_failures_by_type, interrupted, is_unexpected fields.
Bug: angleproject:6854
Change-Id: Ia701f4d4d5d1adc9a24c057bb5c58fd5457cbd65
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3602834
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
e6fc5e62
|
2022-04-22T12:45:33
|
|
Use _FindAdb (cached) directly instead of Adb wrapper.
A little simpler, no need to pass around an additional `apk` arg.
Also fix missed _GetAdbRoot() call.
Bug: angleproject:6854
Change-Id: Id64e5f35fb21a6aef7d99e614dc0eae753c24317
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3600912
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
d988db39
|
2022-04-22T11:14:44
|
|
Fall back to adb on PATH if platform-tools not present.
Log adb --version. Example:
INFO:root:adb --version: Android Debug Bridge version 1.0.41
Version 31.0.3-7562133
Installed as /{snip}/third_party/android_sdk/public/platform-tools/adb
When not found -> "Installed as /usr/bin/adb"
Bug: angleproject:6854
Change-Id: I3c19c4eda2211480f2c93b7d14f72f5641b3bfc2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3602821
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
7c091cb3
|
2022-04-19T21:12:35
|
|
Use adb directly (instead of catapult) in gold tests.
Android detected by angle_perftests apk presence relative to pwd.
All of the restricted_traces/*/*.json files are copied but those are
small. Only the necessary .angledata.gz files are copied.
angle_system_info_test is also handled via adb.
Bug: angleproject:6854
Change-Id: I7a89ff57fcdd8ce5dc63a5e3a8f5c0132f766894
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3595241
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|