|
da0b30f0
|
2024-06-07T10:00:37
|
|
Android perf tests: add temp-based throttling to pixel6 bots.
Seems like we don't have sufficient airflow to run heavy tests at full
throttle. Some shards start >10C hotter, so add custom throttling to
avoid aggressive throttling by the phone itself skewing perf results.
This might noticeably increase shard runtime on bots with poor airflow.
Implementation is similar to https://crrev.com/c/5288455 but relies on
data available on current bot OS build (no IThermal) and does _not_
disable phone throttling.
Bug: b/345514719
Change-Id: I582838c2448c9b014381a4f8ba847343b68e4d3a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5608312
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
3ab0cbfd
|
2024-06-06T15:41:51
|
|
Perf tests: log /dev/thermal temps on Android
Bug: b/345514719
Change-Id: Id72bcffc17454ec0d9c3043a2f1e0b51cbf4b4a3
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5606162
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Auto-Submit: Roman Lavrov <romanl@google.com>
Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
|
b744ee7f
|
2024-03-12T11:00:39
|
|
Trace tests: extract debug files to (CAS) output
Follow up on https://crrev.com/c/5361574 which works for gold test but
not perf tests which use temporary directory for output files
Bug: b/296921272
Change-Id: Idfc75839cf2cadde44da1567b7347be7c6520bdf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5365307
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
|
9a7e8b7b
|
2023-11-07T17:02:02
|
|
Perf tests: log hwmon temps on linux
Syslog on a failed bot indicates possible overheating.
Log all hwmon temps which include CPU temps but also some other system
temps (disk probably not included). We might need to self-throttle if
crashes are correlated with higher temps.
Bug: b/296921272
Change-Id: I25f8dd899dd25880fc1567ecc18e102759fb868f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/5008089
Auto-Submit: Roman Lavrov <romanl@google.com>
Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
|
d6fd7eaf
|
2023-08-25T16:33:27
|
|
Trace/perf tests: remove calibration and most warmup options
This removes calibration and warmup except for --warmup
(which does what --warmup-steps=-1 used to do: render each frame once
with glFinish)
Trace and perf tests now default to no warmup.
Unchanged by this CL:
--fixed-test-time=x runs a single trial for x seconds.
--fixed-test-time-with-warmup=x is effectively the above flag coupled
with --warmup-steps=-1
--run-to-key-frame runs to the key frame (no longer disables warmup as
it's now off by default)
Bug: b/297418214
Change-Id: I7a0d3e490067dbde57c5f519c1b9092ba70b1480
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4812049
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
|
|
|
2c624a5e
|
2023-08-17T15:36:37
|
|
android_helper cleanup: remove smoke test and timeout
This was added to help debug cases of stuck tests on bots, but this
doesn't seem to have happened in over a year and the code (and timeout)
keeps getting in the way.
Bug: angleproject:7242
Change-Id: Ic30d976408bd32a745d830921f6e4d98d17e863c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4789845
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
58055032
|
2023-06-16T10:33:26
|
|
Remove _shardN suffix from tests when it's not necessary.
My understanding is that we only need this for --split-shard-samples.
Otherwise we still get _shardN suffix from the shard but the test only
ran on a single shard so there is no point in that.
_shardN added in https://crrev.com/c/3937044
--split-shard-samples disabled in https://crrev.com/c/4605335
Example of _shardN:
https://ci.chromium.org/ui/p/angle/builders/ci/linux-nvidia-gtx1660-perf/1704/overview
Bug: angleproject:7671
Change-Id: Iaec4eb81e6ebad424cfc2ebfcbc7eda312cd1bba
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4615538
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Auto-Submit: Roman Lavrov <romanl@google.com>
|
|
|
2b1bc4fd
|
2023-01-09T12:17:17
|
|
Upload angle metrics to the gs bucket in skia perf format.
ConvertToSkiaPerf converts the data format from angle_metrics.json
to what skia perf ingests. Injestion happens automatically when the
gs bucket is updated.
Example data from this CL on non-perf bots with upload enabled
(smoke mode so actual values aren't meaningful):
https://angle-perf.skia.org/e/?queries=buildername%3Dlinux-test%26metric%3Dwall_time%26test%3Dtrex_200
Bug: angleproject:7299
Change-Id: Ica700b586e08c205968fbc3c1d15cf742ad537f4
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4148167
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
056f8044
|
2022-10-11T11:58:20
|
|
Perf tests: write metrics to file and read from merge script.
The idea behind is to save metrics in a structured output with
full context so that we don't have to parse stdout and
reconstruct that context (e.g. which test was running when
the output was printed etc).
Note: we already have a "histogram" file (addHistogramSample)
written but it is structured using some UUID-based ids making its
digestion more complicated (https://anglebug.com/7299#c3).
This CL just writes events as one-json-per-line
simplifying downstream handling and debugging.
Just a sanity check / logging in the merge script for now.
Bug: angleproject:7299
Change-Id: I56dada643eceef180ce9bb1aa9ae6a641ea41e4c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3945112
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
71e89320
|
2023-01-03T09:10:08
|
|
Tests: Support multiple filters with ":".
Now we can run multiple traces per invocation, e.g.:
"trex_200:among_us"
Bug: angleproject:7775
Change-Id: I8e8176d444f20ae97234469ab8c553543b81b331
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4132944
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
b205207a
|
2022-11-16T14:39:00
|
|
Lock gpu clocks using nvidia-smi to avoid overheating
For now only applies to Windows Nvidia (GTX 1660) bots
where I tested this. Default clocks boost to 1800+
reporting 110W+ initial power usage which can't be dissipated by
the setup and and gpu quickly hits ~90C and enters aggressive
throttling. Locking to 1410MHz yield a much more modest 50~60W usage
with reasonable gpu temps in the 70C~80C range over long runs at
100% utilization.
Also log gpu temperature before/after running tests.
Bug: angleproject:7671
Change-Id: Id43bede3b1d6a445f8caee6bbbaa43f7f380199b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4031495
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
30439960
|
2022-10-24T17:30:18
|
|
Unbreak run_angle_android_test.
Broken by ListTests removed in https://crrev.com/c/3963651
Move helper to angle_test_util.
Bug: angleproject:7755
Change-Id: I1eec442cf3e2bf6c24335d4c294a68e7e868b814
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3975353
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
|
|
|
bc2e9eea
|
2022-10-15T14:38:15
|
|
Refine sampling of trace perf tests.
- The warmup algorithm now computes an accurate steps estimate.
- Bump up the trials, samples, and trial time in the runner.
Bug: angleproject:7671
Change-Id: I89a73472fd72af73f695c9107119cbc559d2d6e1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3956946
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
a41c6173
|
2022-10-16T08:28:53
|
|
Simplify trace test names.
Originally we had a lot of conditionally enabled test configs
in angle_perftests, that we enabled behind a flag because the
total runtime would be too large if we left everything enabled.
This CL switches to using a single test set, one for each
trace, and using a the same command-line API we use in dEQP
and the ANGLE samples to pick a single configuration. This
makes it easier to pick exactly what configurations you want
from the command line.
Because we only run one configuration each time, we don't need
to specialize the test name to include the configuration info.
We can also simplify TracePerfTest -> TraceTest since these
tests are run for both perf and correctness.
Old name: TracePerfTest.Run/vulkan_trex_200
New name: TraceTest.trex_200
Example for tests that don't use default arguments:
Old args: --enable-all-trace-tests
Old name: TracePerfTest.Run/native_offscreen_trex_200
New args: --use-gl=native --offscreen
New name: TraceTest.trex_200
Bug: angleproject:7755
Change-Id: Id93cf5725eff2e2c6b3b14804bc2902f38306e7e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3966535
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
873650f4
|
2022-10-19T16:51:53
|
|
run_perf_tests.py: Fix test list unexpected error.
The tests were failing because of a failure to pull the test
json output from the device after --list-tests. This failure
was benign because the test lists still came through to the
test runner and it was able to proceed, but it was causing
an error message.
Also changes the runner to fail early if there's any error
during --list-tests.
Bug: angleproject:7755
Change-Id: I67bceec6b69fd2856136313d5e456e3c1f1caccf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3964753
Commit-Queue: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
d6ca85bd
|
2022-10-18T17:10:41
|
|
run_perf_tests.py: Remove unneeded Android args.
These only apply to the Chromium Android test runner, which we
no longer use for this script.
Bug: angleproject:7755
Change-Id: I311ac204e1999a77872a76f6d0b9d149f7ac409e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3963651
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
|
|
|
0e0ea026
|
2022-10-13T18:29:33
|
|
Split trace tests out into angle_trace_tests.
This makes angle_perftests strictly for microbenchmarks.
The trace tests are a better separated target because they
are for both gold correctness tests and performance.
You can now use the GN arg "build_angle_trace_tests" to control
building the trace tests. "build_angle_trace_perf_tests" is
still supported for compatibility.
Updates the infra scripts to use this new trace tests target.
Also updates the build so by default we'll build the trace
tests if you check them out, without needing a custom arg.
Also updates docs.
Bug: angleproject:7755
Change-Id: I2b1ae67799234cc1aefe7cda5f3af945c3f867fd
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3953340
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
d0a024f9
|
2022-10-09T16:47:33
|
|
Lower perf tests resource usage further.
This reduces the warmup trials from 2->1 and the trial time
from 5s -> 4s.
Old: (2 warmup + 3 trials) * 5s / trial ~= 25s per sample
New: (1 warmup + 3 trials) * 4s / trial ~= 16s per sample
The tests should run in about 2/3 the time as it took prior.
A future change will likely have to split up the tests so we
can run shorter shards in sequential groupings between tests.
Bug: angleproject:7738
Change-Id: Ic6c744be838174b61f4eb832189ccad14f89103d
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3939917
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
|
|
|
1b5cf9f0
|
2022-10-07T19:02:16
|
|
run_perf_tests: Reduce warmup trials from 3 to 2.
This is an attempt to reduce utilization on the bots, which
are now taking too long and causing errors.
Bug: angleproject:7738
Change-Id: I1c576b2d0989240482d332712f7d93841e933902
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3937051
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Commit-Queue: Yuxin Hu <yuxinhu@google.com>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
b6cc672c
|
2022-10-06T11:10:42
|
|
Perf Tests: Divide test samples between shards.
Instead of a single test shard running all samples for a test,
we run a single sample on different shards to produce a
combined measurement that can account for variance between
test machines.
Bug: angleproject:7671
Change-Id: I04bbd51de257f2f299783ce3bd65da3d9348c8dc
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3937044
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d8489db5
|
2022-10-05T12:57:06
|
|
Perf Tests: Reduce resource usage.
This halves the amount of time spent running the tests on
desktop platforms. It also reduces the number of shards used
on desktop platforms, to reduce resource utilization. It also
bumps up the test timeout to keep from getting infra errors.
Bug: angleproject:7738
Change-Id: Ie487c06452a39bb9ad1fe29b0d682e9c1bf297d5
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3936448
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
67af8d89
|
2022-09-19T15:19:57
|
|
Perf Tests: Make dynamic step count calibration optional.
This uses a fixed test time for the tests, instead of using an
extra step to calibrate a number of steps to run in an approximate
maximum time.
Before:
1. run a test for <test time>, return the number of steps
2. for <num samples> x <num trials> times, run the tests for that
number of steps
After:
1. for <num samples> x <num trials> times, run the tests for
<test time>
This saves some total time since we don't need to run the tests in
a pre-process to calibrate the number of steps, and it also makes
this test script more like the restricted trace perf script.
Bug: angleproject:7671
Change-Id: I413cbf8d65f9f2f87781d8607f799d2450a71ea7
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3906144
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Roman Lavrov <romanl@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
65d3d0f6
|
2022-09-19T10:42:43
|
|
run_perf_tests: Add option to find test binary in out dir.
This allows us to run the perf test runner script from any
directory and use that as a basis to find the most recently
modified test suite binary. This saves us from chdir()ing into
test directory every time we want to run a test suite.
Ideally this should move into the test utils folder to be used
by the gold tests script as well.
Bug: angleproject:7671
Change-Id: Iead98670bb5f7627ae32f7ac41b4ebcee2726841
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3904427
Reviewed-by: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
1fc6c49c
|
2022-09-23T16:39:26
|
|
Log additional info when "failed to record some samples"
Check each sample instead of total in the end to make it more clear.
Bug: angleproject:7117
Change-Id: I7d6137c535266e4cf47e772b816cfb8bfa4575d3
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3916516
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Auto-Submit: Roman Lavrov <romanl@google.com>
|
|
|
f30cfd18
|
2022-08-16T16:50:33
|
|
Perf and gold tests detect test SKIPs from json results.
Make angle_test_util.RunTestSuite() gets and returns json results
(using --isolated-script-test-output).
perf tests has an additional special case
(filed https://anglebug.com/7578):
Previously when metrics were missing it was assumed that this was due to
tests being skipped - which seems to currently be the case except for
the special case of GLMark2Benchmark* tests. Those produce .fps/.score
metrics instead of the metrics expected by this script. This CL keeps
those tests running (once) but then marks them as SKIP in test results
when no metrics are found (which is the behavior before this CL; it
might actually make more sense to count this as PASS because everything
matches our expectations).
We're still running them once, so if they FAIL we'll get notified.
gold tests:
Use json results file instead of checking for '[ SKIPPED ]' in stdout.
Bug: angleproject:7299
Bug: angleproject:7578
Change-Id: Ia751784ad1aa94dc855c8b58ebfe5ba3e06e462f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3826167
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
7c66d77d
|
2022-08-16T15:06:29
|
|
Avoid logging None introduced in https://crrev.com/c/3831885
Currently prints:
Test 5/97: DrawCallPerfBenchmark.Run/gles_null_uniform: None
Wasn't printing anything before, so this CL goes back to that.
Bug: angleproject:7299
Change-Id: I8270129958a93e5c3fbb6cac321fe7e2396a6376
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3834068
Commit-Queue: Roman Lavrov <romanl@google.com>
Commit-Queue: Amirali Abdolrashidi <abdolrashidi@google.com>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Auto-Submit: Roman Lavrov <romanl@google.com>
|
|
|
8878366f
|
2022-08-15T11:51:00
|
|
Refactor to reduce main loop complexity.
Bug: angleproject:7299
Change-Id: I157e84b7a6718d59eecb00471d4ec7ae5b3317cf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3831885
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
|
|
|
2de8bb42
|
2022-08-03T18:15:52
|
|
Converge run wrappers into angle_test_util.RunTestSuite
Bug: angleproject:7299
Change-Id: Ifa882028a6f49579b4a9c6893053cb3bee8d83fd
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808302
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
146329b8
|
2022-08-05T15:51:18
|
|
run_perf_tests: Allow passing flags to the test.
Bug: None
Change-Id: I0260bafd91d371d653248ea278b80bd579a12a5a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3812560
Reviewed-by: Roman Lavrov <romanl@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
88b05d3c
|
2022-08-03T15:21:00
|
|
Use globals in android_helper to simplify flow.
Explicit .Initialize(suite_name) call to detect Android
Track suite (apk) that was installed. We use the same package
com.android.angle.test in multiple apks (eg system info and perftests)
so keeping track of which one is currently installed avoids the need to
take care of it manually (with PrepareTestSuite).
Bug: angleproject:7299
Change-Id: I4b96f0fb24e38fc42d3814fb0eeda5aae527de5a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3808292
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
19c6600c
|
2022-08-02T12:45:57
|
|
Refactoring: move a few common functions to angle_test_util.
Drop ScriptTest, Jamie says no longer needed.
Bug: angleproject:7299
Change-Id: I32bdd1d70701f63b578b4f79512868daac3ad520
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3805381
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
ebf14fe3
|
2022-07-14T17:19:15
|
|
Return EXIT_FAILURE from run_perf_tests.py if any test failed
Previosly EXIT_FAILURE was returned only for exceeding max_errors,
which made Swarming erroneously believe that the task succeeded,
even though some tests in it could have failed.
Bug: angleproject:7498
Change-Id: I896fe97e7d24549202663351d5b61b206afbe6b1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3762456
Commit-Queue: Roman Lavrov <romanl@google.com>
Auto-Submit: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Roman Lavrov <romanl@google.com>
|
|
|
b71c6427
|
2022-05-31T16:23:51
|
|
Add a wrapper script to run traces using android_helper.
Also add missing `appops set MANAGE_EXTERNAL_STORAGE` that appears to be
necessary on newer builds (catapult does it starting from Android 11).
Add a hash check to skip transferring .gz files that are already there.
Bug: angleproject:6854
Change-Id: Ib612d1235fe1274b8fea47718af8389e8810d34e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679486
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Auto-Submit: Roman Lavrov <romanl@google.com>
|
|
|
b03f2789
|
2022-05-31T20:31:23
|
|
Revert "Use a temporary timeout multipler for trace tests."
This reverts commit f810e998993290f049bbdad4fae975e4867100ad.
Reason for revert: Uploads are now sharded so this shouldn't be needed.
Original change's description:
> Use a temporary timeout multipler for trace tests.
>
> The trace tests upload is frequently timing out on our test infra
> because of the size of the test suite. Until we can find a better
> solution we can increase the timeout non-invasively using an
> environment variable.
>
> Bug: angleproject:6854
> Change-Id: Ifbe43c4d9dc8928cf35df8a26b844f884c90eb9d
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3427560
> Reviewed-by: Cody Northrop <cnorthrop@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Bug: angleproject:6854
Change-Id: Iaad7b63073ed900acdc575fbac927a8463b0c530
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679487
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
c9794d6c
|
2022-04-27T14:20:47
|
|
Run a smoke test in each shard and crash on timeouts.
The smoke test (words_with_friends_2) typically takes only a few seconds
to run. Run it first to check that everything is working as intended. As
we know it's a short test, we can set a small timeout (60s) and fail
early in case something goes wrong. This will speed up tests failures
when something is completely broken.
I also changed the main shard run to crash when timeout is detected
instead of just dumping debug info as if the tests didn't finish in 10
minutes they're also unlikely to finish in the 20 minutes after which
swarming times out the shard.
Bug: angleproject:7242
Change-Id: Ia99ca4b2e1676d09b9ff7d67ab116aba94f66512
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3611446
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
cdcf1715
|
2022-04-26T14:48:10
|
|
Add missing result_skip and result_fail.
Missing result_skip results in tests missing from json.output in perf
CI builders. Not sure if the missing result_fail actually happens.
Bug: angleproject:6854
Change-Id: I0d70a93be0d595c45e23558345481e749ba128bc
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606921
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
ebfb9b9f
|
2022-04-26T10:37:52
|
|
Configure logging in perf and gold tests to include time.
Currently no timestamp is logged:
[I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd}
INFO:root:adb --version ...
Before switching to android_helper relative timestamps in seconds
were printed, set up by catapult. Timestamps proposed in this CL are
more consistent with outer logging but keep prefix short. Example:
[I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd}
I18:45:14.999999Z adb --version ...
Bug: angleproject:6854
Change-Id: I8ef7c9ae44eb5dba564a41456ce9417e3fe0f06c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606913
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
d1788614
|
2022-04-22T15:52:16
|
|
Use adb instead of catapult to run perf tests on Android.
Add support for replacing flags:
--isolated-script-test-output
--isolated-script-test-perf-output
--render-test-output-dir
Always set --isolated-script-test-output to check for test failures
using num_failures_by_type, interrupted, is_unexpected fields.
Bug: angleproject:6854
Change-Id: Ia701f4d4d5d1adc9a24c057bb5c58fd5457cbd65
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3602834
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
4ac3ec13
|
2022-04-22T12:15:51
|
|
Switch py_utils import to pathlib.
os.stat path before adding just to catch future copy-paste mistakes.
Bug: angleproject:6854
Change-Id: If58c46f99aca93e6aeda90a987ceebbbb1b66955
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3600911
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
3739a195
|
2022-03-09T13:56:36
|
|
perf tests: Record perf counter metrics.
This adds a new command line argument that will allow the user to
specify perf counters to record into the test output.
Bug: angleproject:4918
Change-Id: Ia7432ff96eadf13ef681f67d2d503d00fd83e06e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3516970
Reviewed-by: Lingfeng Yang <lfy@google.com>
Reviewed-by: Charlie Lao <cclao@google.com>
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
1cfbe863
|
2022-03-09T13:35:44
|
|
perf tests: Report multiple time metrics.
We can now report cpu/wall/gpu time instead of just a single timing
metric. This is in preparation for reporting other types of metric
including counters.
Also includes some other minor script improvements.
Bug: angleproject:4918
Change-Id: I52a3241893c56a1bfe982990ebac0fa168dbac51
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3516969
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Reviewed-by: Charlie Lao <cclao@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
f810e998
|
2022-01-31T11:39:37
|
|
Use a temporary timeout multipler for trace tests.
The trace tests upload is frequently timing out on our test infra
because of the size of the test suite. Until we can find a better
solution we can increase the timeout non-invasively using an
environment variable.
Bug: angleproject:6854
Change-Id: Ifbe43c4d9dc8928cf35df8a26b844f884c90eb9d
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3427560
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
14af7a8d
|
2021-12-22T15:17:26
|
|
Log subprocess output in run_perf_tests.py
To aid with debugging a flaky hang listing tests on Android.
Bug: chromium:1279512
Change-Id: Icbdf753e1e7c21ab766fdd5767ca05d07b66485b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3353760
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
|
|
|
e0b4cf6c
|
2021-11-26T10:23:24
|
|
Fix Python3 usage with perf tests/gold tests.
Bug: angleproject:6733
Change-Id: I9d1baf78ed182db77c6a6d8e896737413ca417f3
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3304012
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8c634492
|
2021-11-17T11:06:23
|
|
Updates to perf test running script.
- Fixes logging (needed reload)
- Improves logging messages (less spam)
- Reduces time in test calibration & sampling
- Record failures on test errors more robustly
- Redesign the result recording class
Bug: angleproject:6090
Change-Id: I53fd86e9e009bd52ec98507334c69b05e711d83e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3288206
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
c3e0eafa
|
2021-09-22T13:35:04
|
|
Revert "Perf Tests: Split samples into sub-tests on desktop."
This reverts commit 90c981dafc06d6bc0696749dface1337e37784d2.
Reason for revert: Causing timeouts on perf bots.
Original change's description:
> Perf Tests: Split samples into sub-tests on desktop.
>
> This adds an option to the perf test runner to split each sample of
> each test into a sub-test. The sub-tests then are split among the
> shards like ordinary tests, and distributed across machines. The
> intent is that running on different machines will reduce inter-build
> variance because of differences between test machines.
>
> This sampling change is only added to desktop (Linux/Win) because
> Android device variance seems to be lower.
>
> Bug: angleproject:6090
> Change-Id: I4046a012727baa8a0f2595de2349cc34257691f9
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3152752
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Cody Northrop <cnorthrop@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Bug: angleproject:6090
Change-Id: I92bffb98999f35274ec31a3b7567d87453feff94
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3174274
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
90c981da
|
2021-09-10T11:04:19
|
|
Perf Tests: Split samples into sub-tests on desktop.
This adds an option to the perf test runner to split each sample of
each test into a sub-test. The sub-tests then are split among the
shards like ordinary tests, and distributed across machines. The
intent is that running on different machines will reduce inter-build
variance because of differences between test machines.
This sampling change is only added to desktop (Linux/Win) because
Android device variance seems to be lower.
Bug: angleproject:6090
Change-Id: I4046a012727baa8a0f2595de2349cc34257691f9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3152752
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
1b2febb9
|
2021-08-23T12:39:16
|
|
Trace Tests: Refactor calibration logic.
Instead of calling glFinish every step we now run a few warmup loops
to try and figure out the right step interval. On each run we refine
the step interval until we reach a number that completes in the
specific time after a single glFinish call. This method will use the
specified test time much more consistenty.
Also a couple other improvements. Will use more test time on the bots
because previously they were using a calibration/warmup time of 1
second per run loop, and this bumps it up to 3 per run loop.
Long explanation below.
What we did before:
step()
glFinish()
step()
glFinish()
.. repeat for "gCalibrationTime" seconds
then take the number of steps, the actual time ran, and compute an
estimate for "mStepsToRun".
When we run for real, we go:
step()
step()
step()
.. repeat for "mStepsToRun" steps.
glFinish()
this has an obvious performance difference, because after each step()
(one frame in a trace) you had a complete stall.
The new code first does the above to compute a tentative mStepsToRun,
then uses the results of the second run to compute a much more accurate
mStepsToRun by using the "actual" time it took to run the estimate.
Bug: angleproject:6090
Change-Id: I0ff19873580e13f2af5a145a21f5b1c59fb15795
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3114606
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
130a8b63
|
2021-08-16T17:09:10
|
|
Perf Runner: Properly merge histograms.
The prior method was emitting 5 histograms per test. The new script
method merges the 5 histogram samples together and recomputes the
running statistics.
Should improve reporting on the dashboard.
Bug: angleproject:6090
Change-Id: I58a5e020f3a3257a8545df2785bc3eaa811ee38e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3096550
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d25c9d7f
|
2021-07-07T12:15:12
|
|
Add perf tests runner script.
This script will be responsible for running the perf tests
multiple times to try and stabilize measurements. We'll use it on
the bots instead of just running the perf tests directly.
Because the script invokes the binary multiple times, this slows
down execution. Most significantly on Android, where we now need
to use 20 shards, up from 6.
Also marks one test as flaky on OpenGL.
Bug: angleproject:6090
Change-Id: I5280035cb0bdb290a68dc6961a384eaf4b40dd4b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3011422
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Tim Van Patten <timvp@google.com>
|