|
b71c6427
|
2022-05-31T16:23:51
|
|
Add a wrapper script to run traces using android_helper.
Also add missing `appops set MANAGE_EXTERNAL_STORAGE` that appears to be
necessary on newer builds (catapult does it starting from Android 11).
Add a hash check to skip transferring .gz files that are already there.
Bug: angleproject:6854
Change-Id: Ib612d1235fe1274b8fea47718af8389e8810d34e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679486
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Auto-Submit: Roman Lavrov <romanl@google.com>
|
|
|
b03f2789
|
2022-05-31T20:31:23
|
|
Revert "Use a temporary timeout multipler for trace tests."
This reverts commit f810e998993290f049bbdad4fae975e4867100ad.
Reason for revert: Uploads are now sharded so this shouldn't be needed.
Original change's description:
> Use a temporary timeout multipler for trace tests.
>
> The trace tests upload is frequently timing out on our test infra
> because of the size of the test suite. Until we can find a better
> solution we can increase the timeout non-invasively using an
> environment variable.
>
> Bug: angleproject:6854
> Change-Id: Ifbe43c4d9dc8928cf35df8a26b844f884c90eb9d
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3427560
> Reviewed-by: Cody Northrop <cnorthrop@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Bug: angleproject:6854
Change-Id: Iaad7b63073ed900acdc575fbac927a8463b0c530
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3679487
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
c9794d6c
|
2022-04-27T14:20:47
|
|
Run a smoke test in each shard and crash on timeouts.
The smoke test (words_with_friends_2) typically takes only a few seconds
to run. Run it first to check that everything is working as intended. As
we know it's a short test, we can set a small timeout (60s) and fail
early in case something goes wrong. This will speed up tests failures
when something is completely broken.
I also changed the main shard run to crash when timeout is detected
instead of just dumping debug info as if the tests didn't finish in 10
minutes they're also unlikely to finish in the 20 minutes after which
swarming times out the shard.
Bug: angleproject:7242
Change-Id: Ia99ca4b2e1676d09b9ff7d67ab116aba94f66512
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3611446
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
cdcf1715
|
2022-04-26T14:48:10
|
|
Add missing result_skip and result_fail.
Missing result_skip results in tests missing from json.output in perf
CI builders. Not sure if the missing result_fail actually happens.
Bug: angleproject:6854
Change-Id: I0d70a93be0d595c45e23558345481e749ba128bc
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606921
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
ebfb9b9f
|
2022-04-26T10:37:52
|
|
Configure logging in perf and gold tests to include time.
Currently no timestamp is logged:
[I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd}
INFO:root:adb --version ...
Before switching to android_helper relative timestamps in seconds
were printed, set up by catapult. Timestamps proposed in this CL are
more consistent with outer logging but keep prefix short. Example:
[I2022-04-25T18:45:14.761163Z 548 0 cmd_stream.go:336] {cmd}
I18:45:14.999999Z adb --version ...
Bug: angleproject:6854
Change-Id: I8ef7c9ae44eb5dba564a41456ce9417e3fe0f06c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3606913
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
d1788614
|
2022-04-22T15:52:16
|
|
Use adb instead of catapult to run perf tests on Android.
Add support for replacing flags:
--isolated-script-test-output
--isolated-script-test-perf-output
--render-test-output-dir
Always set --isolated-script-test-output to check for test failures
using num_failures_by_type, interrupted, is_unexpected fields.
Bug: angleproject:6854
Change-Id: Ia701f4d4d5d1adc9a24c057bb5c58fd5457cbd65
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3602834
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
|
4ac3ec13
|
2022-04-22T12:15:51
|
|
Switch py_utils import to pathlib.
os.stat path before adding just to catch future copy-paste mistakes.
Bug: angleproject:6854
Change-Id: If58c46f99aca93e6aeda90a987ceebbbb1b66955
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3600911
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
3739a195
|
2022-03-09T13:56:36
|
|
perf tests: Record perf counter metrics.
This adds a new command line argument that will allow the user to
specify perf counters to record into the test output.
Bug: angleproject:4918
Change-Id: Ia7432ff96eadf13ef681f67d2d503d00fd83e06e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3516970
Reviewed-by: Lingfeng Yang <lfy@google.com>
Reviewed-by: Charlie Lao <cclao@google.com>
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
1cfbe863
|
2022-03-09T13:35:44
|
|
perf tests: Report multiple time metrics.
We can now report cpu/wall/gpu time instead of just a single timing
metric. This is in preparation for reporting other types of metric
including counters.
Also includes some other minor script improvements.
Bug: angleproject:4918
Change-Id: I52a3241893c56a1bfe982990ebac0fa168dbac51
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3516969
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
Reviewed-by: Charlie Lao <cclao@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
f810e998
|
2022-01-31T11:39:37
|
|
Use a temporary timeout multipler for trace tests.
The trace tests upload is frequently timing out on our test infra
because of the size of the test suite. Until we can find a better
solution we can increase the timeout non-invasively using an
environment variable.
Bug: angleproject:6854
Change-Id: Ifbe43c4d9dc8928cf35df8a26b844f884c90eb9d
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3427560
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
14af7a8d
|
2021-12-22T15:17:26
|
|
Log subprocess output in run_perf_tests.py
To aid with debugging a flaky hang listing tests on Android.
Bug: chromium:1279512
Change-Id: Icbdf753e1e7c21ab766fdd5767ca05d07b66485b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3353760
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
|
|
|
e0b4cf6c
|
2021-11-26T10:23:24
|
|
Fix Python3 usage with perf tests/gold tests.
Bug: angleproject:6733
Change-Id: I9d1baf78ed182db77c6a6d8e896737413ca417f3
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3304012
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8c634492
|
2021-11-17T11:06:23
|
|
Updates to perf test running script.
- Fixes logging (needed reload)
- Improves logging messages (less spam)
- Reduces time in test calibration & sampling
- Record failures on test errors more robustly
- Redesign the result recording class
Bug: angleproject:6090
Change-Id: I53fd86e9e009bd52ec98507334c69b05e711d83e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3288206
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
c3e0eafa
|
2021-09-22T13:35:04
|
|
Revert "Perf Tests: Split samples into sub-tests on desktop."
This reverts commit 90c981dafc06d6bc0696749dface1337e37784d2.
Reason for revert: Causing timeouts on perf bots.
Original change's description:
> Perf Tests: Split samples into sub-tests on desktop.
>
> This adds an option to the perf test runner to split each sample of
> each test into a sub-test. The sub-tests then are split among the
> shards like ordinary tests, and distributed across machines. The
> intent is that running on different machines will reduce inter-build
> variance because of differences between test machines.
>
> This sampling change is only added to desktop (Linux/Win) because
> Android device variance seems to be lower.
>
> Bug: angleproject:6090
> Change-Id: I4046a012727baa8a0f2595de2349cc34257691f9
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3152752
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Cody Northrop <cnorthrop@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Bug: angleproject:6090
Change-Id: I92bffb98999f35274ec31a3b7567d87453feff94
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3174274
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
90c981da
|
2021-09-10T11:04:19
|
|
Perf Tests: Split samples into sub-tests on desktop.
This adds an option to the perf test runner to split each sample of
each test into a sub-test. The sub-tests then are split among the
shards like ordinary tests, and distributed across machines. The
intent is that running on different machines will reduce inter-build
variance because of differences between test machines.
This sampling change is only added to desktop (Linux/Win) because
Android device variance seems to be lower.
Bug: angleproject:6090
Change-Id: I4046a012727baa8a0f2595de2349cc34257691f9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3152752
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
1b2febb9
|
2021-08-23T12:39:16
|
|
Trace Tests: Refactor calibration logic.
Instead of calling glFinish every step we now run a few warmup loops
to try and figure out the right step interval. On each run we refine
the step interval until we reach a number that completes in the
specific time after a single glFinish call. This method will use the
specified test time much more consistenty.
Also a couple other improvements. Will use more test time on the bots
because previously they were using a calibration/warmup time of 1
second per run loop, and this bumps it up to 3 per run loop.
Long explanation below.
What we did before:
step()
glFinish()
step()
glFinish()
.. repeat for "gCalibrationTime" seconds
then take the number of steps, the actual time ran, and compute an
estimate for "mStepsToRun".
When we run for real, we go:
step()
step()
step()
.. repeat for "mStepsToRun" steps.
glFinish()
this has an obvious performance difference, because after each step()
(one frame in a trace) you had a complete stall.
The new code first does the above to compute a tentative mStepsToRun,
then uses the results of the second run to compute a much more accurate
mStepsToRun by using the "actual" time it took to run the estimate.
Bug: angleproject:6090
Change-Id: I0ff19873580e13f2af5a145a21f5b1c59fb15795
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3114606
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
130a8b63
|
2021-08-16T17:09:10
|
|
Perf Runner: Properly merge histograms.
The prior method was emitting 5 histograms per test. The new script
method merges the 5 histogram samples together and recomputes the
running statistics.
Should improve reporting on the dashboard.
Bug: angleproject:6090
Change-Id: I58a5e020f3a3257a8545df2785bc3eaa811ee38e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3096550
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d25c9d7f
|
2021-07-07T12:15:12
|
|
Add perf tests runner script.
This script will be responsible for running the perf tests
multiple times to try and stabilize measurements. We'll use it on
the bots instead of just running the perf tests directly.
Because the script invokes the binary multiple times, this slows
down execution. Most significantly on Android, where we now need
to use 20 shards, up from 6.
Also marks one test as flaky on OpenGL.
Bug: angleproject:6090
Change-Id: I5280035cb0bdb290a68dc6961a384eaf4b40dd4b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3011422
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Tim Van Patten <timvp@google.com>
|