|
056f8044
|
2022-10-11T11:58:20
|
|
Perf tests: write metrics to file and read from merge script.
The idea behind is to save metrics in a structured output with
full context so that we don't have to parse stdout and
reconstruct that context (e.g. which test was running when
the output was printed etc).
Note: we already have a "histogram" file (addHistogramSample)
written but it is structured using some UUID-based ids making its
digestion more complicated (https://anglebug.com/7299#c3).
This CL just writes events as one-json-per-line
simplifying downstream handling and debugging.
Just a sanity check / logging in the merge script for now.
Bug: angleproject:7299
Change-Id: I56dada643eceef180ce9bb1aa9ae6a641ea41e4c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3945112
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
|
|
e495e7fd
|
2022-10-14T13:52:47
|
|
Redo perf and trace test parameters.
- Moves common argument parsing code into test_util.
- Changes the perf test arg parsing to use the common functions.
- Adds new --use-angle and --use-gl parameters to the trace tests.
- Also adds new --offscreen and --vsync parameters to the traces.
- Removes the now unneeded --enable-all-trace-tests argument.
- Both --arg=value and --arg value work in test suites now.
Now, instead of using --enable-all-trace-tests you can specify the
backend with --use-angle=swiftshader, --offscreen, or combinations
of those parameters. The test names are the same as they were
before, but only the configured tests will run in a session. We
could opt to simplify the test names in later CLs if we want to
simplify the test running. Ideally we'd keep the perf reporting
the same because then we'd keep the time series the same on the
test infra.
This also allows us to split up the trace tests into separate
targets on the bots, which will better allow us to control the
workloads and sampling of the tests. For example:
- angle_perftests becomes
- angle_perftests (microbenchmarks)
- angle_trace_perf_vulkan_tests (traces with vulkan back-end)
- angle_trace_perf_native_tests (traces with system GL)
Bug: angleproject:7755
Change-Id: I537168f3a6de96425dfda05ed98220eff9b19b76
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3956937
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Cody Northrop <cnorthrop@google.com>
|
|
|
b968b6c9
|
2022-07-26T14:34:41
|
|
Test Utils: Get executable path from command line.
This simplifies the process fork test and will allow the test to
work on Fuchsia.
Bug: angleproject:3161
Change-Id: I5bb5424c44957f2faef1970b3cbc5e52b8aff630
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3788297
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Auto-Submit: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
2b43d630
|
2022-07-22T14:19:05
|
|
Remove unused (and incorrect on Android) mTestSuiteName.
On Android it ends up being just: _
Bug: angleproject:7299
Change-Id: I3dabcfcbacc3f43338b6fa91ebd9917e43f783ce
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3780587
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Yuxin Hu <yuxinhu@google.com>
|
|
|
81585fe2
|
2022-06-08T14:37:38
|
|
Use consistent test names with dEQP.
This removes the GTest/dEQP test name distinction.
Using the "RegisterTests" method in GTest we can use the
full dEQP test names. The one limitation is that "-" is
reserved in GoogleTest for exclusion filters, so we need
to do some extra work to sanitize our filters before we
pass them to GoogleTest.
Makes maintaining test expectations easier going forward.
Bug: angleproject:7390
Change-Id: I23893c425ddc934c11fe539b9173ec06c58abcc2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3695518
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
|
|
|
b9bdc53b
|
2022-06-07T14:50:46
|
|
Make TestEventListener and inner class of TestSuite.
Avoids having to pass dependencies to the class as it's automatically a
friend; external linkage sadly but such is life.
Bug: angleproject:7299
Change-Id: I5726dcab5a1a98b93c0713073cd02d6eb3a8e216
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3693826
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
4b911686
|
2022-06-06T22:25:37
|
|
Converts from sprintf() to snprintf().
sprintf() is deprecated in Xcode 14, so update to safer equivalents in
order to keep the compiler happy on iOS and macOS.
Bug: chromium:1331345
Change-Id: Id5348088bf69cbd360d9251e6323596cb710666d
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3690747
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
7a9856f6
|
2022-04-05T19:25:26
|
|
Report durations of all flaky retries in JSON test output
Bug: angleproject:7184
Change-Id: I19295daed7b13919a2b528a43ebe8d9cbac9d0a1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3572713
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
f422f21a
|
2021-11-09T18:42:15
|
|
Reland "Metal: Reintroduce GPU power preference selection code."
This reverts commit 67a8cf07a740c5ce3aafd2ad7fddd370451b3525.
Reason for revert: Landed Chromium-side dependency:
https://chromium-review.googlesource.com/c/chromium/src/+/3271170
Original change's description:
> Revert "Metal: Reintroduce GPU power preference selection code."
>
> This reverts commit 017161701b7dbf70a13f3c180a39e1fa45c27d9f.
>
> Reason for revert: Blocking roller, please re-land with the fix.
>
> Original change's description:
> > Metal: Reintroduce GPU power preference selection code.
> >
> > This CL re-introduces the GPU power preference code to
> > the metal backend. It also reworks EGLDisplay caching
> > in the frontend to cache based on the native display
> > as well as the power preference attribute.
> > A new extension, EGL_ANGLE_display_power_preference is
> > added based on EGL_ANGLE_power_preference. This extension
> > is a client extension that allows selection of GPU on
> > display creation, similar to how GPUs are selected on
> > context creation in EGL_ANGLE_power_preference.
> > This CL adds EGLDisplayPowerPreferenceTest and enables it on
> > the metal backend.
> >
> > Bug: angleproject:6143
> > Change-Id: I0a081dcd2e3f18ab365fdd3498ddcb6e2ba35212
> > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3231986
> > Reviewed-by: Kenneth Russell <kbr@chromium.org>
> > Reviewed-by: Jamie Madill <jmadill@chromium.org>
> > Reviewed-by: Gregg Tavares <gman@chromium.org>
> > Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
>
> TBR=kbr@chromium.org,gman@chromium.org,jonahr@google.com,jmadill@chromium.org,angle-scoped@luci-project-accounts.iam.gserviceaccount.com
>
> Change-Id: I4f775bf7139253a87b033a30e0da2100b3c1bb02
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug: angleproject:6143
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270749
> Reviewed-by: Jamie Madill <jmadill@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
# Not skipping CQ checks because this is a reland.
Bug: angleproject:6143
Change-Id: Id9b0a5cbb76e4dea9e2f2da2b1c47a0587dfdaf5
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270970
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
67a8cf07
|
2021-11-09T15:10:50
|
|
Revert "Metal: Reintroduce GPU power preference selection code."
This reverts commit 017161701b7dbf70a13f3c180a39e1fa45c27d9f.
Reason for revert: Blocking roller, please re-land with the fix.
Original change's description:
> Metal: Reintroduce GPU power preference selection code.
>
> This CL re-introduces the GPU power preference code to
> the metal backend. It also reworks EGLDisplay caching
> in the frontend to cache based on the native display
> as well as the power preference attribute.
> A new extension, EGL_ANGLE_display_power_preference is
> added based on EGL_ANGLE_power_preference. This extension
> is a client extension that allows selection of GPU on
> display creation, similar to how GPUs are selected on
> context creation in EGL_ANGLE_power_preference.
> This CL adds EGLDisplayPowerPreferenceTest and enables it on
> the metal backend.
>
> Bug: angleproject:6143
> Change-Id: I0a081dcd2e3f18ab365fdd3498ddcb6e2ba35212
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3231986
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Reviewed-by: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Gregg Tavares <gman@chromium.org>
> Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
TBR=kbr@chromium.org,gman@chromium.org,jonahr@google.com,jmadill@chromium.org,angle-scoped@luci-project-accounts.iam.gserviceaccount.com
Change-Id: I4f775bf7139253a87b033a30e0da2100b3c1bb02
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:6143
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3270749
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
01716170
|
2021-10-14T13:24:41
|
|
Metal: Reintroduce GPU power preference selection code.
This CL re-introduces the GPU power preference code to
the metal backend. It also reworks EGLDisplay caching
in the frontend to cache based on the native display
as well as the power preference attribute.
A new extension, EGL_ANGLE_display_power_preference is
added based on EGL_ANGLE_power_preference. This extension
is a client extension that allows selection of GPU on
display creation, similar to how GPUs are selected on
context creation in EGL_ANGLE_power_preference.
This CL adds EGLDisplayPowerPreferenceTest and enables it on
the metal backend.
Bug: angleproject:6143
Change-Id: I0a081dcd2e3f18ab365fdd3498ddcb6e2ba35212
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3231986
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Gregg Tavares <gman@chromium.org>
Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
0004ecc2
|
2021-09-22T13:39:55
|
|
Write GL info to test artifact in end2end_tests.
These will show up as glinfo.json in the "cas outputs" of the shard
that runs these tests.
We currently have information for Pixel 4, NVIDIA and Intel Win/Linux,
and SwiftShader Windows. GLES 1.x extensions are treated separately.
Some WebGL/Mac-only/D3D-only extensions show as unsupported anywhere.
In order to capture artifacts on Android, we use a render test output
parameter.
Bug: angleproject:6379
Change-Id: I560b219b5a586cdbb68d272090f38eedc76aba20
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3176780
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Ian Elliott <ianelliott@google.com>
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
f65d6acc
|
2021-08-18T12:20:40
|
|
Test Runner: Determine slow tests from test expectations.
This updates the test harness to handle slow test based on the TIMEOUT
test expectation. It removes the "registerSlowTests" API in favor of
using the test expectation files.
Also updates some of the timeout handling for various angle tests,
including on SwiftShader for multithreading tests.
Increases the timeout for tests in a batch to attempt to fix the
problem of the first test being much slower.
Bug: angleproject:6261
Change-Id: I7427344da5a0c1ea26d8c2d6eb0e9be2557b56e6
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/3104007
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
4194d400
|
2021-06-21T13:08:56
|
|
Test Runner: Add maximum failure count.
This will make the test runner early exit after a certain count
of failures. This will prevent the bots from running tens of
thousands of tests when the whole test run is corrupt.
Bug: angleproject:6075
Change-Id: I8f5c19ea694a5ad3a5d5df1429d2ba4b1fe868e3
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2976179
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
c51c59c7
|
2021-06-15T13:43:50
|
|
Test for missing index dirty bit bug
Bug fixed in
https://chromium-review.googlesource.com/c/angle/angle/+/2961690
triggers only in the following situation:
- Context 1: draw indexed -> clears index dirty bit
- Context 1: change state in such a way that closing the render pass is
deferred to dirty bit handling (for example, change FBO)
- Context 1: issue a non-indexed draw call. This closes the render pass
and starts a new one -> bug was that the index dirty bit
was not set
- Context 2: flush the command buffer, which submits the previous render
pass of context 1 (which contained vkCmdBindIndexBuffer).
The primary command buffer is now reset.
- Context 1: issue an indexed draw call. Since the index dirty bit was
not set, this was missing the vkCmdBindIndexBuffer call.
This change implements a regression test based on the above scenario.
Bug: chromium:1183068
Bug: chromium:1190493
Change-Id: I729bd48cd6df2621ca763f6231023a52ac08b0fb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2963836
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Charlie Lao <cclao@google.com>
Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
|
dd686e48
|
2021-05-11T19:08:01
|
|
Test Runner: Add test expectations parser.
Moves the test expectations from dEQP into the test runner.
Also updates angle_end2end_tests to take an expectations file.
Includes some very simple angle_end2end_tests expectations.
Note that the expectations in the file are less expressive than the
skips we use in the cpp.
Bug: angleproject:5951
Change-Id: Ib92235575bc3ea5f3a977ce416b0e78fe806e39b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2892274
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
9a219a14
|
2021-05-12T09:52:16
|
|
Test Runner: Handle skipped gtests.
GTEST_SKIP() allows us to skip tests using the GoogleTest APIs.
This CL updates our test runner code to handle these skips.
Bug: angleproject:5951
Change-Id: Idef5802c7c81bd92226f5a93281a1cea0f1b24ee
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2892272
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e5c5efc0
|
2021-02-05T02:39:55
|
|
Revert "Test Runner: Pick single/multi-process mode automatically."
This reverts commit 573d7f34a462fa5b2ca767dd84773ae6fa460d8a.
Reason for revert: Breaking ANGLE roll.
No-try: true
Original change's description:
> Test Runner: Pick single/multi-process mode automatically.
>
> This enables multi-process when running with more than one test
> config. This saves developers from remembering to add the arg.
> It can also simplify our test configurations by allowing us to omit
> the default arguments. On Android by default we stay with single
> process. It also changes the --bot-mode flag into --single-process
> and --multi-process which override the default behaviour.
>
> Bug: angleproject:5585
> Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=kbr@chromium.org,syoussefi@chromium.org,jonahr@google.com,jmadill@chromium.org,jonahr@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: angleproject:5585
Change-Id: I11ee1034c301aa246c4b03e29ff9b56235a1005c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2676247
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
573d7f34
|
2021-02-01T10:36:56
|
|
Test Runner: Pick single/multi-process mode automatically.
This enables multi-process when running with more than one test
config. This saves developers from remembering to add the arg.
It can also simplify our test configurations by allowing us to omit
the default arguments. On Android by default we stay with single
process. It also changes the --bot-mode flag into --single-process
and --multi-process which override the default behaviour.
Bug: angleproject:5585
Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
85efb9d5
|
2021-01-26T14:39:19
|
|
Log dEQP QPA files as test artifacts.
This adds artifact output to the test runner. We add a fake test
at the start of a test run that owns the artifacts.
Bug: angleproject:5236
Change-Id: Ice8001bf1f2aafbd8123fee76e0e7fcc3e5a8a0c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2657535
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Mohan Maiya <m.maiya@samsung.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
e4b4ff7a
|
2021-01-12T13:48:20
|
|
Test Runner: Add a slow tests list.
Each slow test in the list will use a 3x longer timeout. The list is
implemented using the same filter wildcard we use in the test
expectations logic.
We can test this out using a slow D3D11 varying test.
Bug: angleproject:5076
Bug: angleproject:5496
Change-Id: I31cf45e6ee8a8bbd6e460d675ff8a0cf5f19a504
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2625172
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
3e5b6f81
|
2020-10-31T12:33:28
|
|
Enable writing histogram-set-json-format.
This uses the protobuf histogram functionality to build up a histogram
set. The test suites then output the histograms to JSON. This is only
implemented for angle_perftests.
Bug: angleproject:5161
Change-Id: Ia5a7868e8d8dcf4f13d83115ae622828c63ef0d9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2482295
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
c06a424f
|
2020-11-02T21:13:39
|
|
Test Runner: Robustly handle unexpected crashes.
This adds the ability for the test runner to parse unexpected crashes
from the stdout. It also processes the stdout to determine which tests
failed. Tests that run after the crash are re-tried in a follow-up
child process.
Will allow for the test runner to handle very crashy test suites and
also processes crashes from win-asan and other configs where it is
harder to intercept crashes.
Bug: angleproject:5251
Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
59fbb989
|
2020-11-02T13:27:52
|
|
Test Runner: Add ability to retry flaky tests.
Bug: angleproject:5273
Change-Id: Ie89559bb0897a04213981aa8fe4e2f2bfe78959a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513287
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
5c4c37dc
|
2020-10-25T15:55:45
|
|
Capture/Replay: Test improvements.
Streamlines some arguments.
Use a common subdirectory in out/.
Show process name when logging.
Clean up environment variable handling.
Bug: angleproject:5247
Change-Id: I85eb5b8194a64f90691c4d03db3ee65c5e9b2313
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497560
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
31868665
|
2020-10-23T10:54:02
|
|
Test Runner: Only run watchdog in child processes.
For single test execution we don't need to be running the watchdog
thread. This only serves to cause timeouts in slow running tests.
Adds a new method to keep the --batch-id argument in the list of
parameters so other functions can process them.
Bug: angleproject:5124
Bug: angleproject:5218
Change-Id: I958230149b7d79830802cd55f67f0190de607200
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2495282
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
26cd1cc6
|
2020-10-18T11:44:27
|
|
Perf Tests: Add stdout printing arg to bot mode.
This can be useful for debugging and also for perf tests. The perf
tests output gets scraped from stdout.
Bug: angleproject:5124
Change-Id: Id5005df3f2b66be9497a9ba422abd47b95a1a330
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2486097
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
b4fb7cc9
|
2020-10-15T08:58:06
|
|
Test Runner: Print reproduction steps on a batch failure.
This prints out a large gtest_filter of the tests in a batch. Makes
reproducing a particular failure much easier.
In the future we could potentially look at using an encoding to
reproduce a specific batch: e.g. a flag that acceps a batch / shard
number and runs those tests based on using the same internal counts.
Bug: angleproject:5172
Change-Id: Iefad8f5ca23761417ad1ceb11fd664c9cf23c8be
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2475455
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
ad80b184
|
2020-10-13T17:15:59
|
|
Test Runner: List tests explicitly without GTest.
This dramatically speeds up Android shards that use batches of
hudreds of tests. Because ANGLE already has the test list, it no
longer needs to call GTest with an enourmous gtest_filter.
Bug: angleproject:5164
Change-Id: I28fee2f36c50006f2a35a4dcd90f44f8ebe4f78c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2468464
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
336202d7
|
2020-10-09T16:43:32
|
|
Test Runner: Pass batch ID to child processes.
This will let both the ANGLETest and dEQP test runners handle child
processes correctly. The existing ANGLETest runner will reuse displays,
and the dEQP test runner will write to a batch-unique filename.
This solves the problem of the multi-process --bot-mode writing to the
same file from multiple children simultaneously.
Long-term it will be good to include the dEQP QPA files into the test
artifacts.
Also disables flushing after every log write when running as a child
process. This hugely improves performance on machines with no SSD.
Test: https://chromium-swarm.appspot.com/task?id=4f2b87c4f8234910
Bug: angleproject:5157
Change-Id: I226d24adf55e0f8b98e5a8e7947862e6906b4c40
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464424
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Ian Elliott <ianelliott@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d3aa1e9c
|
2020-10-03T01:31:33
|
|
Test Runner: Pass dEQP args to child processes.
This fixes the sharding to use the specified back-end. With
the bug ANGLE would start the platform default back-end instead.
Bug: angleproject:3162
Change-Id: Ib2453dd3c58ea40fb36619301865ae0818038d15
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447043
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
acbe145c
|
2020-07-02T14:59:55
|
|
Test Runner: Batch tests by config.
This CL sorts tests into subprocesses organized by a single config.
It will allow for the test runner to use a single EGL display per
process in most cases. It doesn't implement the display sharing right
now, that'll be in a forthcoming CL. It assumes test configs are split
between / and (optional ending) __ characters.
Bug: angleproject:3162
Change-Id: I28e8b4687640da6e501f08b8b9347185c99ce30f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2280403
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fe902f4b
|
2020-06-19T14:28:36
|
|
Test Runner: Stub histogram writer.
The Android tests require a valid output for the perf test
results. The smallest valid output is an empty array.
Bug: angleproject:3162
Change-Id: I1bf7602b02358d634347d587b406c0c2d7c838a2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2254088
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|