|
e5c5efc0
|
2021-02-05T02:39:55
|
|
Revert "Test Runner: Pick single/multi-process mode automatically."
This reverts commit 573d7f34a462fa5b2ca767dd84773ae6fa460d8a.
Reason for revert: Breaking ANGLE roll.
No-try: true
Original change's description:
> Test Runner: Pick single/multi-process mode automatically.
>
> This enables multi-process when running with more than one test
> config. This saves developers from remembering to add the arg.
> It can also simplify our test configurations by allowing us to omit
> the default arguments. On Android by default we stay with single
> process. It also changes the --bot-mode flag into --single-process
> and --multi-process which override the default behaviour.
>
> Bug: angleproject:5585
> Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=kbr@chromium.org,syoussefi@chromium.org,jonahr@google.com,jmadill@chromium.org,jonahr@chromium.org
# Not skipping CQ checks because original CL landed > 1 day ago.
Bug: angleproject:5585
Change-Id: I11ee1034c301aa246c4b03e29ff9b56235a1005c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2676247
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
573d7f34
|
2021-02-01T10:36:56
|
|
Test Runner: Pick single/multi-process mode automatically.
This enables multi-process when running with more than one test
config. This saves developers from remembering to add the arg.
It can also simplify our test configurations by allowing us to omit
the default arguments. On Android by default we stay with single
process. It also changes the --bot-mode flag into --single-process
and --multi-process which override the default behaviour.
Bug: angleproject:5585
Change-Id: I3dd885a8a8e6604c224693c546ff5d15224e016e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2664246
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
85efb9d5
|
2021-01-26T14:39:19
|
|
Log dEQP QPA files as test artifacts.
This adds artifact output to the test runner. We add a fake test
at the start of a test run that owns the artifacts.
Bug: angleproject:5236
Change-Id: Ice8001bf1f2aafbd8123fee76e0e7fcc3e5a8a0c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2657535
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Mohan Maiya <m.maiya@samsung.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
feb470cc
|
2021-01-19T10:10:18
|
|
Test Runner: Prevent race by not resetting timeout.
This instead only sets the timeout when tests start to run. That
way the timeout should always be at least the default timeout
and we won't get into a situation where the watchdog times out
the test immediately when the test starts.
Bug: angleproject:5562
Change-Id: I6b12bb8fe8edcf35f46ba4fb106fdf80ff9402a1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2637182
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e4b4ff7a
|
2021-01-12T13:48:20
|
|
Test Runner: Add a slow tests list.
Each slow test in the list will use a 3x longer timeout. The list is
implemented using the same filter wildcard we use in the test
expectations logic.
We can test this out using a slow D3D11 varying test.
Bug: angleproject:5076
Bug: angleproject:5496
Change-Id: I31cf45e6ee8a8bbd6e460d675ff8a0cf5f19a504
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2625172
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
cb8903b1
|
2020-12-08T01:08:00
|
|
Metal: Ignore OS's internal shader cache when testing.
Internal shader cache caused timeout in some dEQP tests.
Work-around: ignore the cache by hooking fopen function and return null
when the cache related files are accessed.
Bug: angleproject:5354
Change-Id: I12ca228540925e67454bf24ce1ba83d703882c87
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2580918
Commit-Queue: Le Hoang Quyen <le.hoang.q@gmail.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
|
|
c859c0ac
|
2020-11-24T17:21:38
|
|
Batch replace std::unordered_map with angle::HashMap in src/
There are a few places that will remain std::unordered_map due to build
or run-time errors, which will need to be evaluated more closely to
determine if they should remain std::unordered_map or if there is
another Abseil data structure that would be more efficient while still
working correctly.
Bug: angleproject:4873
Change-Id: Ib04253e3ad6398e63f4cc2bfe12c0f9e57cb112b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2558873
Commit-Queue: Tim Van Patten <timvp@google.com>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Charlie Lao <cclao@google.com>
|
|
|
9fe2d211
|
2020-12-01T13:24:33
|
|
Test Runner: Handle flaky failures.
When a test mark as flaky fails consistently we need to avoid marking
the test as flaky. We treat is as a deterministic failure. This means
we output "PASS" as the expected result instead of "PASS FAIL". That
change will produce the correct behaviour with the test scripts.
Also adds missing test runner output when we retry a flaky test.
Bug: chromium:1152515
Change-Id: I9e3cbaf04ea57a198717ee81505b5d63a9183410
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2568231
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
3e5b6f81
|
2020-10-31T12:33:28
|
|
Enable writing histogram-set-json-format.
This uses the protobuf histogram functionality to build up a histogram
set. The test suites then output the histograms to JSON. This is only
implemented for angle_perftests.
Bug: angleproject:5161
Change-Id: Ia5a7868e8d8dcf4f13d83115ae622828c63ef0d9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2482295
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
c06a424f
|
2020-11-02T21:13:39
|
|
Test Runner: Robustly handle unexpected crashes.
This adds the ability for the test runner to parse unexpected crashes
from the stdout. It also processes the stdout to determine which tests
failed. Tests that run after the crash are re-tried in a follow-up
child process.
Will allow for the test runner to handle very crashy test suites and
also processes crashes from win-asan and other configs where it is
harder to intercept crashes.
Bug: angleproject:5251
Change-Id: Iee03130622571580cb7910f4fb097fe3659d75ec
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513288
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
59fbb989
|
2020-11-02T13:27:52
|
|
Test Runner: Add ability to retry flaky tests.
Bug: angleproject:5273
Change-Id: Ie89559bb0897a04213981aa8fe4e2f2bfe78959a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2513287
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Tim Van Patten <timvp@google.com>
|
|
|
5c4c37dc
|
2020-10-25T15:55:45
|
|
Capture/Replay: Test improvements.
Streamlines some arguments.
Use a common subdirectory in out/.
Show process name when logging.
Clean up environment variable handling.
Bug: angleproject:5247
Change-Id: I85eb5b8194a64f90691c4d03db3ee65c5e9b2313
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497560
Reviewed-by: Cody Northrop <cnorthrop@google.com>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
31868665
|
2020-10-23T10:54:02
|
|
Test Runner: Only run watchdog in child processes.
For single test execution we don't need to be running the watchdog
thread. This only serves to cause timeouts in slow running tests.
Adds a new method to keep the --batch-id argument in the list of
parameters so other functions can process them.
Bug: angleproject:5124
Bug: angleproject:5218
Change-Id: I958230149b7d79830802cd55f67f0190de607200
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2495282
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
cd88bccf
|
2020-10-21T08:44:32
|
|
Test Runner: Command line sharding args override env.
Previously ANGLE would error out when both the enviornment and
the command line args are specified at the same time. Because
the perf bots use both at once we need to handle the conflict
in the same way as the prior test runner does. In this case
the command line takes precedence over the environment.
Bug: angleproject:5124
Change-Id: I1ba765b4e75759922bf9fe2db9f153cfc5995f85
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2489722
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
26cd1cc6
|
2020-10-18T11:44:27
|
|
Perf Tests: Add stdout printing arg to bot mode.
This can be useful for debugging and also for perf tests. The perf
tests output gets scraped from stdout.
Bug: angleproject:5124
Change-Id: Id5005df3f2b66be9497a9ba422abd47b95a1a330
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2486097
Reviewed-by: Tim Van Patten <timvp@google.com>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
556d812a
|
2020-10-18T11:48:45
|
|
Test Runner: Disable watchdog when debugger attached.
Bug: angleproject:3162
Change-Id: I3685a4fe068b6e82b8bb04448c212788f563ab8b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2483463
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
b4fb7cc9
|
2020-10-15T08:58:06
|
|
Test Runner: Print reproduction steps on a batch failure.
This prints out a large gtest_filter of the tests in a batch. Makes
reproducing a particular failure much easier.
In the future we could potentially look at using an encoding to
reproduce a specific batch: e.g. a flag that acceps a batch / shard
number and runs those tests based on using the same internal counts.
Bug: angleproject:5172
Change-Id: Iefad8f5ca23761417ad1ceb11fd664c9cf23c8be
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2475455
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
9580f3b4
|
2020-10-10T10:06:04
|
|
Test Runner: Accept GTEST shard environment vars.
This makes interoperation with Chromium tests easier. For instance when
triggering tests with scripts when using sharding. And also using the
run/ wrapper scripts instead of run_gtest_angle_test.
Bug: angleproject:5158
Change-Id: Ia5b1f94c02a081caf576615175791fff4e0d3fb3
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464030
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
ad80b184
|
2020-10-13T17:15:59
|
|
Test Runner: List tests explicitly without GTest.
This dramatically speeds up Android shards that use batches of
hudreds of tests. Because ANGLE already has the test list, it no
longer needs to call GTest with an enourmous gtest_filter.
Bug: angleproject:5164
Change-Id: I28fee2f36c50006f2a35a4dcd90f44f8ebe4f78c
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2468464
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
336202d7
|
2020-10-09T16:43:32
|
|
Test Runner: Pass batch ID to child processes.
This will let both the ANGLETest and dEQP test runners handle child
processes correctly. The existing ANGLETest runner will reuse displays,
and the dEQP test runner will write to a batch-unique filename.
This solves the problem of the multi-process --bot-mode writing to the
same file from multiple children simultaneously.
Long-term it will be good to include the dEQP QPA files into the test
artifacts.
Also disables flushing after every log write when running as a child
process. This hugely improves performance on machines with no SSD.
Test: https://chromium-swarm.appspot.com/task?id=4f2b87c4f8234910
Bug: angleproject:5157
Change-Id: I226d24adf55e0f8b98e5a8e7947862e6906b4c40
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2464424
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Ian Elliott <ianelliott@google.com>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d3aa1e9c
|
2020-10-03T01:31:33
|
|
Test Runner: Pass dEQP args to child processes.
This fixes the sharding to use the specified back-end. With
the bug ANGLE would start the platform default back-end instead.
Bug: angleproject:3162
Change-Id: Ib2453dd3c58ea40fb36619301865ae0818038d15
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447043
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
f0d9471c
|
2020-10-02T23:51:57
|
|
Test Runner: More misc improvements.
- reduces default batch size from 1000 -> 256
- reduces default max processes to a max of 16
- stripes tests when in bot mode
- prints less spam when waiting for child processes
The striping and smaller batch size can help with slow tests.
The max process limit seemed to prevent errors on Linux.
Bug: angleproject:3162
Change-Id: Ibed89ca35db54c62ec8ed63b32bc5aed916faec9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2447037
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
23825b12
|
2020-10-02T16:32:32
|
|
Test Runner: Set interrupted to 'false' normally.
The bug here was to output 'interrupted: true' when running
in --bot-mode. Instead in normal execution we will output
that we were not interrupted.
Bug: angleproject:3162
Bug: chromium:1134619
Change-Id: I3842e90466473ff05b0b90bc4660f88606a3c9f6
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2446851
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
1fd3e5d8
|
2020-10-01T10:00:18
|
|
Test Runner: Fix race in watchdog timeouts.
This fixes a TSAN warning that popped up with the standalone test
runner. The watchdog timer actually was never started so timeouts
were not working as intended. It also switches the timeout mode
to call _Exit which skips all the atexit handlers and avoids some
races on teardown. This change also speeds up the TestSuiteTest.
Also a small fix to GetTempDir that was including an extra path
separator on Windows.
Bug: angleproject:3162
Bug: angleproject:5117
Change-Id: I0e7880a08b61bbb6e30c65665d5c0acec2d78db2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2442381
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
867358b7
|
2020-09-21T23:40:57
|
|
Test Runner: Improve sharding validation.
The prior check was incorrectly flagging index=0/count=2 as
invalid. It also missed a few cases like a shard count of 0.
Bug: angleproject:3162
Change-Id: Ida4c0f454ece15e4304d024d205475acf0f135e6
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2423207
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
5d7de5fd
|
2020-09-20T13:09:07
|
|
Test Runner: Fix default config grouping.
Running "angle_unittests" with --bot-mode would produce noisy
test groupings for each permutation of parameterized tests. Fix
this by only splitting configs that begin with "ES". e.g. ES2_Vulkan,
ES3_OpenGL, etc.
Bug: angleproject:3162
Change-Id: I0b0ebe1c9cef9cbbce0489a2bc3f62761d43a1c7
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419046
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Geoff Lang <geofflang@chromium.org>
|
|
|
763e8b81
|
2020-09-20T21:59:23
|
|
Test Runner: Add more timeout logging.
Will help diagnose a timeout failure on Windows Intel.
Also reduce the significant digits in time output.
Bug: angleproject:3162
Change-Id: I6d6a852f572875f613dec8e4ab6e798dfde0ca9b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2420331
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
dd0a9ec9
|
2020-09-20T12:50:21
|
|
Test Runner: Allow empty sharding parameters.
The prior check was incorrectly validating a shard index of zero
and a shard count of one as inconsistent.
Bug: angleproject:3162
Change-Id: I63184c16e82f08080b275672a43e6dacf291560d
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419045
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Commit-Queue: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
76e90947
|
2020-09-19T10:48:55
|
|
Test Runner: Really reuse displays with --bot-mode.
The prior design didn't pass the correct flag to the subprocess.
Instead add a new --reuse-displays flag that forces display reuse
on in ANGLE tests. Pass the new flag to the child processes when
running with --bot-mode.
Bug: angleproject:3162
Change-Id: I5f62125d83d339b5e8b2506b4ca4656976f7398e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419638
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
3f20c713
|
2020-09-19T11:29:56
|
|
Test Runner: Fix "--bot-mode" error level return value.
Previously the runner was returning "1" on success and "0" on failure.
Swap this so that it correctly returns "0" on success and "1" on fail.
This only affected "--bot-mode" which isn't yet enabled.
Bug: angleproject:3162
Change-Id: I1ee942223272d24eff137279d5a0638d3a65694e
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419639
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
d6806d73
|
2020-09-18T17:56:48
|
|
Test Runner: Fix sharding and --bot-mode.
We were using the incorrect test list to set up initial test
expectations. Fix this by using the already generated test set
in the test suite.
Test: angle_unittests with --bot-mode, sharding, and filter
Bug: angleproject:3162
Change-Id: Id108943494130410caab404faad25115792e794d
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2419596
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
58463573
|
2020-09-16T19:30:56
|
|
Test Runner: Fix dirty thread teardown.
When the standalone tests shut down they would not wait for
the watchdog thread to exit cleanly. Fix this by setting a
flag that shuts down the watchdog and attempting to wait.
This was detected by running angle_end2end_tests with TSAN.
Bug: angleproject:3162
Bug: b/168744561
Change-Id: I765ce7a21c3be11703619470a81adb1882b009e4
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2415175
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
23cff4e1
|
2020-09-16T13:41:49
|
|
Test Runner: Handle GTest sharding args.
These were messing up the test runner when used in combination
with the test runner's specific args.
Make it an error if the arguments are used together.
Bug: angleproject:3162
Change-Id: I6fa1c0fb346a8c376ca455f7f4bf0492cfc0198a
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2414931
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
7fc325cc
|
2020-09-12T13:48:38
|
|
Test Runner: Print total test execution time.
Also has a few other tweaks to test spin-up.
Bug: angleproject:3162
Change-Id: I8753f1afedbcc5c4b5ae257965ce68dc4a02ae06
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2407836
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
2879eb55
|
2020-09-12T09:04:16
|
|
TestRunner: Fix missing "=" on swarming args.
These were missing from debe4683095450940883164ca6.
Also fixes a misnamed "perf" parameter.
Bug: angleproject:3162
Change-Id: Iad4c4d8d35cc8a08d0e30d50dd9bbf159a4a7ab9
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2407833
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
|
|
|
05b6740c
|
2020-08-09T12:23:42
|
|
TestRunner: Fix test result handling.
The bots expect us to explicitly label "is_unexpected" for all
unexpected results (e.g. failures).
Also they construct a test filter from our test names so our result
names have to match the tests exactly. That means removing prefixes
like we have for the test suite name.
Bug: angleproject:3162
Change-Id: I1a02ff8e26545b10a8829a3ee47be91f7459aa65
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2345028
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
debe4683
|
2020-07-15T14:36:33
|
|
Test Runner: Accept Chromium args.
This change lets the test runner accept the same isolated script
parameter syntax as Chromium. It keeps the old parameter names so
the tests accept both. This will let us more easily integrate with
the Chromium Android test_runner.py script.
Also update the README which was missing a few flags.
Bug: angleproject:3162
Change-Id: Ic22b289a095418ffdaa19a04caa1b45c6c657872
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2300530
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
acbe145c
|
2020-07-02T14:59:55
|
|
Test Runner: Batch tests by config.
This CL sorts tests into subprocesses organized by a single config.
It will allow for the test runner to use a single EGL display per
process in most cases. It doesn't implement the display sharing right
now, that'll be in a forthcoming CL. It assumes test configs are split
between / and (optional ending) __ characters.
Bug: angleproject:3162
Change-Id: I28e8b4687640da6e501f08b8b9347185c99ce30f
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2280403
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fe902f4b
|
2020-06-19T14:28:36
|
|
Test Runner: Stub histogram writer.
The Android tests require a valid output for the perf test
results. The smallest valid output is an empty array.
Bug: angleproject:3162
Change-Id: I1bf7602b02358d634347d587b406c0c2d7c838a2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2254088
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
e9a0cb87
|
2020-05-18T17:45:52
|
|
Fix assertion when running tests without suffix on Windows
On Windows, we may omit suffix ".exe" when running an executable.
However, this will trigger an assertion failure in current code and this
CL is to fix this issue.
Bug: angleproject:4640
Change-Id: I7edfdc0b4a7c590c874817530b19a812018b9288
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2206425
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
|
|
|
390be9bb
|
2020-03-01T08:47:37
|
|
Test Runner: Use system path for gtest-internal.
This ensures we use the correct internal path in Chromium builds as
well.
Bug: angleproject:3162
Change-Id: I036dcbd8802f67162e5f5eccb6158910f0a5efcc
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2082134
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
593706bc
|
2020-02-10T16:46:52
|
|
Test Runner: Increase default timeouts.
In particular the Debug timeout was not generous enough. Increase the
test timeout to 20 seconds in Release and 60 seconds in Debug. Increase
the batch timeout to 240 seconds in Release and 600 seconds in Debug.
Bug: angleproject:3162
Change-Id: I0bd0ea18cb5cdab005143b8deed9500d9d2e8ea1
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2047415
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
ee07cb31
|
2020-01-29T18:02:50
|
|
Test Harness: Allow filter with sharding params.
Some tweaks to when we call gtest initialization allows this to work.
Bug: angleproject:3162
Change-Id: I75159108d3f746f666e57c1a6495a8c8616ef1cf
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2024335
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
238adb7a
|
2020-01-13T17:21:53
|
|
Fix test harness running disabled tests.
Adds special handling for the DISABLED_ GTest test name. Fixes the test
for the test suite harness itself when run with the new "--bot-mode"
flag.
Bug: angleproject:3162
Change-Id: Idf34098d2d6bcb78263773be7b01c55df9ec624b
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1998660
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|
|
|
5407aaa0
|
2019-12-16T15:50:12
|
|
Re-land "Add new test runner harness." (#2)
Re-land #2 changes:
* export labels are fixed for the CFI build
* crash test disabled because of flakiness and issues with asan
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Bug: chromium:1030192
Change-Id: I71d66a407ea0e53d73cbe75b5b4bfb9e73791534
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1965091
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
|
|
|
b92ec244
|
2019-12-06T15:08:54
|
|
Revert "Re-land "Add new test runner harness.""
This reverts commit e20560faf1de86c01198143ef7733a12a098a90b.
Reason for revert: Now fails on Linux CFI and also is a bit flaky.
Original change's description:
> Re-land "Add new test runner harness."
>
> Re-land changes:
>
> * Unit test is suppressed in ASAN
> * --deqp-case is fixed
> * Debug layer errors should correctly work with failure expectations
>
> Original message:
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
> Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
TBR=ynovikov@chromium.org,ianelliott@google.com,jonahr@google.com,jmadill@chromium.org
Change-Id: Ibfd65b8b18ead3a232abb6cb75fd6489b0ff5f38
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1954570
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
e20560fa
|
2019-12-04T13:18:36
|
|
Re-land "Add new test runner harness."
Re-land changes:
* Unit test is suppressed in ASAN
* --deqp-case is fixed
* Debug layer errors should correctly work with failure expectations
Original message:
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: I841f2b5dfe51f7f44dac68324bdf6afd418b8bfb
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1948240
Reviewed-by: Jonah Ryan-Davis <jonahr@google.com>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
8f31872c
|
2019-12-03T18:34:52
|
|
Revert "Add new test runner harness."
This reverts commit fb40d231c3e2ee7c38f8445ef5defc0ab0f5f15d.
Reason for revert: Has a bug with the ASan build and also has a few
bugs with ANGLE standalone test expectations an filter.
Bug: chromium:1030192
Bug: angleproject:4193
Original change's description:
> Add new test runner harness.
>
> The ANGLE test harness is a harness around GoogleTest that provides
> functionality similar to the Chromium test harness. It supports:
>
> * splitting a test set into shards
> * catching and reporting crashes and timeouts
> * outputting to the Chromium JSON test results format
> * multi-process execution
>
> Unit tests are added in test_utils_unittest.cpp.
>
> Bug: angleproject:3162
> Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
TBR=ynovikov@chromium.org,jonahr@google.com,jmadill@chromium.org
Change-Id: I647e747571784b1ca7c1d0687193c70a63eb08d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: angleproject:3162
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1947456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
|
|
|
fb40d231
|
2019-12-02T16:39:18
|
|
Add new test runner harness.
The ANGLE test harness is a harness around GoogleTest that provides
functionality similar to the Chromium test harness. It supports:
* splitting a test set into shards
* catching and reporting crashes and timeouts
* outputting to the Chromium JSON test results format
* multi-process execution
Unit tests are added in test_utils_unittest.cpp.
Bug: angleproject:3162
Change-Id: Idb15f113de8eb32db12bc93542de93b08d7c1447
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1478016
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
|