src/tests/capture_replay_tests.py


Log

Author Commit Date CI Message
Gert Wollny 1af7caa4 2021-05-14T11:46:54 Capture/Replay: Add suffix to label for test file search When tests are run with a wildcard filter on the backend, e.g. "TEST/ES2_Vulkan*" the same tests from different backends may end up in the same trace file directory, e.g. TEST_ES2_Vulkan and TEST_ES2_Vulkan_SwiftShader. So far, when searching for the ES2_Vulkan trace files the check for trace files would them also pick up all the _SwiftShader files, and the skript would assume that the file count is wrong and skip the replay. By adding "_capture" the start of the commonly used suffix, this is avoided and replays are no longer skipped because mis-counting capture files. Bug: angleproject:5972 Change-Id: I7af54aaf51ee7070333ffe81974d866bd89dbd6f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2897236 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Gert Wollny 022401f9 2021-05-12T09:21:04 Capture/Replay: Print context diff also with frame gaps The capture/replay context differences may not be in each frame, so if the test signals that comparisons fail, don't bail if the context JSON files are not found for one frame, but continue the search until all possibly captured frames have been checked Bug: angleproject:5953 Change-Id: I8af34cc0fe4de5615fc43ef6c45915055546f507 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2891772 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Gert Wollny 31cd0e98 2021-05-10T10:37:54 Capture/Replay: track robustResourceInit Bug: angleproject:5938 Change-Id: I7c9ab9bebd1ec2dc572a071844a860f99d0befe6 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2882580 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Cody Northrop <cnorthrop@google.com>
Gert Wollny 39cef942 2021-04-27T08:28:13 Capture/Replay: capture WebGL compatibility With that we can also enable the tests DifferentStencilMasksTest.* Bug: angleproject:5851 Change-Id: If2c992e456716f606320e3992f389823d0af050f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2853584 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Gert Wollny 078a2ba6 2021-04-19T11:52:38 Capture/Replay: Print capture and replay exe in verbose mode When running the test script to capture and replay it is helpful to know the locations of the executables that are run to make it simpler to locate them for debugging. Bug: None Change-Id: Ia43bd9863e94058515d9023981863ad6c711285e Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2850640 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Gert Wollny 3e1af85a 2021-04-19T18:12:30 Capture/Replay: Handle bindGeneratesResource in trace meta data Bug: angleproject:5844 Change-Id: I0243cab70b5833efd83b0d75a1801f68025ee291 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2835889 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Gert Wollny c1da2794 2021-04-19T11:49:41 Capture/Replay: Handle ClientArraysEnabled in trace meta data Bug: angleproject:5848 Change-Id: Ieba325e2b5efa3242b24cafa8363e562c258546c Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2835888 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Gert Wollny 41c66442 2021-04-12T16:46:47 Capture/Replay: Print context diff on failure when verbose When run with the --verbose flag, and a test fails because the context comparison fails, then print the unified diff between the JSON serialization of context that was captured and the context when it was replayed. Bug: angleproject:5715 Change-Id: I86210d96e5bbdf63f1ed663ddf8dfcb33764cbea Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2821535 Commit-Queue: Gert Wollny <gert.wollny@collabora.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 78dde332 2021-03-29T17:31:52 Move restricted traces to CIPD. All traces are now stored as DEPS entries in CIPD. The auto-generation script generates the DEPS entries. Note that we don't include DEPS in the list of generated outputs to simplify other rollers. Also we update auto-generation to include full sources list to allow 'gn analyze' to work successfully. Usees a trace fixture for common code. This will enable a more compact trace without as much repeated code. We must land a set of re-trace traces to avoid breakage. Also includes a python script for uploading new traces to CIPD. The script first checks if traces are already present in the cloud, and if so it skips the upload. It will take a while to complete as the number of traces grows larger as it takes a few seconds per trace. The traces in this patch are also re-traced to use the common fixture code instead of including duplicated code in each trace. They now form a simple common interface and the autogenerated cpp is now simply a list of properties. I've also updated the capture/replay tests to use the simpler common entry point integration. There is less auto-generated glue code now. We now use a new serialized string query extension instead of calling directly into ANGLE's internals. Also includes a docs update. The capture/replay sample is broken and we'll need to update it in a follow-up CL. Also includes a few necessary fixes to the retracing script. Bug: angleproject:5811 Change-Id: I977bc6dc56843c8966377fc445ae97e91e17319a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2797833 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com>
Gert Wollny cee0bfd6 2021-03-15T14:39:30 Capture/Replay: serialize context to JSON Bug: angleproject:5715 Change-Id: Ibc6897d7567fedab4a2ba7bf8d2bd5a7e496bd4b Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2760325 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 9a025fd4 2021-03-13T22:48:13 Capture/Replay: add frontend feature to force capture limits Some of the limits set when FrameCapture is enabled are reflected in the serialized context. In order for capture/replay tests to pass these limits may also be needed to be enforced when replaying. Add a fronten feature enable_capture_limits for this and use it in the capture replay test script. Patch authored by gert.wollny@collabora.com. Bug: angleproject:5750 Change-Id: I3d333d22fe65c12dd5aa5f263d19a2c1568541c2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2790301 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill eac2397b 2021-03-25T13:36:50 Trace Tests: Work around autoninja.bat change on Win. A breaking change to autoninja.bat was causing the script to fail to find 'python3.bat' in this configuration. Work around the issue by instead invoking 'ninja.exe' directly with the correct '-j' argument. Bug: angleproject:5766 Change-Id: Ie01d7e8c4caf80661f420ccfc55a8d951a4e1b4f Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2787186 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Lubosz Sarnecki 36770173 2021-03-10T15:23:03 capture_replay_tests: Don't shadow subprocess import. Rename local variable, so using subprocess.Timeout does not result in an exception. This patch fixes the test timeout counter. Bug: angleproject:5742 Change-Id: Ib6991bc58df6ea653f765dfe9f6117add75b85db Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2748339 Commit-Queue: Lubosz Sarnecki <lubosz.sarnecki@collabora.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Lubosz Sarnecki 1330aa2c 2021-02-12T15:21:31 capture_replay_tests: Fix repository path retrieval. Use relation to retrieve angle repository path. The previous way of detecting the root repository folder failed to detect a correct path if the angle directory was in a path that contained the string angle. e.g. `/home/user/code/angle/angle` Other paths that would fail with the current implementation: `/home/user/my-angle-workspace/angle` `/home/tangled/workspace/angle/` `/home/user/code/angle-android` This way the path is constructed with the known relation of the script path inside the repository. Another advantage of this way is that the `angle` repository could be checked out with a different name. The disadvantage of this method is that whenever the script was moved, the relative path must be updated. Bug: angleproject:5645 Change-Id: I9a56aeba449b248273087382adceb0d6fba5a494 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2715323 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill f6925344 2021-01-25T13:13:33 Trace Tests: Update expectations. Bug: angleproject:5530 Change-Id: I48a6dc5753406e6dd6cd1845f1f28fcbfc485e34 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2647871 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 3623868e 2021-01-22T12:24:50 Trace Tests: Add xvfb command line option. This will enable running the tests on Linux. Bug: angleproject:5530 Change-Id: I7cd2155fac89d395a8bebacb2f1a44be0e7d3739 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2644723 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill e60ce3da 2021-01-21T16:42:17 Capture/Replay: Allow testing with multiple versions. This embeds the context version and device type info into the replay. The self-tests then can create the correct display and context types. This fixes testing against SwiftShader which is necessary for the bots. Bug: angleproject:4759 Change-Id: If9da6bfdc1c2b315ccd7e453872fc84063277054 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2643363 Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill b745f6df 2021-01-21T14:41:00 Capture/Replay tests: Fixes for goma & depot tools. Adds the depot tools path to be specified via the command line. This will allow the bots to specify working directory locations. Also adds quotes around the goma path in the gn args. Bug: angleproject:5530 Change-Id: I1291a8cdf4e36e55786403b4d9a6ce8565093352 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2642894 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 0ae61c25 2021-01-20T10:40:24 Capture/Replay tests: Use failure exit code. This will allow the test step to fail when run in CI. Bug: angleproject:5530 Change-Id: I7c156438ded82aa7191b8b597a9a52200c0efe57 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2638989 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Jamie Madill 738cb55b 2021-01-14T17:33:20 Capture/Replay tests: Fixes and improvements. The script was broken after the change to build to a separate shared library. Also fixes issues on Windows and cleans up the code. State comparison is still broken on every test from what I can see. Bug: angleproject:5530 Change-Id: I23e77e56fadedfb85d7afdb59dddac5558a71a56 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2630232 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Jamie Madill 39b777c6 2020-10-29T16:55:06 Capture/Replay: Two cleanups. This changes from returning a vector to directly returning a pointer to the binary data for the serialized state. The second cleanup is to use a ContextID as a wrapped type which simplifies the output formatting code. Bug: angleproject:5247 Change-Id: Ieb8afdb9326a12968dd2d69c05e1ed811b93abff Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2506198 Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 1c53df78 2020-10-26T13:42:14 Capture/Replay Tests: Expand suppressions. Updates the suppression list to include Linux tests and other new failures. Also sort the test list output to make it more readable. Bug: angleproject:5134 Change-Id: I38b5d95ceab3270d1cab13e3fb3c19d3137d10dc Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497568 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 5c4c37dc 2020-10-25T15:55:45 Capture/Replay: Test improvements. Streamlines some arguments. Use a common subdirectory in out/. Show process name when logging. Clean up environment variable handling. Bug: angleproject:5247 Change-Id: I85eb5b8194a64f90691c4d03db3ee65c5e9b2313 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2497560 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill 5ffd778e 2020-10-23T13:16:50 Make capture replay tests exectuable. This makes them runnable directly on Linux. Bug: angleproject:5134 Change-Id: Icf5ccf9d82b8cf09b16c5c60312e33752a9724e4 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2495288 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Jamie Madill be73728d 2020-10-03T17:52:25 Improve capture replay tests. Removes the custom logger class and uses the python logging class. Moves the argument descriptions into argparse help. Keep build directories around by default. Essential for normal use. Print more debug messages. Bug: angleproject:5133 Change-Id: Ida48828512abe60ee7d16873cbdb87c58529a60a Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2446898 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Courtney Goeltzenleuchter <courtneygo@google.com>
Manh Nguyen c65517d6 2020-08-12T17:43:53 Add tests' expectations to capture replay tests Skips failing tests so that they do not affect the results of passing tests. Bug: angleproject:4817 Change-Id: I80678cb62b0b81d6a1aa811908019022efbe1268 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2351394 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Manh Nguyen <nguyenmh@google.com>
Manh Nguyen ce5943c5 2020-08-05T23:24:17 Change test result types + add additional logging New result types: Passed, Failed, TimedOut, Crashed, CompileFailed, Skipped. They divide the results of tests more clearly. Results are now immediately logged to the stdout once a test batch completes. Flags, with the exception of --gtest_filter now use "-" instead of "_". Bug: angleproject:4817 Change-Id: I9616b23ef70026818e22126a251e1e2341de641d Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2340787 Commit-Queue: Manh Nguyen <nguyenmh@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Manh Nguyen 8956bfb9 2020-08-05T13:34:01 Use striping when making test batches When grouping tests to test batches, instead of grouping neighboring tests together, use striping so that slow tests are fanned out into multiple batches. This way, no test batch is too slow. Bug: angleproject:4817 Change-Id: I7825539e401ea53b5a31167f206d1f5bfb07bdbf Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2339540 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Manh Nguyen <nguyenmh@google.com>
Manh Nguyen 6aed2832 2020-07-31T10:52:21 Batch capture run + change how results are logged Before, the run stages of tests in a batch were run separately by multiple subprocesses. Now the run stages of tests in a batch are batched together and run by a single subprocess. Changes how results are logged. Tests in a batch are logged together. Within a batch, tests that fail at the same stage are also logged together. Bug: angleproject:4817 Change-Id: Ie3f992c081de914f1f1f521bec2d72f06ccca238 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2331738 Commit-Queue: Manh Nguyen <nguyenmh@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com>
Manh Nguyen 2c41931c 2020-07-21T14:21:27 Batch-compile and batch-run-replay multiple tests Multiple tests are batch-compiled into 1 replay application instead of multiple replay applications. Replay application now runs generated code of multiple tests instead of 1 test. This reduces overhead cost and brings down runtime. Main process now receives messages sent by workers via a message queue and prints them to the main stdout so that user can know if workers are hanging. Add handle for user interrupt (Ctrl-C) so that processes are properly destroyed and cleaned up. Trace files now have the option not to be deleted. Bug: angleproject:4817 Change-Id: Ic90ae0f430e1d3c261ffea5f963be5a4e94b0ad2 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2310909 Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Manh Nguyen <nguyenmh@google.com>
Manh Nguyen 3b0c9918 2020-07-21T11:52:13 Update CaptureAndReplay.md and Python script with links Links to the Python script in CaptureAndReplay.md Links to CaptureAndReplay.md's testing section in the Python script Bug: angleproject:4817 Change-Id: I584f8508661950e4907dae1adb8f0fe300387198 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2309226 Commit-Queue: Manh Nguyen <nguyenmh@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Manh Nguyen 10e9e9a1 2020-07-13T17:02:38 Apply multiprocessing to capture_replay_tests.py Multiple tests are run in paralell to squeeze the most out of CPUs. Tests are stored in a work queue. Whenever a CPU becomes available, the next unrun test is grabbed and run on a spawn process on that CPU. Each cpu gets their own environment and build folders and trace folders so that tests don't overwrite each other. Bug: angleproject:4817 Change-Id: Ifd35c9c75522e480b0257d090d5af70f2a3428ba Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2296040 Commit-Queue: Manh Nguyen <nguyenmh@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Manh Nguyen e5c18475 2020-07-06T09:50:49 Multi-frames serialization + comparision for capture replay test Changes how context states are serialized during capture run. Now, context states are serialized after each frame until the end frame is reached, or context is destroyed. Adds an api to the trace files to query serialization results of multiple frames. Makes change to CaptureReplayTest to serialize multiple frames then compares the serialization results with the ones retrieved from said api. Adds to capture replay the ability to produce working, compilable trace files even when end frame is not reached. Adds to the generated trace files config information and makes CaptureReplayTest utilize said config information to reproduce the exact environment of the captured application. Bug: angleproject:4817 Change-Id: Ie3d487af2bacf349dc3ff6f6b1b5f89e1169dc84 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2282885 Commit-Queue: Manh Nguyen <nguyenmh@google.com> Reviewed-by: Cody Northrop <cnorthrop@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Manh Nguyen dce37b7d 2020-06-21T22:27:06 Serialize framebuffers + compare contexts for CaptureReplayTests Adds to frame capture the ability to serialize a frame's pre-swap GL state and store it in the binary data file Adds to CaptureReplayTests the ability to compare its serialized GL state with the serialized state pulled from the binary data file Adds a serialization module that serializes framebuffers' GL states and the contents of their color attachments Adds checks to automation script so that it would skips tests that do not produce the expected trace files Adds exception handling to automation script so that it will not crash when a replay build crashes Bug: angleproject:4779 Change-Id: I40a02e018073749e79f0ddbfd3d4065745548f46 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2258295 Commit-Queue: Manh Nguyen <nguyenmh@google.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Tim Van Patten <timvp@google.com>
Manh Nguyen 9dee8e3f 2020-06-15T10:59:18 Automate testing of capture_replay automation script will: 1. compile all tests in angle_end2end with framecapture enabled 2. run each test with frame capture 3. compile CaptureReplayTest with cpp trace files 4. Run CaptureReplayTest Test: run_tests_with_framecapture.py Bug: angleproject:4749 Change-Id: Ieb21c6137653632d73beedea48712e46fc15a0f3 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2246321 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Cody Northrop <cnorthrop@google.com> Commit-Queue: Manh Nguyen <nguyenmh@google.com>