capture_replay_expectations.txt

Show log
Commit

Hash : d6bf8e0d
Author :
Date : 2023-10-02T18:10:36

Improve capture_replay_tests (win-trace) logging

* Get rid of the CRASH expectation as it is always confusing. Replace
  items in expectations with SKIP_FOR_CAPTURE. This will no longer run
  the corresponding tests (which was making logs even more confusing)
* Use CaptureFailed and ReplayFailed instead of Crashed status (the
  actual meaning is a non-zero return code from the test harness)
* Introduce batch names by indexing them as they are created:
  batch_000, batch_001, ...
  This enables easy searching in logs.
* When logging a batch (result) include its name and why it is logged
* Failed batches are printed with full output instead of "abbreviated"
  output. One issue with b/303065176 was that an important part
  of the output (errors) was truncated by this "abbreviation"
* Remove "regression log" and "Summary" as these are often just
  confusing: regression log just duplicates information printed earlier
  and summary may not be clear w.r.t test expectations. The part that
  prints what was unexpected is still there. It also now includes
  batch names so it's much more obvious what comes from where. Example:

Unexpected 'CaptureFailed' (8):
     != Pass: batch_000 FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithBellowBaseLevelLOD/ES3_Vulkan_SwiftShader
     != Pass: batch_000 FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithBellowBaseLevelLOD/ES3_Vulkan_SwiftShader_AsyncCommandQueue
     != Pass: batch_000 FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithBellowBaseLevelLOD/ES3_Vulkan_SwiftShader_EnableParallelCompileAndLink
     != Pass: batch_000 FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithSubImageWithBeyondMaxLevel/ES3_Vulkan_SwiftShader
     != Pass: batch_000 FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithSubImageWithBeyondMaxLevel/ES3_Vulkan_SwiftShader_AsyncCommandQueue
     != Pass: batch_000 FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithSubImageWithBeyondMaxLevel/ES3_Vulkan_SwiftShader_EnableParallelCompileAndLink
     != Pass: batch_000 FramebufferTest_ES3.RenderImmutableTextureWithSubImageWithBeyondMaxLevel/ES3_Vulkan_SwiftShader
     != Pass: batch_000 FramebufferTest_ES3.RenderImmutableTextureWithSubImageWithBeyondMaxLevel/ES3_Vulkan_SwiftShader_AsyncCommandQueue

Unexpected 'FailedToTrace' (1):
     != Pass: batch_001 FramebufferTest_ES3.RenderSampleDepthTextureWithExcludedLevel/ES3_Vulkan_SwiftShader

Bug: angleproject:7519
Bug: b/303065176
Change-Id: Ie0c4734c7c4ac965aa6fd38b0c1783e0b92af7a5
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4908349
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Roman Lavrov <romanl@google.com>
Reviewed-by: Cody Northrop <cnorthrop@google.com>

Download

src/tests/capture_replay_tests/capture_replay_expectations.txt

#
# Copyright 2020 The ANGLE Project Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# Test are entered like
# BUG PLATFORMS : test-name = handling/result
#
#  PLATFORMS is optional and lists the platforms the expectation / run disabling happens
#
# Valid values for "handling/result" are
#
#   SKIP_FOR_CAPTURE : SKIP from capture_replay tests
#
#   NOT_RUN (DEPRECATED https://anglebug.com/7516): already skipped by end2end test suite
#
#   FAIL : the test is expected to fail
#
#   FLAKY : the tests is flaky, but we still run it
#
#   TIMEOUT : the test is expected to time out
#
#   COMPILE_FAIL : the compilation of the capture_replay exec is expected to fail
#
# For test batches that run more than one test TIMEOUT, COMPILE_FAIL, and CRASH
# will have to be set for all tests that run a batch, because one test that fails
# with one of these outcomes will bring down the whole batch.
#

# We don't support the Capture/Replay on OpenGL yet

6345 : *OpenGL = SKIP_FOR_CAPTURE

# ES2

6267 : FramebufferTest.MultiContextNoRenderingFeedbackLoops/* = SKIP_FOR_CAPTURE

6180 : GLSLTest_ES3.GLVertexIDIntegerTextureDrawElementsU8Line/* = NOT_RUN
6180 : GLSLTest_ES3.GLVertexIDIntegerTextureDrawElementsU8LineIds/* = NOT_RUN

6221 : VulkanExternalImageTest.ShouldClearOpaqueFdWithSemaphores/* = TIMEOUT
6221 : VulkanExternalImageTest.ShouldClearOpaqueFdWithSemaphoresMutableNoStorage/* = TIMEOUT
6221 : VulkanExternalImageTest.ShouldClearOpaqueFdWithSemaphoresNoStorage/* = TIMEOUT
6221 : VulkanExternalImageTest.ShouldClearOpaqueFdWithSemaphoresWithFlags/* = TIMEOUT
6221 : VulkanExternalImageTest.ShouldDrawOpaqueFdWithSemaphores/* = TIMEOUT
6221 : VulkanExternalImageTest.WaitSemaphoresRetainsContentOpaqueFd/* = TIMEOUT

6267 : WebGLCompatibilityTest.MultiContextNoRenderingFeedbackLoops/* = SKIP_FOR_CAPTURE

6180 WIN : WebGLCompatibilityTest.DrawElementsBufferOutOfBoundsInVertexBuffer/* = FLAKY

6654 : UniformTest.UnusedStructInlineUniform/* = SKIP_FOR_CAPTURE
6654 : UniformTest.UnusedStructInlineUniformWithSampler/* = SKIP_FOR_CAPTURE

# Fails because of unsupported combined depth/stencil GetImage.
4688 ASAN : DepthStencilTest.DepthOnlyEmulatedWithPacked/* = FAIL

# ES3
6180 : BlitFramebufferTest.BlitDepthStencilPixelByPixel/* = NOT_RUN
6180 : GLSLTest_ES3.UnsuccessfulRelinkWithBindAttribLocation/* = NOT_RUN
3886 : ComputeShaderTest.BindImageTexture3D/* = NOT_RUN
7441 : DrawBaseInstanceTest.DrawElementsInstancedBaseInstance/ES3_D3D11__DynamicDraw* = NOT_RUN

# This test only exists on Win64
6180 WIN : BufferDataOverflowTest.VertexBufferIntegerOverflow/* = NOT_RUN

6180 WIN : TransformFeedbackTest.DrawWithoutTransformFeedbackThenWith/* = FLAKY
6180 : TransformFeedbackTest.TransformFeedbackQueryPausedDrawThenResume/* = FLAKY
6513 : TransformFeedbackTest.MultiContext/* = FLAKY
6180 WIN : TransformFeedbackTest.BufferOutOfMemory/* = FLAKY
6180 WIN : TransformFeedbackTest.EndThenBindNewBufferAndRestart/* = FLAKY
6180 : VertexAttributeTestES3.DrawWithUnalignedData/* = FLAKY

# Skip because crashes
6313 : CopyTextureTestES3.InvalidateBlitThenBlend1000Layers/* = SKIP_FOR_CAPTURE
6313 : CopyTextureTestES3.InvalidateBlitThenBlend3Layers/* = SKIP_FOR_CAPTURE
6314 : FramebufferTest_ES3.AttachmentsWithUnequalDimensions/* = SKIP_FOR_CAPTURE
6314 : FramebufferTest_ES3.ChangeAttachmentThenInvalidateAndDraw/* = SKIP_FOR_CAPTURE
6314 : FramebufferTest_ES3.RenderAndInvalidateImmutableTextureWithBeyondMaxLevel/* = SKIP_FOR_CAPTURE
6399 : Texture2DTestES3.TexStorage2DCycleThroughYuvSourcesNoData/* = SKIP_FOR_CAPTURE
6399 : Texture2DTestES3.TexStorage2DSimultaneousUseOfMultipleYuvSourcesNoData/* = SKIP_FOR_CAPTURE
6400 : Texture2DTestES3.TextureCompletenessChangesWithBaseLevel/* = SKIP_FOR_CAPTURE
6399 : Texture2DTestES3.TexStorage2DLargeYuvTextureCount/* = SKIP_FOR_CAPTURE
6335 DEBUG : VulkanExternalImageTest.ShouldImportSemaphoreOpaqueFd/* = SKIP_FOR_CAPTURE


# Skip because Timeout

# ES3 Flaky
6631 : MultithreadingTest.MultiContextCreateAndDeleteResources/* = SKIP_FOR_CAPTURE
6631 WIN : MultithreadingTestES3.MultithreadFenceDraw/* = SKIP_FOR_CAPTURE
6631 WIN : MultithreadingTestES3.MultithreadFenceTexImage/* = SKIP_FOR_CAPTURE
6631 : MultithreadingTest.MultiContextDrawWithSwapBuffers/* = SKIP_FOR_CAPTURE

# Uses multiple contexts, don't want to run
6752 : OcclusionQueriesNoSurfaceTestES3.SwitchingContextsWithQuery/* = SKIP_FOR_CAPTURE

# ES3_1
6180 : ProgramPipelineTest31.VerifyPpoLinkErrorSignalledCorrectly/* = FAIL

# ES3_1 Crashed
6180 : EGLMultiContextTest.ComputeShaderOkayWithRendering/* = SKIP_FOR_CAPTURE
6180 : EGLMultiContextTest.TestContextDestroySimple/* = SKIP_FOR_CAPTURE

# ES3_1 gtest skips
6180 : GLSLTest_ES31.InterpolateAtWithSwizzle/* = NOT_RUN

6741 : VulkanImageTest.* = SKIP_FOR_CAPTURE
6741 : VulkanMemoryTest.* = SKIP_FOR_CAPTURE

6941 : Texture2DTestES3RobustInit.TextureCOMPRESSEDRGB8A1ETC2/* = NOT_RUN
6941 : Texture2DTestES3RobustInit.TextureCOMPRESSEDSRGB8A1ETC2/* = NOT_RUN

6926 : Texture2DTestES3.TextureRGBUpdateWithPBO/ES3_Vulkan* = NOT_RUN

7112 : TransformFeedbackTest.BaseInstance/* = NOT_RUN

# GLES 3.2 tests that fail because of missing features
5366 : GeometryShaderTestES32.MaxGeometryImageUniforms/* = SKIP_FOR_CAPTURE
5366 : ProgramPipelineTest32.MaxGeometryImageUniforms/* = SKIP_FOR_CAPTURE

7088 : VulkanPerformanceCounterTest* = SKIP_FOR_CAPTURE

7291 : BlitFramebufferTest.drawBlitAndDrawAgain/* = SKIP_FOR_CAPTURE
7291 : BlitFramebufferTest.scissorDrawBlitAndDrawAgain/* = SKIP_FOR_CAPTURE

7357 : FramebufferTest_ES3.FramebufferConditionalFeedbackLoop/* = SKIP_FOR_CAPTURE
7499 : Texture2DTest.TextureMaxSize/* = SKIP_FOR_CAPTURE
7373 : FramebufferTest_ES3.SurfaceDimensionsChangeAndFragCoord/* = SKIP_FOR_CAPTURE

7423 : MultithreadingTest.NoFlushNoContextReturnsTimeout/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadBClientWaitBeforeThreadASyncFinish/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadBClientWaitBeforeThreadASyncFlush/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadBGetStatusBeforeThreadASyncFinish/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadBGetStatusBeforeThreadASyncFlush/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadBServerWaitBeforeThreadASyncFinish/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadBServerWaitBeforeThreadASyncFlush/* = SKIP_FOR_CAPTURE
7423 : MultithreadingTestES3.ThreadCWaitBeforeThreadBSyncFinish/* = SKIP_FOR_CAPTURE

# Also multi-threaded
7423 : EGLMultiContextTest.RepeatedEglInitAndTerminate/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBClientWaitBeforeThreadASyncFlush/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBServerWaitBeforeThreadASyncFlush/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBGetStatusBeforeThreadASyncFlush/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBClientWaitBeforeThreadASyncFinish/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBServerWaitBeforeThreadASyncFinish/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBGetStatusBeforeThreadASyncFinish/* = SKIP_FOR_CAPTURE
7423 : EGLMultiContextTest.ThreadBCanSubmitWhileThreadAWaiting/* = SKIP_FOR_CAPTURE

7419 : MultithreadingTestES3.CreateFramebufferFetchBeforeRenderPass/* = NOT_RUN
7419 : MultithreadingTestES3.CreateFramebufferFetchMidRenderPass/* = NOT_RUN

7389 : Texture2DTest.ManySupersedingTextureUpdates/* = SKIP_FOR_CAPTURE

7449 : MultisampleTest.ContentPresevedAfterInterruption/* = SKIP_FOR_CAPTURE
7449 : MultisampleTest.Line/* = SKIP_FOR_CAPTURE
7449 : MultisampleTest.Point/* = SKIP_FOR_CAPTURE
7449 : MultisampleTest.Triangle/* = SKIP_FOR_CAPTURE

7483 : MultisampleTest.AlphaToSampleCoverage/* = SKIP_FOR_CAPTURE
7483 : MultisampleTestES3.ResolveToFBO/* = SKIP_FOR_CAPTURE

6723 : EGLMultiContextTest.ReuseUnterminatedDisplay/* = SKIP_FOR_CAPTURE

# No support yet for Capture/Replay of External Textures using other attribs
7570 : ImageTest.MipLevels/* = SKIP_FOR_CAPTURE
7570 : ImageTest.Source2DTarget2D_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.Source2DTarget2DArray_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.Source2DTargetExternal_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.SourceCubeTarget2D_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.SourceCubeTargetExternal_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.SourceRenderbufferTargetTexture_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.SourceRenderbufferTargetTextureExternal_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.ValidationGLEGLImage_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTest.ValidationImageBase/* = SKIP_FOR_CAPTURE
7570 : ImageTestES3.Source2DTargetExternalESSL3_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTestES3.SourceCubeTargetExternalESSL3/* = SKIP_FOR_CAPTURE
7570 : ImageTestES3.SourceCubeTargetExternalESSL3_Colorspace/* = SKIP_FOR_CAPTURE
7570 : ImageTestES3.SourceRenderbufferTargetTextureExternalESSL3_Colorspace/* = SKIP_FOR_CAPTURE

7581 : FramebufferTest_ES3.BlitWithDifferentPreRotations/* = SKIP_FOR_CAPTURE

7814 WIN : PBOExtensionTest.PBOWithExistingData/ES2_Vulkan_SwiftShader = FLAKY
7814 WIN : PBOExtensionTest.PBOWithOtherTarget/ES2_Vulkan_SwiftShader = FLAKY

# Related to uninitialized memory/buffers when using image suballocation
8033 WIN : FramebufferTestWithFormatFallback.R4G4B4A4_OutOfRangeStagedUpdateReformated/* = SKIP_FOR_CAPTURE
8033 WIN : ClearTestES3.RepeatedStencilClear/* = SKIP_FOR_CAPTURE
8033 WIN : ClearTestES3.RepeatedStencilClearWithBlitAfterClearAndDrawInBetween/* = SKIP_FOR_CAPTURE
8033 WIN : StateChangeTestES3.StencilReferenceAndCompareMask/* = SKIP_FOR_CAPTURE
8033 WIN : StateChangeTestES3.StencilWriteMask/* = SKIP_FOR_CAPTURE

# Corrupting memory by writing past readBufferSize due to GL_PACK_SKIP_ROWS etc
8070 : Texture2DNorm16TestES3.TextureNorm16R16RenderTest/* = SKIP_FOR_CAPTURE
8070 : Texture2DNorm16TestES3.TextureNorm16RG16RenderTest/* = SKIP_FOR_CAPTURE
8070 : Texture2DNorm16TestES3.TextureNorm16RGBA16RenderTest/* = SKIP_FOR_CAPTURE

# Fails if run after DifferentStencilMasksTest.DrawWithSameEffectiveMask
8079 : DiscardFramebufferEXTTest.ClearDepthThenDrawWithoutDepthTestThenDiscard/* = SKIP_FOR_CAPTURE

Download