run_code_generation.py

Show log
Commit

Hash : 050b124d
Author :
Date : 2019-06-30T03:26:18

Reland "Vulkan: Debug overlay"

This is a reland of e54d0f90d1a165404236fd7abd1b05ddd041a686

This was reverted due to a build failure as a result of a missing
virtual destructor in the widget base class.

Original change's description:
> Vulkan: Debug overlay
>
> A debug overlay system for the Vulkan backend designed with efficiency
> and runtime configurability in mind.  Overlay widgets are of two
> fundamental types:
>
> - Text widgets: A single line of text with small, medium or large font.
> - Graph widgets: A bar graph of data.
>
> Built on these, various overlay widget types are defined that gather
> statistics.  Five such types are defined with one widget per type as
> example:
>
> - Count: A widget that counts something.  VulkanValidationMessageCount
>   is an overlay widget of this type that shows the number of validation
>   messages received from the validation layers.
> - Text: A generic text.  VulkanLastValidationMessage is an overlay
>   widget of this type that shows the last validation message.
> - PerSecond: A value that gets reset every second automatically.  FPS is
>   an overlay widget of this type that simply gets incremented on every
>   swap().
> - RunningGraph: A graph of last N values.  VulkanCommandGraphSize is an
>   overlay of this type.  On every vkQueueSubmit, the number of nodes in
>   the command graph is accumulated.  On every present(), the value is
>   taken as the number of nodes for the whole duration of the frame.
> - RunningHistogram: A histogram of last N values.  Input values are in
>   the [0, 1] range and they are ranked to N buckets for histogram
>   calculation.  VulkanSecondaryCommandBufferPoolWaste is an overlay
>   widget of this type.  On vkQueueSubmit, the memory waste from command
>   buffer pool allocations is recorded in the histogram.
>
> Overlay font is placed in libANGLE/overlay/ which gen_overlay_fonts.py
> processes to create an array of bits, which is processed at runtime to
> create the actual font image (an image with 3 layers).
>
> The overlay widget layout is defined in overlay_widgets.json which
> gen_overlay_widgets.py processes to generate an array of widgetss, each
> of its respective type, and sets their properties, such as color and
> bounding box.  The json file allows widgets to align against other
> widgets as well as against the framebuffer edges.
>
> Two compute shaders are implemented to efficiently render the UI:
>
> - OverlayCull: This shader creates a bitset of Text and Graph widgets
>   whose bounding boxes intersect a corresponding subgroup processed by
>   OverlayDraw.  This is done only when the enabled overlay widgets are
>   changed (a feature that is not yet implemented) or the surface is
>   resized.
> - OverlayDraw: Using the bitsets generated by OverlayCull, values that
>   are uniform for each workgroup (set to be equal to hardware subgroup
>   size), this shader loops over enabled widgets that can possibly
>   intersect the pixel being processed and renders and blends in texts
>   and graphs.  This is done once per frame on present().
>
> Currently, to enable overlay widgets an environment variable is used.
> For example:
>
>     $ export ANGLE_OVERLAY=FPS:VulkanSecondaryCommandBufferPoolWaste
>     $ ./hello_triangle --use-angle=vulkan
>
> Possible future work:
>
> - On Android, add settings in developer options and enable widgets based
>   on those.
> - Spawn a small server in ANGLE and write an application that sends
>   enable/disable commands remotely.
> - Implement overlay for other backends.
>
> Bug: angleproject:3757
> Change-Id: If9c6974d1935c18f460ec569e79b41188bd7afcc
> Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1729440
> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
> Reviewed-by: Jamie Madill <jmadill@chromium.org>

Bug: angleproject:3757
Change-Id: I47915d88b37b6f882c686c2de13fca309a10b572
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1780897
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>

Download

scripts/run_code_generation.py

#!/usr/bin/python2
#
# Copyright 2017 The ANGLE Project Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
#
# run_code_generation.py:
#   Runs ANGLE format table and other script code generation scripts.

import hashlib
import json
import os
import subprocess
import sys
import platform

script_dir = sys.path[0]
root_dir = os.path.abspath(os.path.join(script_dir, '..'))

hash_dir = 'code_generation_hashes'

# auto_script is a standard way for scripts to return their inputs and outputs.


def get_child_script_dirname(script):
    # All script names are relative to ANGLE's root
    return os.path.dirname(os.path.abspath(os.path.join(root_dir, script)))


# Replace all backslashes with forward slashes to be platform independent
def clean_path_slashes(path):
    return path.replace("\\", "/")


# Takes a script file name which is relative to the code generation script's directory and
# changes it to be relative to the angle root directory
def rebase_script_path(script_path, relative_path):
    return os.path.relpath(os.path.join(os.path.dirname(script_path), relative_path), root_dir)


# Check if we need a module from vpython
def get_executable_name(first_line):
    if 'vpython' in first_line:
        return 'vpython.bat' if platform.system() == 'Windows' else 'vpython'
    return 'python'


def grab_from_script(script, param):
    res = ''
    f = open(os.path.basename(script), "r")
    res = subprocess.check_output([get_executable_name(f.readline()), script, param]).strip()
    f.close()
    if res == '':
        return []
    return [clean_path_slashes(rebase_script_path(script, name)) for name in res.split(',')]


def auto_script(script):
    # Set the CWD to the script directory.
    os.chdir(get_child_script_dirname(script))
    base_script = os.path.basename(script)
    info = {
        'inputs': grab_from_script(base_script, 'inputs'),
        'outputs': grab_from_script(base_script, 'outputs')
    }
    # Reset the CWD to the root ANGLE directory.
    os.chdir(root_dir)
    return info


generators = {
    'ANGLE format':
        'src/libANGLE/renderer/gen_angle_format_table.py',
    'ANGLE load functions table':
        'src/libANGLE/renderer/gen_load_functions_table.py',
    'D3D11 blit shader selection':
        'src/libANGLE/renderer/d3d/d3d11/gen_blit11helper.py',
    'D3D11 format':
        'src/libANGLE/renderer/d3d/d3d11/gen_texture_format_table.py',
    'DXGI format':
        'src/libANGLE/renderer/d3d/d3d11/gen_dxgi_format_table.py',
    'DXGI format support':
        'src/libANGLE/renderer/d3d/d3d11/gen_dxgi_support_tables.py',
    'GL copy conversion table':
        'src/libANGLE/gen_copy_conversion_table.py',
    'GL/EGL/WGL loader':
        'scripts/generate_loader.py',
    'GL/EGL entry points':
        'scripts/generate_entry_points.py',
    'GLenum value to string map':
        'scripts/gen_gl_enum_utils.py',
    'GL format map':
        'src/libANGLE/gen_format_map.py',
    'uniform type':
        'src/common/gen_uniform_type_table.py',
    'OpenGL dispatch table':
        'src/libANGLE/renderer/gl/generate_gl_dispatch_table.py',
    'packed enum':
        'src/common/gen_packed_gl_enums.py',
    'proc table':
        'scripts/gen_proc_table.py',
    'Vulkan format':
        'src/libANGLE/renderer/vulkan/gen_vk_format_table.py',
    'Vulkan mandatory format support table':
        'src/libANGLE/renderer/vulkan/gen_vk_mandatory_format_support_table.py',
    'Vulkan internal shader programs':
        'src/libANGLE/renderer/vulkan/gen_vk_internal_shaders.py',
    'overlay fonts':
        'src/libANGLE/gen_overlay_fonts.py',
    'overlay widgets':
        'src/libANGLE/gen_overlay_widgets.py',
    'Emulated HLSL functions':
        'src/compiler/translator/gen_emulated_builtin_function_tables.py',
    'Static builtins':
        'src/compiler/translator/gen_builtin_symbols.py'
}


def md5(fname):
    hash_md5 = hashlib.md5()
    with open(fname, "r") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash_md5.update(chunk)
    return hash_md5.hexdigest()


def get_hash_file_name(name):
    return name.replace(' ', '_').replace('/', '_') + '.json'


def any_hash_dirty(name, filenames, new_hashes, old_hashes):
    found_dirty_hash = False

    for fname in filenames:
        if not os.path.isfile(fname):
            print('File not found: "%s". Code gen dirty for %s' % (fname, name))
            found_dirty_hash = True
        else:
            new_hashes[fname] = md5(fname)
            if (not fname in old_hashes) or (old_hashes[fname] != new_hashes[fname]):
                print('Hash for "%s" dirty for %s generator.' % (fname, name))
                found_dirty_hash = True
    return found_dirty_hash


def any_old_hash_missing(all_new_hashes, all_old_hashes):
    result = False
    for file, old_hashes in all_old_hashes.iteritems():
        if file not in all_new_hashes:
            print('"%s" does not exist. Code gen dirty.' % file)
            result = True
        else:
            for name, _ in old_hashes.iteritems():
                if name not in all_new_hashes[file]:
                    print('Hash for %s is missing from "%s". Code gen is dirty.' % (name, file))
                    result = True
    return result


def update_output_hashes(script, outputs, new_hashes):
    for output in outputs:
        if not os.path.isfile(output):
            print('Output is missing from %s: %s' % (script, output))
            sys.exit(1)
        new_hashes[output] = md5(output)


def load_hashes():
    hashes = {}
    for file in os.listdir(hash_dir):
        hash_fname = os.path.join(hash_dir, file)
        with open(hash_fname) as hash_file:
            hashes[file] = json.load(open(hash_fname))
    return hashes


def main():
    os.chdir(script_dir)

    all_old_hashes = load_hashes()
    all_new_hashes = {}
    any_dirty = False

    verify_only = False
    if len(sys.argv) > 1 and sys.argv[1] == '--verify-no-dirty':
        verify_only = True

    for name, script in sorted(generators.iteritems()):
        info = auto_script(script)
        fname = get_hash_file_name(name)
        filenames = info['inputs'] + info['outputs'] + [script]
        new_hashes = {}
        if fname not in all_old_hashes:
            all_old_hashes[fname] = {}
        if any_hash_dirty(name, filenames, new_hashes, all_old_hashes[fname]):
            any_dirty = True

            if not verify_only:
                # Set the CWD to the script directory.
                os.chdir(get_child_script_dirname(script))

                print('Running ' + name + ' code generator')

                f = open(os.path.basename(script), "r")
                if subprocess.call([get_executable_name(f.readline()),
                                    os.path.basename(script)]) != 0:
                    sys.exit(1)
                f.close()

        # Update the hash dictionary.
        all_new_hashes[fname] = new_hashes

    if any_old_hash_missing(all_new_hashes, all_old_hashes):
        any_dirty = True

    if verify_only:
        sys.exit(any_dirty)

    if any_dirty:
        args = ['git.bat'] if os.name == 'nt' else ['git']
        # The diff can be so large the arguments to clang-format can break the Windows command
        # line length limits. Work around this by calling git cl format with --full.
        args += ['cl', 'format', '--full']
        print('Calling git cl format')
        subprocess.call(args)

        # Update the output hashes again since they can be formatted.
        for name, script in sorted(generators.iteritems()):
            info = auto_script(script)
            fname = get_hash_file_name(name)
            update_output_hashes(name, info['outputs'], all_new_hashes[fname])

        os.chdir(script_dir)

        for fname, new_hashes in all_new_hashes.iteritems():
            hash_fname = os.path.join(hash_dir, fname)
            json.dump(
                new_hashes,
                open(hash_fname, "w"),
                indent=2,
                sort_keys=True,
                separators=(',', ':\n    '))


if __name__ == '__main__':
    sys.exit(main())

Download