kc3-lang/angle/src/program_serialize_data_version.py

Branch - branch -main

• Show log

  Commit

• Hash : 82826be0
  Author :
  Date : 2023-01-06T11:20:14
  

  Reland "Make ANGLE program version only dependent on data that matters"
  
  This is a reland of commit 61728827d2e5ecce685578bc54bb2c744b65fc9a
  
  Changes made on top of previous commit
  1. In src/program_serialize_data_version.py,
  fix the script error on Windows: anglebug.com/7918.
  2. In BUILD.gn, replace the 'write_file' with
  'response_file_content'. See
  https://gn.googlesource.com/gn/+/main/docs/reference.md#var_response_file_contents
  3. In scripts/generate_android.bp.py, add code to
  handle the special GN build flag {{response_file_name}}.
  The code writes the list defined in srcs
  (identified by $(in) in Android blueprint) into a
  temp file named 'gn_response_file', and replaces the
  {{response_file_name}} with 'gn_response_file'.
  
  Original change's description
  
  > Changes made on top of original commit
  > 1. Enable execution permission on python script
  > program_serialize_data_version.py
  > 2. Remove unused list in libGLESv2.gni
  > 3. In angle/BUILD.gn, change file path from
  > "relative to angle_root", to "relative to root_build_dir",
  > so that inside the script program_serialize_data_version.py,
  > we don't have to find the absolute path of the code files for
  > hashing.
  
  > Original change's description
  
  > > This change introduces a new variable ANGLE_PROGRAM_VERSION
  > > to track the version of ANGLE source files that affect shader
  > > program serialization/deserialization. This change include more
  > > source files than necessary, to serve the purpose of a conservative
  > > jumping off point. We will narrow down the list of files for
  > > ANGLE_PROGRAM_VERSION hash generation in the future.
  
  > > Add a new script program_serialize_data_version.py that will
  > > be triggered during the build when the related source files changed.
  > > The script will generate a hash and the hash size from the related
  > > source files. In program serialization/deserialization and cache
  > > key generation, we will use this hash value instead of the entire
  > > ANGLE git hash. When the hash value changed, we know that the
  > > related source files changed, and we should invalidate the program
  > > cache and re-generate the blob cache / program binary.
  
  > > Bug: angleproject:4981
  > > Change-Id: I2fb609416738d459d3289190c232c2d797ba58e3
  > > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4072215
  > > Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
  > > Reviewed-by: Cody Northrop <cnorthrop@google.com>
  > > Reviewed-by: Jamie Madill <jmadill@chromium.org>
  > > Commit-Queue: Yuxin Hu <yuxinhu@google.com>
  
  > Bug: angleproject:4981
  > Change-Id: Iaa9eb0ab33439197bc30d03064fc245ea7ef1ea8
  > Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4113445
  > Reviewed-by: Amirali Abdolrashidi <abdolrashidi@google.com>
  > Reviewed-by: Cody Northrop <cnorthrop@google.com>
  > Commit-Queue: Yuxin Hu <yuxinhu@google.com>
  
  Bug: angleproject:4981
  Change-Id: Ib5bba199be6d08a1e19807026df0e7b747dbc8a9
  Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/4144078
  Reviewed-by: Roman Lavrov <romanl@google.com>
  Commit-Queue: Yuxin Hu <yuxinhu@google.com>
  Reviewed-by: Cody Northrop <cnorthrop@google.com>
  

Download

• src/program_serialize_data_version.py

• #!/usr/bin/env python3
  #  Copyright 2022 The ANGLE Project Authors. All rights reserved.
  #  Use of this source code is governed by a BSD-style license that can be
  #  found in the LICENSE file.
  
  # Generate ANGLEShaderProgramVersion.h with hash of files affecting data
  # used in serializing/deserializing shader programs.
  import hashlib
  import argparse
  import shlex
  
  
  # Generate a hash from a list of files defined in angle_code_files
  def GenerateHashOfAffectedFiles(angle_code_files):
      hash_md5 = hashlib.md5()
      for file in angle_code_files:
          hash_md5.update(file.encode(encoding='utf-8'))
          with open(file, 'r', encoding='utf-8') as f:
              for chunk in iter(lambda: f.read(4096), ""):
                  hash_md5.update(chunk.encode())
      return hash_md5.hexdigest(), hash_md5.digest_size
  
  
  # The script is expecting two mandatory input arguments:
  # 1) output_file: the header file where we will add two constants that indicate
  # the version of the code files that affect shader program compilations
  # 2) response_file_name: a file that includes a list of code files that affect
  # shader program compilations
  parser = argparse.ArgumentParser(description='Generate the file ANGLEShaderProgramVersion.h')
  parser.add_argument(
      'output_file',
      help='path (relative to build directory) to output file name, stores ANGLE_PROGRAM_VERSION and ANGLE_PROGRAM_VERSION_HASH_SIZE'
  )
  parser.add_argument(
      'response_file_name',
      help='path (relative to build directory) to response file name. The response file stores a list of program files that ANGLE_PROGRAM_VERSION hashes over. See https://gn.googlesource.com/gn/+/main/docs/reference.md#var_response_file_contents'
  )
  
  args = parser.parse_args()
  
  output_file = args.output_file
  response_file_name = args.response_file_name
  
  with open(response_file_name, "r") as input_files_for_hash_generation:
      angle_code_files = shlex.split(input_files_for_hash_generation)
  digest, digest_size = GenerateHashOfAffectedFiles(angle_code_files)
  hfile = open(output_file, 'w')
  hfile.write('#define ANGLE_PROGRAM_VERSION "%s"\n' % digest)
  hfile.write('#define ANGLE_PROGRAM_VERSION_HASH_SIZE %d\n' % digest_size)
  hfile.close()
  

Download