scripts/run_code_generation.py


Log

Author Commit Date CI Message
Frank Henigman 569b9cb9 2018-07-07T21:44:35 Fix run_code_generation.py hash calculation. Open files with mode 'r' instead of 'rb' so the hash calculation produces the same results on Linux and Windows. Recalculate the hashes and proc_table_autogen.cpp which was out of date. BUG=angleproject:2711 Change-Id: I31562c96ce36e6df009c44c565fe9a3f1b5ba6c4 Reviewed-on: https://chromium-review.googlesource.com/1128549 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Frank Henigman <fjhenigman@chromium.org>
Geoff Lang b1cc789c 2018-07-05T15:05:21 Finish populating the new_hashes map after finding a dirty input. any_input_dirty would return early when finding a dirty input and not finish filling the new_hashes map. This would require multiple runs of the code generation script to fully generate all the outputs. BUG=angleproject:2695 Change-Id: Ie62190efe2765df432b0a535fb8d33ed2ffa66a7 Reviewed-on: https://chromium-review.googlesource.com/1127439 Commit-Queue: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Geoff Lang 9a257801 2018-07-05T14:31:08 Write relative paths with forward slashes to the code generation hashes. This should avoid generating diffs when running the code generation with different source directories or on different platforms. BUG=angleproject:2695 Change-Id: I67776883bdbeb867a49bea00f16998c04f7857b4 Reviewed-on: https://chromium-review.googlesource.com/1127355 Commit-Queue: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Geoff Lang 5695a5b2 2018-07-05T14:29:30 Allow the run_code_generation.py script to be run from any directory. BUG=angleproject:2695 Change-Id: I9a083bf67b17ca0dc1ee213b75caa4ecece62f00 Reviewed-on: https://chromium-review.googlesource.com/1127354 Commit-Queue: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill a72f400c 2018-06-29T17:05:01 run_code_generation: Compare hashes instead of mtime. Using mtime is fundamentally flawed when working with git. Replace these flaky checks with hash comparisons. The hashes are stored in an autogenerated json file that will be stored in the repository. This makes the run_code_generation script robust against any and all input changes. It also removes the need to track script outputs as dependencies. Bug: angleproject:2697 Change-Id: I60f2a87a8680b1f775ad678b05112f5b16c7dde7 Reviewed-on: https://chromium-review.googlesource.com/1120159 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Frank Henigman 388f991c 2018-06-15T17:18:09 Vulkan: fix third_party path. run_code_generation.py stopped working, likely as a result of the big Vulkan repo reorganization. I fixed a path, regenerated everything (by temporarily hacking the script so it thought everything was dirty), and nothing changed except one comment. BUG=angleproject:2558 Change-Id: I5ac4c040ac1ec207098172303cc2f2507cccdecb Reviewed-on: https://chromium-review.googlesource.com/1103281 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Tobin Ehlis <tobine@google.com>
Jamie Madill d4703d50 2018-05-24T17:31:43 Move packed enum code to common/ This makes it accessible in the utilities files. Bug: angleproject:2574 Bug: angleproject:2169 Change-Id: I0fdd34b4233e72b7534cb2b09f451539c1a394cd Reviewed-on: https://chromium-review.googlesource.com/1067110 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Jamie Madill 1c597eea 2018-05-24T14:19:31 Fix run_code_generation for huge diffs. This uses the --full argument to work around max path limitations on Windows. Also includes a couple minor changes for generators to generate already-formatted code. Bug: angleproject:2578 Change-Id: I1e400b02e828bfdca21cacb73c649f41226bef55 Reviewed-on: https://chromium-review.googlesource.com/1072161 Reviewed-by: Lingfeng Yang <lfy@google.com> Reviewed-by: Luc Ferron <lucferron@chromium.org> Reviewed-by: Brandon1 Jones <brandon1.jones@intel.com> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill d47044ad 2018-04-27T11:45:03 Vulkan: Add framework for internal shaders. Vulkan intenal shaders are stored in a ShaderLibrary, and this is owned by the RendererVk. This way the shaders are reused between all the different Contexts. They are initialized lazily to keep init time low. They also have an associated Serial (called a ProgramSerial) so they can be identified in a PipelineDesc (used by the Pipeline cache). We use a python script to build and invoke the glslang validator, that also produces SPIR-V binary code snippets. These snippets are gathered into an auto-generated file that is exposed via an auto-generated header file. The InternalShaderID enum class gives access to the internal shaders that are shared through the Vulkan back-end. This also adds simple clear shaders to be used in masked color clears. The patch doesn't add any functionality but it is split off from the color clear functionality to keep the code size down. Bug: angleproject:2339 Bug: angleproject:2455 Change-Id: Ie83043eda217c9f013817b198c92a3b7ba0878b4 Reviewed-on: https://chromium-review.googlesource.com/1031372 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org>
Jamie Madill 0ade8e88 2018-04-27T11:45:02 Use time tolerance in run_code_generation check. The mtime values I was seeing on my file system were sometimes floating point numbers that were very very close but not exactly the same. This might be due to numerical errors. Increasing the tolerance to something still very small, precision is less than a second. It will lead to fewer rebuilds of non-dirty files. Bug: angleproject:2455 Change-Id: I95dc3214ee91af7a70a20cc625405e0e2bc18698 Reviewed-on: https://chromium-review.googlesource.com/1032855 Reviewed-by: Luc Ferron <lucferron@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Geoff Lang 8ceea819 2018-04-10T03:07:13 Refactor packed enum generation to support EGL enums. Convert the very simple EGL texture type enum. BUG=angleproject:1618 Change-Id: Ieea382a282a8f2544f2982627e8445e6e5cea826 Reviewed-on: https://chromium-review.googlesource.com/1019386 Commit-Queue: Geoff Lang <geofflang@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org>
Olli Etuaho 5fec7ab2 2018-04-04T11:58:33 Identify functions by unique id in BuiltInFunctionEmulator Now that unique ids of all builtins are compile-time constants, we can use them to look up functions in BuiltInFunctionEmulator. This is simpler than using a custom struct with the name and parameters for identifying functions. This requires that we store a reference to a TFunction in those TIntermUnary nodes that were created based on a function. This decreases shader_translator binary size by about 6 KB on Windows. BUG=angleproject:2267 BUG=chromium:823856 TEST=angle_unittests Change-Id: Idd5a00c772c6f26dd36fdbbfbe161d22ab27c2fe Reviewed-on: https://chromium-review.googlesource.com/995372 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
Olli Etuaho 89398b65 2018-03-21T17:30:50 Avoid mangled name comparisons of 3-parameter functions The hash values used for looking up built-ins now encode whether the mangled name contains arrays, structs or interface blocks in its parameters list. This is written in the most significant bit of the hash value. We check this bit at the start of built-in lookup and if the bit is set we exit early. After that we know that the lookup name doesn't contain array, struct or interface block parameters. When we find a hash that matches a hash of a built-in function, we now know 3 things: 1) the length of the mangled name matches 2) the open parentheses in the mangled name matches 3) the lookup doesn't contain array, struct or block parameters. Additionally, we have an if statement checking whether the function name matches. Collisions are only possible with functions that 1) have the same name 2) have the same number of parameters With these preconditions we can check beforehand whether collisions are possible for 3-parameter functions. If there are no collisions, we don't need to compare the full mangled name. This is similar to what was already being done with functions that had 0 to 2 parameters. This reduces shader_translator binary size by around 4 KB on Windows. Besides increased complexity, the tradeoff is that an exhaustive search of hash values for possible 3-parameter combinations is costly, so the gen_builtin_functions.py code generation script now takes around one minute to run on a high-end workstation. Due to this, the script now exits early if it detects it has already been run with the same inputs based on a hash value stored in builtin_symbols_hash_autogen.txt. BUG=angleproject:2267 BUG=chromium:823856 TEST=angle_unittests Change-Id: I3ff8c6eb85b90d3c4971ac8d73ee171a07a7e55f Reviewed-on: https://chromium-review.googlesource.com/973372 Reviewed-by: Corentin Wallez <cwallez@chromium.org> Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
Olli Etuaho c26214de 2018-03-16T10:43:11 Move AST utilities to a subdirectory Move AST related utilities to compiler/translator/tree_util. BUG=angleproject:2409 TEST=angle_unittests Change-Id: I7567c2f6f2710292029263257c7ac26e2a144ac8 Reviewed-on: https://chromium-review.googlesource.com/966032 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Corentin Wallez <cwallez@chromium.org> Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
Olli Etuaho b391ec40 2018-03-12T17:04:59 Generate code for looking up built-ins Instead of storing built-ins in a std::unordered_map, we now generate a series of switch statements using the hash value of the look-up string. This works similarly to earlier implementation of looking up unmangled built-ins. Those built-ins that need to be initialized at run-time are stored as member variables of TSymbolTable. This increases compiler init performance significantly, as well as increasing compiler perf test scores around 1-2%. Binary size is larger than before though. BUG=angleproject:2267 TEST=angle_unittests Change-Id: If1dcd36f0d2b30c2ed315cdcf6e831ae9fe70c94 Reviewed-on: https://chromium-review.googlesource.com/960031 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
Olli Etuaho 2bfe9f6b 2018-03-02T16:53:29 Use function id to group functions in ParseContext This way we can do numeric comparisons instead of string comparisons. The effect on compiler perf test scores is fairly marginal, but this reduces binary size by a few kilobytes, and there may be a larger effect on shaders calling a lot of texture functions. BUG=angleproject:2267 TEST=angle_unittests Change-Id: I077db97b16b16b70b7e18ee037e06d7450d08dc9 Reviewed-on: https://chromium-review.googlesource.com/947952 Commit-Queue: Olli Etuaho <oetuaho@nvidia.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Luc Ferron 4ea3b450 2018-03-13T11:48:26 run_code_generation.py: Add gen_load_functions_table.py to the list Bug:angleproject:2358 Change-Id: I93bdbbe65ac8109a5a506d88202715c51918613b Reviewed-on: https://chromium-review.googlesource.com/960574 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Olli Etuaho 391bda23 2018-02-23T11:43:14 Generate code for initializing built-in variables gen_builtin_symbols.py now generates code for initializing built-in variable symbols as well. Some of the variable symbols are static, but some of them also get initialized dynamically based on values in ShBuiltInResources. The static symbols have get functions in a header file so they can be referenced from AST traversers as well without doing a lookup. BUG=angleproject:2267 TEST=angle_unittests, angle_end2end_tests Change-Id: Ida7f3aeb06d2bce0f737f1483b1bd5833aeddd2e Reviewed-on: https://chromium-review.googlesource.com/911768 Reviewed-by: Corentin Wallez <cwallez@chromium.org> Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
Olli Etuaho 7618eaf9 2018-03-07T11:25:40 Fix incorrect path in run_code_generation.py Adding missing "src". BUG=angleproject:2267 TEST=python scripts/run_code_generation.py Change-Id: I89d981c8f09e6d0839d0b796a2ea14a8466f98a7 Reviewed-on: https://chromium-review.googlesource.com/952925 Reviewed-by: Olli Etuaho <oetuaho@nvidia.com> Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
Olli Etuaho 065aa863 2018-02-22T15:30:27 Generate code for unmangled name lookup Instead of using an std::map at each symbol table level, use the gen_builtin_symbols.py script to build a function to query unmangled names. BUG=angleproject:2267 TEST=angle_unittests Change-Id: I4f1cf1df1f50fe9d909f3249150ee002ee6efb61 Reviewed-on: https://chromium-review.googlesource.com/931885 Commit-Queue: Olli Etuaho <oetuaho@nvidia.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Olli Etuaho 140152e7 2018-02-08T14:46:44 Statically allocate built-in function symbols A script gen_builtin_symbols.py now generates code for initializing built-in function symbols. The TFunction objects are initialized at C++ compile time. The source file used for the functions is in a format that's similar to how functions are given out in the GLSL spec, so it is easy to maintain. The function symbols are still inserted to the symbol table levels same as before. Getting rid of inserting the symbols at runtime is intended to be done as follow-up. This speeds up angle_unittests on Linux in release mode by a bit less than half, and in debug mode by more than half. BUG=angleproject:2267 TEST=angle_unittests Change-Id: I11c9de98c74d28e7e8cdf024516e2f6ee30ca33e Reviewed-on: https://chromium-review.googlesource.com/924155 Commit-Queue: Olli Etuaho <oetuaho@nvidia.com> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Luc Ferron 0aa1ffe3 2018-02-08T13:42:36 Vulkan: Autogen mandatory texture caps * This commit includes a JS file to execute on the spec and generate the JSON output of all the mandatory texture caps. Bug: angleproject:2348 Change-Id: I57e969915bdd0e7104e00a73fd3743ff1ecf0a6d Reviewed-on: https://chromium-review.googlesource.com/911615 Commit-Queue: Luc Ferron <lucferron@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org>
Frank Henigman dda048cd 2018-01-11T20:09:09 Make scripts executable by python2. Add #!/usr/bin/python2 and the executable permission bit to all scripts where missing. BUG=angleproject:2209 Change-Id: Ib33017c17e579c371b89bbfbdb7136b870027dc5 Reviewed-on: https://chromium-review.googlesource.com/862987 Reviewed-by: Corentin Wallez <cwallez@chromium.org> Commit-Queue: Frank Henigman <fjhenigman@chromium.org>
Jamie Madill 5ad52994 2017-11-14T12:43:40 Add generator for EGL proc table. This should improve ANGLE startup time by avoiding creating a large std::map filled with the entry points. BUG=chromium:781460 Change-Id: I20cfdb10b99844d0f60759dda73b729991dc60fe Reviewed-on: https://chromium-review.googlesource.com/768209 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Antoine Labour <piman@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Jamie Madill a8b73ed0 2017-11-02T09:22:29 Add a meta-script to run code generators. This script calls all the various GL and back-end python scripts to generate our internal format tables and entry points, etc. It uses a GYP-like scheme of inputs/outputs to check modified time before running the generators. It also will automatically call 'git cl format' if any generator was called. Also updates the copyright in a couple of touched files. BUG=angleproject:2207 Change-Id: I4187a7622accc1c97a8d779b8f87fe00b74855ea Reviewed-on: https://chromium-review.googlesource.com/742372 Commit-Queue: Jamie Madill <jmadill@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org>