src/compiler/preprocessor/Tokenizer.cpp


Log

Author Commit Date CI Message
Corentin Wallez dc0fa46a 2017-02-01T14:44:43 preprocessor: Check for line number overflow Also remove dead code in Tokenizer.l BUG=chromium:668842 Change-Id: Ice18313a64f0bb2242299993bfaa882a6578ad54 Reviewed-on: https://chromium-review.googlesource.com/435042 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org> Commit-Queue: Corentin Wallez <cwallez@chromium.org>
Corentin Wallez 168d5e80 2017-02-01T12:18:56 compiler: Regenerate parser with latest flex and bison Flex version is 2.6.4 Bison version is 3.0.4 BUG=chromium:668842 Change-Id: Ia05ae338c9b9e588534f8346ff5c59ed747c56bf Reviewed-on: https://chromium-review.googlesource.com/435553 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
Corentin Wallez 054f7ed0 2016-09-20T17:15:59 preprocessor: Miscellaneous cleanups - Use full header paths in includes - Use ASSERT instead of assert - Use angle::NonCopyable instead of PP_DISALLOW_COPY_AND_ASSIGN - Use range-for in a couple places - Remove pp_utils.h BUG=angleproject:1522 Change-Id: If107fef89e8465bca65cf664926d1051c5d1e232 Reviewed-on: https://chromium-review.googlesource.com/387212 Commit-Queue: Corentin Wallez <cwallez@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org>
Geoff Lang 93561c3a 2016-09-20T10:22:55 Regenerate the preprocessor with the latest versions of flex and bison. BUG=648063 Change-Id: Ia90d12c8cb4ae0a694227145267e78ca02dcc2b0 Reviewed-on: https://chromium-review.googlesource.com/387114 Commit-Queue: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Olli Etuaho 26e355b8 2015-08-14T14:16:19 Add full support for line continuation in the preprocessor Re-landing earlier change with constant signedness fixed (was causing build issues on Linux). Line continuation in ESSL 3.00 needs to be processed before tokenization, since tokens can span the line continuation. On the other hand, ANGLE's tokenizer keeps track of line numbers, and whenever a line continuation appears the line number still needs to be incremented by one, just like on a regular newline. That's why line continuation is now implemented as follows: when the shader strings are concatenated in Input, they are also checked for line continuation. Whenever line continuation is encountered, the string is cut before that point. When the tokenizer asks for more input, the string starting from the character after line continuation is passed to it, and the line number is incremented from Input. This way the tokenizer can parse tokens that span multiple lines - it never sees the line continuation - but still keeps track of the line number correctly. Relevant spec is in ESSL 3.00 section 3.2 "Source strings". Support for line continuation also applies to ESSL 1.00. ESSL 3.00 spec section 1.5 says that line continuation support is mandated when an ESSL 1.00 shader is used with the OpenGL ES 3.0 API, and is optional when ESSL 1.00 is used with the OpenGL ES 2.0 API. TEST=dEQP-GLES3.functional.shaders.preprocessor.line_continuation.* (all pass), angle_unittests BUG=angleproject:1125 Change-Id: Ic086aacac53cd75bf93c0fda782416501d2f842b Reviewed-on: https://chromium-review.googlesource.com/294200 Reviewed-by: Jamie Madill <jmadill@chromium.org> Tested-by: Olli Etuaho <oetuaho@nvidia.com>
Jamie Madill 3b040eb8 2015-08-17T16:51:33 Revert "Add full support for line continuation in the preprocessor" Warning in the Linux/Mac builders: In file included from ../../third_party/angle/src/tests/preprocessor_tests/input_test.cpp:7: In file included from ../../third_party/angle/src/tests/preprocessor_tests/PreprocessorTest.h:7: ../../testing/gtest/include/gtest/gtest.h:1392:16: error: comparison of integers of different signs: 'const int' and 'const unsigned long' [-Werror,-Wsign-compare] if (expected == actual) { ~~~~~~~~ ^ ~~~~~~ ../../testing/gtest/include/gtest/gtest.h:1422:12: note: in instantiation of function template specialization 'testing::internal::CmpHelperEQ<int, unsigned long>' requested here return CmpHelperEQ(expected_expression, actual_expression, expected, ^ ../../third_party/angle/src/tests/preprocessor_tests/input_test.cpp:171:5: note: in instantiation of function template specialization 'testing::internal::EqHelper<false>::Compare<int, unsigned long>' requested here EXPECT_EQ(3, input.read(buf, maxSize, &lineNo)); ^ BUG=angleproject:1125 This reverts commit c1157d1963170c7411eb6c32e2b2fbce02c5a170. Change-Id: Ic6fa286d190b006cccc5154d86e21ecc03175763 Reviewed-on: https://chromium-review.googlesource.com/294080 Reviewed-by: Jamie Madill <jmadill@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org>
Olli Etuaho c1157d19 2015-08-14T14:16:19 Add full support for line continuation in the preprocessor Line continuation in ESSL 3.00 needs to be processed before tokenization, since tokens can span the line continuation. On the other hand, ANGLE's tokenizer keeps track of line numbers, and whenever a line continuation appears the line number still needs to be incremented by one, just like on a regular newline. That's why line continuation is now implemented as follows: when the shader strings are concatenated in Input, they are also checked for line continuation. Whenever line continuation is encountered, the string is cut before that point. When the tokenizer asks for more input, the string starting from the character after line continuation is passed to it, and the line number is incremented from Input. This way the tokenizer can parse tokens that span multiple lines - it never sees the line continuation - but still keeps track of the line number correctly. Relevant spec is in ESSL 3.00 section 3.2 "Source strings". Support for line continuation also applies to ESSL 1.00. ESSL 3.00 spec section 1.5 says that line continuation support is mandated when an ESSL 1.00 shader is used with the OpenGL ES 3.0 API, and is optional when ESSL 1.00 is used with the OpenGL ES 2.0 API. TEST=dEQP-GLES3.functional.shaders.preprocessor.line_continuation.* (all pass), angle_unittests BUG=angleproject:1125 Change-Id: I1c1de49602e7cd755d6072c3c0aa5524cd0313b4 Reviewed-on: https://chromium-review.googlesource.com/293721 Reviewed-by: Jamie Madill <jmadill@chromium.org> Tested-by: Olli Etuaho <oetuaho@nvidia.com>
Scott Graham a897542f 2015-05-01T11:09:12 vs2015: disable warning in generated code, fix another Macro redefinition is for INT8_MIN, etc. which are generated by flex. VS doesn't define __STDC_VERSION__ as >= C99 because it's still only partial support in VS2015. Fix a float->int conversion narrowing warning. Change-Id: I5232eb23426eaf584218137c068e14d74119a1ef Reviewed-on: https://chromium-review.googlesource.com/268821 Reviewed-by: Zhenyao Mo <zmo@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Tested-by: Scott Graham <scottmg@chromium.org>
Jamie Madill 2dc8bf8b 2015-04-30T15:56:52 translator: Fix C++11-deprecated register usage. Define 'register' as an empty string if we're in a modern version of C++. BUG=255186 BUG=angleproject:463 Change-Id: Ied044fb87a9b05c91cb419c54295c39f0f0ab776 Reviewed-on: https://chromium-review.googlesource.com/268512 Reviewed-by: Geoff Lang <geofflang@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Jamie Madill b96687df 2015-04-30T15:56:51 translator: Regenerate with Bison 3.0.4. BUG=angleproject:463 Change-Id: If89a29de8fb006e8e3b0483ac5df21c8833ce974 Reviewed-on: https://chromium-review.googlesource.com/268511 Tested-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Zhenyao Mo <zmo@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org>
Minmin Gong 794e0009 2015-04-07T18:31:54 Fix and enable warning C4244 (Conversion from 'type1' to 'type2', possible loss of data) Change-Id: Id0e06d7d6600344d858f00dabc219d79289bbc82 Reviewed-on: https://chromium-review.googlesource.com/265020 Tested-by: Minmin Gong <mgong@microsoft.com> Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Geoff Lang <geofflang@chromium.org>
Jamie Madill b3584fb4 2015-04-09T17:34:21 Revert "Fix and enable warning C4244 (Conversion from 'type1' to 'type2', possible loss of data)" Causing a build failure on Mac/Clang: ./Tokenizer.cpp:551:7: error: extra tokens at end of #else directive [-Werror,-Wextra-tokens] #else if defined(_MSC_VER) http://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Mac%20Builder/builds/29136 This reverts commit 3b26e231d99154814eb428f75a67bbe7a21adadc. Change-Id: I2d11ddcc18130d908fd2ec3d6f5ab890cfccd5e7 Reviewed-on: https://chromium-review.googlesource.com/264983 Reviewed-by: Jamie Madill <jmadill@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org>
Minmin Gong 3b26e231 2015-04-07T18:31:54 Fix and enable warning C4244 (Conversion from 'type1' to 'type2', possible loss of data) Change-Id: I73d9a2b9ad16f032be974b9c819de0dc1247c2ea Reviewed-on: https://chromium-review.googlesource.com/264533 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 185de884 2014-12-22T15:17:52 Update ANGLE's translator to Bison 3. BUG=angle:462 Change-Id: I2c1c18027dee1c3b4efb87374caaadbf58367841 Reviewed-on: https://chromium-review.googlesource.com/236930 Tested-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Zhenyao Mo d526f989 2014-05-13T14:51:19 Fix code style violation in compiler/preprocessor BUG=angle:650 TEST=no behavior change Change-Id: Ib52f15f6471fc7897b66d11baee11216cf08158a Reviewed-on: https://chromium-review.googlesource.com/199591 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Zhenyao Mo <zmo@chromium.org> Tested-by: Zhenyao Mo <zmo@chromium.org>
Jamie Madill 5508f39d 2014-02-20T13:31:36 Fix breaking the build with missing GetGlobalMaxTokenSize. Because the preprocessor is used independently from the compiler, we need a way to track max token size when we don't have access to the parse context with the current spec. BUG=angle:550 Change-Id: Idf5035ec2c001ee75f264151ab3c4e92f3cd44d7 Reviewed-on: https://chromium-review.googlesource.com/187140 Reviewed-by: Geoff Lang <geofflang@chromium.org> Reviewed-by: Shannon Woods <shannonwoods@chromium.org> Reviewed-by: Nicolas Capens <nicolascapens@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org>
Jamie Madill 88f6e946 2014-02-19T10:27:53 Proper support for token max size in WebGL+ES3. WebGL specifies a maximum token size of 256 characters, while ES3 specifies 1024 characters. We can determine the proper max size to support from the spec. BUG=angle:550 Change-Id: I6aeabe8af3b6184a27b456248ce2f84f361b76e4 Reviewed-on: https://chromium-review.googlesource.com/186973 Reviewed-by: Shannon Woods <shannonwoods@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org>
Shannon Woods 7f2d7945 2013-11-19T15:07:58 Manual merge of Ehsan Akhgari's patch to rename Diagnostics enums to avoid collision with Windows.h (See https://chromium-review.googlesource.com/#/c/177181/3) Change-Id: I2978d06ec96789b3ee1696b65a84c2a9f31f7ba4
Zhenyao Mo f1d723c6 2013-09-23T14:57:07 Clamp numeric overflow rather than failing with an error BUG=249086 ANGLEBUG=468 TEST= R=alokp@chromium.org, kbr@chromium.org Review URL: https://codereview.appspot.com/13195043
shannonwoods@chromium.org 2a5436ff 2013-05-30T00:21:41 Added preprocessor support for parsing the float suffix. TRAC #23185 Signed-off-by: Geoff Lang Signed-off-by: Shannon Woods Author: Nicolas Capens git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2413 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannonwoods@chromium.org 3f83e29c 2013-05-30T00:21:34 Added preprocessor support for parsing the unsigned integer suffix. TRAC #23185 Signed-off-by: Geoff Lang Signed-off-by: Shannon Woods Author: Nicolas Capens git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2412 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannonwoods@chromium.org b0757168 2013-05-30T00:21:27 Restore 64-bit support of the preprocessor tokenizer. TRAC #23185 Signed-off-by: Geoff Lang Signed-off-by: Shannon Woods Author: Nicolas Capens git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2411 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannonwoods@chromium.org c8100b85 2013-05-30T00:20:34 Add support for unsigned integer literals in the shading language. TRAC #23080 Signed-off-by: Nicolas Capens Signed-off-by: Shannon Woods Author: Jamie Madill git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2405 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannonwoods@chromium.org 8ddaf5c0 2013-05-30T00:13:08 Increase the maximum token length in the preprocessor to be 1024, to comply with the GLSL ES 3 specification. TRAC #23077 Signed-off-by: Shannon Woods Signed-off-by: Geoff Lang Author: Jamie Madill git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2349 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannonwoods@chromium.org 4b8a3115 2013-05-30T00:13:01 Regenerate the preprocessor source files with the new versions of the compiler tools. Will have to be resolved with 64-bit compatible bison in the future. TRAC #23077 Signed-off-by: Shannon Woods Signed-off-by: Geoff Lang Author: Jamie Madill git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2348 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannon.woods%transgaming.com@gtempaccount.com bcde56f7 2013-04-13T03:32:12 Added support for line continuations. TRAC #22713 Signed-off-by: Geoff Lang Signed-off-by: Shannon Woods Author: Nicolas Capens git-svn-id: https://angleproject.googlecode.com/svn/branches/es3proto@2086 736b8ea6-26fd-11df-bfd4-992fa37f6226
shannon.woods@transgaming.com d64b3dab 2013-02-28T23:19:26 Fixed 64-bit integer truncation issues in shader translator. This is an incompatible API change, but one which is necessary in order to improve correctness of the code. The API version in ShaderLang.h is updated and, unfortunately, the define renamed to something less ambiguous due to conflicts on some Android buildbots. Temporary patches in Chromium and WebKit will be landed separately to support this upgrade. BUG=403,404,405,406,407,408,409 Review URL: https://codereview.appspot.com/7300058 Conflicts: include/GLSLANG/ShaderLang.h git-svn-id: https://angleproject.googlecode.com/svn/branches/dx11proto@1960 736b8ea6-26fd-11df-bfd4-992fa37f6226
daniel@transgaming.com b3077d08 2013-01-11T04:12:09 Upstream various build fixes from WebKit to ANGLE to make updating ANGLE in WebKit easier. a) http://trac.webkit.org/changeset/127747 b) http://trac.webkit.org/changeset/128539 c) http://trac.webkit.org/changeset/122870 - Specifically, items #3 and #4 in this changeset's commit message. Review URL: https://codereview.appspot.com/7040045 Author: maxvujovic@gmail.com ------ Upodate preprocessor.vcxproj to reflect changes in r1640. Review URL: https://codereview.appspot.com/7061044 git-svn-id: https://angleproject.googlecode.com/svn/branches/dx11proto@1703 736b8ea6-26fd-11df-bfd4-992fa37f6226
daniel@transgaming.com b401a92b 2012-10-26T18:58:24 Move the new preprocessor out of the 'new' directory. TRAC #21966 Signed-off-by: Daniel Koch Author: Nicolas Capens git-svn-id: https://angleproject.googlecode.com/svn/trunk@1326 736b8ea6-26fd-11df-bfd4-992fa37f6226