kmx git

Commit	Date	Message
9b5cce7a	2023-09-21T00:44:50	include: Remove more unnecessary includes
11a1839d	2023-09-20T17:54:48	globals: Move remaining globals back to correct header files This undoes a lot of damage.
4e1c13eb	2023-09-18T14:45:10	debug: Remove debugging code This is barely useful these days and only clutters the code base.
e48f2695	2023-08-29T17:41:18	parser: Remove push parser debugging code
0d24fc0a	2023-08-14T12:53:49	html: Remove encoding hack in htmlCreateFileParserCtxt Switch encoding directly instead of calling htmlCheckEncoding with faked content.
5db5a704	2023-08-09T18:39:14	html: Fix UAF in htmlCurrentChar Short-lived regression found by OSS-Fuzz.
95e81a36	2023-08-08T15:21:31	parser: Decode all data in xmlCharEncInput Even with flush set to true, xmlCharEncInput didn't guarantee to decode all data. This complicated the push parser. Remove the flush flag and always decode all available data. Also fix ICU code where the flush flag has a different meaning. Always set flush to false and retry even with empty input buffers.
834b8123	2023-08-08T15:21:28	parser: Stream data when reading from memory Don't create a copy of the whole input buffer. Read the data chunk by chunk to save memory. Historically, it was probably envisioned to read data from memory without additional copying. This doesn't work reliably with the current design of the XML parser which requires a terminating null byte at the end of input buffers. This lead to xmlReadMemory interfaces, which expect pointer and size arguments, being changed to make a zero-terminated copy of the input buffer. Interfaces based on xmlReadDoc, which actually expect a zero-terminated string and would make zero-copy operation work, were then simplified to rely on xmlReadMemoryi, resulting in an unnecessary copy. To avoid copying (possibly gigabytes) of memory temporarily, we now stream in-memory input just like content read from files in a chunk-by-chunk fashion (using a somewhat outdated INPUT_CHUNK size of 250 bytes). As a side effect, we also avoid another copy of the whole input when handling non-UTF-8 data which was made possible by some earlier commits. Interfaces expecting zero-terminated strings now make use of strnlen which unfortunately isn't part of the standard C library and only mandated since POSIX 2008.
facc2a06	2023-08-08T15:21:21	parser: Don't overwrite EOF parser state
59fa0bb3	2023-08-08T15:21:14	parser: Simplify input pointer updates The base member always points to the beginning of the buffer.
ec7be506	2023-08-08T15:19:46	parser: Rework encoding detection Introduce XML_INPUT_HAS_ENCODING flag for xmlParserInput which is set when xmlSwitchEncoding is called. The parser can use the flag to reliably detect whether an encoding was already set via user override, BOM or other auto-detection. In this case, the encoding declaration won't be used to switch the encoding. Before, an inscrutable mix of ctxt->charset, ctxt->input->encoding and ctxt->input->buf->encoder was used. Introduce private helper functions to switch encodings used by both the XML and HTML parser: - xmlDetectEncoding which skips over the BOM, allowing to remove the BOM checks from other encoding functions. - xmlSetDeclaredEncoding, replacing htmlCheckEncodingDirect, which warns about encoding mismatches. If users override the encoding, store the declared instead of the actual encoding in xmlDoc. In this case, the actual encoding is known and the raw value from the doc is more useful. Also use the input flags to store the ISO-8859-1 fallback state. Restrict the fallback to cases where no encoding was specified. (The fallback is only useful in recovery mode and these days broken UTF-8 is probably more likely than ISO-8859-1, so it might eventually be removed completely.) The 'charset' member of xmlParserCtxt is now unused. The 'encoding' member of xmlParserInput is now unused. The 'standalone' member of xmlParserInput is renamed to 'flags'. A new parser state XML_PARSER_XML_DECL is added for the push parser.
3a64f394	2023-08-08T15:19:25	html: Remove some debugging code in htmlParseTryOrFinish
20f5c734	2023-06-07T14:05:34	parser: Recover more input from encoding errors Don't halt the parser in xmlParserGrow to allow more input to be recovered in case of encoding errors. Fixes #543.
320f5084	2023-04-30T18:25:09	parser: Improve handling of encoding and IO errors Make sure that xmlCharEncInput, xmlParserInputBufferPush and xmlParserInputBufferGrow set the correct error code in the xmlParserInputBuffer. Handle errors when calling these functions.
1061537e	2023-03-26T22:40:54	malloc-fail: Fix buffer overread with HTML doctype declarations Found by OSS-Fuzz, see #344.
7fbd454d	2023-03-21T13:26:36	parser: Grow input buffer earlier when reading characters Make more bytes available after invoking CUR_CHAR or NEXT.
04d1bedd	2023-03-21T13:08:44	parser: Rework shrinking of input buffers Don't try to grow the input buffer in xmlParserShrink. This makes sure that no memory allocations are made and the function always succeeds. Remove unnecessary invocations of SHRINK. Invoke SHRINK at the end of DTD parsing loops. Shrink before growing.
44ecefc8	2023-03-20T15:52:38	malloc-fail: Fix buffer overread after htmlParseScript Found by OSS-Fuzz, see #344.
067986fa	2023-03-18T14:44:28	parser: Fix regressions from previous commits - Fix memory leak in xmlParseNmtoken. - Fix buffer overread after htmlParseCharDataInternal.
9ef2a9ab	2023-03-17T14:06:28	html: Rely on CUR_CHAR to grow the input buffer - Remove useless invocations of GROW. - Add some error checks. - Fix invocations of SHRINK.
62f199ed	2023-03-17T12:40:46	malloc-fail: Add error check in htmlParseHTMLAttribute This function must return NULL is an error occurs. Found by OSS-Fuzz, see #344.
8090e585	2023-03-17T12:27:07	malloc-fail: Fix buffer overread in htmlParseScript Found by OSS-Fuzz, see #344.
ca2bfece	2023-03-15T16:18:11	malloc-fail: Fix buffer overread when reading from input Found by OSS-Fuzz, see #344.
4b3452d1	2023-03-15T16:56:36	html: Fix quadratic behavior in htmlParseTryOrFinish Fix check for end of script content. Found by OSS-Fuzz.
14c62e0d	2023-03-15T16:22:13	html: Use NEXTL in htmlParseHTMLAttribute This is more efficient than NEXT.
2099441f	2023-03-13T17:51:13	parser: Stop calling xmlParserInputShrink Introduce xmlParserShrink which takes a parser context to simplify error handling.
cabde70f	2023-03-12T19:07:23	parser: Simplify calculation of available buffer space
b75976e0	2023-03-12T19:06:19	parser: Use size_t when subtracting input buffer pointers Avoid integer overflows.
9a6ca816	2023-03-12T19:03:11	parser: Check for integer overflow when updating checkIndex Unfortunately, checkIndex is a long, not a size_t. Check for integer overflow before updating the value.
bd63d730	2023-03-12T17:40:55	html: Impose some length limits Impose length limits on names, attribute values, PIs and comments, similar to the XML parser.
3eb6bf03	2023-03-12T16:47:15	parser: Stop calling xmlParserInputGrow Introduce xmlParserGrow which takes a parser context to simplify error handling.
53d1cc98	2023-02-16T15:09:32	malloc-fail: Fix error code in htmlParseChunk Found with libFuzzer, see #344.
15b0ed08	2023-02-16T15:09:02	malloc-fail: Fix infinite loop in htmlParseDocTypeDecl Found with libFuzzer, see #344.
041789d9	2023-02-16T15:02:08	malloc-fail: Fix null deref in htmlnamePush Found with libFuzzer, see #344.
0ec9c910	2023-02-16T14:57:24	malloc-fail: Fix infinite loop in htmlParseStartTag Found with libFuzzer, see #344.
04c29551	2023-02-16T14:53:29	malloc-fail: Fix infinite loop in htmlParseContentInternal Found with libFuzzer, see #344.
f3e62035	2023-02-16T14:49:06	malloc-fail: Fix memory leak in htmlCreatePushParserCtxt Found with libFuzzer, see #344.
fc256953	2023-02-16T14:47:41	malloc-fail: Fix memory leak in htmlCreateMemoryParserCtxt Found with libFuzzer, see #344.
643b4e90	2023-02-16T14:45:06	malloc-fail: Fix infinite loop in htmlParseStartTag Found with libFuzzer, see #344.
59b33661	2022-12-27T14:15:51	error: Limit number of parser errors Reporting errors is expensive and some abusive test cases can generate an error for each invalid input byte. This causes the parser to spend most of the time with error handling. Limit the number of errors and warnings to 100.
4b959ee1	2022-12-01T13:23:09	Remove hacky heuristic from b2dc5675e94aa6b5557ba63f7d66b0f08dd17e4d Checking whether the context is close to the parent context by hardcoding 250 is not portable (I noticed tests were failing on Morello since the value is 288 there due to pointers being 128 bits). Instead we should ensure that the XML_VCTXT_USE_PCTXT flag is not set in cases where the user data is not actually a parser context (or ideally add a separate field but that would be an ABI break. From what I can see in the source, the XML_VCTXT_USE_PCTXT is only set if the userData field points to a valid context, and if this is not the case the flag should be cleared when changing userData rather than relying on the offset between the two. Looking at the history, I think d7cb33cf44aa688f24215c9cd398c1a26f0d25ff fixed most of the need for this workaround, but it looks like there are a few more locations that need updating; This commit changes two more places to set/clear/copy the XML_VCTXT_USE_PCTXT flag, so this heuristic should not be needed anymore. I've also drop two = NULL assignment in xmllint since this is not needed after a call to memset(). There was also an uninitialized vctxt.flags (and other fields) in `xmlShellValidate()`, which I've fixed by adding a memset() call.
c715ded0	2022-12-01T12:53:15	Avoid creating an out-of-bounds pointer by rewriting a check Creating more than one-past-the-end pointers is undefined behaviour in C and while this code is unlikely to be miscompiled, I discovered that an out-of-bounds pointer is being created using UBSan on a CHERI-enabled system.
c7a9b85c	2022-11-30T17:11:33	html: Improve parsing of nested lists Allow ul/ol as immediate children of ul/ol. This is more in line with the HTML5 spec. Fixes #447.
e414f825	2022-11-25T15:01:22	html: Fix htmlInitAutoClose documentation
c9367938	2022-11-20T19:54:34	html: Fix check for end of comment in push parser Make sure to reset checkIndex. Handle case where "--" or "--!" is at the end of the buffer. Fix "avail" check in htmlParseOrTryFinish.
68a6518c	2022-11-15T18:23:33	parser: Rewrite push parser boundary checks Remove inaccurate xmlParseCheckTransition check. Remove non-incremental xmlParseGetLasts check. Add functions that check for several boundary constructs more accurately, keeping track of progress in ctxt->checkIndex. Fixes #439.
6843fc72	2022-09-01T02:58:00	Remove or annotate char casts
2cac6269	2022-09-01T03:14:13	Don't use sizeof(xmlChar) or sizeof(char)
ad338ca7	2022-09-01T01:18:30	Remove explicit integer casts Remove explicit integer casts as final operation - in assignments - when passing arguments - when returning values Remove casts - to the same type - from certain range-bound values The main motivation is that these explicit casts don't change the result of operations and only render UBSan's implicit-conversion checks useless. Removing these casts allows UBSan to detect cases where truncation or sign-changes occur unexpectedly. Document some explicit casts as truncating and add a few missing ones.
65dc8a63	2022-09-01T00:13:19	Make xmlNewSAXParserCtx take a const sax handler Also improve documentation.
0f568c0b	2022-08-26T01:22:33	Consolidate private header files Private functions were previously declared - in header files in the root directory - in public headers guarded with IN_LIBXML - in libxml.h - redundantly in source files that used them. Consolidate all private header files in include/private.
58fc89e8	2022-08-25T20:57:30	Deprecate internal parser functions
a308c0cd	2022-08-25T20:18:16	Deprecate old HTML SAX API
9a82b94a	2022-08-24T04:21:58	Introduce xmlNewSAXParserCtxt and htmlNewSAXParserCtxt Add API functions to create a parser context with a custom SAX handler without having to mess with ctxt->sax manually.
0a04db19	2022-08-24T14:06:00	Don't mess with parser options in htmlParseDocument Don't set ctxt->html. This member should already be initialized. Set ctxt->linenumbers in htmlCtxtUseOptions like the XML parser does.
d45263a2	2022-08-24T14:04:35	Remove useless call to htmlDefaultSAXHandlerInit This function is already called from xmlInitParser.
4b184240	2022-08-22T14:11:15	Remove htmlDefaultSAXHandler from non-SAX1 build This matches long-standing behavior of the XML counterpart.
80bd34c3	2022-08-22T14:06:37	Don't initialize SAX handler in htmlReadMemory The SAX handler is already initialized when creating the parser context.
37cedc0b	2022-08-22T14:04:07	Fix htmlReadMemory mixing up XML and HTML functions Also see fe6890e2.
920753c4	2022-08-22T13:46:50	Don't use default SAX handler to report unrelated errors
38f04779	2022-08-22T13:33:35	Fix HTML parser with threads and --without-legacy If the legacy functions are disabled, the default "V1" HTML SAX handler isn't initialized in threads other than the main thread. htmlInitParserCtxt would later use the empty V1 SAX handler, resulting in NULL documents. Change htmlInitParserCtxt to initialize the HTML SAX handler by calling xmlSAX2InitHtmlDefaultSAXHandler. This removes the ability to change the default handler but is more in line with the XML parser which initializes the SAX handler by calling xmlSAXVersion, ignoring the V1 default handler. Fixes #399.
5b2d07a7	2022-08-20T17:00:50	Use xmlStrlen in *CtxtReadDoc xmlStrlen handles buffers larger than INT_MAX more gracefully.
4ad71c2d	2022-08-20T16:19:34	Fix xmlCtxtReadDoc with encoding xmlCtxtReadDoc used to create an input stream involving xmlNewStringInputStream. This would create a stream without an input buffer, causing problems with encodings (see #34). After commit aab584dc3, an error was returned even with UTF-8 encodings which happened to work before. Make xmlCtxtReadDoc call xmlCtxtReadMemory which doesn't suffer from these issues. Also fix htmlCtxtReadDoc. Fixes #397.
e986d09c	2022-07-15T14:02:26	Skip incorrectly opened HTML comments Commit 4fd69f3e fixed handling of '<' characters not followed by an ASCII letter. But a '<!' sequence followed by invalid characters should be treated as bogus comment and skipped. Fixes #380.
6722d22c	2022-07-15T13:26:41	Reduce indentation in HTMLparser.c No functional change.
a82ea25f	2022-07-28T21:35:17	Also reset nsNr in htmlCtxtReset
44e9118c	2022-04-08T12:33:17	Prevent integer-overflow in htmlSkipBlankChars() and xmlSkipBlankChars() * HTMLparser.c: (htmlSkipBlankChars): * parser.c: (xmlSkipBlankChars): - Cap the return value at INT_MAX. - The commit range that OSS-Fuzz listed for the fix didn't make any changes to xmlSkipBlankChars(), so it seems like this issue may still exist. Found by OSS-Fuzz Issue 44803.
40483d0c	2022-03-06T13:55:48	Deprecate module init and cleanup functions These functions shouldn't be part of the public API. Most init functions are only thread-safe when called from xmlInitParser. Global variables should only be cleaned up by calling xmlCleanupParser.
ebb17970	2022-03-04T02:31:59	Remove unneeded #includes
d7b287b9	2021-07-17T14:36:53	htmlParseComment: handle abruptly-closed comments See guidance provided on abrutply-closed comments here: https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment
776d15d3	2022-03-02T00:29:17	Don't check for standard C89 headers Don't check for - ctype.h - errno.h - float.h - limits.h - math.h - signal.h - stdarg.h - stdlib.h - string.h - time.h Stop including non-standard headers - malloc.h - strings.h
4fd69f3e	2022-02-22T18:15:53	Fix recovery from invalid HTML start tags Only try to parse a start tag if there's a '<' followed by an ASCII letter. This is more in line with HTML5 and the old behavior in recovery mode. Emit a literal '<' if the following character is invalid. Fixes #101. Fixes #339.
346c3a93	2022-02-20T18:46:42	Remove elfgcchack.h The same optimization can be enabled with -fno-semantic-interposition since GCC 5. clang has always used this option by default.
d7cb33cf	2022-01-13T17:06:14	Rework validation context flags Use a bitmask instead of magic values to - keep track whether the validation context is part of a parser context - keep track whether xmlValidateDtdFinal was called This allows to add addtional flags later. Note that this deliberately changes the name of a public struct member, assuming that this was always private data never to be used by client code.
96dc7f4a	2022-02-01T16:15:51	Also register HTML document nodes Fixes #196.
fe6890e2	2021-07-27T13:20:20	Fix htmlReadFd, which was using a mix of xml and html context functions
e7d1c53a	2021-07-14T11:32:57	Fix memory leak in xmlFreeParserInputBuffer Found by Coverity. https://bugzilla.redhat.com/show_bug.cgi?id=1938806
798bdf13	2022-01-10T14:50:20	Different approach to fix quadratic behavior in HTML push parser The old approach introduced a regression, see issue #312 and the previous commit. Disable code that tries to recover from invalid start tags. This only affects "recovery" mode. Add a comment outlining a better fix in accordance with the HTML5 spec.
094fc08a	2022-01-10T14:02:10	Fix regression when parsing invalid HTML tags in push mode Revert part of commit 173a0830 that changed behavior when parsing malformed start tags with the push parser. This reintroduces quadratic behavior in recovery mode which will be worked around in the next commit. Fixes #312.
2732b234	2022-01-10T13:32:14	Fix regression parsing public IDs literals in HTML Fix regression introduced when reworking htmlParsePubidLiteral in commit 93ce33c2. Fixes #318.
7279d236	2021-05-06T10:37:07	Fix htmlTagLookup Fix regression introduced with b25acce8. Some users like libxslt may call the HTML output functions on documents with uppercase tag names, so we must keep case-insensitive string comparison. Fixes #248.
683de7ef	2021-03-04T19:06:04	Fix duplicate xmlStrEqual calls in htmlParseEndTag
8095365b	2021-03-04T18:46:11	Speed up htmlCheckAutoClose Switch to binary search.
b25acce8	2021-03-04T17:44:45	Speed up htmlTagLookup Switch to binary search. This is the first time bsearch is used in the libxml2 code base. But it's a standard library function since C89 and should be portable.
0fb3ae58	2021-02-22T17:31:05	Revert "Improve HTML fuzzer stability" This reverts commit de1b51eddcc17fd7ed1bbcc6d5d7d529407dfbe2.
de1b51ed	2021-02-22T12:25:29	Improve HTML fuzzer stability Call htmlInitAutoClose during fuzzer initialization to fix stability issue. Leave a note concerning problems with this function.
dcb80b92	2021-02-20T20:30:43	Fix slow parsing of HTML with encoding errors Under certain circumstances, the HTML parser would try to guess and switch input encodings multiple times, leading to slow processing of documents with encoding errors. The repeated scanning of the input buffer when guessing encodings could even lead to quadratic behavior. The code htmlCurrentChar probably assumed that if there's an encoding handler, it is guaranteed to produce valid UTF-8. This holds true in general, but if the detected encoding was "UTF-8", the UTF8ToUTF8 encoding handler simply invoked memcpy without checking for invalid UTF-8. This still must be fixed, preferably by not using this handler at all. Also leave a note that switching encodings twice seems impossible to implement correctly. Add a check when handling UTF-8 encoding errors in htmlCurrentChar to avoid this situation, even if encoders produce invalid UTF-8. Found by OSS-Fuzz.
954696e7	2021-02-07T13:23:09	Fix infinite loop in HTML parser introduced with recent commits Check for XML_PARSER_EOF to avoid an infinite loop introduced with recent changes to the HTML push parser. Found by OSS-Fuzz.
a67b63d1	2020-10-11T14:15:37	use new htmlParseLookupCommentEnd to find comment ends Note that the caret in error messages generated during comment parsing may have moved by one byte. See guidance provided on incorrectly-closed comments here: https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment
29f5d20e	2020-08-03T17:36:05	htmlParseComment: treat `--!>` as if it closed the comment See guidance provided on incorrectly-closed comments here: https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment
94c2e415	2020-12-06T16:38:00	Fix quadratic runtime in HTML push parser with null bytes Null bytes in the input stream do not necessarily signal an EOF condition. Check the stream pointers for EOF to avoid quadratic rescanning of input data. Note that the CUR_CHAR macro used in functions like htmlParseCharData calls htmlCurrentChar which translates null bytes. Found by OSS-Fuzz.
438e595a	2020-08-09T14:43:53	Stop counting nbChars in parser context The value was inaccurate and never used.
f6a9541f	2020-08-09T14:29:35	Remove unneeded progress checks in HTML parser The HTML parser should now be guaranteed to make progress, so the checks became unnecessary.
93ce33c2	2020-07-23T17:34:08	Fix several quadratic runtime issues in HTML push parser Fix a few remaining cases where the HTML push parser would scan more content during lookahead than being parsed later. Make sure that htmlParseDocTypeDecl consumes all content up to the final '>' in case of errors. The old comment said "We shouldn't try to resynchronize", but ignoring invalid content is also what the HTML5 spec mandates. Likewise, make htmlParseEndTag skip to the final '>' in invalid end tags even if not in recovery mode. This is probably the most visible change in practice and leads to different output for some tests but is also more in line with HTML5. Make sure that htmlParsePI and htmlParseComment don't abort if invalid characters are encountered but log an error and ignore the character. Change some other end-of-buffer checks to test for a zero byte instead of relying on IS_CHAR. Fix usage of IS_CHAR macro in htmlParseScript.
173a0830	2020-07-22T23:15:35	Fix quadratic runtime when push parsing HTML start tags Make sure that htmlParseStartTag doesn't terminate on characters for which IS_CHAR_CH is false like control chars. In htmlParseTryOrFinish, only switch to START_TAG if the next character starts a valid name. Otherwise, htmlParseStartTag might return without consuming all characters up to the final '>'. Found by OSS-Fuzz.
6995eed0	2020-07-19T13:54:52	Fix quadratic runtime when push parsing HTML entity refs The HTML push parser would look ahead for characters in "; >/" to terminate an entity reference but actual parsing could stop earlier, potentially resulting in quadratic runtime. Parse char data and references alternately in htmlParseTryOrFinish and only look ahead once for a terminating '<' character. Found by OSS-Fuzz.
8e219b15	2020-07-12T21:43:44	Fix HTML push parser lookahead The parsing rules when looking for terminating chars or sequences in the push parser differed from the actual parsing code. This could result in the lookahead to overshoot and data being rescanned, potentially leading to quadratic runtime. Comments must never be handled during lookahead. Attribute values must only be skipped for start tags and doctype declarations, not for end tags, comments, PIs and script content.
e050062c	2020-07-15T14:38:55	Make htmlCurrentChar always translate U+0000 The general assumption is that htmlCurrentChar only returns 0 if the end of the input buffer is reached. The UTF-8 path already logged an error if a zero byte U+0000 was found and returned a space character instead. Make the ASCII code path do the same. htmlParseTryOrFinish skips zero bytes at the beginning of a buffer, so even if 0 was returned from htmlCurrentChar, the push parser would make progress. But rescanning the input could cause performance problems. The pull parser would abort parsing and now handles zero bytes in ASCII mode the same way as the push parser or as in UTF-8 mode. It would be better to return the replacement character U+FFFD instead, but some of the client code assumes that the UTF-8 length of input and output matches.
dfd4e330	2020-07-15T14:22:08	Rework control flow in htmlCurrentChar Don't call xmlCurrentChar after switching encodings. Rearrange code blocks and fall through to normal UTF-8 handling.
1493130e	2020-07-15T12:54:25	Fix UTF-8 decoder in HTML parser Reject sequences starting with a continuation byte as well as overlong sequences like the XML parser. Also fixes an infinite loop in connection with previous commit 50078922 since htmlCurrentChar would return 0 even if not at the end of the buffer. Found by OSS-Fuzz.

9b5cce7a

2023-09-21T00:44:50

include: Remove more unnecessary includes

11a1839d

2023-09-20T17:54:48

globals: Move remaining globals back to correct header files This undoes a lot of damage.

4e1c13eb

2023-09-18T14:45:10

debug: Remove debugging code This is barely useful these days and only clutters the code base.

e48f2695

2023-08-29T17:41:18

parser: Remove push parser debugging code

0d24fc0a

2023-08-14T12:53:49

html: Remove encoding hack in htmlCreateFileParserCtxt Switch encoding directly instead of calling htmlCheckEncoding with faked content.

5db5a704

2023-08-09T18:39:14

html: Fix UAF in htmlCurrentChar Short-lived regression found by OSS-Fuzz.

95e81a36

2023-08-08T15:21:31

parser: Decode all data in xmlCharEncInput Even with flush set to true, xmlCharEncInput didn't guarantee to decode all data. This complicated the push parser. Remove the flush flag and always decode all available data. Also fix ICU code where the flush flag has a different meaning. Always set flush to false and retry even with empty input buffers.

834b8123

2023-08-08T15:21:28

parser: Stream data when reading from memory Don't create a copy of the whole input buffer. Read the data chunk by chunk to save memory. Historically, it was probably envisioned to read data from memory without additional copying. This doesn't work reliably with the current design of the XML parser which requires a terminating null byte at the end of input buffers. This lead to xmlReadMemory interfaces, which expect pointer and size arguments, being changed to make a zero-terminated copy of the input buffer. Interfaces based on xmlReadDoc, which actually expect a zero-terminated string and would make zero-copy operation work, were then simplified to rely on xmlReadMemoryi, resulting in an unnecessary copy. To avoid copying (possibly gigabytes) of memory temporarily, we now stream in-memory input just like content read from files in a chunk-by-chunk fashion (using a somewhat outdated INPUT_CHUNK size of 250 bytes). As a side effect, we also avoid another copy of the whole input when handling non-UTF-8 data which was made possible by some earlier commits. Interfaces expecting zero-terminated strings now make use of strnlen which unfortunately isn't part of the standard C library and only mandated since POSIX 2008.

facc2a06

2023-08-08T15:21:21

parser: Don't overwrite EOF parser state

59fa0bb3

2023-08-08T15:21:14

parser: Simplify input pointer updates The base member always points to the beginning of the buffer.

ec7be506

2023-08-08T15:19:46

parser: Rework encoding detection Introduce XML_INPUT_HAS_ENCODING flag for xmlParserInput which is set when xmlSwitchEncoding is called. The parser can use the flag to reliably detect whether an encoding was already set via user override, BOM or other auto-detection. In this case, the encoding declaration won't be used to switch the encoding. Before, an inscrutable mix of ctxt->charset, ctxt->input->encoding and ctxt->input->buf->encoder was used. Introduce private helper functions to switch encodings used by both the XML and HTML parser: - xmlDetectEncoding which skips over the BOM, allowing to remove the BOM checks from other encoding functions. - xmlSetDeclaredEncoding, replacing htmlCheckEncodingDirect, which warns about encoding mismatches. If users override the encoding, store the declared instead of the actual encoding in xmlDoc. In this case, the actual encoding is known and the raw value from the doc is more useful. Also use the input flags to store the ISO-8859-1 fallback state. Restrict the fallback to cases where no encoding was specified. (The fallback is only useful in recovery mode and these days broken UTF-8 is probably more likely than ISO-8859-1, so it might eventually be removed completely.) The 'charset' member of xmlParserCtxt is now unused. The 'encoding' member of xmlParserInput is now unused. The 'standalone' member of xmlParserInput is renamed to 'flags'. A new parser state XML_PARSER_XML_DECL is added for the push parser.

3a64f394

2023-08-08T15:19:25

html: Remove some debugging code in htmlParseTryOrFinish

20f5c734

2023-06-07T14:05:34

parser: Recover more input from encoding errors Don't halt the parser in xmlParserGrow to allow more input to be recovered in case of encoding errors. Fixes #543.

320f5084

2023-04-30T18:25:09

parser: Improve handling of encoding and IO errors Make sure that xmlCharEncInput, xmlParserInputBufferPush and xmlParserInputBufferGrow set the correct error code in the xmlParserInputBuffer. Handle errors when calling these functions.

1061537e

2023-03-26T22:40:54

malloc-fail: Fix buffer overread with HTML doctype declarations Found by OSS-Fuzz, see #344.

7fbd454d

2023-03-21T13:26:36

parser: Grow input buffer earlier when reading characters Make more bytes available after invoking CUR_CHAR or NEXT.

04d1bedd

2023-03-21T13:08:44

parser: Rework shrinking of input buffers Don't try to grow the input buffer in xmlParserShrink. This makes sure that no memory allocations are made and the function always succeeds. Remove unnecessary invocations of SHRINK. Invoke SHRINK at the end of DTD parsing loops. Shrink before growing.

44ecefc8

2023-03-20T15:52:38

malloc-fail: Fix buffer overread after htmlParseScript Found by OSS-Fuzz, see #344.

067986fa

2023-03-18T14:44:28

parser: Fix regressions from previous commits - Fix memory leak in xmlParseNmtoken. - Fix buffer overread after htmlParseCharDataInternal.

9ef2a9ab

2023-03-17T14:06:28

html: Rely on CUR_CHAR to grow the input buffer - Remove useless invocations of GROW. - Add some error checks. - Fix invocations of SHRINK.

62f199ed

2023-03-17T12:40:46

malloc-fail: Add error check in htmlParseHTMLAttribute This function must return NULL is an error occurs. Found by OSS-Fuzz, see #344.

8090e585

2023-03-17T12:27:07

malloc-fail: Fix buffer overread in htmlParseScript Found by OSS-Fuzz, see #344.

ca2bfece

2023-03-15T16:18:11

malloc-fail: Fix buffer overread when reading from input Found by OSS-Fuzz, see #344.

4b3452d1

2023-03-15T16:56:36

html: Fix quadratic behavior in htmlParseTryOrFinish Fix check for end of script content. Found by OSS-Fuzz.

14c62e0d

2023-03-15T16:22:13

html: Use NEXTL in htmlParseHTMLAttribute This is more efficient than NEXT.

2099441f

2023-03-13T17:51:13

parser: Stop calling xmlParserInputShrink Introduce xmlParserShrink which takes a parser context to simplify error handling.

cabde70f

2023-03-12T19:07:23

parser: Simplify calculation of available buffer space

b75976e0

2023-03-12T19:06:19

parser: Use size_t when subtracting input buffer pointers Avoid integer overflows.

9a6ca816

2023-03-12T19:03:11

parser: Check for integer overflow when updating checkIndex Unfortunately, checkIndex is a long, not a size_t. Check for integer overflow before updating the value.

bd63d730

2023-03-12T17:40:55

html: Impose some length limits Impose length limits on names, attribute values, PIs and comments, similar to the XML parser.

3eb6bf03

2023-03-12T16:47:15

parser: Stop calling xmlParserInputGrow Introduce xmlParserGrow which takes a parser context to simplify error handling.

53d1cc98

2023-02-16T15:09:32

malloc-fail: Fix error code in htmlParseChunk Found with libFuzzer, see #344.

15b0ed08

2023-02-16T15:09:02

malloc-fail: Fix infinite loop in htmlParseDocTypeDecl Found with libFuzzer, see #344.

041789d9

2023-02-16T15:02:08

malloc-fail: Fix null deref in htmlnamePush Found with libFuzzer, see #344.

0ec9c910

2023-02-16T14:57:24

malloc-fail: Fix infinite loop in htmlParseStartTag Found with libFuzzer, see #344.

04c29551

2023-02-16T14:53:29

malloc-fail: Fix infinite loop in htmlParseContentInternal Found with libFuzzer, see #344.

f3e62035

2023-02-16T14:49:06

malloc-fail: Fix memory leak in htmlCreatePushParserCtxt Found with libFuzzer, see #344.

fc256953

2023-02-16T14:47:41

malloc-fail: Fix memory leak in htmlCreateMemoryParserCtxt Found with libFuzzer, see #344.

643b4e90

2023-02-16T14:45:06

malloc-fail: Fix infinite loop in htmlParseStartTag Found with libFuzzer, see #344.

59b33661

2022-12-27T14:15:51

error: Limit number of parser errors Reporting errors is expensive and some abusive test cases can generate an error for each invalid input byte. This causes the parser to spend most of the time with error handling. Limit the number of errors and warnings to 100.

4b959ee1

2022-12-01T13:23:09

Remove hacky heuristic from b2dc5675e94aa6b5557ba63f7d66b0f08dd17e4d Checking whether the context is close to the parent context by hardcoding 250 is not portable (I noticed tests were failing on Morello since the value is 288 there due to pointers being 128 bits). Instead we should ensure that the XML_VCTXT_USE_PCTXT flag is not set in cases where the user data is not actually a parser context (or ideally add a separate field but that would be an ABI break. From what I can see in the source, the XML_VCTXT_USE_PCTXT is only set if the userData field points to a valid context, and if this is not the case the flag should be cleared when changing userData rather than relying on the offset between the two. Looking at the history, I think d7cb33cf44aa688f24215c9cd398c1a26f0d25ff fixed most of the need for this workaround, but it looks like there are a few more locations that need updating; This commit changes two more places to set/clear/copy the XML_VCTXT_USE_PCTXT flag, so this heuristic should not be needed anymore. I've also drop two = NULL assignment in xmllint since this is not needed after a call to memset(). There was also an uninitialized vctxt.flags (and other fields) in `xmlShellValidate()`, which I've fixed by adding a memset() call.

c715ded0

2022-12-01T12:53:15

Avoid creating an out-of-bounds pointer by rewriting a check Creating more than one-past-the-end pointers is undefined behaviour in C and while this code is unlikely to be miscompiled, I discovered that an out-of-bounds pointer is being created using UBSan on a CHERI-enabled system.

c7a9b85c

2022-11-30T17:11:33

html: Improve parsing of nested lists Allow ul/ol as immediate children of ul/ol. This is more in line with the HTML5 spec. Fixes #447.

e414f825

2022-11-25T15:01:22

html: Fix htmlInitAutoClose documentation

c9367938

2022-11-20T19:54:34

html: Fix check for end of comment in push parser Make sure to reset checkIndex. Handle case where "--" or "--!" is at the end of the buffer. Fix "avail" check in htmlParseOrTryFinish.

68a6518c

2022-11-15T18:23:33

parser: Rewrite push parser boundary checks Remove inaccurate xmlParseCheckTransition check. Remove non-incremental xmlParseGetLasts check. Add functions that check for several boundary constructs more accurately, keeping track of progress in ctxt->checkIndex. Fixes #439.

6843fc72

2022-09-01T02:58:00

Remove or annotate char casts

2cac6269

2022-09-01T03:14:13

Don't use sizeof(xmlChar) or sizeof(char)

ad338ca7

2022-09-01T01:18:30

Remove explicit integer casts Remove explicit integer casts as final operation - in assignments - when passing arguments - when returning values Remove casts - to the same type - from certain range-bound values The main motivation is that these explicit casts don't change the result of operations and only render UBSan's implicit-conversion checks useless. Removing these casts allows UBSan to detect cases where truncation or sign-changes occur unexpectedly. Document some explicit casts as truncating and add a few missing ones.

65dc8a63

2022-09-01T00:13:19

Make xmlNewSAXParserCtx take a const sax handler Also improve documentation.

0f568c0b

2022-08-26T01:22:33

Consolidate private header files Private functions were previously declared - in header files in the root directory - in public headers guarded with IN_LIBXML - in libxml.h - redundantly in source files that used them. Consolidate all private header files in include/private.

58fc89e8

2022-08-25T20:57:30

Deprecate internal parser functions

a308c0cd

2022-08-25T20:18:16

Deprecate old HTML SAX API

9a82b94a

2022-08-24T04:21:58

Introduce xmlNewSAXParserCtxt and htmlNewSAXParserCtxt Add API functions to create a parser context with a custom SAX handler without having to mess with ctxt->sax manually.

0a04db19

2022-08-24T14:06:00

Don't mess with parser options in htmlParseDocument Don't set ctxt->html. This member should already be initialized. Set ctxt->linenumbers in htmlCtxtUseOptions like the XML parser does.

d45263a2

2022-08-24T14:04:35

Remove useless call to htmlDefaultSAXHandlerInit This function is already called from xmlInitParser.

4b184240

2022-08-22T14:11:15

Remove htmlDefaultSAXHandler from non-SAX1 build This matches long-standing behavior of the XML counterpart.

80bd34c3

2022-08-22T14:06:37

Don't initialize SAX handler in htmlReadMemory The SAX handler is already initialized when creating the parser context.

37cedc0b

2022-08-22T14:04:07

Fix htmlReadMemory mixing up XML and HTML functions Also see fe6890e2.

920753c4

2022-08-22T13:46:50

Don't use default SAX handler to report unrelated errors

38f04779

2022-08-22T13:33:35

Fix HTML parser with threads and --without-legacy If the legacy functions are disabled, the default "V1" HTML SAX handler isn't initialized in threads other than the main thread. htmlInitParserCtxt would later use the empty V1 SAX handler, resulting in NULL documents. Change htmlInitParserCtxt to initialize the HTML SAX handler by calling xmlSAX2InitHtmlDefaultSAXHandler. This removes the ability to change the default handler but is more in line with the XML parser which initializes the SAX handler by calling xmlSAXVersion, ignoring the V1 default handler. Fixes #399.

5b2d07a7

2022-08-20T17:00:50

Use xmlStrlen in *CtxtReadDoc xmlStrlen handles buffers larger than INT_MAX more gracefully.

4ad71c2d

2022-08-20T16:19:34

Fix xmlCtxtReadDoc with encoding xmlCtxtReadDoc used to create an input stream involving xmlNewStringInputStream. This would create a stream without an input buffer, causing problems with encodings (see #34). After commit aab584dc3, an error was returned even with UTF-8 encodings which happened to work before. Make xmlCtxtReadDoc call xmlCtxtReadMemory which doesn't suffer from these issues. Also fix htmlCtxtReadDoc. Fixes #397.

e986d09c

2022-07-15T14:02:26

Skip incorrectly opened HTML comments Commit 4fd69f3e fixed handling of '<' characters not followed by an ASCII letter. But a '<!' sequence followed by invalid characters should be treated as bogus comment and skipped. Fixes #380.

6722d22c

2022-07-15T13:26:41

Reduce indentation in HTMLparser.c No functional change.

a82ea25f

2022-07-28T21:35:17

Also reset nsNr in htmlCtxtReset

44e9118c

2022-04-08T12:33:17

Prevent integer-overflow in htmlSkipBlankChars() and xmlSkipBlankChars() * HTMLparser.c: (htmlSkipBlankChars): * parser.c: (xmlSkipBlankChars): - Cap the return value at INT_MAX. - The commit range that OSS-Fuzz listed for the fix didn't make any changes to xmlSkipBlankChars(), so it seems like this issue may still exist. Found by OSS-Fuzz Issue 44803.

40483d0c

2022-03-06T13:55:48

Deprecate module init and cleanup functions These functions shouldn't be part of the public API. Most init functions are only thread-safe when called from xmlInitParser. Global variables should only be cleaned up by calling xmlCleanupParser.

ebb17970

2022-03-04T02:31:59

Remove unneeded #includes

d7b287b9

2021-07-17T14:36:53

htmlParseComment: handle abruptly-closed comments See guidance provided on abrutply-closed comments here: https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment

776d15d3

2022-03-02T00:29:17

Don't check for standard C89 headers Don't check for - ctype.h - errno.h - float.h - limits.h - math.h - signal.h - stdarg.h - stdlib.h - string.h - time.h Stop including non-standard headers - malloc.h - strings.h

4fd69f3e

2022-02-22T18:15:53

Fix recovery from invalid HTML start tags Only try to parse a start tag if there's a '<' followed by an ASCII letter. This is more in line with HTML5 and the old behavior in recovery mode. Emit a literal '<' if the following character is invalid. Fixes #101. Fixes #339.

346c3a93

2022-02-20T18:46:42

Remove elfgcchack.h The same optimization can be enabled with -fno-semantic-interposition since GCC 5. clang has always used this option by default.

d7cb33cf

2022-01-13T17:06:14

Rework validation context flags Use a bitmask instead of magic values to - keep track whether the validation context is part of a parser context - keep track whether xmlValidateDtdFinal was called This allows to add addtional flags later. Note that this deliberately changes the name of a public struct member, assuming that this was always private data never to be used by client code.

96dc7f4a

2022-02-01T16:15:51

Also register HTML document nodes Fixes #196.

fe6890e2

2021-07-27T13:20:20

Fix htmlReadFd, which was using a mix of xml and html context functions

e7d1c53a

2021-07-14T11:32:57

Fix memory leak in xmlFreeParserInputBuffer Found by Coverity. https://bugzilla.redhat.com/show_bug.cgi?id=1938806

798bdf13

2022-01-10T14:50:20

Different approach to fix quadratic behavior in HTML push parser The old approach introduced a regression, see issue #312 and the previous commit. Disable code that tries to recover from invalid start tags. This only affects "recovery" mode. Add a comment outlining a better fix in accordance with the HTML5 spec.

094fc08a

2022-01-10T14:02:10

Fix regression when parsing invalid HTML tags in push mode Revert part of commit 173a0830 that changed behavior when parsing malformed start tags with the push parser. This reintroduces quadratic behavior in recovery mode which will be worked around in the next commit. Fixes #312.

2732b234

2022-01-10T13:32:14

Fix regression parsing public IDs literals in HTML Fix regression introduced when reworking htmlParsePubidLiteral in commit 93ce33c2. Fixes #318.

7279d236

2021-05-06T10:37:07

Fix htmlTagLookup Fix regression introduced with b25acce8. Some users like libxslt may call the HTML output functions on documents with uppercase tag names, so we must keep case-insensitive string comparison. Fixes #248.

683de7ef

2021-03-04T19:06:04

Fix duplicate xmlStrEqual calls in htmlParseEndTag

8095365b

2021-03-04T18:46:11

Speed up htmlCheckAutoClose Switch to binary search.

b25acce8

2021-03-04T17:44:45

Speed up htmlTagLookup Switch to binary search. This is the first time bsearch is used in the libxml2 code base. But it's a standard library function since C89 and should be portable.

0fb3ae58

2021-02-22T17:31:05

Revert "Improve HTML fuzzer stability" This reverts commit de1b51eddcc17fd7ed1bbcc6d5d7d529407dfbe2.

de1b51ed

2021-02-22T12:25:29

Improve HTML fuzzer stability Call htmlInitAutoClose during fuzzer initialization to fix stability issue. Leave a note concerning problems with this function.

dcb80b92

2021-02-20T20:30:43

Fix slow parsing of HTML with encoding errors Under certain circumstances, the HTML parser would try to guess and switch input encodings multiple times, leading to slow processing of documents with encoding errors. The repeated scanning of the input buffer when guessing encodings could even lead to quadratic behavior. The code htmlCurrentChar probably assumed that if there's an encoding handler, it is guaranteed to produce valid UTF-8. This holds true in general, but if the detected encoding was "UTF-8", the UTF8ToUTF8 encoding handler simply invoked memcpy without checking for invalid UTF-8. This still must be fixed, preferably by not using this handler at all. Also leave a note that switching encodings twice seems impossible to implement correctly. Add a check when handling UTF-8 encoding errors in htmlCurrentChar to avoid this situation, even if encoders produce invalid UTF-8. Found by OSS-Fuzz.

954696e7

2021-02-07T13:23:09

Fix infinite loop in HTML parser introduced with recent commits Check for XML_PARSER_EOF to avoid an infinite loop introduced with recent changes to the HTML push parser. Found by OSS-Fuzz.

a67b63d1

2020-10-11T14:15:37

use new htmlParseLookupCommentEnd to find comment ends Note that the caret in error messages generated during comment parsing may have moved by one byte. See guidance provided on incorrectly-closed comments here: https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment

29f5d20e

2020-08-03T17:36:05

htmlParseComment: treat `--!>` as if it closed the comment See guidance provided on incorrectly-closed comments here: https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment

94c2e415

2020-12-06T16:38:00

Fix quadratic runtime in HTML push parser with null bytes Null bytes in the input stream do not necessarily signal an EOF condition. Check the stream pointers for EOF to avoid quadratic rescanning of input data. Note that the CUR_CHAR macro used in functions like htmlParseCharData calls htmlCurrentChar which translates null bytes. Found by OSS-Fuzz.

438e595a

2020-08-09T14:43:53

Stop counting nbChars in parser context The value was inaccurate and never used.

f6a9541f

2020-08-09T14:29:35

Remove unneeded progress checks in HTML parser The HTML parser should now be guaranteed to make progress, so the checks became unnecessary.

93ce33c2

2020-07-23T17:34:08

Fix several quadratic runtime issues in HTML push parser Fix a few remaining cases where the HTML push parser would scan more content during lookahead than being parsed later. Make sure that htmlParseDocTypeDecl consumes all content up to the final '>' in case of errors. The old comment said "We shouldn't try to resynchronize", but ignoring invalid content is also what the HTML5 spec mandates. Likewise, make htmlParseEndTag skip to the final '>' in invalid end tags even if not in recovery mode. This is probably the most visible change in practice and leads to different output for some tests but is also more in line with HTML5. Make sure that htmlParsePI and htmlParseComment don't abort if invalid characters are encountered but log an error and ignore the character. Change some other end-of-buffer checks to test for a zero byte instead of relying on IS_CHAR. Fix usage of IS_CHAR macro in htmlParseScript.

173a0830

2020-07-22T23:15:35

Fix quadratic runtime when push parsing HTML start tags Make sure that htmlParseStartTag doesn't terminate on characters for which IS_CHAR_CH is false like control chars. In htmlParseTryOrFinish, only switch to START_TAG if the next character starts a valid name. Otherwise, htmlParseStartTag might return without consuming all characters up to the final '>'. Found by OSS-Fuzz.

6995eed0

2020-07-19T13:54:52

Fix quadratic runtime when push parsing HTML entity refs The HTML push parser would look ahead for characters in "; >/" to terminate an entity reference but actual parsing could stop earlier, potentially resulting in quadratic runtime. Parse char data and references alternately in htmlParseTryOrFinish and only look ahead once for a terminating '<' character. Found by OSS-Fuzz.

8e219b15

2020-07-12T21:43:44

Fix HTML push parser lookahead The parsing rules when looking for terminating chars or sequences in the push parser differed from the actual parsing code. This could result in the lookahead to overshoot and data being rescanned, potentially leading to quadratic runtime. Comments must never be handled during lookahead. Attribute values must only be skipped for start tags and doctype declarations, not for end tags, comments, PIs and script content.

e050062c

2020-07-15T14:38:55

Make htmlCurrentChar always translate U+0000 The general assumption is that htmlCurrentChar only returns 0 if the end of the input buffer is reached. The UTF-8 path already logged an error if a zero byte U+0000 was found and returned a space character instead. Make the ASCII code path do the same. htmlParseTryOrFinish skips zero bytes at the beginning of a buffer, so even if 0 was returned from htmlCurrentChar, the push parser would make progress. But rescanning the input could cause performance problems. The pull parser would abort parsing and now handles zero bytes in ASCII mode the same way as the push parser or as in UTF-8 mode. It would be better to return the replacement character U+FFFD instead, but some of the client code assumes that the UTF-8 length of input and output matches.

dfd4e330

2020-07-15T14:22:08

Rework control flow in htmlCurrentChar Don't call xmlCurrentChar after switching encodings. Rearrange code blocks and fall through to normal UTF-8 handling.

1493130e

2020-07-15T12:54:25

Fix UTF-8 decoder in HTML parser Reject sequences starting with a continuation byte as well as overlong sequences like the XML parser. Also fixes an infinite loop in connection with previous commit 50078922 since htmlCurrentChar would return 0 even if not at the end of the buffer. Found by OSS-Fuzz.

kc3-lang/libxml2/HTMLparser.c

HTMLparser.c

Log