kmx git

Commit	Date	Message
e0dd330b	2023-09-29T00:18:44	parser: Use hash tables to avoid quadratic behavior Use a hash table to lookup namespaces by prefix. The hash table stores an index into the namespace table. Auxiliary data for namespaces is stored in a separate array along the main namespace table. Use a hash table to verify attribute uniqueness. The hash table stores an index into the attribute table. Reuse hash value from the dictionary to avoid computing them twice. See #346.
cb927e85	2023-09-25T14:35:43	parser: Don't skip CR in xmlCurrentChar Skip over carriage returns later in xmlNextChar.
b9db3d7d	2023-09-22T15:45:20	parser: Simplify xmlStringCurrentChar Start to move away from using this function.
8c084ebd	2023-09-21T22:57:33	doc: Make apibuild.py happy
9b5cce7a	2023-09-21T00:44:50	include: Remove more unnecessary includes
699299ca	2023-09-20T18:54:39	globals: Stop including globals.h
11a1839d	2023-09-20T17:54:48	globals: Move remaining globals back to correct header files This undoes a lot of damage.
4e1c13eb	2023-09-18T14:45:10	debug: Remove debugging code This is barely useful these days and only clutters the code base.
e48f2695	2023-08-29T17:41:18	parser: Remove push parser debugging code
ed3bd052	2023-08-20T20:48:10	parser: Allow to set maximum amplification factor
f1c1f5c6	2023-08-16T19:43:02	parser: Revert change to doc->encoding Fixes #579.
61b8e097	2023-08-16T19:20:47	parser: Never use UTF-8 encoding handler
b973ceaf	2023-08-09T18:37:20	parser: Fix mistake in xmlDetectEncoding Short-lived regression.
95e81a36	2023-08-08T15:21:31	parser: Decode all data in xmlCharEncInput Even with flush set to true, xmlCharEncInput didn't guarantee to decode all data. This complicated the push parser. Remove the flush flag and always decode all available data. Also fix ICU code where the flush flag has a different meaning. Always set flush to false and retry even with empty input buffers.
834b8123	2023-08-08T15:21:28	parser: Stream data when reading from memory Don't create a copy of the whole input buffer. Read the data chunk by chunk to save memory. Historically, it was probably envisioned to read data from memory without additional copying. This doesn't work reliably with the current design of the XML parser which requires a terminating null byte at the end of input buffers. This lead to xmlReadMemory interfaces, which expect pointer and size arguments, being changed to make a zero-terminated copy of the input buffer. Interfaces based on xmlReadDoc, which actually expect a zero-terminated string and would make zero-copy operation work, were then simplified to rely on xmlReadMemoryi, resulting in an unnecessary copy. To avoid copying (possibly gigabytes) of memory temporarily, we now stream in-memory input just like content read from files in a chunk-by-chunk fashion (using a somewhat outdated INPUT_CHUNK size of 250 bytes). As a side effect, we also avoid another copy of the whole input when handling non-UTF-8 data which was made possible by some earlier commits. Interfaces expecting zero-terminated strings now make use of strnlen which unfortunately isn't part of the standard C library and only mandated since POSIX 2008.
59fa0bb3	2023-08-08T15:21:14	parser: Simplify input pointer updates The base member always points to the beginning of the buffer.
4ee08155	2023-08-08T15:19:51	encoding: Move rawconsumed accounting to xmlCharEncInput
ec7be506	2023-08-08T15:19:46	parser: Rework encoding detection Introduce XML_INPUT_HAS_ENCODING flag for xmlParserInput which is set when xmlSwitchEncoding is called. The parser can use the flag to reliably detect whether an encoding was already set via user override, BOM or other auto-detection. In this case, the encoding declaration won't be used to switch the encoding. Before, an inscrutable mix of ctxt->charset, ctxt->input->encoding and ctxt->input->buf->encoder was used. Introduce private helper functions to switch encodings used by both the XML and HTML parser: - xmlDetectEncoding which skips over the BOM, allowing to remove the BOM checks from other encoding functions. - xmlSetDeclaredEncoding, replacing htmlCheckEncodingDirect, which warns about encoding mismatches. If users override the encoding, store the declared instead of the actual encoding in xmlDoc. In this case, the actual encoding is known and the raw value from the doc is more useful. Also use the input flags to store the ISO-8859-1 fallback state. Restrict the fallback to cases where no encoding was specified. (The fallback is only useful in recovery mode and these days broken UTF-8 is probably more likely than ISO-8859-1, so it might eventually be removed completely.) The 'charset' member of xmlParserCtxt is now unused. The 'encoding' member of xmlParserInput is now unused. The 'standalone' member of xmlParserInput is renamed to 'flags'. A new parser state XML_PARSER_XML_DECL is added for the push parser.
131d0dc0	2023-08-08T15:19:39	parser: Don't use 'standalone' member of xmlParserInput The standalone declaration is only parsed in the main input stream.
88447447	2023-06-23T23:04:30	parser: Fix typo in previous commit
9d0541dd	2023-06-22T18:06:53	parser: Make xmlSwitchEncoding always skip the BOM Chromium calls xmlSwitchEncoding from the start document handler and relies on this function to skip the BOM. Commit 98840d40 changed the behavior when switching to UTF-16 since inspecting the input buffer at this point is fragile. Revert part of the commit to also skip a potential (decoded UTF-8) BOM when switching to UTF-16. Make sure that we do this only at the start of an input stream to avoid U-FEFF characters being lost. BOM handling should ultimately be moved to the parsing code to avoid such bugs. See https://bugs.chromium.org/p/chromium/issues/detail?id=1451026
b236b7a5	2023-06-08T21:53:05	parser: Halt parser when growing buffer results in OOM Fix short-lived regression from previous commit. It might be safer to make xmlBufSetInputBaseCur use the original buffer even in case of errors. Found by OSS-Fuzz.
20f5c734	2023-06-07T14:05:34	parser: Recover more input from encoding errors Don't halt the parser in xmlParserGrow to allow more input to be recovered in case of encoding errors. Fixes #543.
e0f3016f	2023-05-18T17:31:44	parser: Fix regression when push parsing UTF-8 sequences Partial UTF-8 sequences are allowed when push parsing. Fixes #542.
9dae389c	2023-05-09T13:28:06	parser: Fix "huge input lookup" error with push parser Fix parsing of larger documents without XML_PARSE_HUGE. Should fix #538.
320f5084	2023-04-30T18:25:09	parser: Improve handling of encoding and IO errors Make sure that xmlCharEncInput, xmlParserInputBufferPush and xmlParserInputBufferGrow set the correct error code in the xmlParserInputBuffer. Handle errors when calling these functions.
fc69cf56	2023-04-30T17:51:29	parser: Move xmlFatalErr to parserInternals.c
3ffcc03b	2023-03-13T19:38:41	parser: Deprecate more internal functions
9282b084	2023-04-19T21:55:24	parser: Fix regression in memory pull parser with encoding Revert another change from commit 98840d40. Decode the whole buffer when reading from memory and switching to the initial encoding. Add some comments about potential improvements.
a19fa11e	2023-04-13T15:11:47	parser: Fix regression when switching input encodings Revert some changes from commit 98840d40. WebKit/Chromium can actually switch from ISO-8859-1 to UTF-16 in the middle of parsing. This is a bad idea, but we have to keep supporting this use case.
921796b0	2023-04-12T13:43:28	parser: Don't grow push parser buffers This should fix a short-lived regression when push parsing with encodings.
0e42adce	2023-03-30T13:53:24	parser: Halt parser if switching encodings fails Avoids buffer overread in htmlParseHTMLAttribute. Found by OSS-Fuzz.
36602292	2023-03-26T14:11:31	parser: Fix buffer overread in xmlDetectEBCDIC Short-lived regression found by OSS-Fuzz.
7fbd454d	2023-03-21T13:26:36	parser: Grow input buffer earlier when reading characters Make more bytes available after invoking CUR_CHAR or NEXT.
98840d40	2023-03-21T19:07:12	parser: Rework EBCDIC code page detection To detect EBCDIC code pages, we used to switch the encoding twice and had to be very careful not to decode data after the XML declaration before the second switch. This relied on a hard-coded expected size of the XML declaration and was complicated and unreliable. Now we convert the first 200 bytes to EBCDIC-US and parse the encoding declaration manually.
04d1bedd	2023-03-21T13:08:44	parser: Rework shrinking of input buffers Don't try to grow the input buffer in xmlParserShrink. This makes sure that no memory allocations are made and the function always succeeds. Remove unnecessary invocations of SHRINK. Invoke SHRINK at the end of DTD parsing loops. Shrink before growing.
1a91392c	2023-03-16T17:48:57	parser: More fixes to xmlParserGrow xmlHaltParser must be called after reporting an error. Switch to xmlBufSetInputBaseCur.
ca2bfece	2023-03-15T16:18:11	malloc-fail: Fix buffer overread when reading from input Found by OSS-Fuzz, see #344.
b167c731	2023-03-14T14:42:36	parser: Fix short-lived regression causing infinite loops Fix 3eb6bf03. We really have to halt the parser, so the input buffer gets reset.
e7c3a4ca	2023-03-13T19:19:46	parser: Deprecate some parser input functions
2099441f	2023-03-13T17:51:13	parser: Stop calling xmlParserInputShrink Introduce xmlParserShrink which takes a parser context to simplify error handling.
457fc622	2023-03-13T16:51:14	malloc-fail: Fix null deref in xmlParserInputShrink Found by OSS-Fuzz.
3eb6bf03	2023-03-12T16:47:15	parser: Stop calling xmlParserInputGrow Introduce xmlParserGrow which takes a parser context to simplify error handling.
2355eac5	2023-01-22T14:52:06	malloc-fail: Fix null deref if growing input buffer fails Also add some error checks. Found with libFuzzer, see #344.
077df27e	2022-12-22T15:22:01	parser: Fix integer overflow of input ID Applies a patch from Chromium. Also stop incrementing input ID of subcontexts. This isn't necessary. Fixes #465.
ce76ebfd	2022-12-19T20:56:23	entities: Stop counting entities This was only used in the old version of xmlParserEntityCheck.
463bbeec	2022-12-19T18:39:45	entities: Rework entity amplification checks This commit implements robust detection of entity amplification attacks, better known as the "billion laughs" attack. We now limit the size of the document after substitution of entities to 10 times the size before expansion. This guarantees linear behavior by definition. There already was a similar check before, but the accounting of "sizeentities" (size of external entities) and "sizeentcopy" (size of all copies created by entity references) wasn't accurate. We also need saturation arithmetic since we're historically limited to "unsigned long" which is 32-bit on many platforms. A maximum of 10 MB of substitutions is always allowed. This should make use cases like DITA work which have caused problems in the past. The old checks based on the number of entities were removed. This is accounted for by adding a fixed cost to each entity reference. Entity amplification checks are now enabled even if XML_PARSE_HUGE is set. This option is mainly used to allow larger text nodes. Most users were unaware that it also disabled entity expansion checks. Some of the limits might be adjusted later. If this change turns out to affect legitimate use cases, we can add a separate parser option to disable the checks. Fixes #294. Fixes #345.
a8b31e68	2022-11-21T21:35:01	parser: Fix progress check when parsing character data Skip over zero bytes to guarantee progress. Short-lived regression.
691a7719	2022-11-20T19:55:12	parser: Fix 'consumed' accounting when switching encodings
249cee4b	2022-11-13T20:19:13	io: Fix a few integer overflows in I/O statistics There are still many places where arithmetic on "consumed" stats isn't checked for overflow, affecting platforms with a 32-bit long type.
6b570619	2022-11-13T19:44:00	io: Rearrange code in xmlSwitchInputEncodingInt No functional change.
46cd7d22	2022-11-13T16:30:46	io: Remove xmlInputReadCallbackNop In some cases, for example when using encoders, the read callback was set to NULL, in other cases it was set to xmlInputReadCallbackNop. xmlGROW only tested for xmlInputReadCallbackNop, resulting in errors when parsing large encoded content from memory. Always use a NULL callback for memory buffers to avoid ambiguities. Fixes #262.
9feafbc5	2022-11-13T16:56:10	io: Check for memory buffer early in xmlParserInputGrow
6843fc72	2022-09-01T02:58:00	Remove or annotate char casts
ad338ca7	2022-09-01T01:18:30	Remove explicit integer casts Remove explicit integer casts as final operation - in assignments - when passing arguments - when returning values Remove casts - to the same type - from certain range-bound values The main motivation is that these explicit casts don't change the result of operations and only render UBSan's implicit-conversion checks useless. Removing these casts allows UBSan to detect cases where truncation or sign-changes occur unexpectedly. Document some explicit casts as truncating and add a few missing ones.
65dc8a63	2022-09-01T00:13:19	Make xmlNewSAXParserCtx take a const sax handler Also improve documentation.
0f568c0b	2022-08-26T01:22:33	Consolidate private header files Private functions were previously declared - in header files in the root directory - in public headers guarded with IN_LIBXML - in libxml.h - redundantly in source files that used them. Consolidate all private header files in include/private.
ca3807d9	2022-08-24T15:55:46	Mark more functions setting globals as deprecated
fd85b566	2022-08-24T15:12:24	Mark more parser functions as deprecated No compiler warnings generated yet.
9a82b94a	2022-08-24T04:21:58	Introduce xmlNewSAXParserCtxt and htmlNewSAXParserCtxt Add API functions to create a parser context with a custom SAX handler without having to mess with ctxt->sax manually.
c21e9cd5	2022-08-20T17:02:02	Use xmlStrlen in xmlNewStringInputStream xmlStrlen handles buffers larger than INT_MAX more gracefully.
b1b65417	2022-08-20T15:15:04	Create stream with buffer in xmlNewStringInputStream Create an input stream with a buffer in xmlNewStringInputStream. Otherwise, switching encodings won't work. See #34.
aab584dc	2022-03-06T23:23:43	Clean up encoding switching code - Remove xmlSwitchToEncodingInt which was basically just a wrapper around xmlSwitchInputEncodingInt. - Simplify xmlSwitchEncoding. - Improve error handling in xmlSwitchInputEncodingInt. - Deprecate xmlSwitchInputEncoding.
92bff866	2022-03-29T14:18:31	Fix calls to deprecated init/cleanup functions Only use xmlInitParser/xmlCleanupParser.
4951c462	2022-03-06T02:29:00	Avoid arithmetic on freed pointers
ebb17970	2022-03-04T02:31:59	Remove unneeded #includes
776d15d3	2022-03-02T00:29:17	Don't check for standard C89 headers Don't check for - ctype.h - errno.h - float.h - limits.h - math.h - signal.h - stdarg.h - stdlib.h - string.h - time.h Stop including non-standard headers - malloc.h - strings.h
2489c1d0	2022-02-28T22:42:10	Remove useless __CYGWIN__ checks From what I can tell, some really early Cygwin versions from around 1998-2000 used to erroneously define _WIN32. This was eventually fixed, but these days, the `defined(_WIN32) && !defined(__CYGWIN__)` idiom is unnecessary. Now, we only check for __CYGWIN__ in xmlexports.h when deciding whether to use __declspec.
346c3a93	2022-02-20T18:46:42	Remove elfgcchack.h The same optimization can be enabled with -fno-semantic-interposition since GCC 5. clang has always used this option by default.
d7cb33cf	2022-01-13T17:06:14	Rework validation context flags Use a bitmask instead of magic values to - keep track whether the validation context is part of a parser context - keep track whether xmlValidateDtdFinal was called This allows to add addtional flags later. Note that this deliberately changes the name of a public struct member, assuming that this was always private data never to be used by client code.
328456bf	2021-07-14T14:43:59	Fix memory leak in xmlNewInputFromFile Found by Coverity. https://bugzilla.redhat.com/show_bug.cgi?id=1938806
dcb80b92	2021-02-20T20:30:43	Fix slow parsing of HTML with encoding errors Under certain circumstances, the HTML parser would try to guess and switch input encodings multiple times, leading to slow processing of documents with encoding errors. The repeated scanning of the input buffer when guessing encodings could even lead to quadratic behavior. The code htmlCurrentChar probably assumed that if there's an encoding handler, it is guaranteed to produce valid UTF-8. This holds true in general, but if the detected encoding was "UTF-8", the UTF8ToUTF8 encoding handler simply invoked memcpy without checking for invalid UTF-8. This still must be fixed, preferably by not using this handler at all. Also leave a note that switching encodings twice seems impossible to implement correctly. Add a check when handling UTF-8 encoding errors in htmlCurrentChar to avoid this situation, even if encoders produce invalid UTF-8. Found by OSS-Fuzz.
438e595a	2020-08-09T14:43:53	Stop counting nbChars in parser context The value was inaccurate and never used.
20c60886	2020-03-08T17:19:42	Fix typos Resolves #133.
2a350ee9	2019-09-30T17:04:54	Large batch of typo fixes Closes #109.
3776cb47	2018-11-22T15:27:28	Fix memory leak in xmlSwitchInputEncodingInt error path Found by OSS-Fuzz.
7a1bd7f6	2018-03-17T00:03:24	Revert "Change calls to xmlCharEncInput to set flush false" This reverts commit 6e6ae5daa6cd9640c9a83c1070896273e9b30d14 which broke decoding of larger documents with ICU. See https://bugs.chromium.org/p/chromium/issues/detail?id=820163
6e6ae5da	2018-01-02T21:47:35	Change calls to xmlCharEncInput to set flush false when not final call. Having flush incorrectly set to true causes errors for ICU.
cb5541c9	2017-11-13T17:08:38	Fix libz and liblzma detection If libz or liblzma are detected with pkg-config, AC_CHECK_HEADERS must not be run because the correct CPPFLAGS aren't set. It is actually not required have separate checks for LIBXML_ZLIB_ENABLED and HAVE_ZLIB_H. Only check for LIBXML_ZLIB_ENABLED and remove HAVE_ZLIB_H macro. Fixes bug 764657, bug 787041.
e03f0a19	2017-11-09T16:42:47	Fix hash callback signatures Make sure that all parameters and return values of hash callback functions exactly match the callback function type. This is required to pass clang's Control Flow Integrity checks and to allow compilation to asm.js with Emscripten. Fixes bug 784861.
e3890546	2017-10-09T00:20:01	Fix the Windows header mess Don't include windows.h and wsockcompat.h from config.h but only when needed. Don't define _WINSOCKAPI_ manually. This was apparently done to stop windows.h from including winsock.h which is a problem if winsock2.h wasn't included first. But on MinGW, this causes compiler warnings. Define WIN32_LEAN_AND_MEAN instead which has the same effect. Always use the compiler-defined _WIN32 macro instead of WIN32.
69936b12	2017-08-30T14:16:01	Revert "Print error messages for truncated UTF-8 sequences" This reverts commit 79c8a6b which caused a serious regression in streaming mode. Also reverts part of commit 52ceced "Fix infinite loops with push parser in recovery mode". Fixes bug 786554.
f19385a5	2017-08-28T20:40:19	Fix a couple of misleading indentation errors Raised by gcc as potential error, no semantic change needed but fixed the indentation
3aca7f31	2017-08-21T13:06:29	Fix unwanted warnings when switching encodings Revert part of commit 46dc989 "Don't switch encoding for internal parameter entities" that caused spurious warnings. Fixes bug 786267.
453dff1e	2017-06-19T17:55:20	Remove unnecessary calls to xmlPopInput It's enough if xmlPopInput is called from xmlSkipBlankChars. Since the replacement text of a parameter entity is surrounded with space characters, that's the only place where the replacement can end in a well-formed document. This is also required to get rid of the "blanks wrapper" hack.
aa267cd1	2017-06-18T23:29:51	Simplify handling of parameter entity references There are only two places where parameter entity references must be handled. For the internal subset in xmlParseInternalSubset. For the external subset or content from other external PEs in xmlSkipBlankChars. Make sure that xmlSkipBlankChars skips over sequences of PEs and whitespace. Rely on xmlSkipBlankChars instead of calling xmlParsePEReference directly when in the external subset or a conditional section. xmlParserHandlePEReference is unused now.
46dc9890	2017-06-08T02:24:56	Don't switch encoding for internal parameter entities This is only needed for external entities. Trying to switch the encoding for internal entities could also cause a memory leak in recovery mode.
79c8a6b1	2017-06-10T17:01:27	Print error messages for truncated UTF-8 sequences Before, truncated UTF-8 sequences at the end of a file were treated as EOF. Create an error message containing the offending bytes. xmlStringCurrentChar would also print characters from the input stream, not the string it's working on.
f9e7997e	2017-06-10T00:33:07	Reset parser input pointers on encoding failure Call xmlBufResetInput before bailing out if switching the encoding fails. Otherwise, the input pointers are left in an invalid state. This would typically lead to an internal error in xmlGROW but could also cause other unforeseen problems.
45ce1ee3	2017-06-09T15:27:25	Add TODO comment in xmlSwitchEncoding It would be nice if we could recover from unsupported encodings in external entities.
0db8dc9d	2017-06-07T16:07:33	Stop parser on unsupported encodings Otherwise, the push parser can loop infinitely in recover mode. Found with libFuzzer.
0bcd05c5	2016-03-01T15:18:04	Heap-based buffer overread in htmlCurrentChar For https://bugzilla.gnome.org/show_bug.cgi?id=758606 * parserInternals.c: (xmlNextChar): Add an test to catch other issues on ctxt->input corruption proactively. For non-UTF-8 charsets, xmlNextChar() failed to check for the end of the input buffer and would continuing reading. Fix this by pulling out the check for the end of the input buffer into common code, and return if we reach the end of the input buffer prematurely. * result/HTML/758606.html: Added. * result/HTML/758606.html.err: Added. * result/HTML/758606.html.sax: Added. * result/HTML/758606_2.html: Added. * result/HTML/758606_2.html.err: Added. * result/HTML/758606_2.html.sax: Added. * test/HTML/758606.html: Added test case. * test/HTML/758606_2.html: Added test case.
4472c3a5	2016-05-13T15:13:17	Fix some format string warnings with possible format string vulnerability For https://bugzilla.gnome.org/show_bug.cgi?id=761029 Decorate every method in libxml2 with the appropriate LIBXML_ATTR_FORMAT(fmt,args) macro and add some cleanups following the reports.
d433ea6c	2016-05-18T14:52:59	Integer signed/unsigned type mismatch in xmlParserInputGrow() For https://bugzilla.gnome.org/show_bug.cgi?id=766635 * parserInternals.c: (xmlParserInputGrow): Change 'ret' type to 'int' to match the return type of xmlParserInputBufferGrow().
fdfeecc1	2015-11-20T15:07:38	Bug on creating new stream from entity sometimes the entity could have a lenght of 0, i.e. it wasn't parsed or used yet, and we ended up with an incoherent input state
afd27c21	2015-11-09T18:07:18	Avoid processing entities after encoding conversion failures For https://bugzilla.gnome.org/show_bug.cgi?id=756527 and was also raised by Chromium team in the past When we hit a convwersion failure when switching encoding it is bestter to stop parsing there, this was treated as a fatal error but the parser was continuing to process to extract more errors, unfortunately that makes little sense as the data is obviously corrupt and can potentially lead to unexpected behaviour.
c35af8b1	2014-06-11T16:59:16	Fixes for xmlInitParserCtxt let's make sure that parser options are updated too when a corrsponding global variable or other field of the context is set.
ff76eb28	2013-08-03T22:25:13	Clear up a potential NULL dereference https://bugzilla.gnome.org/show_bug.cgi?id=705399 if ctxt->node_seq.buffer is null then ctxt->node_seq.maximum ought to be zero but it's better to clarify the check in the code directly.
23f05e0c	2013-02-19T10:21:49	Detect excessive entities expansion upon replacement If entities expansion in the XML parser is asked for, it is possble to craft relatively small input document leading to excessive on-the-fly content generation. This patch accounts for those replacement and stop parsing after a given threshold. it can be bypassed as usual with the HUGE parser option.
bf058dce	2013-02-13T18:19:42	Fix the flushing out of raw buffers on encoding conversions https://bugzilla.gnome.org/show_bug.cgi?id=692915 the new set of converting functions tried to limit the encoding conversion of the raw buffer to the consumption one to work in a more progressive fashion. Unfortunately this was bad for performances and led to errors on progressive parsing when a very large chunk was close to the end of the document. Fix the new internal function and switch back to the old way of converting. Fix another bug in the process.

e0dd330b

2023-09-29T00:18:44

parser: Use hash tables to avoid quadratic behavior Use a hash table to lookup namespaces by prefix. The hash table stores an index into the namespace table. Auxiliary data for namespaces is stored in a separate array along the main namespace table. Use a hash table to verify attribute uniqueness. The hash table stores an index into the attribute table. Reuse hash value from the dictionary to avoid computing them twice. See #346.

cb927e85

2023-09-25T14:35:43

parser: Don't skip CR in xmlCurrentChar Skip over carriage returns later in xmlNextChar.

b9db3d7d

2023-09-22T15:45:20

parser: Simplify xmlStringCurrentChar Start to move away from using this function.

8c084ebd

2023-09-21T22:57:33

doc: Make apibuild.py happy

9b5cce7a

2023-09-21T00:44:50

include: Remove more unnecessary includes

699299ca

2023-09-20T18:54:39

globals: Stop including globals.h

11a1839d

2023-09-20T17:54:48

globals: Move remaining globals back to correct header files This undoes a lot of damage.

4e1c13eb

2023-09-18T14:45:10

debug: Remove debugging code This is barely useful these days and only clutters the code base.

e48f2695

2023-08-29T17:41:18

parser: Remove push parser debugging code

ed3bd052

2023-08-20T20:48:10

parser: Allow to set maximum amplification factor

f1c1f5c6

2023-08-16T19:43:02

parser: Revert change to doc->encoding Fixes #579.

61b8e097

2023-08-16T19:20:47

parser: Never use UTF-8 encoding handler

b973ceaf

2023-08-09T18:37:20

parser: Fix mistake in xmlDetectEncoding Short-lived regression.

95e81a36

2023-08-08T15:21:31

parser: Decode all data in xmlCharEncInput Even with flush set to true, xmlCharEncInput didn't guarantee to decode all data. This complicated the push parser. Remove the flush flag and always decode all available data. Also fix ICU code where the flush flag has a different meaning. Always set flush to false and retry even with empty input buffers.

834b8123

2023-08-08T15:21:28

parser: Stream data when reading from memory Don't create a copy of the whole input buffer. Read the data chunk by chunk to save memory. Historically, it was probably envisioned to read data from memory without additional copying. This doesn't work reliably with the current design of the XML parser which requires a terminating null byte at the end of input buffers. This lead to xmlReadMemory interfaces, which expect pointer and size arguments, being changed to make a zero-terminated copy of the input buffer. Interfaces based on xmlReadDoc, which actually expect a zero-terminated string and would make zero-copy operation work, were then simplified to rely on xmlReadMemoryi, resulting in an unnecessary copy. To avoid copying (possibly gigabytes) of memory temporarily, we now stream in-memory input just like content read from files in a chunk-by-chunk fashion (using a somewhat outdated INPUT_CHUNK size of 250 bytes). As a side effect, we also avoid another copy of the whole input when handling non-UTF-8 data which was made possible by some earlier commits. Interfaces expecting zero-terminated strings now make use of strnlen which unfortunately isn't part of the standard C library and only mandated since POSIX 2008.

59fa0bb3

2023-08-08T15:21:14

parser: Simplify input pointer updates The base member always points to the beginning of the buffer.

4ee08155

2023-08-08T15:19:51

encoding: Move rawconsumed accounting to xmlCharEncInput

ec7be506

2023-08-08T15:19:46

parser: Rework encoding detection Introduce XML_INPUT_HAS_ENCODING flag for xmlParserInput which is set when xmlSwitchEncoding is called. The parser can use the flag to reliably detect whether an encoding was already set via user override, BOM or other auto-detection. In this case, the encoding declaration won't be used to switch the encoding. Before, an inscrutable mix of ctxt->charset, ctxt->input->encoding and ctxt->input->buf->encoder was used. Introduce private helper functions to switch encodings used by both the XML and HTML parser: - xmlDetectEncoding which skips over the BOM, allowing to remove the BOM checks from other encoding functions. - xmlSetDeclaredEncoding, replacing htmlCheckEncodingDirect, which warns about encoding mismatches. If users override the encoding, store the declared instead of the actual encoding in xmlDoc. In this case, the actual encoding is known and the raw value from the doc is more useful. Also use the input flags to store the ISO-8859-1 fallback state. Restrict the fallback to cases where no encoding was specified. (The fallback is only useful in recovery mode and these days broken UTF-8 is probably more likely than ISO-8859-1, so it might eventually be removed completely.) The 'charset' member of xmlParserCtxt is now unused. The 'encoding' member of xmlParserInput is now unused. The 'standalone' member of xmlParserInput is renamed to 'flags'. A new parser state XML_PARSER_XML_DECL is added for the push parser.

131d0dc0

2023-08-08T15:19:39

parser: Don't use 'standalone' member of xmlParserInput The standalone declaration is only parsed in the main input stream.

88447447

2023-06-23T23:04:30

parser: Fix typo in previous commit

9d0541dd

2023-06-22T18:06:53

parser: Make xmlSwitchEncoding always skip the BOM Chromium calls xmlSwitchEncoding from the start document handler and relies on this function to skip the BOM. Commit 98840d40 changed the behavior when switching to UTF-16 since inspecting the input buffer at this point is fragile. Revert part of the commit to also skip a potential (decoded UTF-8) BOM when switching to UTF-16. Make sure that we do this only at the start of an input stream to avoid U-FEFF characters being lost. BOM handling should ultimately be moved to the parsing code to avoid such bugs. See https://bugs.chromium.org/p/chromium/issues/detail?id=1451026

b236b7a5

2023-06-08T21:53:05

parser: Halt parser when growing buffer results in OOM Fix short-lived regression from previous commit. It might be safer to make xmlBufSetInputBaseCur use the original buffer even in case of errors. Found by OSS-Fuzz.

20f5c734

2023-06-07T14:05:34

parser: Recover more input from encoding errors Don't halt the parser in xmlParserGrow to allow more input to be recovered in case of encoding errors. Fixes #543.

e0f3016f

2023-05-18T17:31:44

parser: Fix regression when push parsing UTF-8 sequences Partial UTF-8 sequences are allowed when push parsing. Fixes #542.

9dae389c

2023-05-09T13:28:06

parser: Fix "huge input lookup" error with push parser Fix parsing of larger documents without XML_PARSE_HUGE. Should fix #538.

320f5084

2023-04-30T18:25:09

parser: Improve handling of encoding and IO errors Make sure that xmlCharEncInput, xmlParserInputBufferPush and xmlParserInputBufferGrow set the correct error code in the xmlParserInputBuffer. Handle errors when calling these functions.

fc69cf56

2023-04-30T17:51:29

parser: Move xmlFatalErr to parserInternals.c

3ffcc03b

2023-03-13T19:38:41

parser: Deprecate more internal functions

9282b084

2023-04-19T21:55:24

parser: Fix regression in memory pull parser with encoding Revert another change from commit 98840d40. Decode the whole buffer when reading from memory and switching to the initial encoding. Add some comments about potential improvements.

a19fa11e

2023-04-13T15:11:47

parser: Fix regression when switching input encodings Revert some changes from commit 98840d40. WebKit/Chromium can actually switch from ISO-8859-1 to UTF-16 in the middle of parsing. This is a bad idea, but we have to keep supporting this use case.

921796b0

2023-04-12T13:43:28

parser: Don't grow push parser buffers This should fix a short-lived regression when push parsing with encodings.

0e42adce

2023-03-30T13:53:24

parser: Halt parser if switching encodings fails Avoids buffer overread in htmlParseHTMLAttribute. Found by OSS-Fuzz.

36602292

2023-03-26T14:11:31

parser: Fix buffer overread in xmlDetectEBCDIC Short-lived regression found by OSS-Fuzz.

7fbd454d

2023-03-21T13:26:36

parser: Grow input buffer earlier when reading characters Make more bytes available after invoking CUR_CHAR or NEXT.

98840d40

2023-03-21T19:07:12

parser: Rework EBCDIC code page detection To detect EBCDIC code pages, we used to switch the encoding twice and had to be very careful not to decode data after the XML declaration before the second switch. This relied on a hard-coded expected size of the XML declaration and was complicated and unreliable. Now we convert the first 200 bytes to EBCDIC-US and parse the encoding declaration manually.

04d1bedd

2023-03-21T13:08:44

parser: Rework shrinking of input buffers Don't try to grow the input buffer in xmlParserShrink. This makes sure that no memory allocations are made and the function always succeeds. Remove unnecessary invocations of SHRINK. Invoke SHRINK at the end of DTD parsing loops. Shrink before growing.

1a91392c

2023-03-16T17:48:57

parser: More fixes to xmlParserGrow xmlHaltParser must be called after reporting an error. Switch to xmlBufSetInputBaseCur.

ca2bfece

2023-03-15T16:18:11

malloc-fail: Fix buffer overread when reading from input Found by OSS-Fuzz, see #344.

b167c731

2023-03-14T14:42:36

parser: Fix short-lived regression causing infinite loops Fix 3eb6bf03. We really have to halt the parser, so the input buffer gets reset.

e7c3a4ca

2023-03-13T19:19:46

parser: Deprecate some parser input functions

2099441f

2023-03-13T17:51:13

parser: Stop calling xmlParserInputShrink Introduce xmlParserShrink which takes a parser context to simplify error handling.

457fc622

2023-03-13T16:51:14

malloc-fail: Fix null deref in xmlParserInputShrink Found by OSS-Fuzz.

3eb6bf03

2023-03-12T16:47:15

parser: Stop calling xmlParserInputGrow Introduce xmlParserGrow which takes a parser context to simplify error handling.

2355eac5

2023-01-22T14:52:06

malloc-fail: Fix null deref if growing input buffer fails Also add some error checks. Found with libFuzzer, see #344.

077df27e

2022-12-22T15:22:01

parser: Fix integer overflow of input ID Applies a patch from Chromium. Also stop incrementing input ID of subcontexts. This isn't necessary. Fixes #465.

ce76ebfd

2022-12-19T20:56:23

entities: Stop counting entities This was only used in the old version of xmlParserEntityCheck.

463bbeec

2022-12-19T18:39:45

entities: Rework entity amplification checks This commit implements robust detection of entity amplification attacks, better known as the "billion laughs" attack. We now limit the size of the document after substitution of entities to 10 times the size before expansion. This guarantees linear behavior by definition. There already was a similar check before, but the accounting of "sizeentities" (size of external entities) and "sizeentcopy" (size of all copies created by entity references) wasn't accurate. We also need saturation arithmetic since we're historically limited to "unsigned long" which is 32-bit on many platforms. A maximum of 10 MB of substitutions is always allowed. This should make use cases like DITA work which have caused problems in the past. The old checks based on the number of entities were removed. This is accounted for by adding a fixed cost to each entity reference. Entity amplification checks are now enabled even if XML_PARSE_HUGE is set. This option is mainly used to allow larger text nodes. Most users were unaware that it also disabled entity expansion checks. Some of the limits might be adjusted later. If this change turns out to affect legitimate use cases, we can add a separate parser option to disable the checks. Fixes #294. Fixes #345.

a8b31e68

2022-11-21T21:35:01

parser: Fix progress check when parsing character data Skip over zero bytes to guarantee progress. Short-lived regression.

691a7719

2022-11-20T19:55:12

parser: Fix 'consumed' accounting when switching encodings

249cee4b

2022-11-13T20:19:13

io: Fix a few integer overflows in I/O statistics There are still many places where arithmetic on "consumed" stats isn't checked for overflow, affecting platforms with a 32-bit long type.

6b570619

2022-11-13T19:44:00

io: Rearrange code in xmlSwitchInputEncodingInt No functional change.

46cd7d22

2022-11-13T16:30:46

io: Remove xmlInputReadCallbackNop In some cases, for example when using encoders, the read callback was set to NULL, in other cases it was set to xmlInputReadCallbackNop. xmlGROW only tested for xmlInputReadCallbackNop, resulting in errors when parsing large encoded content from memory. Always use a NULL callback for memory buffers to avoid ambiguities. Fixes #262.

9feafbc5

2022-11-13T16:56:10

io: Check for memory buffer early in xmlParserInputGrow

6843fc72

2022-09-01T02:58:00

Remove or annotate char casts

ad338ca7

2022-09-01T01:18:30

Remove explicit integer casts Remove explicit integer casts as final operation - in assignments - when passing arguments - when returning values Remove casts - to the same type - from certain range-bound values The main motivation is that these explicit casts don't change the result of operations and only render UBSan's implicit-conversion checks useless. Removing these casts allows UBSan to detect cases where truncation or sign-changes occur unexpectedly. Document some explicit casts as truncating and add a few missing ones.

65dc8a63

2022-09-01T00:13:19

Make xmlNewSAXParserCtx take a const sax handler Also improve documentation.

0f568c0b

2022-08-26T01:22:33

Consolidate private header files Private functions were previously declared - in header files in the root directory - in public headers guarded with IN_LIBXML - in libxml.h - redundantly in source files that used them. Consolidate all private header files in include/private.

ca3807d9

2022-08-24T15:55:46

Mark more functions setting globals as deprecated

fd85b566

2022-08-24T15:12:24

Mark more parser functions as deprecated No compiler warnings generated yet.

9a82b94a

2022-08-24T04:21:58

Introduce xmlNewSAXParserCtxt and htmlNewSAXParserCtxt Add API functions to create a parser context with a custom SAX handler without having to mess with ctxt->sax manually.

c21e9cd5

2022-08-20T17:02:02

Use xmlStrlen in xmlNewStringInputStream xmlStrlen handles buffers larger than INT_MAX more gracefully.

b1b65417

2022-08-20T15:15:04

Create stream with buffer in xmlNewStringInputStream Create an input stream with a buffer in xmlNewStringInputStream. Otherwise, switching encodings won't work. See #34.

aab584dc

2022-03-06T23:23:43

Clean up encoding switching code - Remove xmlSwitchToEncodingInt which was basically just a wrapper around xmlSwitchInputEncodingInt. - Simplify xmlSwitchEncoding. - Improve error handling in xmlSwitchInputEncodingInt. - Deprecate xmlSwitchInputEncoding.

92bff866

2022-03-29T14:18:31

Fix calls to deprecated init/cleanup functions Only use xmlInitParser/xmlCleanupParser.

4951c462

2022-03-06T02:29:00

Avoid arithmetic on freed pointers

ebb17970

2022-03-04T02:31:59

Remove unneeded #includes

776d15d3

2022-03-02T00:29:17

Don't check for standard C89 headers Don't check for - ctype.h - errno.h - float.h - limits.h - math.h - signal.h - stdarg.h - stdlib.h - string.h - time.h Stop including non-standard headers - malloc.h - strings.h

2489c1d0

2022-02-28T22:42:10

Remove useless __CYGWIN__ checks From what I can tell, some really early Cygwin versions from around 1998-2000 used to erroneously define _WIN32. This was eventually fixed, but these days, the `defined(_WIN32) && !defined(__CYGWIN__)` idiom is unnecessary. Now, we only check for __CYGWIN__ in xmlexports.h when deciding whether to use __declspec.

346c3a93

2022-02-20T18:46:42

Remove elfgcchack.h The same optimization can be enabled with -fno-semantic-interposition since GCC 5. clang has always used this option by default.

d7cb33cf

2022-01-13T17:06:14

Rework validation context flags Use a bitmask instead of magic values to - keep track whether the validation context is part of a parser context - keep track whether xmlValidateDtdFinal was called This allows to add addtional flags later. Note that this deliberately changes the name of a public struct member, assuming that this was always private data never to be used by client code.

328456bf

2021-07-14T14:43:59

Fix memory leak in xmlNewInputFromFile Found by Coverity. https://bugzilla.redhat.com/show_bug.cgi?id=1938806

dcb80b92

2021-02-20T20:30:43

Fix slow parsing of HTML with encoding errors Under certain circumstances, the HTML parser would try to guess and switch input encodings multiple times, leading to slow processing of documents with encoding errors. The repeated scanning of the input buffer when guessing encodings could even lead to quadratic behavior. The code htmlCurrentChar probably assumed that if there's an encoding handler, it is guaranteed to produce valid UTF-8. This holds true in general, but if the detected encoding was "UTF-8", the UTF8ToUTF8 encoding handler simply invoked memcpy without checking for invalid UTF-8. This still must be fixed, preferably by not using this handler at all. Also leave a note that switching encodings twice seems impossible to implement correctly. Add a check when handling UTF-8 encoding errors in htmlCurrentChar to avoid this situation, even if encoders produce invalid UTF-8. Found by OSS-Fuzz.

438e595a

2020-08-09T14:43:53

Stop counting nbChars in parser context The value was inaccurate and never used.

20c60886

2020-03-08T17:19:42

Fix typos Resolves #133.

2a350ee9

2019-09-30T17:04:54

Large batch of typo fixes Closes #109.

3776cb47

2018-11-22T15:27:28

Fix memory leak in xmlSwitchInputEncodingInt error path Found by OSS-Fuzz.

7a1bd7f6

2018-03-17T00:03:24

Revert "Change calls to xmlCharEncInput to set flush false" This reverts commit 6e6ae5daa6cd9640c9a83c1070896273e9b30d14 which broke decoding of larger documents with ICU. See https://bugs.chromium.org/p/chromium/issues/detail?id=820163

6e6ae5da

2018-01-02T21:47:35

Change calls to xmlCharEncInput to set flush false when not final call. Having flush incorrectly set to true causes errors for ICU.

cb5541c9

2017-11-13T17:08:38

Fix libz and liblzma detection If libz or liblzma are detected with pkg-config, AC_CHECK_HEADERS must not be run because the correct CPPFLAGS aren't set. It is actually not required have separate checks for LIBXML_ZLIB_ENABLED and HAVE_ZLIB_H. Only check for LIBXML_ZLIB_ENABLED and remove HAVE_ZLIB_H macro. Fixes bug 764657, bug 787041.

e03f0a19

2017-11-09T16:42:47

Fix hash callback signatures Make sure that all parameters and return values of hash callback functions exactly match the callback function type. This is required to pass clang's Control Flow Integrity checks and to allow compilation to asm.js with Emscripten. Fixes bug 784861.

e3890546

2017-10-09T00:20:01

Fix the Windows header mess Don't include windows.h and wsockcompat.h from config.h but only when needed. Don't define _WINSOCKAPI_ manually. This was apparently done to stop windows.h from including winsock.h which is a problem if winsock2.h wasn't included first. But on MinGW, this causes compiler warnings. Define WIN32_LEAN_AND_MEAN instead which has the same effect. Always use the compiler-defined _WIN32 macro instead of WIN32.

69936b12

2017-08-30T14:16:01

Revert "Print error messages for truncated UTF-8 sequences" This reverts commit 79c8a6b which caused a serious regression in streaming mode. Also reverts part of commit 52ceced "Fix infinite loops with push parser in recovery mode". Fixes bug 786554.

f19385a5

2017-08-28T20:40:19

Fix a couple of misleading indentation errors Raised by gcc as potential error, no semantic change needed but fixed the indentation

3aca7f31

2017-08-21T13:06:29

Fix unwanted warnings when switching encodings Revert part of commit 46dc989 "Don't switch encoding for internal parameter entities" that caused spurious warnings. Fixes bug 786267.

453dff1e

2017-06-19T17:55:20

Remove unnecessary calls to xmlPopInput It's enough if xmlPopInput is called from xmlSkipBlankChars. Since the replacement text of a parameter entity is surrounded with space characters, that's the only place where the replacement can end in a well-formed document. This is also required to get rid of the "blanks wrapper" hack.

aa267cd1

2017-06-18T23:29:51

Simplify handling of parameter entity references There are only two places where parameter entity references must be handled. For the internal subset in xmlParseInternalSubset. For the external subset or content from other external PEs in xmlSkipBlankChars. Make sure that xmlSkipBlankChars skips over sequences of PEs and whitespace. Rely on xmlSkipBlankChars instead of calling xmlParsePEReference directly when in the external subset or a conditional section. xmlParserHandlePEReference is unused now.

46dc9890

2017-06-08T02:24:56

Don't switch encoding for internal parameter entities This is only needed for external entities. Trying to switch the encoding for internal entities could also cause a memory leak in recovery mode.

79c8a6b1

2017-06-10T17:01:27

Print error messages for truncated UTF-8 sequences Before, truncated UTF-8 sequences at the end of a file were treated as EOF. Create an error message containing the offending bytes. xmlStringCurrentChar would also print characters from the input stream, not the string it's working on.

f9e7997e

2017-06-10T00:33:07

Reset parser input pointers on encoding failure Call xmlBufResetInput before bailing out if switching the encoding fails. Otherwise, the input pointers are left in an invalid state. This would typically lead to an internal error in xmlGROW but could also cause other unforeseen problems.

45ce1ee3

2017-06-09T15:27:25

Add TODO comment in xmlSwitchEncoding It would be nice if we could recover from unsupported encodings in external entities.

0db8dc9d

2017-06-07T16:07:33

Stop parser on unsupported encodings Otherwise, the push parser can loop infinitely in recover mode. Found with libFuzzer.

0bcd05c5

2016-03-01T15:18:04

Heap-based buffer overread in htmlCurrentChar For https://bugzilla.gnome.org/show_bug.cgi?id=758606 * parserInternals.c: (xmlNextChar): Add an test to catch other issues on ctxt->input corruption proactively. For non-UTF-8 charsets, xmlNextChar() failed to check for the end of the input buffer and would continuing reading. Fix this by pulling out the check for the end of the input buffer into common code, and return if we reach the end of the input buffer prematurely. * result/HTML/758606.html: Added. * result/HTML/758606.html.err: Added. * result/HTML/758606.html.sax: Added. * result/HTML/758606_2.html: Added. * result/HTML/758606_2.html.err: Added. * result/HTML/758606_2.html.sax: Added. * test/HTML/758606.html: Added test case. * test/HTML/758606_2.html: Added test case.

4472c3a5

2016-05-13T15:13:17

Fix some format string warnings with possible format string vulnerability For https://bugzilla.gnome.org/show_bug.cgi?id=761029 Decorate every method in libxml2 with the appropriate LIBXML_ATTR_FORMAT(fmt,args) macro and add some cleanups following the reports.

d433ea6c

2016-05-18T14:52:59

Integer signed/unsigned type mismatch in xmlParserInputGrow() For https://bugzilla.gnome.org/show_bug.cgi?id=766635 * parserInternals.c: (xmlParserInputGrow): Change 'ret' type to 'int' to match the return type of xmlParserInputBufferGrow().

fdfeecc1

2015-11-20T15:07:38

Bug on creating new stream from entity sometimes the entity could have a lenght of 0, i.e. it wasn't parsed or used yet, and we ended up with an incoherent input state

afd27c21

2015-11-09T18:07:18

Avoid processing entities after encoding conversion failures For https://bugzilla.gnome.org/show_bug.cgi?id=756527 and was also raised by Chromium team in the past When we hit a convwersion failure when switching encoding it is bestter to stop parsing there, this was treated as a fatal error but the parser was continuing to process to extract more errors, unfortunately that makes little sense as the data is obviously corrupt and can potentially lead to unexpected behaviour.

c35af8b1

2014-06-11T16:59:16

Fixes for xmlInitParserCtxt let's make sure that parser options are updated too when a corrsponding global variable or other field of the context is set.

ff76eb28

2013-08-03T22:25:13

Clear up a potential NULL dereference https://bugzilla.gnome.org/show_bug.cgi?id=705399 if ctxt->node_seq.buffer is null then ctxt->node_seq.maximum ought to be zero but it's better to clarify the check in the code directly.

23f05e0c

2013-02-19T10:21:49

Detect excessive entities expansion upon replacement If entities expansion in the XML parser is asked for, it is possble to craft relatively small input document leading to excessive on-the-fly content generation. This patch accounts for those replacement and stop parsing after a given threshold. it can be bypassed as usual with the HUGE parser option.

bf058dce

2013-02-13T18:19:42

Fix the flushing out of raw buffers on encoding conversions https://bugzilla.gnome.org/show_bug.cgi?id=692915 the new set of converting functions tried to limit the encoding conversion of the raw buffer to the consumption one to work in a more progressive fashion. Unfortunately this was bad for performances and led to errors on progressive parsing when a very large chunk was close to the end of the document. Fix the new internal function and switch back to the old way of converting. Fix another bug in the process.

kc3-lang/libxml2/parserInternals.c

parserInternals.c

Log