thodg/libgit2/src/buffer.c

---

src/buffer.c

---

Log

Author	Commit	Date	CI	Message
	3e8a17b0	2019-09-21T15:18:42		buffer: fix memory leak if unable to grow buffer If growing a buffer fails, we set its pointer to the static `git_buf__oom` structure. While we correctly free the old pointer if `git__malloc` returned an error, we do not free it if there was an integer overflow while calculating the new allocation size. Fix this issue by freeing the pointer to plug the memory leak.
	174b7a32	2019-09-19T12:24:06		buffer: fix printing into out-of-memory buffer Before printing into a `git_buf` structure, we always call `ENSURE_SIZE` first. This macro will reallocate the buffer as-needed depending on whether the current amount of allocated bytes is sufficient or not. If `asize` is big enough, then it will just do nothing, otherwise it will call out to `git_buf_try_grow`. But in fact, it is insufficient to only check `asize`. When we fail to allocate any more bytes e.g. via `git_buf_try_grow`, then we set the buffer's pointer to `git_buf__oom`. Note that we touch neither `asize` nor `size`. So if we just check `asize > targetsize`, then we will happily let the caller of `ENSURE_SIZE` proceed with an out-of-memory buffer. As a result, we will print all bytes into the out-of-memory buffer instead, resulting in an out-of-bounds write. Fix the issue by having `ENSURE_SIZE` verify that the buffer is not marked as OOM. Add a test to verify that we're not writing into the OOM buffer.
	208f1d7a	2019-09-19T12:46:37		buffer: fix infinite loop when growing buffers When growing buffers, we repeatedly multiply the currently allocated number of bytes by 1.5 until it exceeds the requested number of bytes. This has two major problems: 1. If the current number of bytes is tiny and one wishes to resize to a comparatively huge number of bytes, then we may need to loop thousands of times. 2. If resizing to a value close to `SIZE_MAX` (which would fail anyway), then we probably hit an infinite loop as multiplying the current amount of bytes will repeatedly result in integer overflows. When reallocating buffers, one typically chooses values close to 1.5 to enable re-use of resulting memory holes in later reallocations. But because of this, it really only makes sense to use a factor of 1.5 _once_, but not looping until we finally are able to fit it. Thus, we can completely avoid the loop and just opt for the much simpler algorithm of multiplying with 1.5 once and, if the result doesn't fit, just use the target size. This avoids both problems of looping extensively and hitting overflows. This commit also adds a test that would've previously resulted in an infinite loop.
	e5e2fac8	2019-01-21T00:57:39		buffer: explicitly cast Quiet down a warning from MSVC about how we're potentially losing data. This is safe since we've explicitly tested it.
	f673e232	2018-12-27T13:47:34		git_error: use new names in internal APIs and usage Move to the `git_error` name in the internal API for error-related functions.
	ecf4f33a	2018-02-08T11:14:48		Convert usage of `git_buf_free` to new `git_buf_dispose`
	56ffdfc6	2018-02-08T11:14:30		buffer: deprecate `git_buf_free` in favor of `git_buf_dispose`
	8070a357	2018-03-03T18:47:35		Introduce `git_buf_decode_percent` Introduce a function to take a percent-encoded string (URI encoded, described by RFC 1738) and decode it into a `git_buf`.
	4796c916	2017-06-07T09:56:31		buffer: return errors for `git_buf_init` and `git_buf_attach` Both the `git_buf_init` and `git_buf_attach` functions may call `git_buf_grow` in case they were given an allocation length as parameter. As such, it is possible for these functions to fail when we run out of memory. While it won't probably be used anytime soon, it does indeed make sense to also record this fact by returning an error code from both functions. As they belong to the internal API only, this change does not break our interface.
	9a8386a2	2017-06-07T09:50:54		buffer: consistently use `ENSURE_SIZE` to grow buffers on-demand The `ENSURE_SIZE` macro can be used to grow a buffer if its currently allocated size does not suffice a required target size. While most of the code already uses this macro, the `git_buf_join` and `git_buf_join3` functions do not yet use it. Due to the macro first checking whether we have to grow the buffer at all, this has the benefit of saving a function call when it is not needed. While this is nice to have, it will probably not matter at all performance-wise -- instead, this only serves for consistency across the code.
	e82dd813	2017-06-08T11:52:32		buffer: fix `ENSURE_SIZE` macro referencing wrong variable While the `ENSURE_SIZE` macro gets a reference to both the buffer that is to be resized and a new size, we were not consistently referencing the passed buffer, but instead a variable `buf`, which is not passed in. Funnily enough, we never noticed because our buffers seem to always be named `buf` whenever the macro was being used. Fix the macro by always using the passed-in buffer. While at it, add braces around all mentions of passed-in variables as should be done with macros to avoid subtle errors. Found-by: Edward Thompson
	909d5494	2016-12-29T12:25:15		giterr_set: consistent error messages Error messages should be sentence fragments, and therefore: 1. Should not begin with a capital letter, 2. Should not conclude with punctuation, and 3. Should not end a sentence and begin a new one
	8cb27223	2015-09-25T10:48:19		git_buf_quote/unquote: handle > \177 Parse values up to and including `\377` (`0xff`) when unquoting. Print octal values as an unsigned char when quoting, lest `printf` think we're talking about negatives.
	d3d95d5a	2015-09-23T16:30:48		git_buf_quote: quote ugly characters
	5b78dbdb	2015-07-09T13:04:10		git_buf: decode base85 inputs
	d34f6826	2014-04-08T17:18:47		Patch parsing from patch files
	a6599235	2015-06-24T19:32:56		buffer: make use of EINVALID for growing a borrowed buffer This explains more closely what happens. While here, set an error message.
	caab22c0	2015-06-23T15:41:58		buffer: don't allow growing borrowed buffers When we don't own a buffer (asize=0) we currently allow the usage of grow to copy the memory into a buffer we do own. This muddles the meaning of grow, and lets us be a bit cavalier with ownership semantics. Don't allow this any more. Usage of grow should be restricted to buffers which we know own their own memory. If unsure, we must not attempt to modify it.
	d4cf1675	2015-02-19T10:05:33		buffer: introduce git_buf_attach_notowned Provide a convenience function that creates a buffer that can be provided to callers but will not be freed via `git_buf_free`, so the buffer creator maintains the allocation lifecycle of the buffer's contents.
	f1453c59	2015-02-12T12:19:37		Make our overflow check look more like gcc/clang's Make our overflow checking look more like gcc and clang's, so that we can substitute it out with the compiler instrinsics on platforms that support it. This means dropping the ability to pass `NULL` as an out parameter. As a result, the macros also get updated to reflect this as well.
	2884cc42	2015-02-11T09:39:38		overflow checking: don't make callers set oom Have the ALLOC_OVERFLOW testing macros also simply set_oom in the case where a computation would overflow, so that callers don't need to.
	4aa664ae	2015-02-10T23:55:07		git_buf_grow_by: increase buf asize incrementally Introduce `git_buf_grow_by` to incrementally increase the size of a `git_buf`, performing an overflow calculation on the growth.
	392702ee	2015-02-09T23:41:13		allocations: test for overflow of requested size Introduce some helper macros to test integer overflow from arithmetic and set error message appropriately.
	4b1018d2	2015-01-08T17:24:12		Fix crash in free() when git_buf_grow() fails.
	92e0b679	2014-11-21T13:31:30		buffer: Do not `put` anything if len is 0
	e003f83a	2014-07-31T15:14:56		Introduce git_buf_decode_base64 Decode base64-encoded text into a git_buf
	b3af2d80	2014-07-16T13:34:25		Just put it all in buffer.
	1e4976cb	2014-05-08T10:17:14		Be more careful with user-supplied buffers This adds in missing calls to `git_buf_sanitize` and fixes a number of places where `git_buf` APIs could inadvertently write NUL terminator bytes into invalid buffers. This also changes the behavior of `git_buf_sanitize` to NUL terminate a buffer if it can and of `git_buf_shorten` to do nothing if it can. Adds tests of filtering code with zeroed (i.e. unsanitized) buffer which was previously triggering a segfault.
	e349ed50	2014-04-22T14:58:33		patch: emit binary patches (optionally)
	b3b36a68	2014-04-10T12:43:16		Introduce git_buf_putcn Allows for inserting the same character n amount of times
	18234b14	2014-02-21T09:14:16		Add efficient git_buf join3 API There are a few places where we need to join three strings to assemble a path. This adds a simple join3 function to avoid the comparatively expensive join_n (which calls strlen on each string twice).
	4f46a98b	2014-02-24T23:32:25		Remove now-duplicated stdarg.h include
	7cbc6241	2014-01-20T11:41:21		fix corner cases and an undefined behavior
	6adcaab7	2014-01-08T10:07:30		Handle git_buf's from users more liberally
	00e85927	2013-09-23T21:52:42		Clean up unnecessary git_buf_printf calls This replaces some git_buf_printf calls with simple calls to git_buf_put instead. Also, it fixes a missing va_end inside the git_buf_vprintf implementation.
	92d19d16	2013-09-21T09:34:03		Merge pull request #1840 from linquize/warning Fix warning
	66566516	2013-09-08T17:15:42		Fix warning
	a9f51e43	2013-09-11T22:00:36		Merge git_buf and git_buffer This makes the git_buf struct that was used internally into an externally available structure and eliminates the git_buffer. As part of that, some of the special cases that arose with the externally used git_buffer were blended into the git_buf, such as being careful about git_buf objects that may have a NULL ptr and allowing for bufs with a valid ptr and size but zero asize as a way of referring to externally owned data.
	40cb40fa	2013-09-11T14:23:39		Add functions to manipulate filter lists Extend the git2/sys/filter API with functions to look up a filter and add it manually to a filter list. This requires some trickery because the regular attribute lookups and checks are bypassed when this happens, but in the right hands, it will allow a user to have granular control over applying filters.
	2a7d224f	2013-09-10T16:33:32		Extend public filter api with filter lists This moves the git_filter_list into the public API so that users can create, apply, and dispose of filter lists. This allows more granular application of filters to user data outside of libgit2 internals. This also converts all the internal usage of filters to the public APIs along with a few small tweaks to make it easier to use the public git_buffer stuff alongside the internal git_buf.
	0cf77103	2013-08-26T23:17:07		Start of filter API + git_blob_filtered_content This begins the process of exposing git_filter objects to the public API. This includes: * new public type and API for `git_buffer` through which an allocated buffer can be passed to the user * new API `git_blob_filtered_content` * make the git_filter type and GIT_FILTER_TO_... constants public
	278ce746	2013-07-01T10:20:38		Add helpful buffer shorten function
	359fc2d2	2013-01-08T17:07:25		update copyrights
	7bf87ab6	2012-11-28T09:58:48		Consolidate text buffer functions There are many scattered functions that look into the contents of buffers to do various text manipulations (such as escaping or unescaping data, calculating text stats, guessing if content is binary, etc). This groups all those functions together into a new file and converts the code to use that. This has two enhancements to existing functionality. The old text stats function is significantly rewritten and the BOM detection code was extended (although largely we can't deal with anything other than a UTF8 BOM).
	3a14d3e2	2012-10-01T11:58:15		buf: introduce git_buf_splice()
	9d9288f4	2012-10-14T12:29:05		Fix buffer overrun in git_buf_put_base64
	2d3579be	2012-10-10T14:54:31		Add git_buf_put_base64 to buffer API
	0c8858de	2012-08-03T14:28:07		Fix valgrind issues and leaks This fixes up a number of problems flagged by valgrind and also cleans up the internal `git_submodule` allocation handling overall with a simpler model.
	02a0d651	2012-07-12T16:31:59		Add git_buf_unescape and git__unescape to unescape all characters in a string (in-place)
	54e29b93	2012-07-10T23:51:32		Fix missing NUL termination of buffer
	039fc406	2012-07-10T15:10:14		Add a couple of useful git_buf utilities * `git_buf_rfind` (with tests and tests for `git_buf_rfind_next`) * `git_buf_puts_escaped` and `git_buf_puts_escaped_regex` (with tests) to copy strings into a buffer while injecting an escape sequence (e.g. '\') in front of particular characters.
	3f035860	2012-06-07T22:43:03		misc: Fix warnings from PVS Studio trial
	e3557172	2012-05-17T14:44:17		No point in keeping commented out fn
	a0d95962	2012-05-17T13:14:17		Other optimization and warning fixes This fixes a warning left by the earlier optimization and addresses one of the other hotspots identified by GProf.
	b59c73d3	2012-05-17T13:06:20		Optimize away git_text_gather_stats in diff GProf shows `git_text_gather_stats` as the most expensive call in large diffs. The function calculates a lot of information that is not actually used and does not do so in a optimal order. This introduces a tuned `git_buf_is_binary` function that executes the same algorithm in a fraction of the time.
	2c833917	2012-05-15T16:33:05		Document git_buf_common_prefix This function fills in a git_buf with the common prefix of an array of strings, but let's make that a little more clear.
	41a82592	2012-05-15T14:17:39		Ranged iterators and rewritten git_status_file The goal of this work is to rewrite git_status_file to use the same underlying code as git_status_foreach. This is done in 3 phases: 1. Extend iterators to allow ranged iteration with start and end prefixes for the range of file names to be covered. 2. Improve diff so that when there is a pathspec and there is a common non-wildcard prefix of the pathspec, it will use ranged iterators to minimize excess iteration. 3. Rewrite git_status_file to call git_status_foreach_ext with a pathspec that covers just the one file being checked. Since ranged iterators underlie the status & diff implementation, this is actually fairly efficient. The workdir iterator does end up loading the contents of all the directories down to the single file, which should ideally be avoided, but it is pretty good.
	0f49200c	2012-05-09T04:37:02		msvc: Do not use `isspace` Locale-aware bullshit bitting my ass again yo
	baaf1c47	2012-05-02T23:44:22		buffer: Add `git_buf_vprintf`
	2bc8fa02	2012-04-17T10:14:24		Implement git_pool paged memory allocator This adds a `git_pool` object that can do simple paged memory allocation with free for the entire pool at once. Using this, you can replace many small allocations with large blocks that can then cheaply be doled out in small pieces. This is best used when you plan to free the small blocks all at once - for example, if they represent the parsed state from a file or data stream that are either all kept or all discarded. There are two real patterns of usage for `git_pools`: either for "string" allocation, where the item size is a single byte and you end up just packing the allocations in together, or for "fixed size" allocation where you are allocating a large object (e.g. a `git_oid`) and you generally just allocation single objects that can be tightly packed. Of course, you can use it for other things, but those two cases are the easiest.
	952f94c8	2012-03-30T14:42:23		Fix bug when join_n refers to original buffer There was a bug in git_buf_join_n when the contents of the original buffer were joined into itself and the realloc moved the pointer to the original buffer.
	0d0fa7c3	2012-03-16T15:56:01		Convert attr, ignore, mwindow, status to new errors Also cleaned up some previously converted code that still had little things to polish.
	deafee7b	2012-03-14T17:36:15		Continue error conversion This converts blob.c, fileops.c, and all of the win32 files. Also, various minor cleanups throughout the code. Plus, in testing the win32 build, I cleaned up a bunch (although not all) of the warnings with the 64-bit build.
	cb8a7961	2012-03-07T00:02:55		error-handling: Repository This also includes droping `git_buf_lasterror` because it makes no sense in the new system. Note that in most of the places were it has been dropped, the code needs cleanup. I.e. GIT_ENOMEM is going away, so instead it should return a generic `-1` and obviously not throw anything.
	ce49c7a8	2012-03-02T15:09:40		Add filter tests and fix some bugs This adds some initial unit tests for file filtering and fixes some simple bugs in filter application.
	eb8f90e5	2012-02-27T17:22:51		buffer: Null terminate on rtrim
	44b1ff4c	2012-02-27T04:31:05		filter: Apply filters before writing a file to the ODB Initial implementation. The relevant code is in `blob.c`: the blob write function has been split into smaller functions. - Directly write a file to the ODB in streaming mode - Directly write a symlink to the ODB in direct mode - Apply a filter, and write a file to the ODB in direct mode When trying to write a file, we first call `git_filter__load_for_file`, which populates a filters array with the required filters based on the filename. If no filters are resolved to the filename, we can write to the ODB in streaming mode straight from disk. Otherwise, we load the whole file in memory and use double-buffering to apply the filter chain. We finish by writing the file as a whole to the ODB.
	13224ea4	2012-02-27T04:28:31		buffer: Unify `git_fbuffer` and `git_buf` This makes so much sense that I can't believe it hasn't been done before. Kill the old `git_fbuffer` and read files straight into `git_buf` objects. Also: In order to fully support 4GB files in 32-bit systems, the `git_buf` implementation has been changed from using `ssize_t` for storage and storing negative values on allocation failure, to using `size_t` and changing the buffer pointer to a magical pointer on allocation failure. Hopefully this won't break anything.
	b6c93aef	2012-02-21T14:46:24		Uniform iterators for trees, index, and workdir This create a new git_iterator type of object that provides a uniform interface for iterating over the index, an arbitrary tree, or the working directory of a repository. As part of this, git ignore support was extended to support push and pop of directory-based ignore files as the working directory is being traversed (so the array of ignores does not have to be recreated at each directory during traveral). There are a number of other small utility functions in buffer, path, vector, and fileops that are included in this patch that made the iterator implementation cleaner.
	5e0de328	2012-02-13T17:10:24		Update Copyright header Signed-off-by: schu <schu-github@schulog.org>
	1dbcc9fc	2012-01-11T21:07:16		Fix several memory issues This contains fixes for several issues discovered by MSVC and by valgrind, including some bad data access, some memory leakage (in where certain files were not being successfully added to the cache), and some code simplification.
	df743c7d	2012-01-09T15:37:19		Initial implementation of gitignore support Adds support for .gitignore files to git_status_foreach() and git_status_file(). This includes refactoring the gitattributes code to share logic where possible. The GIT_STATUS_IGNORED flag will now be passed in for files that are ignored (provided they are not already in the index or the head of repo).
	b5daae68	2011-12-14T12:34:43		Allow git_buf_joinpath to accept self-joins It was not safe for git_buf_joinpath to be used with a pointer into the buf itself because a reallocation could invalidate the input parameter that pointed into the buffer. This patch makes it safe to self join, at least for the leading input to the join, which is the common "append" case for self joins. Also added unit tests to explicitly cover this case. This should actually fix #511
	bf6d2717	2011-12-14T03:27:53		buffer: inline `git_buf_cstr`
	97769280	2011-11-30T11:27:15		Use git_buf for path storage instead of stack-based buffers This converts virtually all of the places that allocate GIT_PATH_MAX buffers on the stack for manipulating paths to use git_buf objects instead. The patch is pretty careful not to touch the public API for libgit2, so there are a few places that still use GIT_PATH_MAX. This extends and changes some details of the git_buf implementation to add a couple of extra functions and to make error handling easier. This includes serious alterations to all the path.c functions, and several of the fileops.c ones, too. Also, there are a number of new functions that parallel existing ones except that use a git_buf instead of a stack-based buffer (such as git_config_find_global_r that exists alongsize git_config_find_global). This also modifies the win32 version of p_realpath to allocate whatever buffer size is needed to accommodate the realpath instead of hardcoding a GIT_PATH_MAX limit, but that change needs to be tested still.
	969d588d	2011-11-30T13:10:47		Optimized of git_buf_join. This streamlines git_buf_join and removes the join-append behavior, opting instead for a very compact join-replace of the git_buf contents. The unit tests had to be updated to remove the join-append tests and have a bunch more exhaustive tests added.
	309113c9	2011-11-29T23:45:17		Make initial value of git_buf ptr always be a valid empty string. Taking a page from core git's strbuf, this introduces git_buf_initbuf which is an empty string that is used to initialize the git_buf ptr value even for new buffers. Now the git_buf ptr will always point to a valid NUL-terminated string. This change required jumping through a few hoops for git_buf_grow and git_buf_free to distinguish between a actual allocated buffer and the global initial value. Also, this moves the allocation related functions to be next to each other near the top of buffer.c.
	c63728cd	2011-11-29T16:39:49		Make git_buf functions always maintain a valid cstr. At a tiny cost of 1 extra byte per allocation, this makes git_buf_cstr into basically a noop, which simplifies error checking when trying to convert things to use dynamic allocation. This patch also adds a new function (git_buf_copy_cstr) for copying the cstr data directly into an external buffer.
	679b69c4	2011-11-28T13:05:25		Resolve remaining feedback * replace some ints with size_ts * update NULL checks in various places
	3aa294fd	2011-11-28T10:42:57		Add two string git_buf_join and tweak input error checking. This commit addresses two of the comments: * renamed existing n-input git_buf_join to git_buf_join_n * added new git_buf_join that always takes two inputs * moved some parameter error checking to asserts * extended unit tests to cover new version of git_buf_join
	8c74d22e	2011-11-27T21:47:58		Extend git_buf with new utility functions and unit tests. Add new functions to git_buf for: * initializing a buffer from a string * joining one or more strings onto a buffer with separators * swapping two buffers in place * extracting data from a git_buf (leaving it empty) Also, make git_buf_free leave a git_buf back in its initted state, and slightly tweak buffer allocation sizes and thresholds. Finally, port unit tests to clay and extend with lots of new tests for the various git_buf functions.
	3286c408	2011-10-28T14:51:13		global: Properly use `git__` memory wrappers Ensure that all memory related functions (malloc, calloc, strdup, free, etc) are using their respective `git__` wrappers.
	8114ee4c	2011-09-22T10:17:43		Merge pull request #405 from carlosmn/http-ls Implement ls-remote over HTTP
	bb742ede	2011-09-19T01:54:32		Cleanup legal data 1. The license header is technically not valid if it doesn't have a copyright signature. 2. The COPYING file has been updated with the different licenses used in the project. 3. The full GPLv2 header in each file annoys me.
	b87600cb	2011-09-05T02:33:02		buffer: add git_buf_clear Set the size to zero so the memory that has already been allocated can be reused Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
	c7c30513	2011-09-05T21:38:56		buffer: add git_buf_consume Moves the content after 'end' to the beginning of the buffer Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
	afeecf4f	2011-07-09T02:10:46		odb: Direct writes are back DIRECT WRITES ARE BACK AND FASTER THAN EVER. The streaming writer to the ODB was an overkill for the smaller objects like Commit and Tags; most of the streaming logic was taking too long. This commit makes Commits, Tags and Trees to be built-up in memory, and then written to disk in 2 pushes (header + data), instead of streaming everything. This is *always* faster, even for big files (since the git_filebuf class still does streaming writes when the memory cache overflows). This is also a gazillion lines of code smaller, because we don't have to precompute the final size of the object before starting the stream (this was kind of defeating the point of streaming, anyway). Blobs are still written with full streaming instead of loading them in memory, since this is still the fastest way. A new `git_buf` class has been added. It's missing some features, but it'll get there.