src/mwindow.c


Log

Author Commit Date CI Message
Dimitris Apostolou 90df4302 2022-01-05T12:18:05 Fix typos
Calvin Buckley fcfaea4d 2021-07-07T23:29:28 Remove sus unsigned long constant
Calvin Buckley 52505ab5 2021-07-07T19:12:02 Convert long long constant specifiers to stdint macros
lhchavez 2f382ab7 2021-01-20T04:40:40 mwindow: Fix a bug in the LRU window finding code This change now updates the `lru_window` variable to match the current file's MRU window. This makes it such that it doesn't always choose the file that happened to come last in the list of window files, and instead should now correctly choose the file with the least-recently-used one.
Edward Thomson 37763d38 2020-12-05T15:26:59 threads: rename git_atomic to git_atomic32 Clarify the `git_atomic` type and functions now that we have a 64 bit version as well (`git_atomic64`).
lhchavez 322c15ee 2020-08-01T18:24:41 Make the pack and mwindow implementations data-race-free This change fixes a packfile heap corruption that can happen when interacting with multiple packfiles concurrently across multiple threads. This is exacerbated by setting a lower mwindow open file limit. This change: * Renames most of the internal methods in pack.c to clearly indicate that they expect to be called with a certain lock held, making reasoning about the state of locks a bit easier. * Splits the `git_pack_file` lock in two: the one in `git_pack_file` only protects the `index_map`. The protection to `git_mwindow_file` is now in that struct. * Explicitly checks for freshness of the `git_pack_file` in `git_packfile_unpack_header`: this allows the mwindow implementation to close files whenever there is enough cache pressure, and `git_packfile_unpack_header` will reopen the packfile if needed. * After a call to `p_munmap()`, the `data` and `len` fields are poisoned with `NULL` to make use-after-frees more evident and crash rather than being open to the possibility of heap corruption. * Adds a test case to prevent this from regressing in the future. Fixes: #5591
Edward Thomson a3e8b7cd 2020-04-05T17:18:20 mwindow: use GIT_ASSERT
Edward Thomson e316b0d3 2020-05-15T11:47:09 runtime: move init/shutdown into the "runtime" Provide a mechanism for system components to register for initialization and shutdown of the libgit2 runtime.
Edward Thomson 8aa69f88 2020-05-13T10:48:13 mwindow: localize mutex Move the mwindow mutex into the mwindow code itself, initializing it in the mwindow global initialization function instead of in the global initializer.
Andrew Hickman 04d59466 2020-08-05T22:04:15 Preserve behaviour of `git_mwindow_scan_recently_used` with `*out_window` set
Andrew Hickman 8ea9187a 2020-08-05T20:22:55 Fix `git_mwindow_scan_recently_used` spuriously returning true
lhchavez 92d42eb3 2020-07-12T09:53:10 Minor nits and style formatting
lhchavez eab2b044 2020-06-26T16:10:30 Review feedback * Change the default of the file limit to 0 (unlimited). * Changed the heuristic to close files to be the file that contains the least-recently-used window such that the window is the most-recently-used in the file, and the file does not have in-use windows. * Parameterized the filelimit test to check for a limit of 1 and 100 open windows.
lhchavez 9679df57 2020-02-08T20:47:24 mwindow: set limit on number of open files There are some cases in which repositories accrue a large number of packfiles. The existing mwindow limit applies only to the total size of mmap'd files, not on their number. This leads to a situation in which having lots of small packfiles could exhaust the allowed number of open files, particularly on macOS, where the default ulimit is very low (256). This change adds a new configuration parameter (GIT_OPT_SET_MWINDOW_FILE_LIMIT) that sets the maximum number of open packfiles, with a default of 128. This is low enough so that even macOS users should not hit it during normal use. Based on PR #5386, originally written by @josharian. Fixes: #2758
Edward Thomson 6460e8ab 2019-06-23T18:13:29 internal: use off64_t instead of git_off_t Prefer `off64_t` internally.
Patrick Steinhardt e54343a4 2019-06-29T09:17:32 fileops: rename to "futils.h" to match function signatures Our file utils functions all have a "futils" prefix, e.g. `git_futils_touch`. One would thus naturally guess that their definitions and implementation would live in files "futils.h" and "futils.c", respectively, but in fact they live in "fileops.h". Rename the files to match expectations.
Patrick Steinhardt 84a089da 2018-12-01T08:50:36 maps: provide return value when deleting entries Currently, the delete functions of maps do not provide a return value. Like this, it is impossible to tell whether the entry has really been deleted or not. Change the implementation to provide either a return value of zero if the entry has been successfully deleted or `GIT_ENOTFOUND` if the key could not be found. Convert callers to the `delete_at` functions to instead use this higher-level interface.
Patrick Steinhardt 03555830 2019-01-23T10:44:33 strmap: introduce high-level setter for key/value pairs Currently, one would use the function `git_strmap_insert` to insert key/value pairs into a map. This function has historically been a macro, which is why its syntax is kind of weird: instead of returning an error code directly, it instead has to be passed a pointer to where the return value shall be stored. This does not match libgit2's common idiom of directly returning error codes. Introduce a new function `git_strmap_set`, which takes as parameters the map, key and value and directly returns an error code. Convert all callers of `git_strmap_insert` to make use of it.
Patrick Steinhardt ef507bc7 2019-01-23T10:44:02 strmap: introduce `git_strmap_get` and use it throughout the tree The current way of looking up an entry from a map is tightly coupled with the map implementation, as one first has to look up the index of the key and then retrieve the associated value by using the index. As a caller, you usually do not care about any indices at all, though, so this is more complicated than really necessary. Furthermore, it invites for errors to happen if the correct error checking sequence is not being followed. Introduce a new high-level function `git_strmap_get` that takes a map and a key and returns a pointer to the associated value if such a key exists. Otherwise, a `NULL` pointer is returned. Adjust all callers that can trivially be converted.
Patrick Steinhardt 351eeff3 2019-01-23T10:42:46 maps: use uniform lifecycle management functions Currently, the lifecycle functions for maps (allocation, deallocation, resize) are not named in a uniform way and do not have a uniform function signature. Rename the functions to fix that, and stick to libgit2's naming scheme of saying `git_foo_new`. This results in the following new interface for allocation: - `int git_<t>map_new(git_<t>map **out)` to allocate a new map, returning an error code if we ran out of memory - `void git_<t>map_free(git_<t>map *map)` to free a map - `void git_<t>map_clear(git<t>map *map)` to remove all entries from a map This commit also fixes all existing callers.
Edward Thomson f673e232 2018-12-27T13:47:34 git_error: use new names in internal APIs and usage Move to the `git_error` name in the internal API for error-related functions.
Patrick Steinhardt 852bc9f4 2018-11-23T19:26:24 khash: remove intricate knowledge of khash types Instead of using the `khiter_t`, `git_strmap_iter` and `khint_t` types, simply use `size_t` instead. This decouples code from the khash stuff and makes it possible to move the khash includes into the implementation files.
Patrick Steinhardt 0c7f49dd 2017-06-30T13:39:01 Make sure to always include "common.h" first Next to including several files, our "common.h" header also declares various macros which are then used throughout the project. As such, we have to make sure to always include this file first in all implementation files. Otherwise, we might encounter problems or even silent behavioural differences due to macros or defines not being defined as they should be. So in fact, our header and implementation files should make sure to always include "common.h" first. This commit does so by establishing a common include pattern. Header files inside of "src" will now always include "common.h" as its first other file, separated by a newline from all the other includes to make it stand out as special. There are two cases for the implementation files. If they do have a matching header file, they will always include this one first, leading to "common.h" being transitively included as first file. If they do not have a matching header file, they instead include "common.h" as first file themselves. This fixes the outlined problems and will become our standard practice for header and source files inside of the "src/" from now on.
Patrick Steinhardt 13c3bc9a 2017-01-27T14:32:23 strmap: remove GIT__USE_STRMAP macro
Patrick Steinhardt 73028af8 2017-01-27T14:20:24 khash: avoid using macro magic to get return address
Edward Thomson 909d5494 2016-12-29T12:25:15 giterr_set: consistent error messages Error messages should be sentence fragments, and therefore: 1. Should not begin with a capital letter, 2. Should not conclude with punctuation, and 3. Should not end a sentence and begin a new one
Edward Thomson 2381d9e4 2016-08-03T17:01:48 mwindow: init mwindow files in git_libgit2_init
Carlos Martín Nieto d50fd571 2016-03-09T11:16:16 mwindow: free unused windows if we fail to mmap The first time may be due to memory fragmentation or just bad luck on a 32-bit system. When we hit the mmap error for the first time, free up the unused windows and try again.
Stefan Widgren c8e02b87 2015-02-15T21:07:05 Remove extra semicolon outside of a function Without this change, compiling with gcc and pedantic generates warning: ISO C does not allow extra ‘;’ outside of a function.
Edward Thomson 8c8ca730 2014-10-25T19:53:07 mwindow: clean up pack map at shutdown
Carlos Martín Nieto c19b1c04 2014-06-25T21:35:58 pack: clean up error returns Set a message when we fail to lock. Also make the put function void, since it's called from free, which cannot report errors. The only errors we can experience here are internal state corruption, so we assert that we are trying to put a pack which we have previously got.
Carlos Martín Nieto 5e0f47c3 2014-06-25T21:20:39 pack: free the new pack struct if we fail to insert If we fail to insert the packfile in the map, make sure to free it. This makes the free function only attempt to remove its mwindows from the global list if we have opened the packfile to avoid accessing the list unlocked.
Carlos Martín Nieto b3b66c57 2014-06-18T17:13:12 Share packs across repository instances Opening the same repository multiple times will currently open the same file multiple times, as well as map the same region of the file multiple times. This is not necessary, as the packfile data is immutable. Instead of opening and closing packfiles directly, introduce an indirection and allocate packfiles globally. This does mean locking on each packfile open, but we already use this lock for the global mwindow list so it doesn't introduce a new contention point.
Russell Belfer 53607868 2013-04-15T00:09:03 Further threading fixes This builds on the earlier thread safety work to make it so that setting the odb, index, refdb, or config for a repository is done in a threadsafe manner with minimized locking time. This is done by adding a lock to the repository object and using it to guard the assignment of the above listed pointers. The lock is only held to assign the pointer value. This also contains some minor fixes to the other work with pack files to reduce the time that locks are being held to and fix an apparently memory leak.
Arkadiy Shapkin 10c06114 2013-03-17T04:46:46 Several warnings detected by static code analyzer fixed Implicit type conversion argument of function to size_t type Suspicious sequence of types castings: size_t -> int -> size_t Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)' Unsigned type is never < 0
Vicent Marti 59853eff 2013-01-23T02:58:58 Global options setter
Edward Thomson 359fc2d2 2013-01-08T17:07:25 update copyrights
Scott J. Goldman f9b55bcb 2013-01-05T18:20:42 git_mwindow_file_deregister() shouldn't return errors As a function that appears to only be called on error paths, I don't think it makes sense for it to return an error, or clobber the global giterr. Note that no existing callsites actually check the return code. In my own application, there are errors where the real error ends up being hidden, as git_mwindow_file_deregister() clobbers the global giterr. I'm not sure this error is even relevant?
Justin Spahr-Summers a35b3864 2012-12-09T02:31:39 Always check the result of git_mutex_lock
Carlos Martín Nieto 8cef828d 2012-08-18T22:11:49 Make the memory-window conrol structures global Up to now, the idea was that the user would do all the operations for one repository in the same thread. Thus we could have the memory-mapped window information thread-local and avoid any locking. This is not practical in a few environments, such as Apple's GCD which allocates threads arbitrarily or the .NET CLR, where the OS-level thread can change at any moment. Make the control structure global and protect it with a mutex so we don't depend on the thread currently executing the code.
Carlos Martin Nieto 1d8943c6 2012-06-28T12:05:49 mwindow: allow memory-window files to deregister Once a file is registered, there is no way to deregister it, even after the structure that contains it is no longer needed and has been freed. This may be the source of #624. Allow and use the deregister function to remove our file from the global list.
Chris Young 2aeadb9c 2012-06-12T19:25:09 Actually do the mmap... unsurprisingly, this makes the indexer work on SFS On RAM: the .idx and .pack files become links to a .lock and the original download respectively. Assume some feature (such as record locking) supported by SFS but not JXFS or RAM: is required.
Carlos Martín Nieto 8bf10dba 2012-05-23T12:59:21 Remove left-over debugging output
Vicent Martí 946a6dc4 2012-05-02T16:14:30 Update test suite
Vicent Martí 40879fac 2012-05-02T15:59:02 Merge branch 'new-error-handling' into development Conflicts: .travis.yml include/git2/diff.h src/config_file.c src/diff.c src/diff_output.c src/mwindow.c src/path.c tests-clar/clar_helpers.c tests-clar/object/tree/frompath.c tests/t00-core.c tests/t03-objwrite.c tests/t08-tag.c tests/t10-refs.c tests/t12-repo.c tests/t18-status.c tests/test_helpers.c tests/test_main.c
Carlos Martín Nieto 3f93e16c 2012-03-29T17:49:57 indexer: start writing the stream indexer This will allow us to index a packfile as soon as we receive it from the network as well as storing it with its final name so we don't need to pass temporary file names around.
Carlos Martín Nieto 31e80290 2012-04-04T16:21:52 mwindow: make sure the whole range is contained inside the same window Looking through the open windows to check whether we can re-use an open window should take into account whether both `offset` and `offset + extra` are contained within the same window. Failure to do so can lead to invalid memory accesses. This closes #614. While we're in the area remove an outdated assert.
Russell Belfer 0d0fa7c3 2012-03-16T15:56:01 Convert attr, ignore, mwindow, status to new errors Also cleaned up some previously converted code that still had little things to polish.
Russell Belfer deafee7b 2012-03-14T17:36:15 Continue error conversion This converts blob.c, fileops.c, and all of the win32 files. Also, various minor cleanups throughout the code. Plus, in testing the win32 build, I cleaned up a bunch (although not all) of the warnings with the 64-bit build.
Russell Belfer e1de726c 2012-03-12T22:55:40 Migrate ODB files to new error handling This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to the new style of error handling. Also got the unix and win32 versions of map.c. There are some minor changes to other files but no others were completely converted. This also contains an update to filebuf so that a zeroed out filebuf will not think that the fd (== 0) is actually open (and inadvertently call close() on fd 0 if cleaned up). Lastly, this was built and tested on win32 and contains a bunch of fixes for the win32 build which was pretty broken.
schu 5e0de328 2012-02-13T17:10:24 Update Copyright header Signed-off-by: schu <schu-github@schulog.org>
Vicent Marti a15c550d 2011-11-16T14:09:44 threads: Fix the shared global state with TLS See `global.c` for a description of what we're doing. When libgit2 is built with GIT_THREADS support, the threading system must be explicitly initialized with `git_threads_init()`.
Vicent Marti 3286c408 2011-10-28T14:51:13 global: Properly use `git__` memory wrappers Ensure that all memory related functions (malloc, calloc, strdup, free, etc) are using their respective `git__` wrappers.
Carlos Martín Nieto 5fa1bed0 2011-10-15T23:09:05 mwindow: close LRU window properly Remove a wrong call to git_mwindow_close which caused a segfault if it ever did run. In that same piece of code, if the LRU was from the first wiindow in the list in a different file, we didn't update that list, so the first element had been freed. Fix these two issues. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Vicent Martí 71a4c1f1 2011-09-18T20:07:59 Merge pull request #384 from kiryl/warnings Add more -W flags to CFLAGS
Vicent Marti 87d9869f 2011-09-19T03:34:49 Tabify everything There were quite a few places were spaces were being used instead of tabs. Try to catch them all. This should hopefully not break anything. Except for `git blame`. Oh well.
Vicent Marti bb742ede 2011-09-19T01:54:32 Cleanup legal data 1. The license header is technically not valid if it doesn't have a copyright signature. 2. The COPYING file has been updated with the different licenses used in the project. 3. The full GPLv2 header in each file annoys me.
Sebastian Schuberth 18136d83 2011-09-08T16:43:58 Fix an integral overflow on 64-bit
Kirill A. Shutemov d568d585 2011-08-30T23:55:22 CMakefile: add -Wmissing-prototypes and fix warnings Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
schu 33e9ee8f 2011-08-09T12:55:51 mwindow.c: fix -Wmissing-field-initializers Signed-off-by: schu <schu-github@schulog.org>
Vicent Marti f6867e63 2011-08-08T16:56:28 Fix compilation in Windows
Carlos Martín Nieto 7d0cdf82 2011-07-09T02:25:01 Make packfile_unpack_header more generic On the way, store the fd and the size in the mwindow file. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Carlos Martín Nieto 7bfdb3d2 2011-06-28T20:39:30 Factor out the mmap window code This code is useful for more things than just the packfile handling code. Factor it out so it can be reused. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>