|
37763d38
|
2020-12-05T15:26:59
|
|
threads: rename git_atomic to git_atomic32
Clarify the `git_atomic` type and functions now that we have a 64 bit
version as well (`git_atomic64`).
|
|
322c15ee
|
2020-08-01T18:24:41
|
|
Make the pack and mwindow implementations data-race-free
This change fixes a packfile heap corruption that can happen when
interacting with multiple packfiles concurrently across multiple
threads. This is exacerbated by setting a lower mwindow open file limit.
This change:
* Renames most of the internal methods in pack.c to clearly indicate
that they expect to be called with a certain lock held, making
reasoning about the state of locks a bit easier.
* Splits the `git_pack_file` lock in two: the one in `git_pack_file`
only protects the `index_map`. The protection to `git_mwindow_file` is
now in that struct.
* Explicitly checks for freshness of the `git_pack_file` in
`git_packfile_unpack_header`: this allows the mwindow implementation
to close files whenever there is enough cache pressure, and
`git_packfile_unpack_header` will reopen the packfile if needed.
* After a call to `p_munmap()`, the `data` and `len` fields are poisoned
with `NULL` to make use-after-frees more evident and crash rather than
being open to the possibility of heap corruption.
* Adds a test case to prevent this from regressing in the future.
Fixes: #5591
|
|
|
a3e8b7cd
|
2020-04-05T17:18:20
|
|
mwindow: use GIT_ASSERT
|
|
|
e316b0d3
|
2020-05-15T11:47:09
|
|
runtime: move init/shutdown into the "runtime"
Provide a mechanism for system components to register for initialization
and shutdown of the libgit2 runtime.
|
|
|
8aa69f88
|
2020-05-13T10:48:13
|
|
mwindow: localize mutex
Move the mwindow mutex into the mwindow code itself, initializing it in
the mwindow global initialization function instead of in the global
initializer.
|
|
04d59466
|
2020-08-05T22:04:15
|
|
Preserve behaviour of `git_mwindow_scan_recently_used` with `*out_window` set
|
|
|
8ea9187a
|
2020-08-05T20:22:55
|
|
Fix `git_mwindow_scan_recently_used` spuriously returning true
|
|
|
92d42eb3
|
2020-07-12T09:53:10
|
|
Minor nits and style formatting
|
|
|
eab2b044
|
2020-06-26T16:10:30
|
|
Review feedback
* Change the default of the file limit to 0 (unlimited).
* Changed the heuristic to close files to be the file that contains the
least-recently-used window such that the window is the
most-recently-used in the file, and the file does not have in-use
windows.
* Parameterized the filelimit test to check for a limit of 1 and 100
open windows.
|
|
|
9679df57
|
2020-02-08T20:47:24
|
|
mwindow: set limit on number of open files
There are some cases in which repositories accrue a large number of
packfiles. The existing mwindow limit applies only to the total size of
mmap'd files, not on their number. This leads to a situation in which
having lots of small packfiles could exhaust the allowed number of open
files, particularly on macOS, where the default ulimit is very low
(256).
This change adds a new configuration parameter
(GIT_OPT_SET_MWINDOW_FILE_LIMIT) that sets the maximum number of open
packfiles, with a default of 128. This is low enough so that even macOS
users should not hit it during normal use.
Based on PR #5386, originally written by @josharian.
Fixes: #2758
|
|
|
6460e8ab
|
2019-06-23T18:13:29
|
|
internal: use off64_t instead of git_off_t
Prefer `off64_t` internally.
|
|
e54343a4
|
2019-06-29T09:17:32
|
|
fileops: rename to "futils.h" to match function signatures
Our file utils functions all have a "futils" prefix, e.g.
`git_futils_touch`. One would thus naturally guess that their
definitions and implementation would live in files "futils.h" and
"futils.c", respectively, but in fact they live in "fileops.h".
Rename the files to match expectations.
|
|
|
84a089da
|
2018-12-01T08:50:36
|
|
maps: provide return value when deleting entries
Currently, the delete functions of maps do not provide a return value. Like
this, it is impossible to tell whether the entry has really been deleted or not.
Change the implementation to provide either a return value of zero if the entry
has been successfully deleted or `GIT_ENOTFOUND` if the key could not be found.
Convert callers to the `delete_at` functions to instead use this higher-level
interface.
|
|
|
03555830
|
2019-01-23T10:44:33
|
|
strmap: introduce high-level setter for key/value pairs
Currently, one would use the function `git_strmap_insert` to insert key/value
pairs into a map. This function has historically been a macro, which is why its
syntax is kind of weird: instead of returning an error code directly, it instead
has to be passed a pointer to where the return value shall be stored. This does
not match libgit2's common idiom of directly returning error codes.
Introduce a new function `git_strmap_set`, which takes as parameters the map,
key and value and directly returns an error code. Convert all callers of
`git_strmap_insert` to make use of it.
|
|
|
ef507bc7
|
2019-01-23T10:44:02
|
|
strmap: introduce `git_strmap_get` and use it throughout the tree
The current way of looking up an entry from a map is tightly coupled with the
map implementation, as one first has to look up the index of the key and then
retrieve the associated value by using the index. As a caller, you usually do
not care about any indices at all, though, so this is more complicated than
really necessary. Furthermore, it invites for errors to happen if the correct
error checking sequence is not being followed.
Introduce a new high-level function `git_strmap_get` that takes a map and a key
and returns a pointer to the associated value if such a key exists. Otherwise,
a `NULL` pointer is returned. Adjust all callers that can trivially be
converted.
|
|
|
351eeff3
|
2019-01-23T10:42:46
|
|
maps: use uniform lifecycle management functions
Currently, the lifecycle functions for maps (allocation, deallocation, resize)
are not named in a uniform way and do not have a uniform function signature.
Rename the functions to fix that, and stick to libgit2's naming scheme of saying
`git_foo_new`. This results in the following new interface for allocation:
- `int git_<t>map_new(git_<t>map **out)` to allocate a new map, returning an
error code if we ran out of memory
- `void git_<t>map_free(git_<t>map *map)` to free a map
- `void git_<t>map_clear(git<t>map *map)` to remove all entries from a map
This commit also fixes all existing callers.
|
|
|
f673e232
|
2018-12-27T13:47:34
|
|
git_error: use new names in internal APIs and usage
Move to the `git_error` name in the internal API for error-related
functions.
|
|
|
852bc9f4
|
2018-11-23T19:26:24
|
|
khash: remove intricate knowledge of khash types
Instead of using the `khiter_t`, `git_strmap_iter` and `khint_t` types,
simply use `size_t` instead. This decouples code from the khash stuff
and makes it possible to move the khash includes into the implementation
files.
|
|
|
0c7f49dd
|
2017-06-30T13:39:01
|
|
Make sure to always include "common.h" first
Next to including several files, our "common.h" header also declares
various macros which are then used throughout the project. As such, we
have to make sure to always include this file first in all
implementation files. Otherwise, we might encounter problems or even
silent behavioural differences due to macros or defines not being
defined as they should be. So in fact, our header and implementation
files should make sure to always include "common.h" first.
This commit does so by establishing a common include pattern. Header
files inside of "src" will now always include "common.h" as its first
other file, separated by a newline from all the other includes to make
it stand out as special. There are two cases for the implementation
files. If they do have a matching header file, they will always include
this one first, leading to "common.h" being transitively included as
first file. If they do not have a matching header file, they instead
include "common.h" as first file themselves.
This fixes the outlined problems and will become our standard practice
for header and source files inside of the "src/" from now on.
|
|
|
13c3bc9a
|
2017-01-27T14:32:23
|
|
strmap: remove GIT__USE_STRMAP macro
|
|
|
73028af8
|
2017-01-27T14:20:24
|
|
khash: avoid using macro magic to get return address
|
|
909d5494
|
2016-12-29T12:25:15
|
|
giterr_set: consistent error messages
Error messages should be sentence fragments, and therefore:
1. Should not begin with a capital letter,
2. Should not conclude with punctuation, and
3. Should not end a sentence and begin a new one
|
|
|
2381d9e4
|
2016-08-03T17:01:48
|
|
mwindow: init mwindow files in git_libgit2_init
|
|
d50fd571
|
2016-03-09T11:16:16
|
|
mwindow: free unused windows if we fail to mmap
The first time may be due to memory fragmentation or just bad luck on a
32-bit system. When we hit the mmap error for the first time, free up
the unused windows and try again.
|
|
c8e02b87
|
2015-02-15T21:07:05
|
|
Remove extra semicolon outside of a function
Without this change, compiling with gcc and pedantic generates warning:
ISO C does not allow extra ‘;’ outside of a function.
|
|
|
8c8ca730
|
2014-10-25T19:53:07
|
|
mwindow: clean up pack map at shutdown
|
|
|
c19b1c04
|
2014-06-25T21:35:58
|
|
pack: clean up error returns
Set a message when we fail to lock.
Also make the put function void, since it's called from free, which
cannot report errors. The only errors we can experience here are
internal state corruption, so we assert that we are trying to put a
pack which we have previously got.
|
|
|
5e0f47c3
|
2014-06-25T21:20:39
|
|
pack: free the new pack struct if we fail to insert
If we fail to insert the packfile in the map, make sure to free it.
This makes the free function only attempt to remove its mwindows from
the global list if we have opened the packfile to avoid accessing the
list unlocked.
|
|
|
b3b66c57
|
2014-06-18T17:13:12
|
|
Share packs across repository instances
Opening the same repository multiple times will currently open the same
file multiple times, as well as map the same region of the file multiple
times. This is not necessary, as the packfile data is immutable.
Instead of opening and closing packfiles directly, introduce an
indirection and allocate packfiles globally. This does mean locking on
each packfile open, but we already use this lock for the global mwindow
list so it doesn't introduce a new contention point.
|
|
53607868
|
2013-04-15T00:09:03
|
|
Further threading fixes
This builds on the earlier thread safety work to make it so that
setting the odb, index, refdb, or config for a repository is done
in a threadsafe manner with minimized locking time. This is done
by adding a lock to the repository object and using it to guard
the assignment of the above listed pointers. The lock is only
held to assign the pointer value.
This also contains some minor fixes to the other work with pack
files to reduce the time that locks are being held to and fix an
apparently memory leak.
|
|
10c06114
|
2013-03-17T04:46:46
|
|
Several warnings detected by static code analyzer fixed
Implicit type conversion argument of function to size_t type
Suspicious sequence of types castings: size_t -> int -> size_t
Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)'
Unsigned type is never < 0
|
|
59853eff
|
2013-01-23T02:58:58
|
|
Global options setter
|
|
|
359fc2d2
|
2013-01-08T17:07:25
|
|
update copyrights
|
|
f9b55bcb
|
2013-01-05T18:20:42
|
|
git_mwindow_file_deregister() shouldn't return errors
As a function that appears to only be called on error paths, I don't
think it makes sense for it to return an error, or clobber the global
giterr. Note that no existing callsites actually check the return
code.
In my own application, there are errors where the real error ends
up being hidden, as git_mwindow_file_deregister() clobbers the
global giterr. I'm not sure this error is even relevant?
|
|
a35b3864
|
2012-12-09T02:31:39
|
|
Always check the result of git_mutex_lock
|
|
8cef828d
|
2012-08-18T22:11:49
|
|
Make the memory-window conrol structures global
Up to now, the idea was that the user would do all the operations for
one repository in the same thread. Thus we could have the
memory-mapped window information thread-local and avoid any locking.
This is not practical in a few environments, such as Apple's GCD which
allocates threads arbitrarily or the .NET CLR, where the OS-level
thread can change at any moment.
Make the control structure global and protect it with a mutex so we
don't depend on the thread currently executing the code.
|
|
1d8943c6
|
2012-06-28T12:05:49
|
|
mwindow: allow memory-window files to deregister
Once a file is registered, there is no way to deregister it, even
after the structure that contains it is no longer needed and has been
freed. This may be the source of #624.
Allow and use the deregister function to remove our file from the
global list.
|
|
2aeadb9c
|
2012-06-12T19:25:09
|
|
Actually do the mmap... unsurprisingly, this makes the indexer work on SFS
On RAM: the .idx and .pack files become links to a .lock and the original download respectively.
Assume some feature (such as record locking) supported by SFS but not JXFS or RAM: is required.
|
|
|
8bf10dba
|
2012-05-23T12:59:21
|
|
Remove left-over debugging output
|
|
|
946a6dc4
|
2012-05-02T16:14:30
|
|
Update test suite
|
|
|
40879fac
|
2012-05-02T15:59:02
|
|
Merge branch 'new-error-handling' into development
Conflicts:
.travis.yml
include/git2/diff.h
src/config_file.c
src/diff.c
src/diff_output.c
src/mwindow.c
src/path.c
tests-clar/clar_helpers.c
tests-clar/object/tree/frompath.c
tests/t00-core.c
tests/t03-objwrite.c
tests/t08-tag.c
tests/t10-refs.c
tests/t12-repo.c
tests/t18-status.c
tests/test_helpers.c
tests/test_main.c
|
|
|
3f93e16c
|
2012-03-29T17:49:57
|
|
indexer: start writing the stream indexer
This will allow us to index a packfile as soon as we receive it from
the network as well as storing it with its final name so we don't need
to pass temporary file names around.
|
|
|
31e80290
|
2012-04-04T16:21:52
|
|
mwindow: make sure the whole range is contained inside the same window
Looking through the open windows to check whether we can re-use an
open window should take into account whether both `offset` and `offset
+ extra` are contained within the same window. Failure to do so can
lead to invalid memory accesses. This closes #614.
While we're in the area remove an outdated assert.
|
|
0d0fa7c3
|
2012-03-16T15:56:01
|
|
Convert attr, ignore, mwindow, status to new errors
Also cleaned up some previously converted code that still had
little things to polish.
|
|
|
deafee7b
|
2012-03-14T17:36:15
|
|
Continue error conversion
This converts blob.c, fileops.c, and all of the win32 files.
Also, various minor cleanups throughout the code. Plus, in
testing the win32 build, I cleaned up a bunch (although not
all) of the warnings with the 64-bit build.
|
|
|
e1de726c
|
2012-03-12T22:55:40
|
|
Migrate ODB files to new error handling
This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to
the new style of error handling. Also got the unix and win32
versions of map.c. There are some minor changes to other
files but no others were completely converted.
This also contains an update to filebuf so that a zeroed out
filebuf will not think that the fd (== 0) is actually open
(and inadvertently call close() on fd 0 if cleaned up).
Lastly, this was built and tested on win32 and contains a
bunch of fixes for the win32 build which was pretty broken.
|
|
5e0de328
|
2012-02-13T17:10:24
|
|
Update Copyright header
Signed-off-by: schu <schu-github@schulog.org>
|
|
|
a15c550d
|
2011-11-16T14:09:44
|
|
threads: Fix the shared global state with TLS
See `global.c` for a description of what we're doing.
When libgit2 is built with GIT_THREADS support, the threading system
must be explicitly initialized with `git_threads_init()`.
|
|
|
3286c408
|
2011-10-28T14:51:13
|
|
global: Properly use `git__` memory wrappers
Ensure that all memory related functions (malloc, calloc, strdup, free,
etc) are using their respective `git__` wrappers.
|
|
|
5fa1bed0
|
2011-10-15T23:09:05
|
|
mwindow: close LRU window properly
Remove a wrong call to git_mwindow_close which caused a segfault if it
ever did run. In that same piece of code, if the LRU was from the
first wiindow in the list in a different file, we didn't update that
list, so the first element had been freed.
Fix these two issues.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
71a4c1f1
|
2011-09-18T20:07:59
|
|
Merge pull request #384 from kiryl/warnings
Add more -W flags to CFLAGS
|
|
|
87d9869f
|
2011-09-19T03:34:49
|
|
Tabify everything
There were quite a few places were spaces were being used instead of
tabs. Try to catch them all. This should hopefully not break anything.
Except for `git blame`. Oh well.
|
|
|
bb742ede
|
2011-09-19T01:54:32
|
|
Cleanup legal data
1. The license header is technically not valid if it doesn't have a
copyright signature.
2. The COPYING file has been updated with the different licenses used in
the project.
3. The full GPLv2 header in each file annoys me.
|
|
18136d83
|
2011-09-08T16:43:58
|
|
Fix an integral overflow on 64-bit
|
|
d568d585
|
2011-08-30T23:55:22
|
|
CMakefile: add -Wmissing-prototypes and fix warnings
Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
|
|
|
33e9ee8f
|
2011-08-09T12:55:51
|
|
mwindow.c: fix -Wmissing-field-initializers
Signed-off-by: schu <schu-github@schulog.org>
|
|
|
f6867e63
|
2011-08-08T16:56:28
|
|
Fix compilation in Windows
|
|
|
7d0cdf82
|
2011-07-09T02:25:01
|
|
Make packfile_unpack_header more generic
On the way, store the fd and the size in the mwindow file.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
7bfdb3d2
|
2011-06-28T20:39:30
|
|
Factor out the mmap window code
This code is useful for more things than just the packfile handling
code. Factor it out so it can be reused.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|