src/pack.c


Log

Author Commit Date CI Message
Russell Belfer 1a42dd17 2013-05-31T14:13:11 Mutex init can fail It is obviously quite a serious problem if this happens, but mutex initialization can fail and we should detect it. It's a bit like a memory allocation failure, in that you're probably pretty screwed if this occurs, but at least we'll catch it.
Russell Belfer f658dc43 2013-05-31T14:09:58 Zero memory for major objects before freeing By zeroing out the memory when we free larger objects (i.e. those that serve as collections of other data, such as repos, odb, refdb), I'm hoping that it will be easier for libgit2 bindings to find errors in their object management code.
Carlos Martín Nieto 0ddfcb40 2013-05-02T18:06:14 Switch to index_version as "git_pack_file is ready" flag We use p->index_map.data to check whether the struct has been set up and all the information about the index is stored there. This variable gets set up halfway through the setup process, however, and a thread can come along and use fields that haven't been written to yet. Crucially, pack_entry_find_offset() needs to read the index version (which is written after index_map) to know the offset and stride length to pass to sha1_entry_pos(). If these values are wrong, assertions in it will fail, as it will be reading bogus data. Make index_version the last field to be written and switch from using p->index_map.data to p->index_version as "git_pack_file is ready" flag as we can use it to know if every field has been written.
Carlos Martín Nieto 34bd5999 2013-05-02T17:14:05 Revert "Protect sha1_entry_pos call with mutex" This reverts commit 8c535f3f6879c6796d8107d7eb80dd8b2105621b.
Russell Belfer 8c535f3f 2013-05-02T03:34:56 Protect sha1_entry_pos call with mutex There is an occasional assertion failure in sha1_entry_pos from pack_entry_find_index when running threaded. Holding the mutex around the code that grabs the index_map data and processes it makes this assertion failure go away.
Russell Belfer 9d2f841a 2013-05-02T03:03:54 Add extra locking around packfile open We were still seeing a few issues in threaded access to packs. This adds extra locks around the opening of the mwindow to avoid a different race.
Russell Belfer b7f167da 2013-04-29T13:52:12 Make git_oid_cmp public and add git_oid__cmp
Russell Belfer 5d2d21e5 2013-04-16T15:00:43 Consolidate packfile allocation further Rename git_packfile_check to git_packfile_alloc since it is now being used more in that capacity. Fix the various places that use it. Consolidate some repeated code in odb_pack.c related to the allocation of a new pack_backend.
Russell Belfer 38eef611 2013-04-16T14:19:27 Make indexer use shared packfile open code The indexer was creating a packfile object separately from the code in pack.c which was a problem since I put a call to git_mutex_init into just pack.c. This commit updates the pack function for creating a new pack object (i.e. git_packfile_check()) so that it can be used in both places and then makes indexer.c use the shared initialization routine. There are also a few minor formatting and warning message fixes.
Russell Belfer 53607868 2013-04-15T00:09:03 Further threading fixes This builds on the earlier thread safety work to make it so that setting the odb, index, refdb, or config for a repository is done in a threadsafe manner with minimized locking time. This is done by adding a lock to the repository object and using it to guard the assignment of the above listed pointers. The lock is only held to assign the pointer value. This also contains some minor fixes to the other work with pack files to reduce the time that locks are being held to and fix an apparently memory leak.
Russell Belfer 24c70804 2013-04-12T12:59:38 Add mutex around mapping and unmapping pack files When I was writing threading tests for the new cache, the main error I kept running into was a pack file having it's content unmapped underneath the running thread. This adds a lock around the routines that map and unmap the pack data so that threads can effectively reload the data when they need it. This also required reworking the error handling paths in a couple places in the code which I tried to make consistent.
Carlos Martín Nieto 0e040c03 2013-03-03T14:50:47 indexer: use a hashtable for keeping track of offsets These offsets are needed for REF_DELTA objects, which encode which object they use as a base, but not where it lies in the packfile, so we need a list. These objects are mostly from older packfiles, before OFS_DELTA was widely spread. The time spent in indexing these packfiles is greatly reduced, though remains above what git is able to do.
Philip Kelley 11d9f6b3 2013-01-27T14:17:07 Vector improvements and their fallout
Philip Kelley aa3bf89d 2013-01-26T15:12:53 Fix a mutex leak in pack.c
Carlos Martín Nieto 9c62aaab 2013-01-14T17:42:12 pack: evict all of the pages at once Somewhat surprisingly, this can increase the speed considerably, as we don't bother trying to decide what to evict, and the most used entries are quickly back into the cache.
Carlos Martín Nieto ed6648ba 2013-01-14T16:39:22 pack: evict objects from the cache in groups of eight This drops the cache eviction below libcrypto and zlib in the perf output. The number has been chosen empirically.
Carlos Martín Nieto 09e29e47 2013-01-12T19:31:07 pack: fixes to the cache The offset should be git_off_t, and we should check the return value of the mutex lock function.
Carlos Martín Nieto 96c9b9f0 2013-01-12T18:38:19 indexer: properly free the packfile resources The indexer needs to call the packfile's free function so it takes care of freeing the caches. We still need to close the mwf descriptor manually so we can rename the packfile into its final name on Windows.
Carlos Martín Nieto 80d647ad 2013-01-11T20:15:06 Revert "pack: packfile_free -> git_packfile_free and use it in the indexers" This reverts commit f289f886cb81bb570bed747053d5ebf8aba6bef7, which makes the tests fail on Windows. Revert until we can figure out a solution.
nulltoken 090d5e1f 2013-01-11T14:40:09 Fix MSVC compilation warnings
Carlos Martín Nieto f289f886 2013-01-11T17:24:52 pack: packfile_free -> git_packfile_free and use it in the indexers It turns out the indexers have been ignoring the pack's free function and leaking data. Plug that.
Carlos Martín Nieto 0ed75620 2012-12-21T13:46:48 pack: limit the amount of memory the base delta cache can use Currently limited to 16MB (like git) and to objects up to 1MB in size.
Carlos Martín Nieto c8f79c2b 2012-12-21T10:59:10 pack: abstract out the cache into its own functions
Carlos Martín Nieto 525d961c 2012-12-20T07:55:51 pack: refcount entries and add a mutex around cache access
Carlos Martín Nieto c0f4a011 2012-12-19T16:48:12 pack: introduce a delta base cache Many delta bases are re-used. Cache them to avoid inflating the same data repeatedly. This version doesn't limit the amount of entries to store, so it can end up using a considerable amound of memory.
Edward Thomson 359fc2d2 2013-01-08T17:07:25 update copyrights
Vicent Martí 0249a503 2012-12-07T09:40:21 Merge pull request #1091 from carlosmn/stream-object Indexer speedup with large objects
David Michael Barr 44f9f547 2012-11-30T13:33:30 pack: add git_packfile_resolve_header To paraphrase @peff: You can get both size and type from a packed object reasonably cheaply. If you have: * An object that is not a delta; both type and size are available in the packfile header. * An object that is a delta. The packfile type will be OBJ_*_DELTA, and you have to resolve back to the base to find the real type. That means potentially a lot of packfile index lookups, but each one is relatively cheap. For the size, you inflate the first few bytes of the delta, whose header will tell you the resulting size of applying the delta to the base. For simplicity, we just decompress the whole delta for now.
Carlos Martín Nieto 46635339 2012-11-19T22:22:33 pack: introduce a streaming API for raw objects This allows us to take objects from the packfile as a stream instead of having to keep it all in memory.
Russell Belfer c3fb7d04 2012-11-27T15:00:49 Make git_odb_foreach_cb take const param This makes the first OID param of the ODB callback a const pointer and also propogates that change all the way to the backends.
Sven Strickroth fcb48e06 2012-11-24T15:48:17 Set p->mwf.fd to -1 on error If p->mwf.fd is e.g. -2 then it is closed in packfile_free and an exception might be thrown. Signed-off-by: Sven Strickroth <email@cs-ware.de>
Martin Woodward 826bc4a8 2012-11-23T13:31:22 Remove use of English expletives Remove words such as fuck, crap, shit etc. Remove other potentially offensive words from comments. Tidy up other geopolicital terms in comments.
David Michael Barr 60ecdf59 2012-09-10T11:48:21 pack: iterate objects in offset order Compute the ordering on demand and persist until the index is freed.
Vicent Marti 51e1d808 2012-08-06T12:41:08 Merge remote-tracking branch 'arrbee/tree-walk-fixes' into development Conflicts: src/notes.c src/transports/git.c src/transports/http.c src/transports/local.c tests-clar/odb/foreach.c
Russell Belfer 5dca2010 2012-08-03T17:08:01 Update iterators for consistency across library This updates all the `foreach()` type functions across the library that take callbacks from the user to have a consistent behavior. The rules are: * A callback terminates the loop by returning any non-zero value * Once the callback returns non-zero, it will not be called again (i.e. the loop stops all iteration regardless of state) * If the callback returns non-zero, the parent fn returns GIT_EUSER * Although the parent returns GIT_EUSER, no error will be set in the library and `giterr_last()` will return NULL if called. This commit makes those changes across the library and adds tests for most of the iteration APIs to make sure that they follow the above rules.
nulltoken b8457baa 2012-07-24T07:57:58 portability: Improve x86/amd64 compatibility
Carlos Martín Nieto 521aedad 2012-06-05T14:48:51 odb: add git_odb_foreach() Go through each backend and list every objects that exists in them. This allows fsck-like uses.
Carlos Martin Nieto 1d8943c6 2012-06-28T12:05:49 mwindow: allow memory-window files to deregister Once a file is registered, there is no way to deregister it, even after the structure that contains it is no longer needed and has been freed. This may be the source of #624. Allow and use the deregister function to remove our file from the global list.
Chris Young 2aeadb9c 2012-06-12T19:25:09 Actually do the mmap... unsurprisingly, this makes the indexer work on SFS On RAM: the .idx and .pack files become links to a .lock and the original download respectively. Assume some feature (such as record locking) supported by SFS but not JXFS or RAM: is required.
Chris Young 90490113 2012-06-10T18:08:15 Basic mmap/munmap compatiblity
Vicent Martí 904b67e6 2012-05-18T01:48:50 errors: Rename error codes
Vicent Martí e172cf08 2012-05-18T01:21:06 errors: Rename the generic return codes
Russell Belfer 282283ac 2012-05-04T16:46:46 Fix valgrind issues There are three changes here: - correctly propogate error code from failed object lookups - make zlib inflate use our allocators - add OID to notfound error in ODB lookups
Vicent Martí f9f2344b 2012-04-23T17:28:11 Merge pull request #632 from arrbee/win64-cleanup Code clean up, including fixing warnings on Windows 64-bit build
Russell Belfer 44ef8b1b 2012-04-13T13:00:10 Fix warnings on 64-bit windows builds This fixes all the warnings on win64 except those in deps, which come from the regex code.
Carlos Martín Nieto 45d773ef 2012-04-05T23:36:14 pack: signal a short buffer when needed
Russell Belfer 0d0fa7c3 2012-03-16T15:56:01 Convert attr, ignore, mwindow, status to new errors Also cleaned up some previously converted code that still had little things to polish.
Russell Belfer e1de726c 2012-03-12T22:55:40 Migrate ODB files to new error handling This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to the new style of error handling. Also got the unix and win32 versions of map.c. There are some minor changes to other files but no others were completely converted. This also contains an update to filebuf so that a zeroed out filebuf will not think that the fd (== 0) is actually open (and inadvertently call close() on fd 0 if cleaned up). Lastly, this was built and tested on win32 and contains a bunch of fixes for the win32 build which was pretty broken.
Vicent Martí 1a481123 2012-02-17T00:13:34 error-handling: References Yes, this is error handling solely for `refs.c`, but some of the abstractions leak all ofer the code base.
Vicent Martí 0c3bae62 2012-02-15T16:56:56 zlib: Remove custom `git2/zlib.h` header This is legacy compat stuff for when `deflateBound` is not defined, but we're not embedding zlib and that function is always available. Kill that with fire.
schu 5e0de328 2012-02-13T17:10:24 Update Copyright header Signed-off-by: schu <schu-github@schulog.org>
Russell Belfer 1744fafe 2012-01-17T15:49:47 Move path related functions from fileops to path This takes all of the functions that look up simple data about paths (such as `git_futils_isdir`) and moves them over to path.h (becoming `git_path_isdir`). This leaves fileops.h just with functions that actually manipulate the filesystem or look at the file contents in some way. As part of this, the dir.h header which is really just for win32 support was moved into win32 (with some minor changes).
Vicent Martí 1af56d7d 2012-01-15T15:48:36 Fix #534: 64-bit issues in Windows off_t is always 32 bits in Windows, which is beyond stupid, but we just don't care anymore because we're using `git_off_t` which is assured to be 64 bits on all platforms, regardless of compilation mode. Just ensure that no casts to `off_t` are performed. Also, the check for `off_t` overflows has been dropped, once again, because the size of our offsets is always 64 bits on all platforms. Fixes #534
Vicent Marti a15c550d 2011-11-16T14:09:44 threads: Fix the shared global state with TLS See `global.c` for a description of what we're doing. When libgit2 is built with GIT_THREADS support, the threading system must be explicitly initialized with `git_threads_init()`.
Vicent Marti 3286c408 2011-10-28T14:51:13 global: Properly use `git__` memory wrappers Ensure that all memory related functions (malloc, calloc, strdup, free, etc) are using their respective `git__` wrappers.
Brodie Rao b2a2702d 2011-10-11T22:05:12 odb_pack: don't do ambiguity checks for fully qualified SHA1 hashes This makes libgit2 more closely match Git, which only checks for ambiguous pack entries when given short hashes. Note that the only time this is ever relevant is when a pack has the same object more than once (it's happened in the wild, I promise).
Vicent Marti 87d9869f 2011-09-19T03:34:49 Tabify everything There were quite a few places were spaces were being used instead of tabs. Try to catch them all. This should hopefully not break anything. Except for `git blame`. Oh well.
Vicent Marti bb742ede 2011-09-19T01:54:32 Cleanup legal data 1. The license header is technically not valid if it doesn't have a copyright signature. 2. The COPYING file has been updated with the different licenses used in the project. 3. The full GPLv2 header in each file annoys me.
Sebastian Schuberth 1c3fac4d 2011-09-08T14:31:37 Add casts to get rid of some warnings when filling zlib structures
Kirill A. Shutemov 932669b8 2011-08-25T14:22:57 Drop STRLEN() macros There is no need in STRLEN macros. Compilers can do this trivial optimization on its own. Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
Carlos Martín Nieto 061047cc 2011-08-05T18:20:00 Rethrow pack entry offset error Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Carlos Martín Nieto c1af5a39 2011-08-06T00:35:20 Implement cooperative caching When indexing a file with ref deltas, a temporary cache for the offsets has to be built, as we don't have an index file yet. If the user takes the responsiblity for filling the cache, the packing code will look there first when it finds a ref delta. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Carlos Martín Nieto 97f40a0d 2011-08-04T22:51:46 Check for error calculating the delta base Don't assume that it's always going to work. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Carlos Martín Nieto 7d0cdf82 2011-07-09T02:25:01 Make packfile_unpack_header more generic On the way, store the fd and the size in the mwindow file. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Carlos Martín Nieto b5b474dd 2011-07-28T11:45:46 Modify the given offset in git_packfile_unpack The callers immediately throw away the offset, so we don't need any logical changes in any of them. This will be useful for the indexer, as it does need to know where the compressed data ends. Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Carlos Martín Nieto a070f152 2011-07-29T01:08:02 Move pack functions to their own file