|
0ddfcb40
|
2013-05-02T18:06:14
|
|
Switch to index_version as "git_pack_file is ready" flag
We use p->index_map.data to check whether the struct has been set up
and all the information about the index is stored there. This variable
gets set up halfway through the setup process, however, and a thread
can come along and use fields that haven't been written to yet.
Crucially, pack_entry_find_offset() needs to read the index version
(which is written after index_map) to know the offset and stride
length to pass to sha1_entry_pos(). If these values are wrong,
assertions in it will fail, as it will be reading bogus data.
Make index_version the last field to be written and switch from using
p->index_map.data to p->index_version as "git_pack_file is ready" flag
as we can use it to know if every field has been written.
|
|
|
34bd5999
|
2013-05-02T17:14:05
|
|
Revert "Protect sha1_entry_pos call with mutex"
This reverts commit 8c535f3f6879c6796d8107d7eb80dd8b2105621b.
|
|
8c535f3f
|
2013-05-02T03:34:56
|
|
Protect sha1_entry_pos call with mutex
There is an occasional assertion failure in sha1_entry_pos from
pack_entry_find_index when running threaded. Holding the mutex
around the code that grabs the index_map data and processes it
makes this assertion failure go away.
|
|
|
9d2f841a
|
2013-05-02T03:03:54
|
|
Add extra locking around packfile open
We were still seeing a few issues in threaded access to packs.
This adds extra locks around the opening of the mwindow to
avoid a different race.
|
|
|
b7f167da
|
2013-04-29T13:52:12
|
|
Make git_oid_cmp public and add git_oid__cmp
|
|
|
5d2d21e5
|
2013-04-16T15:00:43
|
|
Consolidate packfile allocation further
Rename git_packfile_check to git_packfile_alloc since it is now
being used more in that capacity. Fix the various places that use
it. Consolidate some repeated code in odb_pack.c related to the
allocation of a new pack_backend.
|
|
|
38eef611
|
2013-04-16T14:19:27
|
|
Make indexer use shared packfile open code
The indexer was creating a packfile object separately from the
code in pack.c which was a problem since I put a call to
git_mutex_init into just pack.c. This commit updates the pack
function for creating a new pack object (i.e. git_packfile_check())
so that it can be used in both places and then makes indexer.c
use the shared initialization routine.
There are also a few minor formatting and warning message fixes.
|
|
|
53607868
|
2013-04-15T00:09:03
|
|
Further threading fixes
This builds on the earlier thread safety work to make it so that
setting the odb, index, refdb, or config for a repository is done
in a threadsafe manner with minimized locking time. This is done
by adding a lock to the repository object and using it to guard
the assignment of the above listed pointers. The lock is only
held to assign the pointer value.
This also contains some minor fixes to the other work with pack
files to reduce the time that locks are being held to and fix an
apparently memory leak.
|
|
|
24c70804
|
2013-04-12T12:59:38
|
|
Add mutex around mapping and unmapping pack files
When I was writing threading tests for the new cache, the main
error I kept running into was a pack file having it's content
unmapped underneath the running thread. This adds a lock around
the routines that map and unmap the pack data so that threads can
effectively reload the data when they need it.
This also required reworking the error handling paths in a couple
places in the code which I tried to make consistent.
|
|
|
0e040c03
|
2013-03-03T14:50:47
|
|
indexer: use a hashtable for keeping track of offsets
These offsets are needed for REF_DELTA objects, which encode which
object they use as a base, but not where it lies in the packfile, so
we need a list.
These objects are mostly from older packfiles, before OFS_DELTA was
widely spread. The time spent in indexing these packfiles is greatly
reduced, though remains above what git is able to do.
|
|
11d9f6b3
|
2013-01-27T14:17:07
|
|
Vector improvements and their fallout
|
|
|
aa3bf89d
|
2013-01-26T15:12:53
|
|
Fix a mutex leak in pack.c
|
|
9c62aaab
|
2013-01-14T17:42:12
|
|
pack: evict all of the pages at once
Somewhat surprisingly, this can increase the speed considerably, as we
don't bother trying to decide what to evict, and the most used entries
are quickly back into the cache.
|
|
|
ed6648ba
|
2013-01-14T16:39:22
|
|
pack: evict objects from the cache in groups of eight
This drops the cache eviction below libcrypto and zlib in the perf
output. The number has been chosen empirically.
|
|
|
09e29e47
|
2013-01-12T19:31:07
|
|
pack: fixes to the cache
The offset should be git_off_t, and we should check the return value
of the mutex lock function.
|
|
|
96c9b9f0
|
2013-01-12T18:38:19
|
|
indexer: properly free the packfile resources
The indexer needs to call the packfile's free function so it takes care of
freeing the caches.
We still need to close the mwf descriptor manually so we can rename the
packfile into its final name on Windows.
|
|
|
80d647ad
|
2013-01-11T20:15:06
|
|
Revert "pack: packfile_free -> git_packfile_free and use it in the indexers"
This reverts commit f289f886cb81bb570bed747053d5ebf8aba6bef7, which
makes the tests fail on Windows. Revert until we can figure out a
solution.
|
|
090d5e1f
|
2013-01-11T14:40:09
|
|
Fix MSVC compilation warnings
|
|
|
f289f886
|
2013-01-11T17:24:52
|
|
pack: packfile_free -> git_packfile_free and use it in the indexers
It turns out the indexers have been ignoring the pack's free function
and leaking data. Plug that.
|
|
|
0ed75620
|
2012-12-21T13:46:48
|
|
pack: limit the amount of memory the base delta cache can use
Currently limited to 16MB (like git) and to objects up to 1MB in
size.
|
|
|
c8f79c2b
|
2012-12-21T10:59:10
|
|
pack: abstract out the cache into its own functions
|
|
|
525d961c
|
2012-12-20T07:55:51
|
|
pack: refcount entries and add a mutex around cache access
|
|
|
c0f4a011
|
2012-12-19T16:48:12
|
|
pack: introduce a delta base cache
Many delta bases are re-used. Cache them to avoid inflating the same
data repeatedly.
This version doesn't limit the amount of entries to store, so it can
end up using a considerable amound of memory.
|
|
359fc2d2
|
2013-01-08T17:07:25
|
|
update copyrights
|
|
0249a503
|
2012-12-07T09:40:21
|
|
Merge pull request #1091 from carlosmn/stream-object
Indexer speedup with large objects
|
|
44f9f547
|
2012-11-30T13:33:30
|
|
pack: add git_packfile_resolve_header
To paraphrase @peff:
You can get both size and type from a packed object reasonably cheaply.
If you have:
* An object that is not a delta; both type and size are available in the
packfile header.
* An object that is a delta. The packfile type will be OBJ_*_DELTA, and
you have to resolve back to the base to find the real type. That means
potentially a lot of packfile index lookups, but each one is
relatively cheap. For the size, you inflate the first few bytes of the
delta, whose header will tell you the resulting size of applying the
delta to the base.
For simplicity, we just decompress the whole delta for now.
|
|
|
46635339
|
2012-11-19T22:22:33
|
|
pack: introduce a streaming API for raw objects
This allows us to take objects from the packfile as a stream instead
of having to keep it all in memory.
|
|
|
c3fb7d04
|
2012-11-27T15:00:49
|
|
Make git_odb_foreach_cb take const param
This makes the first OID param of the ODB callback a const pointer
and also propogates that change all the way to the backends.
|
|
fcb48e06
|
2012-11-24T15:48:17
|
|
Set p->mwf.fd to -1 on error
If p->mwf.fd is e.g. -2 then it is closed in packfile_free and an exception might be thrown.
Signed-off-by: Sven Strickroth <email@cs-ware.de>
|
|
826bc4a8
|
2012-11-23T13:31:22
|
|
Remove use of English expletives
Remove words such as fuck, crap, shit etc.
Remove other potentially offensive words from comments.
Tidy up other geopolicital terms in comments.
|
|
|
60ecdf59
|
2012-09-10T11:48:21
|
|
pack: iterate objects in offset order
Compute the ordering on demand and persist until the index is freed.
|
|
51e1d808
|
2012-08-06T12:41:08
|
|
Merge remote-tracking branch 'arrbee/tree-walk-fixes' into development
Conflicts:
src/notes.c
src/transports/git.c
src/transports/http.c
src/transports/local.c
tests-clar/odb/foreach.c
|
|
|
5dca2010
|
2012-08-03T17:08:01
|
|
Update iterators for consistency across library
This updates all the `foreach()` type functions across the library
that take callbacks from the user to have a consistent behavior.
The rules are:
* A callback terminates the loop by returning any non-zero value
* Once the callback returns non-zero, it will not be called again
(i.e. the loop stops all iteration regardless of state)
* If the callback returns non-zero, the parent fn returns GIT_EUSER
* Although the parent returns GIT_EUSER, no error will be set in
the library and `giterr_last()` will return NULL if called.
This commit makes those changes across the library and adds tests
for most of the iteration APIs to make sure that they follow the
above rules.
|
|
|
b8457baa
|
2012-07-24T07:57:58
|
|
portability: Improve x86/amd64 compatibility
|
|
|
521aedad
|
2012-06-05T14:48:51
|
|
odb: add git_odb_foreach()
Go through each backend and list every objects that exists in
them. This allows fsck-like uses.
|
|
|
1d8943c6
|
2012-06-28T12:05:49
|
|
mwindow: allow memory-window files to deregister
Once a file is registered, there is no way to deregister it, even
after the structure that contains it is no longer needed and has been
freed. This may be the source of #624.
Allow and use the deregister function to remove our file from the
global list.
|
|
2aeadb9c
|
2012-06-12T19:25:09
|
|
Actually do the mmap... unsurprisingly, this makes the indexer work on SFS
On RAM: the .idx and .pack files become links to a .lock and the original download respectively.
Assume some feature (such as record locking) supported by SFS but not JXFS or RAM: is required.
|
|
|
90490113
|
2012-06-10T18:08:15
|
|
Basic mmap/munmap compatiblity
|
|
|
904b67e6
|
2012-05-18T01:48:50
|
|
errors: Rename error codes
|
|
|
e172cf08
|
2012-05-18T01:21:06
|
|
errors: Rename the generic return codes
|
|
|
282283ac
|
2012-05-04T16:46:46
|
|
Fix valgrind issues
There are three changes here:
- correctly propogate error code from failed object lookups
- make zlib inflate use our allocators
- add OID to notfound error in ODB lookups
|
|
|
f9f2344b
|
2012-04-23T17:28:11
|
|
Merge pull request #632 from arrbee/win64-cleanup
Code clean up, including fixing warnings on Windows 64-bit build
|
|
|
44ef8b1b
|
2012-04-13T13:00:10
|
|
Fix warnings on 64-bit windows builds
This fixes all the warnings on win64 except those in deps, which
come from the regex code.
|
|
45d773ef
|
2012-04-05T23:36:14
|
|
pack: signal a short buffer when needed
|
|
0d0fa7c3
|
2012-03-16T15:56:01
|
|
Convert attr, ignore, mwindow, status to new errors
Also cleaned up some previously converted code that still had
little things to polish.
|
|
|
e1de726c
|
2012-03-12T22:55:40
|
|
Migrate ODB files to new error handling
This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to
the new style of error handling. Also got the unix and win32
versions of map.c. There are some minor changes to other
files but no others were completely converted.
This also contains an update to filebuf so that a zeroed out
filebuf will not think that the fd (== 0) is actually open
(and inadvertently call close() on fd 0 if cleaned up).
Lastly, this was built and tested on win32 and contains a
bunch of fixes for the win32 build which was pretty broken.
|
|
|
1a481123
|
2012-02-17T00:13:34
|
|
error-handling: References
Yes, this is error handling solely for `refs.c`, but some of the
abstractions leak all ofer the code base.
|
|
|
0c3bae62
|
2012-02-15T16:56:56
|
|
zlib: Remove custom `git2/zlib.h` header
This is legacy compat stuff for when `deflateBound` is not defined, but
we're not embedding zlib and that function is always available. Kill
that with fire.
|
|
5e0de328
|
2012-02-13T17:10:24
|
|
Update Copyright header
Signed-off-by: schu <schu-github@schulog.org>
|
|
|
1744fafe
|
2012-01-17T15:49:47
|
|
Move path related functions from fileops to path
This takes all of the functions that look up simple data about
paths (such as `git_futils_isdir`) and moves them over to path.h
(becoming `git_path_isdir`). This leaves fileops.h just with
functions that actually manipulate the filesystem or look at
the file contents in some way.
As part of this, the dir.h header which is really just for win32
support was moved into win32 (with some minor changes).
|
|
|
1af56d7d
|
2012-01-15T15:48:36
|
|
Fix #534: 64-bit issues in Windows
off_t is always 32 bits in Windows, which is beyond stupid, but we just
don't care anymore because we're using `git_off_t` which is assured to
be 64 bits on all platforms, regardless of compilation mode. Just
ensure that no casts to `off_t` are performed.
Also, the check for `off_t` overflows has been dropped, once again,
because the size of our offsets is always 64 bits on all platforms.
Fixes #534
|
|
|
a15c550d
|
2011-11-16T14:09:44
|
|
threads: Fix the shared global state with TLS
See `global.c` for a description of what we're doing.
When libgit2 is built with GIT_THREADS support, the threading system
must be explicitly initialized with `git_threads_init()`.
|
|
|
3286c408
|
2011-10-28T14:51:13
|
|
global: Properly use `git__` memory wrappers
Ensure that all memory related functions (malloc, calloc, strdup, free,
etc) are using their respective `git__` wrappers.
|
|
b2a2702d
|
2011-10-11T22:05:12
|
|
odb_pack: don't do ambiguity checks for fully qualified SHA1 hashes
This makes libgit2 more closely match Git, which only checks for
ambiguous pack entries when given short hashes.
Note that the only time this is ever relevant is when a pack has the
same object more than once (it's happened in the wild, I promise).
|
|
|
87d9869f
|
2011-09-19T03:34:49
|
|
Tabify everything
There were quite a few places were spaces were being used instead of
tabs. Try to catch them all. This should hopefully not break anything.
Except for `git blame`. Oh well.
|
|
|
bb742ede
|
2011-09-19T01:54:32
|
|
Cleanup legal data
1. The license header is technically not valid if it doesn't have a
copyright signature.
2. The COPYING file has been updated with the different licenses used in
the project.
3. The full GPLv2 header in each file annoys me.
|
|
1c3fac4d
|
2011-09-08T14:31:37
|
|
Add casts to get rid of some warnings when filling zlib structures
|
|
932669b8
|
2011-08-25T14:22:57
|
|
Drop STRLEN() macros
There is no need in STRLEN macros. Compilers can do this trivial
optimization on its own.
Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
|
|
|
c1af5a39
|
2011-08-06T00:35:20
|
|
Implement cooperative caching
When indexing a file with ref deltas, a temporary cache for the
offsets has to be built, as we don't have an index file yet. If the
user takes the responsiblity for filling the cache, the packing code
will look there first when it finds a ref delta.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
061047cc
|
2011-08-05T18:20:00
|
|
Rethrow pack entry offset error
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
97f40a0d
|
2011-08-04T22:51:46
|
|
Check for error calculating the delta base
Don't assume that it's always going to work.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
7d0cdf82
|
2011-07-09T02:25:01
|
|
Make packfile_unpack_header more generic
On the way, store the fd and the size in the mwindow file.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
a070f152
|
2011-07-29T01:08:02
|
|
Move pack functions to their own file
|
|
|
b5b474dd
|
2011-07-28T11:45:46
|
|
Modify the given offset in git_packfile_unpack
The callers immediately throw away the offset, so we don't need any
logical changes in any of them. This will be useful for the indexer,
as it does need to know where the compressed data ends.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|