|
6481a68d
|
2012-12-07T19:23:16
|
|
indexer: move the temporary buffers into the indexer object
Storing 4kB or 8kB in the stack is not very gentle. As this part has
to be linear, put the buffer into the indexer object so we allocate it
once in the heap.
|
|
3908c254
|
2012-11-30T17:25:50
|
|
indexer: correctly deal with objects larger than the window size
A mmap-window is not guaranteed to give you the whole object, but the
indexer currently assumes so.
Loop asking for more data until we've successfully CRC'd all of the
packed data.
|
|
|
5a3ad89d
|
2012-11-20T07:03:56
|
|
indexer: make use of streaming also for deltas
Up to now, deltas needed to be enterily in the packfile, and we tried
to decompress then in their entirety over and over again.
Adjust the logic so we read them as they come, just as we do for full
objects. This also allows us to simplify the logic and have less
nested code. The delta resolving phase still needs to decompress the
whole object into memory, as there is not yet any streaming
delta-apply support, but it helps in speeding up the downloading
process and reduces the amount of memory allocations we need to do.
|
|
|
f56f8585
|
2012-11-19T22:23:16
|
|
indexer: use the packfile streaming API
The new API allows us to read the object bit by bit from the packfile,
instead of needing it all at once in the packfile. This also allows us
to hash the object as it comes in from the network instead of having
to try to read it all and failing repeatedly for larger objects.
This is only the first step, but it already shows huge improvements
when dealing with objects over a few megabytes in size. It reduces the
memory needs in some cases, but delta objects still need to be
completely in memory and the old inefficent method is still used for
that.
|
|
839c5f57
|
2012-11-26T12:04:07
|
|
API updates for indexer.h
|
|
826bc4a8
|
2012-11-23T13:31:22
|
|
Remove use of English expletives
Remove words such as fuck, crap, shit etc.
Remove other potentially offensive words from comments.
Tidy up other geopolicital terms in comments.
|
|
4cc7342e
|
2012-11-18T09:07:35
|
|
Indexer: Avoid a possible double-deletion in error case
|
|
603bee07
|
2012-11-12T19:22:49
|
|
Remove git_hash_ctx_new - callers now _ctx_init()
|
|
d6fb0924
|
2012-11-05T12:37:15
|
|
Win32 CryptoAPI and CNG support for SHA1
|
|
|
81eecc34
|
2012-10-29T13:34:14
|
|
Fetch: don't clobber received count
This memset was being reached after the entire packfile under
WinHttp, so the byte count was being lost for small repos.
|
|
|
7d222e13
|
2012-10-24T13:29:14
|
|
Network progress: rename things
git_indexer_stats and friends -> git_transfer_progress*
Also made git_transfer_progress members more sanely
named.
|
|
|
909f6265
|
2012-10-18T15:28:09
|
|
Indexing progress now goes to 100%
|
|
|
216863c4
|
2012-10-17T14:02:24
|
|
Fetch/indexer: progress callbacks
|
|
e3f8d58d
|
2012-08-14T23:07:54
|
|
indexer: do not require absolute path
|
|
2b175ca9
|
2012-08-26T00:35:52
|
|
indexer: kill git_indexer_stats.data_received
It's not really needed with the current code as we have EOS and the
sideband's flush to tell us we're done.
Keep the distinction between processed and received objects.
|
|
|
7a57ae54
|
2012-08-25T23:31:29
|
|
indexer: don't segfault when freeing an unused indexer
Make sure that idx->pack isn't NULL before trying to free resources
under it.
|
|
|
bffa852f
|
2012-07-13T12:01:11
|
|
indexer: recognize and mark when all of the packfile has been downloaded
We can't always rely on the network telling us when the download is
finished. Recognize it from the indexer itself.
|
|
|
d1af70b0
|
2012-07-13T20:43:56
|
|
indexer: delay resolving deltas
Not all delta bases are available on the first try. By delaying
resolving all deltas until the end, we avoid decompressing some of the
data twice or even more times, saving effort and time.
|
|
|
1d8943c6
|
2012-06-28T12:05:49
|
|
mwindow: allow memory-window files to deregister
Once a file is registered, there is no way to deregister it, even
after the structure that contains it is no longer needed and has been
freed. This may be the source of #624.
Allow and use the deregister function to remove our file from the
global list.
|
|
|
37159957
|
2012-06-28T09:33:08
|
|
indexer: don't use '/objects/pack/' unconditionally
Not everyone who indexes a packfile wants to put it in the standard
git repository location.
|
|
|
f9fd7105
|
2012-06-25T15:26:38
|
|
indexer: start parsing input data immediately
Currently, the first call of git_indexer_stream_add adds the data to the
underlying pack file and opens it for later use, but doesn't start
parsing the already available data.
This means, git_indexer_stream_finalize only works if
git_indexer_stream_add was called at least twice. Kill this limitation
by parsing available data immediately.
|
|
a21bb1aa
|
2012-06-13T23:28:51
|
|
Merge remote-tracking branch 'source/development' into development
|
|
|
2aeadb9c
|
2012-06-12T19:25:09
|
|
Actually do the mmap... unsurprisingly, this makes the indexer work on SFS
On RAM: the .idx and .pack files become links to a .lock and the original download respectively.
Assume some feature (such as record locking) supported by SFS but not JXFS or RAM: is required.
|
|
3f035860
|
2012-06-07T22:43:03
|
|
misc: Fix warnings from PVS Studio trial
|
|
|
54db1a18
|
2012-05-19T13:20:55
|
|
Cleanup
* indexer: remove leftover printf
* commit: remove unused macros COMMIT_BASIC_PARSE, COMMIT_FULL_PARSE
and COMMIT_PRINT
|
|
|
904b67e6
|
2012-05-18T01:48:50
|
|
errors: Rename error codes
|
|
|
e172cf08
|
2012-05-18T01:21:06
|
|
errors: Rename the generic return codes
|
|
|
6a9d61ef
|
2012-05-15T15:08:54
|
|
indexer: add more consistency checks
Error out in finalize if there is junk after the packfile hash or we
couldn't process all the objects.
|
|
|
73d87a09
|
2012-05-15T21:42:01
|
|
Introduce GITERR_INDEXER
|
|
|
a640d79e
|
2012-05-09T13:11:50
|
|
indexer: close the pack's fd before renaming it
Windows gets upset if we rename a file with an open descriptor.
|
|
fa6420f7
|
2012-04-29T21:46:33
|
|
buf: deploy git_buf_len()
|
|
821f6bc7
|
2012-04-26T13:04:54
|
|
Fix Win32 warnings
|
|
|
dee5515a
|
2012-04-14T18:34:50
|
|
transports: buffer the git requests before sending them
Trying to send every single line immediately won't give us any speed
improvement and duplicates the code we need for other transports. Make
the git transport use the same buffer functions as HTTP.
|
|
|
1c9c081a
|
2012-04-13T19:25:06
|
|
indexer: add git_indexer_stream_free() and _hash()
|
|
|
453ab98d
|
2012-04-11T12:55:34
|
|
indexer: Add git_indexer_stream_finalize()
Resolve any lingering deltas, write out the index file and rename the
packfile.
|
|
|
3f93e16c
|
2012-03-29T17:49:57
|
|
indexer: start writing the stream indexer
This will allow us to index a packfile as soon as we receive it from
the network as well as storing it with its final name so we don't need
to pass temporary file names around.
|
|
4aa7de15
|
2012-03-19T17:49:46
|
|
Convert indexer, notes, sha1_lookup, and signature
More files moved to new error handling style.
|
|
|
deafee7b
|
2012-03-14T17:36:15
|
|
Continue error conversion
This converts blob.c, fileops.c, and all of the win32 files.
Also, various minor cleanups throughout the code. Plus, in
testing the win32 build, I cleaned up a bunch (although not
all) of the warnings with the 64-bit build.
|
|
|
e1de726c
|
2012-03-12T22:55:40
|
|
Migrate ODB files to new error handling
This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to
the new style of error handling. Also got the unix and win32
versions of map.c. There are some minor changes to other
files but no others were completely converted.
This also contains an update to filebuf so that a zeroed out
filebuf will not think that the fd (== 0) is actually open
(and inadvertently call close() on fd 0 if cleaned up).
Lastly, this was built and tested on win32 and contains a
bunch of fixes for the win32 build which was pretty broken.
|
|
|
cb8a7961
|
2012-03-07T00:02:55
|
|
error-handling: Repository
This also includes droping `git_buf_lasterror` because it makes no sense
in the new system. Note that in most of the places were it has been
dropped, the code needs cleanup. I.e. GIT_ENOMEM is going away, so
instead it should return a generic `-1` and obviously not throw
anything.
|
|
|
0c3bae62
|
2012-02-15T16:56:56
|
|
zlib: Remove custom `git2/zlib.h` header
This is legacy compat stuff for when `deflateBound` is not defined, but
we're not embedding zlib and that function is always available. Kill
that with fire.
|
|
5e0de328
|
2012-02-13T17:10:24
|
|
Update Copyright header
Signed-off-by: schu <schu-github@schulog.org>
|
|
|
18e5b854
|
2012-02-10T19:47:02
|
|
odb: Add internal `git_odb__hashfd`
|
|
|
d0ec3fb8
|
2012-01-19T17:07:49
|
|
indexer: save the pack index with the right name
Truncate at the slash; otherwise we get ppack-*.idx filenames.
|
|
|
97769280
|
2011-11-30T11:27:15
|
|
Use git_buf for path storage instead of stack-based buffers
This converts virtually all of the places that allocate GIT_PATH_MAX
buffers on the stack for manipulating paths to use git_buf objects
instead. The patch is pretty careful not to touch the public API
for libgit2, so there are a few places that still use GIT_PATH_MAX.
This extends and changes some details of the git_buf implementation
to add a couple of extra functions and to make error handling easier.
This includes serious alterations to all the path.c functions, and
several of the fileops.c ones, too. Also, there are a number of new
functions that parallel existing ones except that use a git_buf
instead of a stack-based buffer (such as git_config_find_global_r
that exists alongsize git_config_find_global).
This also modifies the win32 version of p_realpath to allocate whatever
buffer size is needed to accommodate the realpath instead of hardcoding
a GIT_PATH_MAX limit, but that change needs to be tested still.
|
|
|
89fb8f02
|
2011-10-28T19:04:23
|
|
Merge pull request #456 from brodie/perm-fixes
Create objects, indexes, and directories with the right file permissions
|
|
|
3286c408
|
2011-10-28T14:51:13
|
|
global: Properly use `git__` memory wrappers
Ensure that all memory related functions (malloc, calloc, strdup, free,
etc) are using their respective `git__` wrappers.
|
|
01ad7b3a
|
2011-09-06T15:48:45
|
|
*: correct and codify various file permissions
The following files now have 0444 permissions:
- loose objects
- pack indexes
- pack files
- packs downloaded by fetch
- packs downloaded by the HTTP transport
And the following files now have 0666 permissions:
- config files
- repository indexes
- reflogs
- refs
This brings libgit2 more in line with Git.
Note that git_filebuf_commit() and git_filebuf_commit_at() have both
gained a new mode parameter.
The latter change fixes an important issue where filebufs created with
GIT_FILEBUF_TEMPORARY received 0600 permissions (due to mkstemp(3)
usage). Now we chmod() the file before renaming it into place.
Tests have been added to confirm that new commit, tag, and tree
objects are created with the right permissions. I don't have access to
Windows, so for now I've guarded the tests with "#ifndef GIT_WIN32".
|
|
|
72d6a20b
|
2011-10-05T19:59:34
|
|
indexer: NUL-terminate the filename
As we no longer use the STRLEN macro, the NUL-terminator in the string
was not copied over. Fix this.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
92be7908
|
2011-10-01T14:46:30
|
|
indexer: return immediately if passed a NULL value
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
bb742ede
|
2011-09-19T01:54:32
|
|
Cleanup legal data
1. The license header is technically not valid if it doesn't have a
copyright signature.
2. The COPYING file has been updated with the different licenses used in
the project.
3. The full GPLv2 header in each file annoys me.
|
|
26e74c6a
|
2011-09-08T14:21:17
|
|
Fix some random size_t vs. int conversion warnings
|
|
932669b8
|
2011-08-25T14:22:57
|
|
Drop STRLEN() macros
There is no need in STRLEN macros. Compilers can do this trivial
optimization on its own.
Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
|
|
|
c85e08b1
|
2011-08-16T13:05:05
|
|
odb: Do not pass around a header when hashing
|
|
|
ade3c9bb
|
2011-08-07T10:26:33
|
|
Assert a filename in indexer creation
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
c1af5a39
|
2011-08-06T00:35:20
|
|
Implement cooperative caching
When indexing a file with ref deltas, a temporary cache for the
offsets has to be built, as we don't have an index file yet. If the
user takes the responsiblity for filling the cache, the packing code
will look there first when it finds a ref delta.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
2133c44f
|
2011-08-09T17:08:18
|
|
Merge pull request #355 from jdavid/fix-build
Fix "redefinition of typedef git_indexer" build error
|
|
|
f6867e63
|
2011-08-08T16:56:28
|
|
Fix compilation in Windows
|
|
2d3e417e
|
2011-08-05T15:11:25
|
|
Fix "redefinition of typedef git_indexer" build error
Signed-off-by: J. David Ibáñez <jdavid@itaapy.com>
|
|
|
48b3ad4f
|
2011-08-01T14:02:09
|
|
Move pack index writing to a public function
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
b7c44096
|
2011-07-28T23:35:39
|
|
Implement the indexer
Only v2 index files are supported.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
a070f152
|
2011-07-29T01:08:02
|
|
Move pack functions to their own file
|
|
|
7d0cdf82
|
2011-07-09T02:25:01
|
|
Make packfile_unpack_header more generic
On the way, store the fd and the size in the mwindow file.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
b5b474dd
|
2011-07-28T11:45:46
|
|
Modify the given offset in git_packfile_unpack
The callers immediately throw away the offset, so we don't need any
logical changes in any of them. This will be useful for the indexer,
as it does need to know where the compressed data ends.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
ab525a74
|
2011-07-07T19:20:13
|
|
Rename stuff to git_indexer_
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
f23c4a66
|
2011-07-07T19:08:45
|
|
Start the runner
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
3412391d
|
2011-07-07T11:47:31
|
|
Intial indexer code
|