thodg/libgit2/src/odb_loose.c

src/odb_loose.c

Log

Author Commit Date CI Message
nulltoken 209f9b67 2013-09-08T18:25:17 odb: Teach loose backend to return EAMBIGUOUS
nulltoken 4047950f 2013-08-29T14:19:34 odb: Prevent stream_finalize_write() from overwriting Now that #1785 is merged, git_odb_stream_finalize_write() calculates the object id before invoking the odb backend. This commit gives a chance to the backend to check if it already knows this object.
Vicent Martí 520287f6 2013-08-19T02:17:00 Merge pull request #1785 from libgit2/cmn/odb-hash-frontend odb: move hashing to the frontend for streaming
nulltoken d19dd9cf 2013-08-18T23:38:51 odb: Straighten oid prefix handling
Carlos Martín Nieto fe0c6d4e 2013-08-17T01:41:08 odb: make it clearer that the id is calculated in the frontend The frontend is in charge of calculating the id of the objects. Thus the backends should treat it as a read-only value. The positioning in the function signature made it seem as though it was an output parameter. Make the id const and move it from the front to behind the subject (backend or stream).
Carlos Martín Nieto d4e6cf0c 2013-08-15T14:32:47 odb: remove a duplicate object header formatting function
Carlos Martín Nieto 8380b39a 2013-08-15T14:29:39 odb: perform the stream hashing in the frontend Hash the data as it's coming into the stream and tell the backend what its name is when finalizing the write. This makes it consistent with the way a plain git_odb_write() performs the write.
Russell Belfer 8294e8cf 2013-06-22T17:15:31 Constrain mkdir calls to avoid extra mkdirs This updates the calls that make the subdirectories for objects to use a base directory above which git_futils_mkdir won't walk any higher. This prevents attempts to mkdir all the way up to the root of the filesystem. Also, this moves the objects_dir into the loose backend structure and removes the separate allocation, plus does some preformatting of the objects_dir value to guarantee a trailing slash, etc.
Russell Belfer 83cc70d9 2013-04-19T12:48:33 Move odb_backend implementors stuff into git2/sys This moves some of the odb_backend stuff that is related to the internals of an odb_backend implementation into include/git2/sys. Some of the stuff related to streaming I left in include/git2 because it seemed like it would be reasonably needed by a normal user who wanted to stream objects into and out of the ODB. Also, I added APIs for traversing the list of backends so that some of the tests would not need to access ODB internals.
Edward Thomson 359fc2d2 2013-01-08T17:07:25 update copyrights
David Michael Barr 4d185dd9 2012-12-19T14:30:06 odb: check if object exists before writing Update the procondition of git_odb_backend::write. It may now be assumed that the object has already been hashed.
Ben Straub 55f6f21b 2012-11-29T19:59:18 Deploy versioned git_odb_backend structure
Russell Belfer c3fb7d04 2012-11-27T15:00:49 Make git_odb_foreach_cb take const param This makes the first OID param of the ODB callback a const pointer and also propogates that change all the way to the backends.
Vicent Marti 51e1d808 2012-08-06T12:41:08 Merge remote-tracking branch 'arrbee/tree-walk-fixes' into development Conflicts: src/notes.c src/transports/git.c src/transports/http.c src/transports/local.c tests-clar/odb/foreach.c
Russell Belfer 5dca2010 2012-08-03T17:08:01 Update iterators for consistency across library This updates all the `foreach()` type functions across the library that take callbacks from the user to have a consistent behavior. The rules are: * A callback terminates the loop by returning any non-zero value * Once the callback returns non-zero, it will not be called again (i.e. the loop stops all iteration regardless of state) * If the callback returns non-zero, the parent fn returns GIT_EUSER * Although the parent returns GIT_EUSER, no error will be set in the library and `giterr_last()` will return NULL if called. This commit makes those changes across the library and adds tests for most of the iteration APIs to make sure that they follow the above rules.
nulltoken b8457baa 2012-07-24T07:57:58 portability: Improve x86/amd64 compatibility
Carlos Martín Nieto b7158c53 2012-07-12T20:48:46 Use GIT_INLINE instead of inline
Carlos Martín Nieto 521aedad 2012-06-05T14:48:51 odb: add git_odb_foreach() Go through each backend and list every objects that exists in them. This allows fsck-like uses.
Vicent Martí 904b67e6 2012-05-18T01:48:50 errors: Rename error codes
Vicent Martí e172cf08 2012-05-18T01:21:06 errors: Rename the generic return codes
Russell Belfer 282283ac 2012-05-04T16:46:46 Fix valgrind issues There are three changes here: - correctly propogate error code from failed object lookups - make zlib inflate use our allocators - add OID to notfound error in ODB lookups
nulltoken fa6420f7 2012-04-29T21:46:33 buf: deploy git_buf_len()
Carlos Martín Nieto 8e8b6b01 2012-04-04T13:13:43 Clean up valgrind warnings
Russell Belfer 0d0fa7c3 2012-03-16T15:56:01 Convert attr, ignore, mwindow, status to new errors Also cleaned up some previously converted code that still had little things to polish.
Russell Belfer e1de726c 2012-03-12T22:55:40 Migrate ODB files to new error handling This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to the new style of error handling. Also got the unix and win32 versions of map.c. There are some minor changes to other files but no others were completely converted. This also contains an update to filebuf so that a zeroed out filebuf will not think that the fd (== 0) is actually open (and inadvertently call close() on fd 0 if cleaned up). Lastly, this was built and tested on win32 and contains a bunch of fixes for the win32 build which was pretty broken.
Vicent Martí cb8a7961 2012-03-07T00:02:55 error-handling: Repository This also includes droping `git_buf_lasterror` because it makes no sense in the new system. Note that in most of the places were it has been dropped, the code needs cleanup. I.e. GIT_ENOMEM is going away, so instead it should return a generic `-1` and obviously not throw anything.
Vicent Martí 9d160ba8 2012-03-06T01:37:56 diff: Fix rebase breackage
Vicent Martí 1a481123 2012-02-17T00:13:34 error-handling: References Yes, this is error handling solely for `refs.c`, but some of the abstractions leak all ofer the code base.
Vicent Martí 13224ea4 2012-02-27T04:28:31 buffer: Unify `git_fbuffer` and `git_buf` This makes so much sense that I can't believe it hasn't been done before. Kill the old `git_fbuffer` and read files straight into `git_buf` objects. Also: In order to fully support 4GB files in 32-bit systems, the `git_buf` implementation has been changed from using `ssize_t` for storage and storing negative values on allocation failure, to using `size_t` and changing the buffer pointer to a magical pointer on allocation failure. Hopefully this won't break anything.
Vicent Martí 0c3bae62 2012-02-15T16:56:56 zlib: Remove custom `git2/zlib.h` header This is legacy compat stuff for when `deflateBound` is not defined, but we're not embedding zlib and that function is always available. Kill that with fire.
schu 5e0de328 2012-02-13T17:10:24 Update Copyright header Signed-off-by: schu <schu-github@schulog.org>
Russell Belfer 1744fafe 2012-01-17T15:49:47 Move path related functions from fileops to path This takes all of the functions that look up simple data about paths (such as `git_futils_isdir`) and moves them over to path.h (becoming `git_path_isdir`). This leaves fileops.h just with functions that actually manipulate the filesystem or look at the file contents in some way. As part of this, the dir.h header which is really just for win32 support was moved into win32 (with some minor changes).
Russell Belfer 97769280 2011-11-30T11:27:15 Use git_buf for path storage instead of stack-based buffers This converts virtually all of the places that allocate GIT_PATH_MAX buffers on the stack for manipulating paths to use git_buf objects instead. The patch is pretty careful not to touch the public API for libgit2, so there are a few places that still use GIT_PATH_MAX. This extends and changes some details of the git_buf implementation to add a couple of extra functions and to make error handling easier. This includes serious alterations to all the path.c functions, and several of the fileops.c ones, too. Also, there are a number of new functions that parallel existing ones except that use a git_buf instead of a stack-based buffer (such as git_config_find_global_r that exists alongsize git_config_find_global). This also modifies the win32 version of p_realpath to allocate whatever buffer size is needed to accommodate the realpath instead of hardcoding a GIT_PATH_MAX limit, but that change needs to be tested still.
Russell Belfer b762e576 2011-11-17T15:10:27 filebuf: add GIT_FILEBUF_INIT and protect multiple opens and cleanups Update all stack allocations of git_filebuf to use GIT_FILEBUF_INIT and make git_filebuf_open and git_filebuf_cleanup safe to be called multiple times on the same buffer. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Carlos Martín Nieto 472d4d85 2011-11-17T20:32:04 Don't overwrite existing objects It's redundant to do this (git doesn't) and Windows doesn't allow us to overwrite a read-only file (which objects are). Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
Vicent Martí 89fb8f02 2011-10-28T19:04:23 Merge pull request #456 from brodie/perm-fixes Create objects, indexes, and directories with the right file permissions
Vicent Marti 3286c408 2011-10-28T14:51:13 global: Properly use `git__` memory wrappers Ensure that all memory related functions (malloc, calloc, strdup, free, etc) are using their respective `git__` wrappers.
Roberto Tyley c51065e3 2011-10-24T14:39:03 Tolerate zlib deflation with window size < 32Kb libgit2 currently identifies loose objects as corrupt if they've been deflated using a window size less than 32Kb, because the is_zlib_compressed_data() function doesn't recognise the header byte as a zlib header. This patch makes the method tolerant of all valid window sizes (15-bit to 8-bit) - but doesn't sacrifice it's accuracy in distingushing the standard loose-object format from the experimental (now abandoned) format. It's based on a patch which has been merged into C-Git master branch: https://github.com/git/git/commit/7f684a2aff636f44a506 On memory constrained systems zlib may use a much smaller window size - working on Agit, I found that Android uses a 4KB window; giving a header byte of 0x48, not 0x78. Consequently all loose objects generated by the Android platform appear 'corrupt' :( It might appear that this patch changes isStandardFormat() to the point where it could incorrectly identify the experimental format as the standard one, but the two criteria (bitmask & checksum) can only give a false result for an experimental object where both of the following are true: 1) object size is exactly 8 bytes when uncompressed (bitmask) 2) [single-byte in-pack git type&size header] * 256 + [1st byte of the following zlib header] % 31 = 0 (checksum) As it happens, for all possible combinations of valid object type (1-4) and window bits (0-7), the only time when the checksum will be divisible by 31 is for 0x1838 - ie object type *1*, a Commit - which, due the fields all Commit objects must contain, could never be as small as 8 bytes in size. Given this, the combination of the two criteria (bitmask & checksum) always correctly determines the buffer format, and is more tolerant than the previous version. References: Android uses a 4KB window for deflation: http://android.git.kernel.org/?p=platform/libcore.git;a=blob;f=luni/src/main/native/java_util_zip_Deflater.cpp;h=c0b2feff196e63a7b85d97cf9ae5bb258 Code snippet searching for false positives with the zlib checksum: https://gist.github.com/1118177 Change-Id: Ifd84cd2bd6b46f087c9984fb4cbd8309f483dec0
Brodie Rao 01ad7b3a 2011-09-06T15:48:45 *: correct and codify various file permissions The following files now have 0444 permissions: - loose objects - pack indexes - pack files - packs downloaded by fetch - packs downloaded by the HTTP transport And the following files now have 0666 permissions: - config files - repository indexes - reflogs - refs This brings libgit2 more in line with Git. Note that git_filebuf_commit() and git_filebuf_commit_at() have both gained a new mode parameter. The latter change fixes an important issue where filebufs created with GIT_FILEBUF_TEMPORARY received 0600 permissions (due to mkstemp(3) usage). Now we chmod() the file before renaming it into place. Tests have been added to confirm that new commit, tag, and tree objects are created with the right permissions. I don't have access to Windows, so for now I've guarded the tests with "#ifndef GIT_WIN32".
Brodie Rao ce8cd006 2011-09-07T15:32:44 fileops/repository: create (most) directories with 0777 permissions To further match how Git behaves, this change makes most of the directories libgit2 creates in a git repo have a file mode of 0777. Specifically: - Intermediate directories created with git_futils_mkpath2file() have 0777 permissions. This affects odb_loose, reflog, and refs. - The top level folder for bare repos is created with 0777 permissions. - The top level folder for non-bare repos is created with 0755 permissions. - /objects/info/, /objects/pack/, /refs/heads/, and /refs/tags/ are created with 0777 permissions. Additionally, the following changes have been made: - fileops functions that create intermediate directories have grown a new dirmode parameter. The only exception to this is filebuf's lock_file(), which unconditionally creates intermediate directories with 0777 permissions when GIT_FILEBUF_FORCE is set. - The test runner now sets the umask to 0 before running any tests. This ensurses all file mode checks are consistent across systems. - t09-tree.c now does a directory permissions check. I've avoided adding this check to other tests that might reuse existing directories from the prefabricated test repos. Because they're checked into the repo, they have 0755 permissions. - Other assorted directories created by tests have 0777 permissions.
Vicent Marti c103d7b4 2011-09-29T15:49:28 odb: Pass compression settings to filebuf
Vicent Marti 8af4d074 2011-09-29T15:34:17 odb: Let users decide compression level for the loose ODB
Vicent Martí 71a4c1f1 2011-09-18T20:07:59 Merge pull request #384 from kiryl/warnings Add more -W flags to CFLAGS
Vicent Marti 87d9869f 2011-09-19T03:34:49 Tabify everything There were quite a few places were spaces were being used instead of tabs. Try to catch them all. This should hopefully not break anything. Except for `git blame`. Oh well.
Vicent Marti bb742ede 2011-09-19T01:54:32 Cleanup legal data 1. The license header is technically not valid if it doesn't have a copyright signature. 2. The COPYING file has been updated with the different licenses used in the project. 3. The full GPLv2 header in each file annoys me.
Sebastian Schuberth 1c3fac4d 2011-09-08T14:31:37 Add casts to get rid of some warnings when filling zlib structures
Kirill A. Shutemov d568d585 2011-08-30T23:55:22 CMakefile: add -Wmissing-prototypes and fix warnings Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
Vicent Marti afeecf4f 2011-07-09T02:10:46 odb: Direct writes are back DIRECT WRITES ARE BACK AND FASTER THAN EVER. The streaming writer to the ODB was an overkill for the smaller objects like Commit and Tags; most of the streaming logic was taking too long. This commit makes Commits, Tags and Trees to be built-up in memory, and then written to disk in 2 pushes (header + data), instead of streaming everything. This is *always* faster, even for big files (since the git_filebuf class still does streaming writes when the memory cache overflows). This is also a gazillion lines of code smaller, because we don't have to precompute the final size of the object before starting the stream (this was kind of defeating the point of streaming, anyway). Blobs are still written with full streaming instead of loading them in memory, since this is still the fastest way. A new `git_buf` class has been added. It's missing some features, but it'll get there.
Vicent Marti f79026b4 2011-07-04T11:43:34 fileops: Cleanup Cleaned up the structure of the whole OS-abstraction layer. fileops.c now contains a set of utility methods for file management used by the library. These are abstractions on top of the original POSIX calls. There's a new file called `posix.c` that contains emulations/reimplementations of all the POSIX calls the library uses. These are prefixed with `p_`. There's a specific posix file for each platform (win32 and unix). All the path-related methods have been moved from `utils.c` to `path.c` and have their own prefix.
Vicent Marti f1d01851 2011-06-16T02:48:48 oid: Uniformize ncmp methods Drop redundant methods. The ncmp method is now public
Vicent Marti fa48608e 2011-06-16T02:36:21 oid: Rename methods Yeah. Finally. Fuck the old names, this ain't POSIX and they don't make any sense at all.
Marc Pegon c09093cc 2011-06-06T10:55:36 Renamed git_oid_match to git_oid_ncmp. As suggested by carlosmn, git_oid_ncmp would probably be a better name than git_oid_match, for it does the same as git_oid_cmp but only up to a certain amount of hex digits.
Marc Pegon aea8a638 2011-05-29T18:00:35 Implemented read_unique_short_oid method for loose backend.
Vicent Marti d0323a5f 2011-06-01T21:25:56 short-oid: Cleanup
Marc Pegon ecd6fdf1 2011-05-27T18:49:09 Added a read_unique_short_oid method to backends, to make it possible to find objects from sha1 prefixes in the future. Default implementations throw GIT_ENOTIMPLEMENTED for strict prefixes (i.e. length < GIT_OID_HEXSZ).
Vicent Marti 60e1b49a 2011-05-23T21:12:18 odb_loose: Reword errors
Jakob Pfender dfb12cd5 2011-05-19T12:28:46 odb_loose.c: Move to new error handling mechanism
Jakob Pfender f93f8ec5 2011-05-19T12:27:43 odb_loose.c: Return GIT_ENOMEM when allocation fails When trying to inflate a buffer, a GIT_ERROR was returned when malloc() failed. Fix this to return GIT_ENOMEM.
Carlos Martín Nieto d8e1d038 2011-05-06T12:47:21 Fix two warnings from Clang Both are about not reading the value stored in a variable. Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
Sergey Nikishin a3ced637 2011-04-22T17:36:28 Fix going into infinite loop in read_header_loose() read_header_loose causes infinite loop on this file: $ cat ../libcppgit/bin/sample-repo/test_mailbox/.git/objects/8f/e274605cbc740a2a957f44b2722a8a73915a09 | base64 eAErKUpNVTAzYzA0MDAzMVHISUxKzSlmWLgkuyN5+rxr6juMPR2EmN8s7Vl9D6oiN7UkkcHJdLbl 7Z3N/oxfE0W8wrSbuFRkAwDFfBn1
Vicent Marti f6f72d7e 2011-03-23T18:44:53 Improve the ODB writing backend Temporary files when doing streaming writes are now stored inside the Objects folder, to prevent issues when moving files between disks/partitions. Add support for block writes to the ODB again (for those backends that cannot implement streaming).
Vicent Marti 72a3fe42 2011-03-18T19:38:49 I broke your bindings Hey. Apologies in advance -- I broke your bindings. This is a major commit that includes a long-overdue redesign of the whole object-database structure. This is expected to be the last major external API redesign of the library until the first non-alpha release. Please get your bindings up to date with these changes. They will be included in the next minor release. Sorry again! Major features include: - Real caching and refcounting on parsed objects - Real caching and refcounting on objects read from the ODB - Streaming writes & reads from the ODB - Single-method writes for all object types - The external API is now partially thread-safe The speed increases are significant in all aspects, specially when reading an object several times from the ODB (revwalking) and when writing big objects to the ODB. Here's a full changelog for the external API: blob.h ------ - Remove `git_blob_new` - Remove `git_blob_set_rawcontent` - Remove `git_blob_set_rawcontent_fromfile` - Rename `git_blob_writefile` -> `git_blob_create_fromfile` - Change `git_blob_create_fromfile`: The `path` argument is now relative to the repository's working dir - Add `git_blob_create_frombuffer` commit.h -------- - Remove `git_commit_new` - Remove `git_commit_add_parent` - Remove `git_commit_set_message` - Remove `git_commit_set_committer` - Remove `git_commit_set_author` - Remove `git_commit_set_tree` - Add `git_commit_create` - Add `git_commit_create_v` - Add `git_commit_create_o` - Add `git_commit_create_ov` tag.h ----- - Remove `git_tag_new` - Remove `git_tag_set_target` - Remove `git_tag_set_name` - Remove `git_tag_set_tagger` - Remove `git_tag_set_message` - Add `git_tag_create` - Add `git_tag_create_o` tree.h ------ - Change `git_tree_entry_2object`: New signature is `(git_object **object_out, git_repository *repo, git_tree_entry *entry)` - Remove `git_tree_new` - Remove `git_tree_add_entry` - Remove `git_tree_remove_entry_byindex` - Remove `git_tree_remove_entry_byname` - Remove `git_tree_clearentries` - Remove `git_tree_entry_set_id` - Remove `git_tree_entry_set_name` - Remove `git_tree_entry_set_attributes` object.h ------------ - Remove `git_object_new - Remove `git_object_write` - Change `git_object_close`: This method is now *mandatory*. Not closing an object causes a memory leak. odb.h ----- - Remove type `git_rawobj` - Remove `git_rawobj_close` - Rename `git_rawobj_hash` -> `git_odb_hash` - Change `git_odb_hash`: New signature is `(git_oid *id, const void *data, size_t len, git_otype type)` - Add type `git_odb_object` - Add `git_odb_object_close` - Change `git_odb_read`: New signature is `(git_odb_object **out, git_odb *db, const git_oid *id)` - Change `git_odb_read_header`: New signature is `(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id)` - Remove `git_odb_write` - Add `git_odb_open_wstream` - Add `git_odb_open_rstream` odb_backend.h ------------- - Change type `git_odb_backend`: New internal signatures are as follows int (* read)(void **, size_t *, git_otype *, struct git_odb_backend *, const git_oid *) int (* read_header)(size_t *, git_otype *, struct git_odb_backend *, const git_oid *) int (* writestream)(struct git_odb_stream **, struct git_odb_backend *, size_t, git_otype) int (* readstream)( struct git_odb_stream **, struct git_odb_backend *, const git_oid *) - Add type `git_odb_stream` - Add enum `git_odb_streammode` Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 19a30a3f 2011-03-03T19:53:17 Add new move function, `gitfo_mv_force` Forces a move by creating the folder for the destination file, if it doesn't exist. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti d4b5a4e2 2011-02-09T19:49:02 Internal changes on the backend system The priority value for different backends has been removed from the public `git_odb_backend` struct. We handle that internally. The priority value is specified on the `git_odb_add_alternate`. This is convenient because it allows us to poll a backend twice with different priorities without having to instantiate it twice. We also differentiate between main backends and alternates; alternates have lower priority and cannot be written to. These changes come with some unit tests to make sure that the backend sorting is consistent. The libgit2 version has been bumped to 0.4.0. This commit changes the external API: CHANGED: struct git_odb_backend No longer has a `priority` attribute; priority for the backend in managed internally by the library. git_odb_add_backend(git_odb *odb, git_odb_backend *backend, int priority) Now takes an additional priority parameter, the priority that will be given to the backend. ADDED: git_odb_add_alternate(git_odb *odb, git_odb_backend *backend, int priority) Add a backend as an alternate. Alternate backends have always lower priority than main backends, and writing is disabled on them. Signed-off-by: Vicent Marti <tanoku@gmail.com> Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti f725931b 2011-02-05T12:42:41 Fix directory/path manipulation methods The `dirname` and `dirbase` methods have been replaced with the Android implementation, which is actually compilant to some kind of standard. A new method `topdir` has been added, which returns the topmost directory in a path. These changes fix issue #49: `gitfo_prettify_dir_path` converts "./.git/" to ".git/", so the code at src/repository.c:190 goes out of bounds when trying to find the topmost directory. The new `git__topdir` method handles this gracefully, and the fixed `git__dirname` now returns the proper value for the repository's working dir. E.g. /repo/.git/ ==> working dir '/repo/' .git/ ==> working dir '.' Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 44908fe7 2010-12-06T23:03:16 Change the library include file Libgit2 is now officially include as #include "<git2.h>" or indidividual files may be included as #include <git2/index.h> Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti d12299fe 2010-12-03T22:22:10 Change include structure for the project The maze with include dependencies has been fixed. There is now a global include: #include <git.h> The git_odb_backend API has been exposed. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 7d7cd885 2010-12-03T18:01:30 Decouple storage from ODB logic Comes with two default backends: loose object and packfiles. Signed-off-by: Vicent Marti <tanoku@gmail.com>
kmx.io     mail     discord kmxgit v0.4.0