src/odb.c


Log

Author Commit Date CI Message
nulltoken 8cf80525 2013-09-11T20:13:59 errors: Fix format of some error messages
nulltoken 031f3f80 2013-09-07T22:39:05 odb: Error when streaming in too [few|many] bytes
nulltoken 4047950f 2013-08-29T14:19:34 odb: Prevent stream_finalize_write() from overwriting Now that #1785 is merged, git_odb_stream_finalize_write() calculates the object id before invoking the odb backend. This commit gives a chance to the backend to check if it already knows this object.
nulltoken b1a6c316 2013-08-30T17:36:00 odb: Move the auto refresh logic to the pack backend Previously, `git_object_read()`, `git_object_read_prefix()` and `git_object_exists()` were implementing an auto refresh logic. When the expected object couldn't be found in any backend, a call to `git_odb_refresh()` was triggered and the lookup was once again performed against all backends. This commit removes this auto-refresh logic from the odb layer and pushes it down into the pack-backend (as it's the only one currently exposing a `refresh()` endpoint).
nulltoken a12e069a 2013-08-30T16:31:52 odb: Honor the non refreshing capability of a backend
Carlos Martín Nieto 090a07d2 2013-08-17T02:12:04 odb: avoid hashing twice in and edge case If none of the backends support direct writes and we must stream the whole file, we already know what the object's id should be; so use the stream's functions directly, bypassing the frontend's hashing and overwriting of our existing id.
Carlos Martín Nieto fe0c6d4e 2013-08-17T01:41:08 odb: make it clearer that the id is calculated in the frontend The frontend is in charge of calculating the id of the objects. Thus the backends should treat it as a read-only value. The positioning in the function signature made it seem as though it was an output parameter. Make the id const and move it from the front to behind the subject (backend or stream).
Carlos Martín Nieto 8380b39a 2013-08-15T14:29:39 odb: perform the stream hashing in the frontend Hash the data as it's coming into the stream and tell the backend what its name is when finalizing the write. This makes it consistent with the way a plain git_odb_write() performs the write.
Carlos Martín Nieto 376e6c9f 2013-08-15T13:48:35 odb: wrap the stream reading and writing functions This is in preparation for moving the hashing to the frontend, which requires us to handle the incoming data before passing it to the backend's stream.
Carlos Martín Nieto e54cfb9b 2013-08-12T11:50:27 odb: free object data when id is ambiguous By the time we recognise this as an ambiguous id, the object's data has been loaded into memory. Free it when returning EABMIGUOUS.
Rémi Duraffort c6451624 2013-07-15T16:00:07 Fix some more memory leaks in error path
Vicent Marti 6de9b2ee 2013-06-12T21:10:33 util: It's called `memzero`
Russell Belfer 3e9e6cda 2013-06-07T09:54:33 Add safe memset and use it This adds a `git__memset` routine that will not be optimized away and updates the places where I memset() right before a free() call to use it.
Russell Belfer f658dc43 2013-05-31T14:09:58 Zero memory for major objects before freeing By zeroing out the memory when we free larger objects (i.e. those that serve as collections of other data, such as repos, odb, refdb), I'm hoping that it will be easier for libgit2 bindings to find errors in their object management code.
Vicent Martí 03c28d92 2013-05-06T06:45:53 Merge pull request #1526 from arrbee/cleanup-error-return-without-msg Make sure error messages are set for most error returns
Vicent Marti dfec726b 2013-05-03T23:30:54 odb: Do not error out if an alternate ODB is missing
Russell Belfer f063f578 2013-05-01T14:48:35 Catch some odd odb backend corner case errors There are some cases, particularly where no loaded ODB backends support a particular operation, where we would return an error code without having set an error. This catches those cases and reports that no ODB backends support the operation in question.
Vicent Martí cd2ed9f0 2013-04-30T04:02:52 Merge pull request #1518 from arrbee/export-oid-comparison Remove most inlines from the public API
Russell Belfer b7f167da 2013-04-29T13:52:12 Make git_oid_cmp public and add git_oid__cmp
Edward Thomson c8a4e8a5 2013-04-29T11:14:56 don't use uninitialized struct stat in win32
Russell Belfer 78606263 2013-04-15T00:05:44 Add callback to git_objects_table This adds create and free callback to the git_objects_table so that more of the creation and destruction of objects can be table driven instead of using switch statements. This also makes the semantics of certain object creation functions consistent so that we can make better use of function pointers. This also fixes a theoretical error case where an object allocation fails and we end up storing NULL into the cache.
Vicent Marti 8842c75f 2013-04-03T22:30:07 What has science done.
Vicent Marti 5df18424 2013-04-01T19:38:23 lol this worked first try wtf
Vicent Marti 0edad3cc 2013-04-22T16:41:56 Merge branch 'development' into vmg/dupe-odb-backends Conflicts: src/odb.c
Vicent Marti 4ef2c79c 2013-04-22T16:37:40 odb: Disable inode checks for Win32
Russell Belfer 83cc70d9 2013-04-19T12:48:33 Move odb_backend implementors stuff into git2/sys This moves some of the odb_backend stuff that is related to the internals of an odb_backend implementation into include/git2/sys. Some of the stuff related to streaming I left in include/git2 because it seemed like it would be reasonably needed by a normal user who wanted to stream objects into and out of the ODB. Also, I added APIs for traversing the list of backends so that some of the tests would not need to access ODB internals.
Vicent Marti a29c6b5f 2013-04-19T23:51:18 odb: Do not allow duplicate on-disk backends
Michael Schubert f5e28202 2013-03-25T13:38:43 opts: allow configuration of odb cache size Currently, the odb cache has a fixed size of 128 slots as defined by GIT_DEFAULT_CACHE_SIZE. Allow users to set the size of the cache via git_libgit2_opts(). Fixes #1035.
Arkadiy Shapkin 10c06114 2013-03-17T04:46:46 Several warnings detected by static code analyzer fixed Implicit type conversion argument of function to size_t type Suspicious sequence of types castings: size_t -> int -> size_t Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)' Unsigned type is never < 0
Vicent Marti 8fe6bc5c 2013-01-10T15:43:08 odb: Refresh on `exists` query too
Vicent Marti 891a4681 2013-01-04T17:42:41 dat errorcode
Vicent Marti 4a863c06 2013-01-03T20:36:26 Sane refresh logic All the ODB backends have a specific refresh interface. When reading an object, first we attempt every single backend: if the read fails, then we refresh all the backends and retry the read one more time to see if the object has appeared.
Edward Thomson 359fc2d2 2013-01-08T17:07:25 update copyrights
David Michael Barr 4d185dd9 2012-12-19T14:30:06 odb: check if object exists before writing Update the procondition of git_odb_backend::write. It may now be assumed that the object has already been hashed.
Vicent Martí 0249a503 2012-12-07T09:40:21 Merge pull request #1091 from carlosmn/stream-object Indexer speedup with large objects
Ben Straub c7231c45 2012-11-30T16:31:42 Deploy GITERR_CHECK_VERSION
Ben Straub 55f6f21b 2012-11-29T19:59:18 Deploy versioned git_odb_backend structure
Carlos Martín Nieto f56f8585 2012-11-19T22:23:16 indexer: use the packfile streaming API The new API allows us to read the object bit by bit from the packfile, instead of needing it all at once in the packfile. This also allows us to hash the object as it comes in from the network instead of having to try to read it all and failing repeatedly for larger objects. This is only the first step, but it already shows huge improvements when dealing with objects over a few megabytes in size. It reduces the memory needs in some cases, but delta objects still need to be completely in memory and the old inefficent method is still used for that.
Vicent Marti 9507a434 2012-11-28T10:47:10 odb: Add `git_odb_add_disk_alternate` Loads a disk alternate by path to the ODB. Mimics the `GIT_ALTERNATE_OBJECT_DIRECTORIES` shell var.
Ben Straub 2e76b5fc 2012-11-27T09:49:16 API updates for odb.h
Carlos Martín Nieto 85e7efa1 2012-11-14T13:35:43 odb: recursively load alternates The maximum depth is 5, like in git
Edward Thomson 603bee07 2012-11-12T19:22:49 Remove git_hash_ctx_new - callers now _ctx_init()
Edward Thomson d6fb0924 2012-11-05T12:37:15 Win32 CryptoAPI and CNG support for SHA1
Edward Thomson 09cc0b92 2012-11-05T11:33:10 create callback to handle packs from fetch, move the indexer to odb_pack
Michael Schubert edca6c8f 2012-07-01T19:44:22 git_odb_object_free: don't segfault w/ arg == NULL
Jameson Miller addc9be4 2012-09-26T17:21:32 Fix error hashing empty file.
Carlos Martín Nieto e8776d30 2012-09-16T00:10:07 odb: don't overflow the link path buffer Allocate a buffer large enough to store the path plus the terminator instead of letting readlink write beyond the end.
Vicent Martí 9be2261e 2012-09-13T09:24:12 Merge pull request #927 from arrbee/hashfile-with-filters Add git_repository_hashfile to hash with filters
Michael Schubert 13faa77c 2012-09-13T17:57:45 Fix -Wuninitialized warning
Russell Belfer a13fb55a 2012-09-11T17:26:21 Add tests and improve param checks Fixed some minor `git_repository_hashfile` issues: - Fixed incorrect doc (saying that repo could be NULL) - Added checking of object type value to acceptable ones - Added more tests for various parameter permutations
Vicent Marti c859184b 2012-09-11T23:05:24 Properly handle p_reads
Russell Belfer c6ac28fd 2012-09-10T12:24:05 Reorg internal odb read header and object lookup Often `git_odb_read_header` will "fail" and have to read the entire object into memory instead of just the header. When this happens, the object is loaded and then disposed of immediately, which makes it difficult to efficiently use the header information to decide if the object should be loaded (since attempting to do so will often result in loading the object twice). This commit takes the existing code and reorganizes it to have two new functions: - `git_odb__read_header_or_object` which acts just like the old read header function except that it returns the object, too, if it was forced to load the whole thing. It then becomes the callers responsibility to free the `git_odb_object`. - `git_object__from_odb_object` which was extracted from the old `git_object_lookup` and creates a subclass of `git_object` from an existing `git_odb_object` (separating the ODB lookup from the `git_object` creation). This allows you to use the first header reading function efficiently without instantiating the `git_odb_object` twice. There is no net change to the behavior of any of the existing functions, but this allows internal code to tap into the ODB lookup and object creation to be more efficient.
Russell Belfer 60b9d3fc 2012-09-05T15:00:40 Implement filters for status/diff blobs This adds support to diff and status for running filters (a la crlf) on blobs in the workdir before computing SHAs and before generating text diffs. This ended up being a bit more code change than I had thought since I had to reorganize some of the diff logic to minimize peak memory use when filtering blobs in a diff. This also adds a cap on the maximum size of data that will be loaded to diff. I set it at 512Mb which should match core git. Right now it is a #define in src/diff.h but it could be moved into the public API if desired.
Michael Schubert 0e9f2fce 2012-09-06T11:35:09 odb: mark unused variable
Philip Kelley c49d328c 2012-08-27T09:59:13 Expose a malloc function to 3rd party ODB backends
Vicent Marti c07d9c95 2012-08-09T15:33:04 oid: Explicitly include `oid.h` for the inlined CMP
Vicent Marti 51e1d808 2012-08-06T12:41:08 Merge remote-tracking branch 'arrbee/tree-walk-fixes' into development Conflicts: src/notes.c src/transports/git.c src/transports/http.c src/transports/local.c tests-clar/odb/foreach.c
Russell Belfer 5dca2010 2012-08-03T17:08:01 Update iterators for consistency across library This updates all the `foreach()` type functions across the library that take callbacks from the user to have a consistent behavior. The rules are: * A callback terminates the loop by returning any non-zero value * Once the callback returns non-zero, it will not be called again (i.e. the loop stops all iteration regardless of state) * If the callback returns non-zero, the parent fn returns GIT_EUSER * Although the parent returns GIT_EUSER, no error will be set in the library and `giterr_last()` will return NULL if called. This commit makes those changes across the library and adds tests for most of the iteration APIs to make sure that they follow the above rules.
nulltoken b8457baa 2012-07-24T07:57:58 portability: Improve x86/amd64 compatibility
Carlos Martín Nieto 521aedad 2012-06-05T14:48:51 odb: add git_odb_foreach() Go through each backend and list every objects that exists in them. This allows fsck-like uses.
Carlos Martín Nieto c06e0003 2012-06-20T01:41:30 odb: don't leak when detecting id ambiguity If we find several objects with the same prefix, we need to free the memory where we stored the earlier object. Keep track of the raw.data pointer across read_prefix calls and free it if we find another object.
Vicent Martí 904b67e6 2012-05-18T01:48:50 errors: Rename error codes
Vicent Martí e172cf08 2012-05-18T01:21:06 errors: Rename the generic return codes
Han-Wen Nienhuys 24634c6f 2012-05-12T15:01:39 Handle duplicate objects from different backends in git_odb_read_prefix().
Russell Belfer 282283ac 2012-05-04T16:46:46 Fix valgrind issues There are three changes here: - correctly propogate error code from failed object lookups - make zlib inflate use our allocators - add OID to notfound error in ODB lookups
Russell Belfer 2bc8fa02 2012-04-17T10:14:24 Implement git_pool paged memory allocator This adds a `git_pool` object that can do simple paged memory allocation with free for the entire pool at once. Using this, you can replace many small allocations with large blocks that can then cheaply be doled out in small pieces. This is best used when you plan to free the small blocks all at once - for example, if they represent the parsed state from a file or data stream that are either all kept or all discarded. There are two real patterns of usage for `git_pools`: either for "string" allocation, where the item size is a single byte and you end up just packing the allocations in together, or for "fixed size" allocation where you are allocating a large object (e.g. a `git_oid`) and you generally just allocation single objects that can be tightly packed. Of course, you can use it for other things, but those two cases are the easiest.
Russell Belfer 4aa7de15 2012-03-19T17:49:46 Convert indexer, notes, sha1_lookup, and signature More files moved to new error handling style.
Russell Belfer deafee7b 2012-03-14T17:36:15 Continue error conversion This converts blob.c, fileops.c, and all of the win32 files. Also, various minor cleanups throughout the code. Plus, in testing the win32 build, I cleaned up a bunch (although not all) of the warnings with the 64-bit build.
Russell Belfer e1de726c 2012-03-12T22:55:40 Migrate ODB files to new error handling This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to the new style of error handling. Also got the unix and win32 versions of map.c. There are some minor changes to other files but no others were completely converted. This also contains an update to filebuf so that a zeroed out filebuf will not think that the fd (== 0) is actually open (and inadvertently call close() on fd 0 if cleaned up). Lastly, this was built and tested on win32 and contains a bunch of fixes for the win32 build which was pretty broken.
Russell Belfer 998f7b3d 2012-03-07T10:52:17 Fix issues raised on pull request This resolves the comments on pull request #590
Russell Belfer ae9e29fd 2012-03-06T16:14:31 Migrating diff to new error handling Ended up migrating a bunch of upstream functions as well including vector, attr_file, and odb in order to get this to work right.
Vicent Martí 1a481123 2012-02-17T00:13:34 error-handling: References Yes, this is error handling solely for `refs.c`, but some of the abstractions leak all ofer the code base.
Vicent Martí 13224ea4 2012-02-27T04:28:31 buffer: Unify `git_fbuffer` and `git_buf` This makes so much sense that I can't believe it hasn't been done before. Kill the old `git_fbuffer` and read files straight into `git_buf` objects. Also: In order to fully support 4GB files in 32-bit systems, the `git_buf` implementation has been changed from using `ssize_t` for storage and storing negative values on allocation failure, to using `size_t` and changing the buffer pointer to a magical pointer on allocation failure. Hopefully this won't break anything.
Russell Belfer 1ec1de6d 2012-02-23T11:15:45 Fix warnings about type conversion on win32
Vicent Martí 0c3bae62 2012-02-15T16:56:56 zlib: Remove custom `git2/zlib.h` header This is legacy compat stuff for when `deflateBound` is not defined, but we're not embedding zlib and that function is always available. Kill that with fire.
schu 5e0de328 2012-02-13T17:10:24 Update Copyright header Signed-off-by: schu <schu-github@schulog.org>
Vicent Martí f19e3ca2 2012-02-10T20:16:42 odb: Proper symlink hashing
Vicent Martí 18e5b854 2012-02-10T19:47:02 odb: Add internal `git_odb__hashfd`
Russell Belfer 1744fafe 2012-01-17T15:49:47 Move path related functions from fileops to path This takes all of the functions that look up simple data about paths (such as `git_futils_isdir`) and moves them over to path.h (becoming `git_path_isdir`). This leaves fileops.h just with functions that actually manipulate the filesystem or look at the file contents in some way. As part of this, the dir.h header which is really just for win32 support was moved into win32 (with some minor changes).
Russell Belfer 97769280 2011-11-30T11:27:15 Use git_buf for path storage instead of stack-based buffers This converts virtually all of the places that allocate GIT_PATH_MAX buffers on the stack for manipulating paths to use git_buf objects instead. The patch is pretty careful not to touch the public API for libgit2, so there are a few places that still use GIT_PATH_MAX. This extends and changes some details of the git_buf implementation to add a couple of extra functions and to make error handling easier. This includes serious alterations to all the path.c functions, and several of the fileops.c ones, too. Also, there are a number of new functions that parallel existing ones except that use a git_buf instead of a stack-based buffer (such as git_config_find_global_r that exists alongsize git_config_find_global). This also modifies the win32 version of p_realpath to allocate whatever buffer size is needed to accommodate the realpath instead of hardcoding a GIT_PATH_MAX limit, but that change needs to be tested still.
Vicent Marti 45e79e37 2011-11-26T04:59:21 Rename all `_close` methods There's no difference between `_free` and `_close` semantics: keep everything with the same name to avoid confusions.
Vicent Marti 9462c471 2011-11-25T08:16:26 repository: Change ownership semantics The ownership semantics have been changed all over the library to be consistent. There are no more "borrowed" or duplicated references. Main changes: - `git_repository_open2` and `3` have been dropped. - Added setters and getters to hotswap all the repository owned objects: `git_repository_index` `git_repository_set_index` `git_repository_odb` `git_repository_set_odb` `git_repository_config` `git_repository_set_config` `git_repository_workdir` `git_repository_set_workdir` Now working directories/index files/ODBs and so on can be hot-swapped after creating a repository and between operations. - All these objects now have proper ownership semantics with refcounting: they all require freeing after they are no longer needed (the repository always keeps its internal reference). - Repository open and initialization has been updated to keep in mind the configuration files. Bare repositories are now always detected, and a default config file is created on init. - All the tests affected by these changes have been dropped from the old test suite and ported to the new one.
Vicent Marti 3286c408 2011-10-28T14:51:13 global: Properly use `git__` memory wrappers Ensure that all memory related functions (malloc, calloc, strdup, free, etc) are using their respective `git__` wrappers.
Vicent Marti 8af4d074 2011-09-29T15:34:17 odb: Let users decide compression level for the loose ODB
Vicent Marti 87d9869f 2011-09-19T03:34:49 Tabify everything There were quite a few places were spaces were being used instead of tabs. Try to catch them all. This should hopefully not break anything. Except for `git blame`. Oh well.
Vicent Marti bb742ede 2011-09-19T01:54:32 Cleanup legal data 1. The license header is technically not valid if it doesn't have a copyright signature. 2. The COPYING file has been updated with the different licenses used in the project. 3. The full GPLv2 header in each file annoys me.
Vicent Marti 84dd3820 2011-08-18T02:13:51 posix: Properly handle `snprintf` in all platforms
Vicent Marti c85e08b1 2011-08-16T13:05:05 odb: Do not pass around a header when hashing
nulltoken b21fb849 2011-07-09T06:36:18 Fix MSVC compilation warning
Vicent Marti c52736fa 2011-07-09T15:05:14 status: Cleanup The `hashfile` function has been moved to ODB, next to `git_odb_hash`. Global state has been removed from the dirent call in `status.c`, because global state is killing the rainforest and causing global warming.
Vicent Marti de18f276 2011-07-07T01:46:20 vector: Timsort all of the things Drop the GLibc implementation of Merge Sort and replace it with Timsort. The algorithm has been tuned to work on arrays of pointers (void **), so there's no longer a need to abstract the byte-width of each element in the array. All the comparison callbacks now take pointers-to-elements, not pointers-to-pointers, so there's now one less level of dereferencing. E.g. int index_cmp(const void *a, const void *b) { - const git_index_entry *entry_a = *(const git_index_entry **)(a); + const git_index_entry *entry_a = (const git_index_entry *)(a); The result is up to a 40% speed-up when sorting vectors. Memory usage remains lineal. A new `bsearch` implementation has been added, whose callback also supplies pointer-to-elements, to uniform the Vector API again.
Vicent Marti f79026b4 2011-07-04T11:43:34 fileops: Cleanup Cleaned up the structure of the whole OS-abstraction layer. fileops.c now contains a set of utility methods for file management used by the library. These are abstractions on top of the original POSIX calls. There's a new file called `posix.c` that contains emulations/reimplementations of all the POSIX calls the library uses. These are prefixed with `p_`. There's a specific posix file for each platform (win32 and unix). All the path-related methods have been moved from `utils.c` to `path.c` and have their own prefix.
Kirill A. Shutemov 932d1baf 2011-06-30T19:52:34 cleanup: remove trailing spaces Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
Vicent Marti 984ed6b6 2011-06-19T12:48:16 odb: Add GIT_EPASSTHROUGH Allows a custom user backend to passthrough one of the callbacks. Used for e.g. caching backends.
Vicent Marti 0291b5b7 2011-06-03T19:59:16 odb: Fix loading ODB alternates Fixed an issue with the `strtokz implementation and added support for comments and relative paths in the alternates file.
Vicent Martí 1e9b7a09 2011-06-02T15:12:37 Merge pull request #144 from nordsturm/fix_fakewstream Fix fake wstream write
Marc Pegon 6c8ca697 2011-05-29T17:57:25 Fixed some error messages related to searching objects from a short oid. Fixed forgot to check that prefix length is greater than minimum prefix length in read_unique_short_oid method from pack backend.
Vicent Marti d0323a5f 2011-06-01T21:25:56 short-oid: Cleanup
Marc Pegon dd453c4d 2011-05-27T22:46:41 Added git.git sha1 lookup method to replace simple binary search in pack backend. Implemented find_unique_short_oid for pack backend, based on git sha1 lookup method; finding an object given its full oid is just a particular case of searching the unique object matching an oid prefix (short oid). Added git_odb_read_unique_short_oid, which iterates over all the backends to find and read the unique object matching the given oid prefix. Added a git_object_lookup_short_oid method to find the unique object in the repository matching a given oid prefix : it generalizes git_object_lookup which now does nothing but calls git_object_lookup_short_oid.
Vicent Marti 1e85d1aa 2011-05-23T21:09:07 odb: Reword errors