src/filebuf.c


Log

Author Commit Date CI Message
Vicent Marti afeecf4f 2011-07-09T02:10:46 odb: Direct writes are back DIRECT WRITES ARE BACK AND FASTER THAN EVER. The streaming writer to the ODB was an overkill for the smaller objects like Commit and Tags; most of the streaming logic was taking too long. This commit makes Commits, Tags and Trees to be built-up in memory, and then written to disk in 2 pushes (header + data), instead of streaming everything. This is *always* faster, even for big files (since the git_filebuf class still does streaming writes when the memory cache overflows). This is also a gazillion lines of code smaller, because we don't have to precompute the final size of the object before starting the stream (this was kind of defeating the point of streaming, anyway). Blobs are still written with full streaming instead of loading them in memory, since this is still the fastest way. A new `git_buf` class has been added. It's missing some features, but it'll get there.
Vicent Marti 2fc78e70 2011-07-08T23:01:37 posix: Portable `vsnprintf` Our good, lovely folks at Microsoft decided that there was no good reason to make `vsnprintf` compilant with the C standard, so that function in Windows returns -1 on overflow, instead of returning the actual byte count needed to write the full string. We now handle this situation more gracefully with the POSIX compatibility layer, by returning the needed byte size using an auxiliary method instead of blindly resizing the target buffer until it fits. This means we can now support `printf`s of any size by allocating a temporary buffer. That's good.
Vicent Marti f79026b4 2011-07-04T11:43:34 fileops: Cleanup Cleaned up the structure of the whole OS-abstraction layer. fileops.c now contains a set of utility methods for file management used by the library. These are abstractions on top of the original POSIX calls. There's a new file called `posix.c` that contains emulations/reimplementations of all the POSIX calls the library uses. These are prefixed with `p_`. There's a specific posix file for each platform (win32 and unix). All the path-related methods have been moved from `utils.c` to `path.c` and have their own prefix.
Kirill A. Shutemov 932d1baf 2011-06-30T19:52:34 cleanup: remove trailing spaces Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
Kirill A. Shutemov fe5babac 2011-06-30T00:16:23 filebuf: fix endless loop on writing buf > WRITE_BUFFER_SIZE Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
Carlos Martín Nieto bb9272dd 2011-06-07T17:03:07 filebuf cleanup: only unlink lockfile if we've opened it Add a check for the file descriptor in git_filebuf_cleanup. Without it, an existing lockfile would be deleted if we tried to acquire it (but failed, as the lockfile already existed). Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
Vicent Marti f9213015 2011-06-06T01:54:59 filebuf: Fix printf buffer overflows The filebuf was not being properly written after a flush. This should cut it now. Fixes #228
Romain Geissler 1549cba9 2011-06-03T21:18:24 Filebuf: Fixed a TODO in filebuf (real lock in lock_file) Added gitfo_creat_locked and gitfo_creat_locked_force
Vicent Marti f6328611 2011-05-23T21:00:19 filebuf: Reword errors
Jakob Pfender 374db5f9 2011-05-17T17:43:00 filebuf.c: Move to new error handling mechanism
Vicent Marti a6359408 2011-04-10T12:23:55 Use Z_BEST_SPEED for filebuf deflating This is what Git uses by default for all deflating.
Vicent Marti f6f72d7e 2011-03-23T18:44:53 Improve the ODB writing backend Temporary files when doing streaming writes are now stored inside the Objects folder, to prevent issues when moving files between disks/partitions. Add support for block writes to the ODB again (for those backends that cannot implement streaming).
Vicent Marti 72a3fe42 2011-03-18T19:38:49 I broke your bindings Hey. Apologies in advance -- I broke your bindings. This is a major commit that includes a long-overdue redesign of the whole object-database structure. This is expected to be the last major external API redesign of the library until the first non-alpha release. Please get your bindings up to date with these changes. They will be included in the next minor release. Sorry again! Major features include: - Real caching and refcounting on parsed objects - Real caching and refcounting on objects read from the ODB - Streaming writes & reads from the ODB - Single-method writes for all object types - The external API is now partially thread-safe The speed increases are significant in all aspects, specially when reading an object several times from the ODB (revwalking) and when writing big objects to the ODB. Here's a full changelog for the external API: blob.h ------ - Remove `git_blob_new` - Remove `git_blob_set_rawcontent` - Remove `git_blob_set_rawcontent_fromfile` - Rename `git_blob_writefile` -> `git_blob_create_fromfile` - Change `git_blob_create_fromfile`: The `path` argument is now relative to the repository's working dir - Add `git_blob_create_frombuffer` commit.h -------- - Remove `git_commit_new` - Remove `git_commit_add_parent` - Remove `git_commit_set_message` - Remove `git_commit_set_committer` - Remove `git_commit_set_author` - Remove `git_commit_set_tree` - Add `git_commit_create` - Add `git_commit_create_v` - Add `git_commit_create_o` - Add `git_commit_create_ov` tag.h ----- - Remove `git_tag_new` - Remove `git_tag_set_target` - Remove `git_tag_set_name` - Remove `git_tag_set_tagger` - Remove `git_tag_set_message` - Add `git_tag_create` - Add `git_tag_create_o` tree.h ------ - Change `git_tree_entry_2object`: New signature is `(git_object **object_out, git_repository *repo, git_tree_entry *entry)` - Remove `git_tree_new` - Remove `git_tree_add_entry` - Remove `git_tree_remove_entry_byindex` - Remove `git_tree_remove_entry_byname` - Remove `git_tree_clearentries` - Remove `git_tree_entry_set_id` - Remove `git_tree_entry_set_name` - Remove `git_tree_entry_set_attributes` object.h ------------ - Remove `git_object_new - Remove `git_object_write` - Change `git_object_close`: This method is now *mandatory*. Not closing an object causes a memory leak. odb.h ----- - Remove type `git_rawobj` - Remove `git_rawobj_close` - Rename `git_rawobj_hash` -> `git_odb_hash` - Change `git_odb_hash`: New signature is `(git_oid *id, const void *data, size_t len, git_otype type)` - Add type `git_odb_object` - Add `git_odb_object_close` - Change `git_odb_read`: New signature is `(git_odb_object **out, git_odb *db, const git_oid *id)` - Change `git_odb_read_header`: New signature is `(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id)` - Remove `git_odb_write` - Add `git_odb_open_wstream` - Add `git_odb_open_rstream` odb_backend.h ------------- - Change type `git_odb_backend`: New internal signatures are as follows int (* read)(void **, size_t *, git_otype *, struct git_odb_backend *, const git_oid *) int (* read_header)(size_t *, git_otype *, struct git_odb_backend *, const git_oid *) int (* writestream)(struct git_odb_stream **, struct git_odb_backend *, size_t, git_otype) int (* readstream)( struct git_odb_stream **, struct git_odb_backend *, const git_oid *) - Add type `git_odb_stream` - Add enum `git_odb_streammode` Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 55ffebe3 2011-03-05T14:34:32 Fix creation of deeply-rooted references Use a new `gitfo_creat_force` that will create the full path to a file before creating it. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 19a30a3f 2011-03-03T19:53:17 Add new move function, `gitfo_mv_force` Forces a move by creating the folder for the destination file, if it doesn't exist. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 86194b24 2011-02-18T21:57:53 Split packed from unpacked references These two reference types are now stored separately to eventually allow the removal/renaming of loose references and rewriting of the refs packfile. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti c3be1468 2011-02-24T19:31:12 Fix double-freeing file descriptors Was crashing the Windows build. Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 5591ea15 2011-02-22T14:58:54 Add printf method to the File Buffer Signed-off-by: Vicent Marti <tanoku@gmail.com>
Vicent Marti 817c2820 2011-02-21T17:05:16 Rewrite all file IO for more performance The new `git_filebuf` structure provides atomic high-performance writes to disk by using a write cache, and optionally a double-buffered scheme through a worker thread (not enabled yet). Writes can be done 3-layered, like in git.git (user code -> write cache -> disk), or 2-layered, by writing directly on the cache. This makes index writing considerably faster. The `git_filebuf` structure contains all the old functionality of `git_filelock` for atomic file writes and reads. The `git_filelock` structure has been removed. Additionally, the `git_filebuf` API allows to automatically hash (SHA1) all the data as it is written to disk (hashing is done smartly on big chunks to improve performance). Signed-off-by: Vicent Marti <tanoku@gmail.com>