|
3fa764ed
|
2015-11-04T09:20:14
|
|
filebuf: allow using a custom buffer size
Allow setting the buffer size on open in order to use this data
structure more generally as a spill buffer, with larger buffer sizes for
specific use-cases.
|
|
6ff8a7c4
|
2016-03-10T17:05:30
|
|
filebuf: handle write error in `lock_file`
When writing to a file with locking not check if writing the
locked file actually succeeds. Fix the issue by returning error
code and message when writing fails.
|
|
ec50b23a
|
2015-11-03T17:02:07
|
|
filebuf: detect directories in our way
When creating a filebuf, detect a directory that exists in our
target file location. This prevents a failure later, when we try
to move the lock file to the destination.
|
|
|
d83b2e9f
|
2015-09-05T03:54:06
|
|
filebuf: follow symlinks when creating a lock file
We create a lockfile to update files under GIT_DIR. Sometimes these
files are actually located elsewhere and a symlink takes their place. In
that case we should lock and update the file at its final location
rather than overwrite the symlink.
|
|
|
19d9beb7
|
2015-07-24T19:22:41
|
|
filebuf: remove lockfile upon rename errors
When we have an error renaming the lockfile, we need to make sure
that we remove it upon cleanup. For this, we need to keep track of
whether we opened the file and whether the rename succeeded.
If we did create the lockfile but the rename did not succeed, we
remove the lockfile. This won't protect against all errors, but
the most common ones (target file is open) does get handled.
|
|
7dd22538
|
2015-05-11T10:19:25
|
|
centralizing all IO buffer size values
|
|
8aab36a3
|
2015-02-12T22:14:53
|
|
filebuf: use an int for return check
|
|
|
f1453c59
|
2015-02-12T12:19:37
|
|
Make our overflow check look more like gcc/clang's
Make our overflow checking look more like gcc and clang's, so that
we can substitute it out with the compiler instrinsics on platforms
that support it. This means dropping the ability to pass `NULL` as
an out parameter.
As a result, the macros also get updated to reflect this as well.
|
|
|
ec3b4d35
|
2015-02-11T11:20:05
|
|
Use `size_t` to hold size of arrays
Use `size_t` to hold the size of arrays to ease overflow checking,
lest we check for overflow of a `size_t` then promptly truncate
by packing the length into a smaller type.
|
|
|
392702ee
|
2015-02-09T23:41:13
|
|
allocations: test for overflow of requested size
Introduce some helper macros to test integer overflow from arithmetic
and set error message appropriately.
|
|
|
e58281aa
|
2014-04-04T14:40:38
|
|
filebuf: make unlocking atomic
When renaming a lock file to its final location, we need to make sure
that it is replaced atomically.
We currently have a workaround for Windows by removing the target file.
This means that the target file, which may be a ref or a packfile, may
cease to exist for a short wile, which shold be avoided.
Implement the workaround only in Windows, by making sure that the file
we want to replace is writable.
|
|
|
4f46a98b
|
2014-02-24T23:32:25
|
|
Remove now-duplicated stdarg.h include
|
|
|
1d3a8aeb
|
2013-11-04T18:28:57
|
|
move mode_t to filebuf_open instead of _commit
|
|
|
f966acd1
|
2013-11-04T15:46:32
|
|
Take umask into account in filebuf_commit
|
|
|
3d276874
|
2013-08-19T10:30:44
|
|
index: report when it's locked
Report the index being locked with its own error code in order to be
able to differentiate, as a locked index is typically the result of a
crashed process or concurrent access, both of which often require user
intervention to fix.
|
|
|
359fc2d2
|
2013-01-08T17:07:25
|
|
update copyrights
|
|
|
2a612fe3
|
2012-11-13T14:57:35
|
|
filebuf now has a git_hash_ctx instead of a ctx*
|
|
|
d6fb0924
|
2012-11-05T12:37:15
|
|
Win32 CryptoAPI and CNG support for SHA1
|
|
|
603bee07
|
2012-11-12T19:22:49
|
|
Remove git_hash_ctx_new - callers now _ctx_init()
|
|
744cc03e
|
2012-10-30T12:10:36
|
|
Add git_config_refresh() API to reload config
This adds a new API that allows users to reload the config if the
file has changed on disk. A new config callback function to
refresh the config was added.
The modified time and file size are used to test if the file needs
to be reloaded (and are now stored in the disk backend object).
In writing tests, just using mtime was a problem / race, so I
wanted to check file size as well. To support that, I extended
`git_futils_readbuffer_updated` to optionally check file size in
addition to mtime, and I added a new function `git_filebuf_stats`
to fetch the mtime and size for an open filebuf (so that the
config could be easily refreshed after a write).
Lastly, I moved some similar file checking code for attributes
into filebuf. It is still only being used for attrs, but it
seems potentially reusable, so I thought I'd move it over.
|
|
c859184b
|
2012-09-11T23:05:24
|
|
Properly handle p_reads
|
|
|
97a17e4e
|
2012-08-24T12:19:22
|
|
Fix valgrind warnings and spurious error messages
Just clean up valgrind warnings about uninitialized memory
and also clear out errno in some cases where it results in
a false error message being generated at a later point.
|
|
|
5daca042
|
2012-08-03T01:01:21
|
|
filebuf: Check the return value for `close`
|
|
9bea8e85
|
2012-05-27T19:54:53
|
|
filebuf: add git_filebuf_flush()
|
|
4d53f3e2
|
2012-04-05T23:37:38
|
|
filebuf: add option not to buffer the contents at all
The new indexer needs to be able to bypass any kind of buffering, as
it's trying to map data that it has just written to disk.
|
|
0d0fa7c3
|
2012-03-16T15:56:01
|
|
Convert attr, ignore, mwindow, status to new errors
Also cleaned up some previously converted code that still had
little things to polish.
|
|
|
e1de726c
|
2012-03-12T22:55:40
|
|
Migrate ODB files to new error handling
This migrates odb.c, odb_loose.c, odb_pack.c and pack.c to
the new style of error handling. Also got the unix and win32
versions of map.c. There are some minor changes to other
files but no others were completely converted.
This also contains an update to filebuf so that a zeroed out
filebuf will not think that the fd (== 0) is actually open
(and inadvertently call close() on fd 0 if cleaned up).
Lastly, this was built and tested on win32 and contains a
bunch of fixes for the win32 build which was pretty broken.
|
|
|
dda708e7
|
2012-03-09T19:55:50
|
|
error-handling: On-disk config file backend
Includes:
- Proper error reporting when encountering syntax errors in a
config file (file, line number, column).
- Rewritten `config_write`, now with 99% less goto-spaghetti
- Error state in `git_filebuf`: filebuf write functions no longer
need to be checked for error returns. If any of the writes performed
on a buffer fail, the last call to `git_filebuf_commit` or
`git_filebuf_hash` will fail accordingly and set the appropiate error
message. Baller!
|
|
|
ae9e29fd
|
2012-03-06T16:14:31
|
|
Migrating diff to new error handling
Ended up migrating a bunch of upstream functions as well
including vector, attr_file, and odb in order to get this
to work right.
|
|
|
1a481123
|
2012-02-17T00:13:34
|
|
error-handling: References
Yes, this is error handling solely for `refs.c`, but some of the
abstractions leak all ofer the code base.
|
|
1db9d2c3
|
2012-02-23T17:11:20
|
|
Ensure that commits don't fail if committing content that already exists
Making a commit that results in a blob that already exists in the ODB (i.e.
committing something, then making a revert commit) will result in us trying
to p_rename -> MoveFileExW a temp file into the existing ODB entry. Despite
the MOVEFILE_REPLACE_EXISTING flag is passed in, Win32 does not care and
fails it with STATUS_ACCESS_DENIED.
To fix this, we p_unlink the ODB entry before attempting to rename it. This
call will typically fail, but we don't care, we'll let the p_rename fail if
the file actually does exist and we couldn't delete it for some reason (ACLs,
etc).
|
|
5e0de328
|
2012-02-13T17:10:24
|
|
Update Copyright header
Signed-off-by: schu <schu-github@schulog.org>
|
|
|
1744fafe
|
2012-01-17T15:49:47
|
|
Move path related functions from fileops to path
This takes all of the functions that look up simple data about
paths (such as `git_futils_isdir`) and moves them over to path.h
(becoming `git_path_isdir`). This leaves fileops.h just with
functions that actually manipulate the filesystem or look at
the file contents in some way.
As part of this, the dir.h header which is really just for win32
support was moved into win32 (with some minor changes).
|
|
|
97769280
|
2011-11-30T11:27:15
|
|
Use git_buf for path storage instead of stack-based buffers
This converts virtually all of the places that allocate GIT_PATH_MAX
buffers on the stack for manipulating paths to use git_buf objects
instead. The patch is pretty careful not to touch the public API
for libgit2, so there are a few places that still use GIT_PATH_MAX.
This extends and changes some details of the git_buf implementation
to add a couple of extra functions and to make error handling easier.
This includes serious alterations to all the path.c functions, and
several of the fileops.c ones, too. Also, there are a number of new
functions that parallel existing ones except that use a git_buf
instead of a stack-based buffer (such as git_config_find_global_r
that exists alongsize git_config_find_global).
This also modifies the win32 version of p_realpath to allocate whatever
buffer size is needed to accommodate the realpath instead of hardcoding
a GIT_PATH_MAX limit, but that change needs to be tested still.
|
|
|
b762e576
|
2011-11-17T15:10:27
|
|
filebuf: add GIT_FILEBUF_INIT and protect multiple opens and cleanups
Update all stack allocations of git_filebuf to use GIT_FILEBUF_INIT
and make git_filebuf_open and git_filebuf_cleanup safe to be called
multiple times on the same buffer.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
0c49ec2d
|
2011-11-07T19:34:24
|
|
Implement p_rename
Move the callers of git_futils_mv_atomic to use p_rename.
Signed-off-by: Carlos Martín Nieto <carlos@cmartin.tk>
|
|
|
89fb8f02
|
2011-10-28T19:04:23
|
|
Merge pull request #456 from brodie/perm-fixes
Create objects, indexes, and directories with the right file permissions
|
|
|
3286c408
|
2011-10-28T14:51:13
|
|
global: Properly use `git__` memory wrappers
Ensure that all memory related functions (malloc, calloc, strdup, free,
etc) are using their respective `git__` wrappers.
|
|
01ad7b3a
|
2011-09-06T15:48:45
|
|
*: correct and codify various file permissions
The following files now have 0444 permissions:
- loose objects
- pack indexes
- pack files
- packs downloaded by fetch
- packs downloaded by the HTTP transport
And the following files now have 0666 permissions:
- config files
- repository indexes
- reflogs
- refs
This brings libgit2 more in line with Git.
Note that git_filebuf_commit() and git_filebuf_commit_at() have both
gained a new mode parameter.
The latter change fixes an important issue where filebufs created with
GIT_FILEBUF_TEMPORARY received 0600 permissions (due to mkstemp(3)
usage). Now we chmod() the file before renaming it into place.
Tests have been added to confirm that new commit, tag, and tree
objects are created with the right permissions. I don't have access to
Windows, so for now I've guarded the tests with "#ifndef GIT_WIN32".
|
|
|
ce8cd006
|
2011-09-07T15:32:44
|
|
fileops/repository: create (most) directories with 0777 permissions
To further match how Git behaves, this change makes most of the
directories libgit2 creates in a git repo have a file mode of
0777. Specifically:
- Intermediate directories created with git_futils_mkpath2file() have
0777 permissions. This affects odb_loose, reflog, and refs.
- The top level folder for bare repos is created with 0777
permissions.
- The top level folder for non-bare repos is created with 0755
permissions.
- /objects/info/, /objects/pack/, /refs/heads/, and /refs/tags/ are
created with 0777 permissions.
Additionally, the following changes have been made:
- fileops functions that create intermediate directories have grown a
new dirmode parameter. The only exception to this is filebuf's
lock_file(), which unconditionally creates intermediate directories
with 0777 permissions when GIT_FILEBUF_FORCE is set.
- The test runner now sets the umask to 0 before running any
tests. This ensurses all file mode checks are consistent across
systems.
- t09-tree.c now does a directory permissions check. I've avoided
adding this check to other tests that might reuse existing
directories from the prefabricated test repos. Because they're
checked into the repo, they have 0755 permissions.
- Other assorted directories created by tests have 0777 permissions.
|
|
|
c103d7b4
|
2011-09-29T15:49:28
|
|
odb: Pass compression settings to filebuf
|
|
|
3125929b
|
2011-09-18T19:54:18
|
|
Merge pull request #393 from schu/unused-but-set-variable
filebuf.c: fix unused-but-set warning
|
|
|
bb742ede
|
2011-09-19T01:54:32
|
|
Cleanup legal data
1. The license header is technically not valid if it doesn't have a
copyright signature.
2. The COPYING file has been updated with the different licenses used in
the project.
3. The full GPLv2 header in each file annoys me.
|
|
|
c8f16bfe
|
2011-09-09T14:05:32
|
|
filebuf.c: fix unused-but-set warning
write_deflate() used to ignore errors by zlib's deflate function when
not compiling in DEBUG mode. Always read $result and throw an error
instead.
Signed-off-by: schu <schu-github@schulog.org>
|
|
1c3fac4d
|
2011-09-08T14:31:37
|
|
Add casts to get rid of some warnings when filling zlib structures
|
|
|
353560b4
|
2011-09-08T14:29:54
|
|
Get rid of a superfluous pointer cast
|
|
|
45e93ef3
|
2011-09-08T14:22:29
|
|
Fix minor indentation issues (spaces to tabs)
|
|
05a62d1a
|
2011-07-18T05:11:18
|
|
filebuf: update git_filebuf.write signature to take non-const buffer
z_stream.next_in is non-const. Although currently Zlib doesn't modify
buffer content on deflate(), it might be change in the future. gzwrite()
already modify it.
To avoid this let's change signature of git_filebuf.write and rework
git_filebuf_write() accordingly.
Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
|
|
|
afeecf4f
|
2011-07-09T02:10:46
|
|
odb: Direct writes are back
DIRECT WRITES ARE BACK AND FASTER THAN EVER. The streaming writer to the
ODB was an overkill for the smaller objects like Commit and Tags; most
of the streaming logic was taking too long.
This commit makes Commits, Tags and Trees to be built-up in memory, and
then written to disk in 2 pushes (header + data), instead of streaming
everything.
This is *always* faster, even for big files (since the git_filebuf class
still does streaming writes when the memory cache overflows). This is
also a gazillion lines of code smaller, because we don't have to
precompute the final size of the object before starting the stream (this
was kind of defeating the point of streaming, anyway).
Blobs are still written with full streaming instead of loading them in
memory, since this is still the fastest way.
A new `git_buf` class has been added. It's missing some features, but
it'll get there.
|
|
|
2fc78e70
|
2011-07-08T23:01:37
|
|
posix: Portable `vsnprintf`
Our good, lovely folks at Microsoft decided that there was no good
reason to make `vsnprintf` compilant with the C standard, so that
function in Windows returns -1 on overflow, instead of returning the
actual byte count needed to write the full string.
We now handle this situation more gracefully with the POSIX
compatibility layer, by returning the needed byte size using an
auxiliary method instead of blindly resizing the target buffer until it
fits.
This means we can now support `printf`s of any size by allocating a
temporary buffer. That's good.
|
|
|
f79026b4
|
2011-07-04T11:43:34
|
|
fileops: Cleanup
Cleaned up the structure of the whole OS-abstraction layer.
fileops.c now contains a set of utility methods for file management used
by the library. These are abstractions on top of the original POSIX
calls.
There's a new file called `posix.c` that contains
emulations/reimplementations of all the POSIX calls the library uses.
These are prefixed with `p_`. There's a specific posix file for each
platform (win32 and unix).
All the path-related methods have been moved from `utils.c` to `path.c`
and have their own prefix.
|
|
|
932d1baf
|
2011-06-30T19:52:34
|
|
cleanup: remove trailing spaces
Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
|
|
|
fe5babac
|
2011-06-30T00:16:23
|
|
filebuf: fix endless loop on writing buf > WRITE_BUFFER_SIZE
Signed-off-by: Kirill A. Shutemov <kirill@shutemov.name>
|
|
bb9272dd
|
2011-06-07T17:03:07
|
|
filebuf cleanup: only unlink lockfile if we've opened it
Add a check for the file descriptor in git_filebuf_cleanup. Without
it, an existing lockfile would be deleted if we tried to acquire it
(but failed, as the lockfile already existed).
Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
|
|
|
f9213015
|
2011-06-06T01:54:59
|
|
filebuf: Fix printf buffer overflows
The filebuf was not being properly written after a flush. This should
cut it now.
Fixes #228
|
|
1549cba9
|
2011-06-03T21:18:24
|
|
Filebuf: Fixed a TODO in filebuf (real lock in lock_file)
Added gitfo_creat_locked and gitfo_creat_locked_force
|
|
|
f6328611
|
2011-05-23T21:00:19
|
|
filebuf: Reword errors
|
|
374db5f9
|
2011-05-17T17:43:00
|
|
filebuf.c: Move to new error handling mechanism
|
|
|
a6359408
|
2011-04-10T12:23:55
|
|
Use Z_BEST_SPEED for filebuf deflating
This is what Git uses by default for all deflating.
|
|
|
f6f72d7e
|
2011-03-23T18:44:53
|
|
Improve the ODB writing backend
Temporary files when doing streaming writes are now stored inside the
Objects folder, to prevent issues when moving files between
disks/partitions.
Add support for block writes to the ODB again (for those backends that
cannot implement streaming).
|
|
|
72a3fe42
|
2011-03-18T19:38:49
|
|
I broke your bindings
Hey. Apologies in advance -- I broke your bindings.
This is a major commit that includes a long-overdue redesign of the
whole object-database structure. This is expected to be the last major
external API redesign of the library until the first non-alpha release.
Please get your bindings up to date with these changes. They will be
included in the next minor release. Sorry again!
Major features include:
- Real caching and refcounting on parsed objects
- Real caching and refcounting on objects read from the ODB
- Streaming writes & reads from the ODB
- Single-method writes for all object types
- The external API is now partially thread-safe
The speed increases are significant in all aspects, specially when
reading an object several times from the ODB (revwalking) and when
writing big objects to the ODB.
Here's a full changelog for the external API:
blob.h
------
- Remove `git_blob_new`
- Remove `git_blob_set_rawcontent`
- Remove `git_blob_set_rawcontent_fromfile`
- Rename `git_blob_writefile` -> `git_blob_create_fromfile`
- Change `git_blob_create_fromfile`:
The `path` argument is now relative to the repository's working dir
- Add `git_blob_create_frombuffer`
commit.h
--------
- Remove `git_commit_new`
- Remove `git_commit_add_parent`
- Remove `git_commit_set_message`
- Remove `git_commit_set_committer`
- Remove `git_commit_set_author`
- Remove `git_commit_set_tree`
- Add `git_commit_create`
- Add `git_commit_create_v`
- Add `git_commit_create_o`
- Add `git_commit_create_ov`
tag.h
-----
- Remove `git_tag_new`
- Remove `git_tag_set_target`
- Remove `git_tag_set_name`
- Remove `git_tag_set_tagger`
- Remove `git_tag_set_message`
- Add `git_tag_create`
- Add `git_tag_create_o`
tree.h
------
- Change `git_tree_entry_2object`:
New signature is `(git_object **object_out, git_repository *repo, git_tree_entry *entry)`
- Remove `git_tree_new`
- Remove `git_tree_add_entry`
- Remove `git_tree_remove_entry_byindex`
- Remove `git_tree_remove_entry_byname`
- Remove `git_tree_clearentries`
- Remove `git_tree_entry_set_id`
- Remove `git_tree_entry_set_name`
- Remove `git_tree_entry_set_attributes`
object.h
------------
- Remove `git_object_new
- Remove `git_object_write`
- Change `git_object_close`:
This method is now *mandatory*. Not closing an object causes a
memory leak.
odb.h
-----
- Remove type `git_rawobj`
- Remove `git_rawobj_close`
- Rename `git_rawobj_hash` -> `git_odb_hash`
- Change `git_odb_hash`:
New signature is `(git_oid *id, const void *data, size_t len, git_otype type)`
- Add type `git_odb_object`
- Add `git_odb_object_close`
- Change `git_odb_read`:
New signature is `(git_odb_object **out, git_odb *db, const git_oid *id)`
- Change `git_odb_read_header`:
New signature is `(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id)`
- Remove `git_odb_write`
- Add `git_odb_open_wstream`
- Add `git_odb_open_rstream`
odb_backend.h
-------------
- Change type `git_odb_backend`:
New internal signatures are as follows
int (* read)(void **, size_t *, git_otype *, struct git_odb_backend *, const git_oid *)
int (* read_header)(size_t *, git_otype *, struct git_odb_backend *, const git_oid *)
int (* writestream)(struct git_odb_stream **, struct git_odb_backend *, size_t, git_otype)
int (* readstream)( struct git_odb_stream **, struct git_odb_backend *, const git_oid *)
- Add type `git_odb_stream`
- Add enum `git_odb_streammode`
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
55ffebe3
|
2011-03-05T14:34:32
|
|
Fix creation of deeply-rooted references
Use a new `gitfo_creat_force` that will create the full path to a file
before creating it.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
19a30a3f
|
2011-03-03T19:53:17
|
|
Add new move function, `gitfo_mv_force`
Forces a move by creating the folder for the destination file, if it
doesn't exist.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
86194b24
|
2011-02-18T21:57:53
|
|
Split packed from unpacked references
These two reference types are now stored separately to eventually allow
the removal/renaming of loose references and rewriting of the refs
packfile.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
c3be1468
|
2011-02-24T19:31:12
|
|
Fix double-freeing file descriptors
Was crashing the Windows build.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
5591ea15
|
2011-02-22T14:58:54
|
|
Add printf method to the File Buffer
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|
|
|
817c2820
|
2011-02-21T17:05:16
|
|
Rewrite all file IO for more performance
The new `git_filebuf` structure provides atomic high-performance writes
to disk by using a write cache, and optionally a double-buffered scheme
through a worker thread (not enabled yet).
Writes can be done 3-layered, like in git.git (user code -> write cache
-> disk), or 2-layered, by writing directly on the cache. This makes
index writing considerably faster.
The `git_filebuf` structure contains all the old functionality of
`git_filelock` for atomic file writes and reads. The `git_filelock`
structure has been removed.
Additionally, the `git_filebuf` API allows to automatically hash (SHA1)
all the data as it is written to disk (hashing is done smartly on big
chunks to improve performance).
Signed-off-by: Vicent Marti <tanoku@gmail.com>
|