src/fileops.c


Log

Author Commit Date CI Message
Patrick Steinhardt 38b6e700 2017-04-12T08:09:08 fileops: fix leaking fd in `mmap_ro_file` When the `git_futils_mmap_ro_file` function encounters an error after the file has been opened, it will do a simple returns. Instead, we should close the opened file descriptor to avoid a leak. This commit fixes the issue.
Patrick Steinhardt 9daba9f4 2017-03-28T10:12:23 fileops: do not overwrite correct error message on mmap When executing `git_futils_mmap_ro_file`, we first try to guess whether the file is mmapable at all. Part of this check is whether the file is too large to be mmaped, which can be true on systems with 32 bit `size_t` types. The check is performed by first getting the file size wtih `git_futils_filesize` and then checking whether the returned size can be represented as `size_t`, returning an error if so. While this test also catches the case where the function returned an error (as `-1` is not representable by `size_t`), we will set the misleading error message "file too large to mmap". But in fact, a negative return value from `git_futils_filesize` will be caused by the inability to fstat the file. Fix the error message by handling negative return values separately and not overwriting the error message in that case.
Patrick Steinhardt 1d39a603 2017-03-24T08:26:33 Merge pull request #4175 from libgit2/ethomson/dont_trunc_and_excl git_futils: don't O_EXCL and O_TRUNC
Edward Thomson 6fd6c678 2017-03-22T20:29:22 Merge pull request #4030 from libgit2/ethomson/fsync fsync all the things
Edward Thomson 4a26915d 2017-03-22T19:48:50 git_futils: don't O_EXCL and O_TRUNC `O_EXCL` and `O_TRUNC` are mutually exclusive flags to open(2); you can't truncate a file if you're asserting that it can't exist in the first place. Drop `O_TRUNC`.
Edward Thomson 11c8e756 2017-03-09T12:26:23 git_futils_readbuffer: don't compute sha-1 Don't compute the sha-1 in `git_futils_readbuffer_updated` unless the checksum was requested. This means that `git_futils_readbuffer` will not calculate the checksum unnecessarily.
Edward Thomson 3ac05d11 2017-02-17T16:48:03 win32: don't fsync parent directories on Windows Windows doesn't support it.
Edward Thomson 1229e1c4 2017-02-17T16:36:53 fsync parent directories when fsyncing When fsync'ing files, fsync the parent directory in the case where we rename a file into place, or create a new file, to ensure that the directory entry is flushed correctly.
Edward Thomson 5a747e0c 2016-12-27T23:07:07 git_futils_writebuffer: default only when flags==0 Only use defaults for `git_futils_writebuffer` when flags == 0, lest (1 << 31) be treated as the defaults.
Edward Thomson 5312621b 2016-12-15T10:51:02 git_futils_writebuffer: optionally fsync Add a custom `O_FSYNC` bit (if it's not been defined by the operating system`) so that `git_futils_writebuffer` can optionally do an `fsync` when it's done writing. We call `fsync` ourselves, even on systems that define `O_FSYNC` because its definition is no guarantee of its actual support. Mac, for instance, defines it but doesn't support it in an `open(2)` call.
Patrick Steinhardt 13c3bc9a 2017-01-27T14:32:23 strmap: remove GIT__USE_STRMAP macro
Patrick Steinhardt 73028af8 2017-01-27T14:20:24 khash: avoid using macro magic to get return address
Edward Thomson 909d5494 2016-12-29T12:25:15 giterr_set: consistent error messages Error messages should be sentence fragments, and therefore: 1. Should not begin with a capital letter, 2. Should not conclude with punctuation, and 3. Should not end a sentence and begin a new one
Carlos Martín Nieto f94825c1 2015-12-24T17:21:51 fileops: save errno and report file existence We need to save the errno, lest we clobber it in the giterr_set() call. Also add code for reporting that a path component is missing, which is a distinct failure mode.
Edward Thomson 27051d4e 2016-07-22T13:34:19 odb: only freshen pack files every 2 seconds Since writing multiple objects may all already exist in a single packfile, avoid freshening that packfile repeatedly in a tight loop. Instead, only freshen pack files every 2 seconds.
Edward Thomson 8f09a98e 2016-07-14T16:23:24 odb: freshen existing objects when writing When writing an object, we calculate its OID and see if it exists in the object database. If it does, we need to freshen the file that contains it.
Edward Thomson 3d6a42d1 2016-02-25T11:23:19 nsec: support NDK's crazy nanoseconds Android NDK does not have a `struct timespec` in its `struct stat` for nanosecond support, instead it has a single nanosecond member inside the struct stat itself. We will use that and use a macro to expand to the `st_mtim` / `st_mtimespec` definition on other systems (much like the existing `st_mtime` backcompat definition).
Edward Thomson 9447b9e5 2016-02-09T10:40:33 xplat: use st_mtimespec everywhere on mac
Jacques Germishuys eb11fac6 2015-11-20T18:57:13 Detect stat's structure
Carlos Martín Nieto 75a0ccf5 2015-11-12T19:53:09 Merge pull request #3170 from CmdrMoozy/nsec_fix git_index_entry__init_from_stat: set nsec fields in entry stats
Carlos Martín Nieto 3547b122 2015-10-30T21:36:51 filebuf: use an internal buffer This reduces the chances of a crash in the thread tests. This shouldn't affect general usage too much, since the main usage of these functions are to read into an empty buffer.
Carlos Martín Nieto eb597799 2015-10-29T21:12:37 filebuf: use a checksum to detect file changes Instead of relying on the size and timestamp, which can hide changes performed in the same second, hash the file content's when we care about detecting changes.
Axel Rasmussen 973a09a4 2015-10-01T18:35:21 index: don't populate nsec values if GIT_USE_NSEC is off
Axel Rasmussen 28cdb315 2015-09-18T23:48:22 apple: work around non-POSIX struct stat on OS X.
Axel Rasmussen 0226f7dd 2015-08-29T13:59:20 diff/index: respect USE_NSEC for racily clean file detection
Edward Thomson 9ce2e7b3 2015-09-17T12:48:37 `mkdir`: cope with root path on win32
Edward Thomson 81aaf370 2015-09-17T11:26:38 mkdir: chmod existing paths with `GIT_MKDIR_CHMOD`
Edward Thomson e24c60db 2015-09-17T09:42:05 mkdir: find component paths for mkdir_relative `git_futils_mkdir` does not blindly call `git_futils_mkdir_relative`. `git_futils_mkdir_relative` is used when you have some base directory and want to create some path inside of it, potentially removing blocking symlinks and files in the process. This is not suitable for a general recursive mkdir within the filesystem. Instead, when `mkdir` is being recursive, locate the first existent parent directory and use that as the base for `mkdir_relative`.
Edward Thomson ac2fba0e 2015-09-16T15:07:27 git_futils_mkdir_*: make a relative-to-base mkdir Untangle git_futils_mkdir from git_futils_mkdir_ext - the latter assumes that we own everything beneath the base, as if it were being called with a base of the repository or working directory, and is tailored towards checkout and ensuring that there is no bogosity beneath the base that must be cleaned up. This is (at best) slow and (at worst) unsafe in the larger context of a filesystem where we do not own things and cannot do things like unlink symlinks that are in our way.
Carlos Martín Nieto edef91ee 2015-05-15T12:23:27 fileops: set an error on write error for file copy We set an error if we get an error when reading, but we don't bother setting an error message for write failing. This causes a cryptic error to be shown to the user when the target filesystem is full.
Carlos Martín Nieto 54738368 2015-05-15T12:18:05 fileops: set an error message if we fail to link a file Now that `git_path_direach` lets us specify an error message to report, set an appropriate error message while linking.
J Wyman 7dd22538 2015-05-11T10:19:25 centralizing all IO buffer size values
Vicent Marti d88e6e9b 2015-03-19T18:38:13 mkdir-ext: Retry lstat on EEXIST race
Stefan Widgren c8e02b87 2015-02-15T21:07:05 Remove extra semicolon outside of a function Without this change, compiling with gcc and pedantic generates warning: ISO C does not allow extra ‘;’ outside of a function.
Carlos Martín Nieto a7fa970f 2015-02-15T05:13:50 Merge pull request #2895 from ethomson/alloc_overflow allocations: test for overflow of requested size
Edward Thomson f1453c59 2015-02-12T12:19:37 Make our overflow check look more like gcc/clang's Make our overflow checking look more like gcc and clang's, so that we can substitute it out with the compiler instrinsics on platforms that support it. This means dropping the ability to pass `NULL` as an out parameter. As a result, the macros also get updated to reflect this as well.
Carlos Martín Nieto 3ca91107 2015-02-13T08:30:21 Merge pull request #2897 from leoyanggit/fix_fileop_warning Fix build warning on Android
Edward Thomson 8d534b47 2015-02-11T13:01:00 p_read: ensure requested len is ssize_t Ensure that the given length to `p_read` is of ssize_t and ensure that callers test the return as if it were an `ssize_t`.
Edward Thomson 392702ee 2015-02-09T23:41:13 allocations: test for overflow of requested size Introduce some helper macros to test integer overflow from arithmetic and set error message appropriately.
Leo Yang 3556a82a 2015-02-09T17:35:25 Fix build warning on Android Always do a time_t cast on st_mtime. st_mtime on Android is not the type of time_t but has the same meaning which is the number of seconds past epoch.
Carlos Martín Nieto d24a5312 2015-02-12T02:34:58 Merge pull request #2866 from ethomson/checkout_perf2 Checkout performance
Edward Thomson 9cb5b0f7 2015-02-04T23:45:22 mkdir: respect the root path Don't try to strip trailing paths from the root directory on Windows (trying to create `C:` will fail).
Edward Thomson 500ec543 2015-02-03T01:46:01 checkout: hold seen dir paths in a map
Edward Thomson e74340b0 2015-01-14T18:47:00 checkout: remove files before writing new ones On case insensitive filesystems, we may have files in the working directory that case fold to a name we want to write. Remove those files (by default) so that we will not end up with a filename that has the unexpected case.
Edward Thomson fe598f09 2015-01-13T11:18:02 mkdir: walk up tree to mkdir Walk up the tree to mkdir, which is less immediately efficient, but allows us to look at intermediate directories that may need attention.
Edward Thomson 1d50b364 2015-01-12T16:16:27 checkout: introduce git_checkout_perfdata Checkout can now provide performance data about the number of (some) syscalls performed using an optional callback.
The rugged tests are fragile bbb988a5 2014-09-17T14:52:31 path: Fix `git_path_walk_up` to work with non-rooted paths
Russell Belfer 668ae2dd 2014-08-22T10:05:09 Allow mkdir helper to skip parent errors Our mkdir helper was failing is a parent directory was not accessible even if the child directory could be created. This changes the helper to keep trying child directories even when the parent is unwritable.
Carlos Martín Nieto 94f742ba 2014-05-28T10:18:05 fileops: allow linking files when copying directory structures When passed the LINK_FILES flag, the recursive copy will hardlink files instead of copying them.
Russell Belfer 823c0e9c 2014-04-17T11:53:13 Fix broken logic for attr cache invalidation The checks to see if files were out of date in the attibute cache was wrong because the cache-breaker data wasn't getting stored correctly. Additionally, when the cache-breaker triggered, the old file data was being leaked.
Russell Belfer 7d490872 2014-04-10T22:31:01 Attribute file cache refactor This is a big refactoring of the attribute file cache to be a bit simpler which in turn makes it easier to enforce a lock around any updates to the cache so that it can be used in a threaded env. Tons of changes to the attributes and ignores code.
Edward Thomson 83634d38 2014-02-24T17:43:10 Move system directory cache out of utils
Russell Belfer 9cfce273 2013-12-12T12:11:38 Cleanups, renames, and leak fixes This renames git_vector_free_all to the better git_vector_free_deep and also contains a couple of memory leak fixes based on valgrind checks. The fixes are specifically: failure to free global dir path variables when not compiled with threading on and failure to free filters from the filter registry that had not be initialized fully.
Russell Belfer 25e0b157 2013-12-06T15:07:57 Remove converting user error to GIT_EUSER This changes the behavior of callbacks so that the callback error code is not converted into GIT_EUSER and instead we propagate the return value through to the caller. Instead of using the giterr_capture and giterr_restore functions, we now rely on all functions to pass back the return value from a callback. To avoid having a return value with no error message, the user can call the public giterr_set_str or some such function to set an error message. There is a new helper 'giterr_set_callback' that functions can invoke after making a callback which ensures that some error message was set in case the callback did not set one. In places where the sign of the callback return value is meaningful (e.g. positive to skip, negative to abort), only the negative values are returned back to the caller, obviously, since the other values allow for continuing the loop. The hardest parts of this were in the checkout code where positive return values were overloaded as meaningful values for checkout. I fixed this by adding an output parameter to many of the internal checkout functions and removing the overload. This added some code, but it is probably a better implementation. There is some funkiness in the network code where user provided callbacks could be returning a positive or a negative value and we want to rely on that to cancel the loop. There are still a couple places where an user error might get turned into GIT_EUSER there, I think, though none exercised by the tests.
Russell Belfer 96869a4e 2013-12-03T16:45:39 Improve GIT_EUSER handling This adds giterr_user_cancel to return GIT_EUSER and clear any error message that is sitting around. As a result of using that in places, we need to be more thorough with capturing errors that happen inside a callback when used internally. To help with that, this also adds giterr_capture and giterr_restore so that when we internally use a foreach-type function that clears errors and converts them to GIT_EUSER, it is easier to restore not just the return value, but the actual error message text.
nulltoken 39b1ad7f 2013-11-05T16:14:20 Plug configuration file search paths leaks
Edward Thomson 1d3a8aeb 2013-11-04T18:28:57 move mode_t to filebuf_open instead of _commit
Edward Thomson c2408a69 2013-11-01T17:03:01 preserve windows error numbers as well
Vicent Martí 2c2b0ebb 2013-10-11T09:47:05 Merge pull request #1562 from libgit2/cmn/refs-namespace-lookup Provide the user with a more useful error code when a looking up a reference which name points to a namepace
Carlos Martín Nieto 7c9f5bec 2013-08-17T07:11:31 futils: return GIT_ENOTFOUND when trying to read a directory This lets the reference code return not-found when the user asks to look up a reference when in fact they pass a namespace.
Russell Belfer 14997dc5 2013-10-08T12:45:43 More filemode cleanups for FAT on MacOS This cleans up some additional issues. The main change is that on a filesystem that doesn't support mode bits, libgit2 will now create new blobs with GIT_FILEMODE_BLOB always instead of being at the mercy to the filesystem driver to report executable or not. This means that if "core.filemode" lies and claims that filemode is not supported, then we will ignore the executable bit from the filesystem. Previously we would have allowed it. This adds an option to the new git_repository_reset_filesystem to recurse through submodules if desired. There may be other types of APIs that would like a "recurse submodules" option, but this one is particularly useful. This also has a number of cleanups, etc., for related things including trying to give better error messages when problems come up from the filesystem. For example, the FAT filesystem driver on MacOS appears to return errno EINVAL if you attempt to write a filename with invalid UTF-8 in it. We try to capture that with a better error message now.
Russell Belfer 219d3457 2013-10-01T16:12:15 Initial iconv hookup for precomposed unicode This hooks up git_path_direach and git_path_dirload so that they will take a flag indicating if directory entry names should be tested and converted from decomposed unicode to precomposed form. This code will only come into play on the Apple platform and even then, only when certain types of filesystems are used. This involved adding a flag to these functions which involved changing a lot of places in the code. This was an opportunity to do a bit of code cleanup here and there, for example, getting rid of the git_futils_cleanupdir_r function in favor of a simple flag to git_futils_rmdir_r to not remove the top level entry. That ended up adding depth tracking during rmdir_r which led to a safety check for infinite directory recursion. Yay. This hasn't actually been tested on the Mac filesystems where the issue occurs. I still need to get test environment for that.
Vicent Martí 3d4f1698 2013-09-17T10:21:22 Merge pull request #1858 from linquize/win32-template-dir Configurable template dir for Win32
Russell Belfer 13f36ffb 2013-09-13T16:30:21 Add clar helpers for testing file equality These are a couple of new clar helpers for testing that a file has expected contents that I extracted from the checkout code. Actually wrote this as part of an abandoned earlier attempt at a new filters API, but it will be useful now for some of the tests I'm going to write.
Russell Belfer a3aa5f4d 2013-09-11T12:45:20 Add simple global shutdown hooks Increasingly there are a number of components that want to do some cleanup at global shutdown time (at least if there are not going to be memory leaks). This creates a very simple system of shutdown hooks that will be invoked by git_threads_shutdown. Right now, the maximum number of hooks is hardcoded, but since adding a hook is not a public API, it should be fine and I thought it was better to start off with really simple code.
Linquize a025907e 2013-09-17T23:55:11 Can load default template directory
Linquize 7e8934bb 2013-09-17T23:29:02 Can guess win32 git template dir
Linquize f84bc388 2013-09-17T23:24:44 Refactor git_win32__find_system_dirs() to extract "etc\\" as subpath parameter
nulltoken d0cd6c42 2013-09-08T18:22:28 path: Make direach() return EUSER on callback error
Russell Belfer f240acce 2013-09-05T11:20:12 Add more file mode permissions macros This adds some more macros for some standard operations on file modes, particularly related to permissions, and then updates a number of places around the code base to use the new macros.
Carlos Martín Nieto 3d276874 2013-08-19T10:30:44 index: report when it's locked Report the index being locked with its own error code in order to be able to differentiate, as a locked index is typically the result of a crashed process or concurrent access, both of which often require user intervention to fix.
Vicent Martí 89f6d84c 2013-08-14T06:40:38 Merge pull request #1781 from brodie/brodie/stat-before-open fileops: stat() before open()ing in git_futils_readbuffer_updated()
Brodie Rao 9ccdb211 2013-08-13T10:55:37 fileops: stat() before open()ing in git_futils_readbuffer_updated() This reverts refactoring done in 13224ea4aad9a1b3c9cc4c992ceaea9af623e047 that introduces a performance regression for NFS when reading files that don't exist. open() forces a cache invalidation on NFS, while stat()ing a file just uses the cache and is very quick. To give a specific example, say you have a repo with a thousand packed refs. Before this change, looking up every single one ould incur a thousand slow open() calls. With this change, it's a thousand fast stat() calls.
Russell Belfer d4cff0cb 2013-08-13T09:40:32 Rename git__win32_path fns to git_win32_path
Vicent Marti abf37327 2013-08-13T09:15:39 windows: Path conversion with better semantics
Ben Straub aaefbdee 2013-08-08T08:48:57 Discriminate path-specific and general UTF-X conversions
Ben Straub 2c0128ee 2013-08-07T19:29:33 Rename git_win_str_utf* to git_win32_path_utf*
Ben Straub 9c38f7a6 2013-08-07T13:22:41 Add typedefs for win32 utf-8 and utf-16 buffers ...and normalize the signatures of the two conversion functions.
Ben Straub d8563619 2013-08-05T11:41:39 Split UTF-16 and UTF-8 buffer sizes for win32 Also fixed up call-sites to use the correct buffer sizes, especially when converting to utf-8.
Sven Strickroth f5254d78 2013-07-27T20:15:06 Fix possible double close Signed-off-by: Sven Strickroth <email@cs-ware.de>
Russell Belfer 989710d9 2013-07-22T11:22:55 Fix warning message about mismatched types
Edward Thomson 0a1c8f55 2013-07-11T17:09:15 preload configuration paths
Sven Strickroth 022a45e0 2013-06-25T16:43:15 Revert "Work around reparse point stat issues" This reverts commit 32c12ea6a9cafd76a746af2e2be9366c95752f5b.
Vicent Martí 09ee60c6 2013-06-24T11:21:09 Merge pull request #1670 from arrbee/open-cloexec Add O_CLOEXEC to open calls
Russell Belfer 32c12ea6 2013-06-24T09:19:24 Work around reparse point stat issues In theory, p_stat should never return an S_ISLNK result, but due to the current implementation on Windows with mount points it is possible that it will. For now, work around that by allowing a link in the path to a directory being created. If it is really a problem, then the issue will be caught on the next iteration of the loop, but typically this will be the right thing to do.
Russell Belfer 3d3ea4dc 2013-06-22T20:58:32 Add O_CLOEXEC to open calls
yorah 2da72fb2 2013-06-14T12:10:13 fileops: fix invalid read
Russell Belfer f7e56150 2013-06-05T15:41:42 Make mkdir early exit cases clearer There are two places where git_futils_mkdir should exit early or at least do less. The first is when using GIT_MKDIR_SKIP_LAST and having that flag leave no directory left to create; it was being handled previously, but the behavior was subtle. Now I put in a clear explicit check that exits early in that case. The second is when there is no directory to create, but there is a valid path that should be verified. I shifted the logic a bit so we'll be better about not entering the loop than that happens.
Russell Belfer 999d4405 2013-06-05T12:02:28 Simplify git_futils_mkdir This routine was (is) pretty complicated, but given the recent changes, it seemed like it could be simplified a bit.
Vicent Marti 2e1fa15f 2013-06-05T19:00:16 I'm a dick
Russell Belfer b832ecf7 2013-06-05T09:46:51 Ensure git_futils_mkdir won't mkdir root This makes sure that git_futils_mkdir always skips over the root directory at a minimum, even on platforms where the root is not simply '/'. Also, this removes the GIT_WIN32 ifdef in favor of making EACCES as a potentially recoverable error on all platforms.
Jameson Miller daf98cb2 2013-06-04T14:49:39 Allow creation of directories under the volume root in Win32 We ran into an issue where cloning a repository to a folder directly underneath the root of a volume (e.g. 'd:\libgit2') would fail with an access denied error. This was traced down to a call to make a directory that is the root (e.g. 'd:') could return an error indicated access denied instead of an error indicating the path already exists. This change now handles the access denied error on Win32 and checks for the existence of the folder.
Russell Belfer 4742148d 2013-05-20T13:41:39 Add more diff rename detection tests This adds a bunch more rename detection tests including checks vs the working directory, the new exact match options, some more whitespace variants, etc. This also adds a git_futils_writebuffer helper function and uses it in checkout. This is mainly added because I wanted an easy way to write out a git_buf to disk inside my test code.
Vicent Martí 03c28d92 2013-05-06T06:45:53 Merge pull request #1526 from arrbee/cleanup-error-return-without-msg Make sure error messages are set for most error returns
Edward Thomson e09d18ee 2013-05-03T18:39:44 allow checkout to proceed when a dir to be removed is in use (win32)
Russell Belfer e830c020 2013-05-01T13:50:39 Report stat error when checking if file modified
Vicent Martí 5b27bf7e 2013-03-18T16:17:14 Merge pull request #1417 from arrbee/opts-for-paths Implement opts interface for global/system file search paths
Russell Belfer 32460251 2013-03-18T15:54:35 Fixes and cleanups Get rid of some dead code, tighten things up a bit, and fix a bug with core::env test.
Russell Belfer 41954a49 2013-03-18T14:19:35 Switch search paths to classic delimited strings This switches the APIs for setting and getting the global/system search paths from using git_strarray to using a simple string with GIT_PATH_LIST_SEPARATOR delimited paths, just as the environment PATH variable would contain. This makes it simpler to get and set the value. I also added code to expand "$PATH" when setting a new value to embed the old value of the path. This means that I no longer require separate actions to PREPEND to the value.
Arkadiy Shapkin 10c06114 2013-03-17T04:46:46 Several warnings detected by static code analyzer fixed Implicit type conversion argument of function to size_t type Suspicious sequence of types castings: size_t -> int -> size_t Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)' Unsigned type is never < 0