src/diff.h


Log

Author Commit Date CI Message
Patrick Steinhardt d8896bda 2018-01-03T16:07:36 diff_generate: avoid excessive stats of .gitattribute files When generating a diff between two trees, for each file that is to be diffed we have to determine whether it shall be treated as text or as binary files. While git has heuristics to determine which kind of diff to generate, users can also that default behaviour by setting or unsetting the 'diff' attribute for specific files. Because of that, we have to query gitattributes in order to determine how to diff the current files. Instead of hitting the '.gitattributes' file every time we need to query an attribute, which can get expensive especially on networked file systems, we try to cache them instead. This works perfectly fine for every '.gitattributes' file that is found, but we hit cache invalidation problems when we determine that an attribuse file is _not_ existing. We do create an entry in the cache for missing '.gitattributes' files, but as soon as we hit that file again we invalidate it and stat it again to see if it has now appeared. In the case of diffing large trees with each other, this behaviour is very suboptimal. For each pair of files that is to be diffed, we will repeatedly query every directory component leading towards their respective location for an attributes file. This leads to thousands or even hundreds of thousands of wasted syscalls. The attributes cache already has a mechanism to help in that scenario in form of the `git_attr_session`. As long as the same attributes session is still active, we will not try to re-query the gitmodules files at all but simply retain our currently cached results. To fix our problem, we can create a session at the top-most level, which is the initialization of the `git_diff` structure, and use it in order to look up the correct diff driver. As the `git_diff` structure is used to generate patches for multiple files at once, this neatly solves our problem by retaining the session until patches for all files have been generated. The fix has been tested with linux.git by calling `git_diff_tree_to_tree` and `git_diff_to_buf` with v4.10^{tree} and v4.14^{tree}. | time | .gitattributes stats without fix | 33.201s | 844614 with fix | 30.327s | 4441 While execution only improved by roughly 10%, the stat(3) syscalls for .gitattributes files decreased by 99.5%. The benchmarks were quite simple with best-of-three timings on Linux ext4 systems. One can assume that for network based file systems the performance gain will be a lot larger due to a much higher latency.
Patrick Steinhardt 0c7f49dd 2017-06-30T13:39:01 Make sure to always include "common.h" first Next to including several files, our "common.h" header also declares various macros which are then used throughout the project. As such, we have to make sure to always include this file first in all implementation files. Otherwise, we might encounter problems or even silent behavioural differences due to macros or defines not being defined as they should be. So in fact, our header and implementation files should make sure to always include "common.h" first. This commit does so by establishing a common include pattern. Header files inside of "src" will now always include "common.h" as its first other file, separated by a newline from all the other includes to make it stand out as special. There are two cases for the implementation files. If they do have a matching header file, they will always include this one first, leading to "common.h" being transitively included as first file. If they do not have a matching header file, they instead include "common.h" as first file themselves. This fixes the outlined problems and will become our standard practice for header and source files inside of the "src/" from now on.
Edward Thomson b859faa6 2016-08-23T23:38:39 Teach `git_patch_from_diff` about parsed diffs Ensure that `git_patch_from_diff` can return the patch for parsed diffs, not just generate a patch for a generated diff.
Edward Thomson 7166bb16 2016-04-25T00:35:48 introduce `git_diff_from_buffer` to parse diffs Parse diff files into a `git_diff` structure.
Edward Thomson 9be638ec 2016-04-19T15:12:18 git_diff_generated: abstract generated diffs
Edward Thomson 25e84f95 2015-11-23T15:49:54 checkout: only consider nsecs when built that way When examining the working directory and determining whether it's up-to-date, only consider the nanoseconds in the index entry when built with `GIT_USE_NSEC`. This prevents us from believing that the working directory is always dirty when the index was originally written with a git client that uinderstands nsecs (like git 2.x).
Pierre-Olivier Latour c2e1b058 2015-06-05T18:26:49 Only write index if updated when passing GIT_DIFF_UPDATE_INDEX When diffing the index with the workdir and GIT_DIFF_UPDATE_INDEX has been passed, the previous implementation was always writing the index to disk even if it wasn't modified.
Edward Thomson 90177111 2015-06-23T16:27:33 stash: save the workdir file when deleted in index When stashing the workdir tree, examine the index as well. Using a mechanism similar to `git_diff_tree_to_workdir_with_index` allows us to determine that a file was added in the index and subsequently modified in the working directory. Without examining the index, we would erroneously believe that this file was untracked and fail to include it in the working directory tree. Use a slightly modified `git_diff_tree_to_workdir_with_index` in order to avoid some of the behavior custom to `git diff`. In particular, be sure to include the working directory side of a file when it was deleted in the index.
Edward Thomson 5ef43d41 2015-06-23T10:29:59 git_diff__merge: allow pluggable diff merges
Edward Thomson 96dd171e 2015-06-19T08:32:26 diff: preserve original mode in the index When updating the index during a diff, preserve the original mode, which prevents us from dropping the mode to what we have interpreted as on our system (eg, what the working directory claims it to be, which may be a lie on some systems.)
Russell Belfer 9c8ed499 2014-04-29T15:05:58 Remove trace / add git_diff_perfdata struct + api
Russell Belfer cd424ad5 2014-04-28T16:39:53 Add GIT_STATUS_OPT_UPDATE_INDEX and use trace API This adds an option to refresh the stat cache while generating status. It also rips out the GIT_PERF stuff I had an makes use of the trace API to keep statistics about what happens during diff.
Russell Belfer 94fb4aad 2014-04-28T14:48:41 Add diff option to update index stat cache When diff is scanning the working directory, if it finds a file where it is not sure if the index entry matches the working dir, it will recalculate the OID (which is pretty expensive). This adds a new flag to diff so that if the OID calculation finds that the file actually has not changed (i.e. just the modified time was altered or such), then it will refresh the stat cache in the index so that future calls to diff will not have to check the oid again.
Russell Belfer 0fc8e1f6 2014-04-28T14:34:55 Lay groundwork for updating stat cache in diff This reorganized the diff OID calculation to make it easier to correctly update the stat cache during a diff once the flags to do so are enabled. This includes marking the path of a git_index_entry as const so we can make a "fake" git_index_entry with a "const char *" path and not get warnings. I was a little surprised at how unobtrusive this change was, but I think it's probably a good thing.
Russell Belfer 240f4af3 2014-04-28T14:04:29 Add build option for diff internal statistics
Jacques Germishuys d8cc1fb6 2014-04-11T19:15:15 Introduce git_diff_format_email and git_diff_commit_as_email
Carlos Martín Nieto 9950bb4e 2014-01-24T20:23:17 diff: rename the file's 'oid' to 'id' In the same vein as the previous commits in this series.
Russell Belfer 3e57069e 2013-11-01T13:49:43 Fix --assume-unchanged support This was never really working right because we were checking the wrong flag and not checking it in all the places that we need to be checking it. I finally got around to writing a test and adding actual support for it.
Russell Belfer 3ff1d123 2013-10-11T14:51:54 Rename diff objects and split patch.h This makes no functional change to diff but renames a couple of the objects and splits the new git_patch (formerly git_diff_patch) into a new header file.
Ben Straub 5e96f316 2013-08-08T08:54:38 Merge pull request #1738 from libgit2/diff-patch-content-size Add API for getting at git_diff_patch->content_size
Russell Belfer effdbeb3 2013-07-24T17:48:37 Make rename detection file size fix better The previous fix for checking file sizes with rename detection always loads the blob. In this version, if the odb backend can get the object header without loading the whole thing into memory, then we'll just use that, so that we can eliminate possible rename sources & targets without loading them.
Russell Belfer 197b8966 2013-07-23T14:34:31 Add hunk/file headers to git_diff_patch_size This allows git_diff_patch_size to account for hunk headers and file headers in the returned size. This required some refactoring of the code that is used to print file headers so that it could be invoked by the git_diff_patch_size API. Also this increases the test coverage and fixes an off-by-one bug in the size calculation when newline changes happen at the end of the file.
Russell Belfer 2b672d5b 2013-07-08T22:46:36 Add git_pathspec_match_diff API This adds an additional pathspec API that will match a pathspec against a diff object. This is convenient if you want to handle renames (so you need the whole diff and can't use the pathspec constraint built into the diff API) but still want to tell if the diff had any files that matched the pathspec. When the pathspec is matched against a diff, instead of keeping a list of filenames that matched, instead the API keeps the list of git_diff_deltas that matched and they can be retrieved via a new API git_pathspec_match_list_diff_entry. There are a couple of other minor API extensions here that were mostly for the sake of convenience and to reduce dependencies on knowing the internal data structure between files inside the library.
Russell Belfer a1683f28 2013-06-14T16:18:04 More tests and bug fixes for status with rename This changes the behavior of the status RENAMED flags so that they will be combined with the MODIFIED flags if appropriate. If a file is modified in the index and also renamed, then the status code will have both the GIT_STATUS_INDEX_MODIFIED and INDEX_RENAMED bits set. If it is renamed but the OID has not changed, then just the GIT_STATUS_INDEX_RENAMED bit will be set. Similarly, the flags GIT_STATUS_WT_MODIFIED and GIT_STATUS_WT_RENAMED can both be set independently of one another. This fixes a serious bug where the check for unmodified files that was done at data load time could end up erasing the RENAMED state of a file that was renamed with no changes. Lastly, this contains a bunch of new tests for status with renames, including tests where the only rename changes are case changes. The expected results of these tests have to vary by whether the platform uses a case sensitive filesystem or not, so the expected data covers those platform differences separately.
Russell Belfer 351888cf 2013-06-13T15:37:06 Improve case handling in git_diff__paired_foreach This commit reinstates some changes to git_diff__paired_foreach that were discarded during the rebase (because the diff_output.c file had gone away), and also adjusts the case insensitively logic slightly to hopefully deal with either mismatched icase diffs and other case insensitivity scenarios.
Russell Belfer 114f5a6c 2013-06-10T10:10:39 Reorganize diff and add basic diff driver This is a significant reorganization of the diff code to break it into a set of more clearly distinct files and to document the new organization. Hopefully this will make the diff code easier to understand and to extend. This adds a new `git_diff_driver` object that looks of diff driver information from the attributes and the config so that things like function content in diff headers can be provided. The full driver spec is not implemented in the commit - this is focused on the reorganization of the code and putting the driver hooks in place. This also removes a few #includes from src/repository.h that were overbroad, but as a result required extra #includes in a variety of places since including src/repository.h no longer results in pulling in the whole world.
Russell Belfer 67db583d 2013-05-23T15:06:07 More diff rename tests; better split swap handling This adds a couple more tests of different rename scenarios. Also, this fixes a problem with the case where you have two "split" deltas and the left half of one matches the right half of the other. That case was already being handled, but in the wrong order in a way that could result in bad output. Also, if the swap also happened to put the other two halves into the correct place (i.e. two files exchanged places with each other), then the second delta was left with the SPLIT flag set when it really should be cleared.
Russell Belfer a21cbb12 2013-05-22T10:37:12 Significant rename detection rewrite This flips rename detection around so instead of creating a forward mapping from deltas to possible rename targets, instead it creates a reverse mapping, looking at possible targets and trying to find a source that they could have been renamed or copied from. This is important because each output can only have a single source, but a given source could map to multiple outputs (in the form of COPIED records). Additionally, this makes a couple of tweaks to the public rename detection APIs, mostly renaming a couple of options that control the behavior to make more sense and to be more like core Git. I walked through the tests looking at the exact results and updated the expectations based on what I saw. The new code is different from the old because it cannot give some nonsense results (like A was renamed to both B and C) which were part of the outputs previously.
Russell Belfer e35e2684 2013-05-07T04:32:17 Add GIT_DIFF_LINE_CONTEXT_EOFNL This adds a new line origin constant for the special line that is used when both files end without a newline. In the course of writing the tests for this, I was having problems with modifying a file but not having diff notice because it was the same size and modified less than one second from the start of the test, so I decided to start working on nanosecond timestamp support. This commit doesn't contain the nanosecond support, but it contains the reorganization of maybe_modified and the hooks so that if the nanosecond data were being read by stat() (or rather being copied by git_index_entry__init_from_stat), then the nsec would be taken into account. This new stuff could probably use some more tests, although there is some amount of it here.
Edward Thomson 0462fba5 2013-04-30T14:56:41 renames!
Russell Belfer 71a3d27e 2013-02-08T10:06:47 Replace diff delta binary with flags Previously the git_diff_delta recorded if the delta was binary. This replaces that (with no net change in structure size) with a full set of flags. The flag values that were already in use for individual git_diff_file objects are reused for the delta flags, too (along with renaming those flags to make it clear that they are used more generally). This (a) makes things somewhat more consistent (because I was using a -1 value in the "boolean" binary field to indicate unset, whereas now I can just use the flags that are easier to understand), and (b) will make it easier for me to add some additional flags to the delta object in the future, such as marking the results of a copy/rename detection or other deltas that might want a special indicator. While making this change, I officially moved some of the flags that were internal only into the private diff header. This also allowed me to remove a gross hack in rename/copy detect code where I was overwriting the status field with an internal value.
Edward Thomson 359fc2d2 2013-01-08T17:07:25 update copyrights
Russell Belfer 9950d27a 2012-12-06T13:26:58 Clean up iterator APIs This removes the need to explicitly pass the repo into iterators where the repo is implied by the other parameters. This moves the repo to be owned by the parent struct. Also, this has some iterator related updates to the internal diff API to lay the groundwork for checkout improvements.
Ben Straub c7231c45 2012-11-30T16:31:42 Deploy GITERR_CHECK_VERSION
Ben Straub 2f8d30be 2012-11-29T15:05:04 Deploy GIT_DIFF_OPTIONS_INIT
Russell Belfer bbe6dbec 2012-11-14T23:29:48 Add explicit git_index ptr to diff and checkout A number of diff APIs and the `git_checkout_index` API take a `git_repository` object an operate on the index. This updates them to take a `git_index` pointer explicitly and only fall back on the `git_repository` index if the index input is NULL. This makes it easier to operate on a temporary index.
Russell Belfer bad68c0a 2012-11-13T14:02:59 Add iterator for git_index object The index iterator could previously only be created from a repo object, but this allows creating an iterator from a `git_index` object instead (while keeping, though renaming, the old function).
Russell Belfer 55cbd05b 2012-11-08T16:56:34 Some diff refactorings to help code reuse There are some diff functions that are useful in a rewritten checkout and this lays some groundwork for that. This contains three main things: 1. Share the function diff uses to calculate the OID for a file in the working directory (now named `git_diff__oid_for_file` 2. Add a `git_diff__paired_foreach` function to iterator over two diff lists concurrently. Convert status to use it. 3. Move all the string/prefix/index entry comparisons into function pointers inside the `git_diff_list` object so they can be switched between case sensitive and insensitive versions. This makes them easier to reuse in various functions without replicating logic. As part of this, move a couple of index functions out of diff.c and into index.c.
Russell Belfer 0f3def71 2012-11-09T11:19:46 Fix various cross-platform build issues This fixes a number of warnings and problems with cross-platform builds. Among other things, it's not safe to name a member of a structure "strcmp" because that may be #defined.
Russell Belfer db106d01 2012-10-30T09:40:50 Move rename detection into new file This improves the naming for the rename related functionality moving it to be called `git_diff_find_similar()` and renaming all the associated constants, etc. to make more sense. I also moved the new code (plus the existing `git_diff_merge`) into a new file `diff_tform.c` where I can put new functions related to manipulating git diff lists. This also updates the implementation significantly from the last revision fixing some ordering issues (where break-rewrite needs to be handled prior to copy and rename detection) and improving config option handling.
Russell Belfer b4f5bb07 2012-10-23T16:40:51 Initial implementation of diff rename detection This implements the basis for diff rename and copy detection, although it is based on simple SHA comparison right now instead of using a matching algortihm. Just as `git_diff_merge` can be used as a post-pass on diffs to emulate certain command line behaviors, there is a new API `git_diff_detect` which will update a diff list in-place, adjusting some deltas to RENAMED or COPIED state (and also, eventually, splitting MODIFIED deltas where the change is too large into DELETED/ADDED pairs). This also adds a new test repo that will hold rename/copy/split scenarios. Right now, it just has exact-match rename and copy, but the tests are written to use tree diffs, so we should be able to add new test scenarios easily without breaking tests.
Russell Belfer 52a61bb8 2012-10-17T14:10:23 Fix minor bugs Fixed no-submodule speedup of new checkout code. Fixed missing final update to progress (which may go away, I realize). Fixed unused structure in header and incorrect comment.
Russell Belfer bae957b9 2012-09-25T16:31:46 Add const to all shared pointers in diff API There are a lot of places where the diff API gives the user access to internal data structures and many of these were being exposed through non-const pointers. This replaces them all with const pointers for any object that the user can access but is still owned internally to the git_diff_list or git_diff_patch objects. This will probably break some bindings... Sorry!
Russell Belfer 5f69a31f 2012-09-24T20:52:34 Initial implementation of new diff patch API Replacing the `git_iterator` object, this creates a simple API for accessing the "patch" for any file pair in a diff list and then gives indexed access to the hunks in the patch and the lines in the hunk. This is the initial implementation of this revised API - it is still broken, but at least builds cleanly.
Russell Belfer 64286308 2012-09-25T10:48:50 Fix bugs in new diff patch code This fixes all the bugs in the new diff patch code. The only really interesting one is that when we merge two diffs, we now have to actually exclude diff delta records that are not supposed to be tracked, as opposed to before where they could be included because they would be skipped silently by `git_diff_foreach()`. Other than that, there are just minor errors.
Russell Belfer b36effa2 2012-09-10T09:59:14 Replace git_diff_iterator_num_files with progress The `git_diff_iterator_num_files` API was problematic, since we don't actually know the exact number of files to be iterated over until we load those files into memory. This replaces it with a new `git_diff_iterator_progress` API that goes from 0 to 1, and moves and renamed the old API for the internal places that can tolerate a max value instead of an exact value.
Russell Belfer 60b9d3fc 2012-09-05T15:00:40 Implement filters for status/diff blobs This adds support to diff and status for running filters (a la crlf) on blobs in the workdir before computing SHAs and before generating text diffs. This ended up being a bit more code change than I had thought since I had to reorganize some of the diff logic to minimize peak memory use when filtering blobs in a diff. This also adds a cap on the maximum size of data that will be loaded to diff. I set it at 512Mb which should match core git. Right now it is a #define in src/diff.h but it could be moved into the public API if desired.
Russell Belfer f335ecd6 2012-08-30T14:24:16 Diff iterators This refactors the diff output code so that an iterator object can be used to traverse and generate the diffs, instead of just the `foreach()` style with callbacks. The code has been rearranged so that the two styles can still share most functions. This also replaces `GIT_REVWALKOVER` with `GIT_ITEROVER` and uses that as a common error code for marking the end of iteration when using a iterator style of object.
Russell Belfer 0abd7244 2012-06-04T16:17:41 Fix filemode comparison in diffs File modes were both not being ignored properly on platforms where they should be ignored, nor be diffed consistently on platforms where they are supported. This change adds a number of diff and status filemode change tests. This also makes sure that filemode-only changes are included in the diff output when they occur and that filemode changes are ignored successfully when core.filemode is false. There is no code that automatically toggles core.filemode based on the capabilities of the current platform, so the user still needs to be careful in their .git/config file.
Russell Belfer 16b83019 2012-03-04T23:28:36 Fix usage of "new" for fieldname in public header This should restore the ability to include libgit2 headers in C++ projects. Cherry picked 2de60205dfea2c4a422b2108a5e8605f97c2e895 from development into new-error-handling.
nulltoken eb3d71a5 2012-04-25T22:23:35 diff: fix generation of the header of a removal patch
Russell Belfer 19fa2bc1 2012-04-17T15:12:50 Convert attrs and diffs to use string pools This converts the git attr related code (including ignores) and the git diff related code (and implicitly the status code) to use `git_pools` for storing strings. This reduces the number of small blocks allocated dramatically.
Russell Belfer 14a513e0 2012-04-13T15:00:29 Add support for pathspec to diff and status This adds preliminary support for pathspecs to diff and status. The implementation is not very optimized (it still looks at every single file and evaluated the the pathspec match against them), but it works.
Russell Belfer 95dfb031 2012-03-30T14:40:50 Improve config handling for diff,submodules,attrs This adds support for a bunch of core.* settings that affect diff and status, plus fixes up some incorrect implementations of those settings from before. Also, this cleans up the handling of config settings in the new submodules code and in the old attrs/ignore code.
Russell Belfer 1db12b00 2012-03-25T23:04:26 Eliminate hairy COITERATE macro I decided that the COITERATE macro was, in the end causing more confusion that it would save and decided just to write out the loops that I needed for parallel diff list iteration. It is not that much code and this just feels less obfuscated.
Russell Belfer a48ea31d 2012-03-21T12:33:09 Reimplment git_status_foreach using git diff This is an initial reimplementation of status using diff a la the way that core git does it.
Russell Belfer 74fa4bfa 2012-02-28T16:14:47 Update diff to use iterators This is a major reorganization of the diff code. This changes the diff functions to use the iterators for traversing the content. This allowed a lot of code to be simplified. Also, this moved the functions relating to outputting a diff into a new file (diff_output.c). This includes a number of other changes - adding utility functions, extending iterators, etc. plus more tests for the diff code. This also takes the example diff.c program much further in terms of emulating git-diff command line options.
Russell Belfer e47329b6 2012-02-13T17:29:30 First pass of diff index to workdir implementation This is an initial version of git_diff_workdir_to_index. It also includes renaming some structures and some refactoring of the existing code so that it could be shared better with the new function. This is not complete since it needs a rebase to get some new odb functions from the upstream branch.
Russell Belfer a2e895be 2012-02-07T12:14:28 Continue implementation of git-diff * Implemented git_diff_index_to_tree * Reworked git_diff_options structure to handle more options * Made most of the options in git_diff_options actually work * Reorganized code a bit to remove some redundancy * Added option parsing to examples/diff.c to test most options
Russell Belfer 65b09b1d 2012-02-02T18:03:43 Implement diff lists and formatters This reworks the diff API to separate the steps of producing a diff descriptions from formatting the diff. This will allow us to share diff output code with the various diff creation scenarios and will allow us to implement rename detection as an optional pass that can be run on a diff list.