src/diff_file.c


Log

Author Commit Date CI Message
Patrick Steinhardt d8896bda 2018-01-03T16:07:36 diff_generate: avoid excessive stats of .gitattribute files When generating a diff between two trees, for each file that is to be diffed we have to determine whether it shall be treated as text or as binary files. While git has heuristics to determine which kind of diff to generate, users can also that default behaviour by setting or unsetting the 'diff' attribute for specific files. Because of that, we have to query gitattributes in order to determine how to diff the current files. Instead of hitting the '.gitattributes' file every time we need to query an attribute, which can get expensive especially on networked file systems, we try to cache them instead. This works perfectly fine for every '.gitattributes' file that is found, but we hit cache invalidation problems when we determine that an attribuse file is _not_ existing. We do create an entry in the cache for missing '.gitattributes' files, but as soon as we hit that file again we invalidate it and stat it again to see if it has now appeared. In the case of diffing large trees with each other, this behaviour is very suboptimal. For each pair of files that is to be diffed, we will repeatedly query every directory component leading towards their respective location for an attributes file. This leads to thousands or even hundreds of thousands of wasted syscalls. The attributes cache already has a mechanism to help in that scenario in form of the `git_attr_session`. As long as the same attributes session is still active, we will not try to re-query the gitmodules files at all but simply retain our currently cached results. To fix our problem, we can create a session at the top-most level, which is the initialization of the `git_diff` structure, and use it in order to look up the correct diff driver. As the `git_diff` structure is used to generate patches for multiple files at once, this neatly solves our problem by retaining the session until patches for all files have been generated. The fix has been tested with linux.git by calling `git_diff_tree_to_tree` and `git_diff_to_buf` with v4.10^{tree} and v4.14^{tree}. | time | .gitattributes stats without fix | 33.201s | 844614 with fix | 30.327s | 4441 While execution only improved by roughly 10%, the stat(3) syscalls for .gitattributes files decreased by 99.5%. The benchmarks were quite simple with best-of-three timings on Linux ext4 systems. One can assume that for network based file systems the performance gain will be a lot larger due to a much higher latency.
Patrick Steinhardt 2388a9e2 2017-12-15T10:47:01 diff_file: properly refcount blobs when initializing file contents When initializing a `git_diff_file_content` from a source whose data is derived from a blob, we simply assign the blob's pointer to the resulting struct without incrementing its refcount. Thus, the structure can only be used as long as the blob is kept alive by the caller. Fix the issue by using `git_blob_dup` instead of a direct assignment. This function will increment the refcount of the blob without allocating new memory, so it does exactly what we want. As `git_diff_file_content__unload` already frees the blob when `GIT_DIFF_FLAG__FREE_BLOB` is set, we don't need to add new code handling the free but only have to set that flag correctly.
Patrick Steinhardt 0c7f49dd 2017-06-30T13:39:01 Make sure to always include "common.h" first Next to including several files, our "common.h" header also declares various macros which are then used throughout the project. As such, we have to make sure to always include this file first in all implementation files. Otherwise, we might encounter problems or even silent behavioural differences due to macros or defines not being defined as they should be. So in fact, our header and implementation files should make sure to always include "common.h" first. This commit does so by establishing a common include pattern. Header files inside of "src" will now always include "common.h" as its first other file, separated by a newline from all the other includes to make it stand out as special. There are two cases for the implementation files. If they do have a matching header file, they will always include this one first, leading to "common.h" being transitively included as first file. If they do not have a matching header file, they instead include "common.h" as first file themselves. This fixes the outlined problems and will become our standard practice for header and source files inside of the "src/" from now on.
Edward Thomson 909d5494 2016-12-29T12:25:15 giterr_set: consistent error messages Error messages should be sentence fragments, and therefore: 1. Should not begin with a capital letter, 2. Should not conclude with punctuation, and 3. Should not end a sentence and begin a new one
Edward Thomson 9be638ec 2016-04-19T15:12:18 git_diff_generated: abstract generated diffs
Edward Thomson d68cb736 2015-09-22T18:25:03 diff: include oid length in deltas Now that `git_diff_delta` data can be produced by reading patch file data, which may have an abbreviated oid, allow consumers to know that the id is abbreviated.
Edward Thomson 6b0fc6ab 2015-11-03T09:43:18 diff: on win32, treat fake "symlinks" specially On platforms that lack `core.symlinks`, we should not go looking for symbolic links and `p_readlink` their target. Instead, we should examine the file's contents.
Carlos Martín Nieto c2418f46 2015-06-25T12:48:44 Rename FALLBACK to UNSPECIFIED Fallback describes the mechanism, while unspecified explains what the user is thinking.
Carlos Martín Nieto c6f489c9 2015-05-04T17:29:12 submodule: add an ignore option to status This lets us specify in the status call which ignore rules we want to use (optionally falling back to whatever the submodule has in its configuration). This removes one of the reasons for having `_set_ignore()` set the value in-memory. We re-use the `IGNORE_RESET` value for this as it is no longer relevant but has a similar purpose to `IGNORE_FALLBACK`. Similarly, we remove `IGNORE_DEFAULT` which does not have use outside of initializers and move that to fall back to the configuration as well.
Carlos Martín Nieto 64bbd47a 2015-05-04T17:09:21 submodule: don't let status change an existing instance As submodules are becomes more like values, we should not let a status check to update its properties. Instead of taking a submodule, have status take a repo and submodule name.
Edward Thomson 8147b1af 2015-05-25T20:03:59 diff: introduce binary diff callbacks Introduce a new binary diff callback to provide the actual binary delta contents to callers. Create this data from the diff contents (instead of directly from the ODB) to support binary diffs including the workdir, not just things coming out of the ODB.
Edward Thomson 795eaccd 2015-02-19T11:09:54 git_filter_opt_t -> git_filter_flag_t For consistency with the rest of the library, where an opt is an options *structure*.
Alan Rogers 61bef72d 2014-05-20T23:57:40 Start adding GIT_DELTA_UNREADABLE and GIT_STATUS_WT_UNREADABLE.
Russell Belfer 5269008c 2014-05-06T16:01:49 Add filter options and ALLOW_UNSAFE Diff and status do not want core.safecrlf to actually raise an error regardless of the setting, so this extends the filter API with an additional options flags parameter and adds a flag so that filters can be applied with GIT_FILTER_OPT_ALLOW_UNSAFE, indicating that unsafe filter application should be downgraded from a failure to a warning.
Russell Belfer 591e8295 2014-03-25T16:52:01 Fix submodule leaks and invalid references This cleans up some places I missed that could hold onto submodule references and cleans up the way in which the repository cache is both reloaded and released so that existing submodule references aren't destroyed inappropriately.
Russell Belfer a15c7802 2014-03-25T09:14:48 Make submodules externally refcounted `git_submodule` objects were already refcounted internally in case the submodule name was different from the path at which it was stored. This makes that refcounting externally used as well, so `git_submodule_lookup` and `git_submodule_add_setup` return an object that requires a `git_submodule_free` when done.
Russell Belfer 6789b7a7 2014-02-27T14:13:22 Add buffer to buffer diff and patch APIs This adds `git_diff_buffers` and `git_patch_from_buffers`. This also includes a bunch of internal refactoring to increase the shared code between these functions and the blob-to-blob and blob-to-buffer APIs, as well as some higher level assert helpers in the tests to also remove redundancy.
Carlos Martín Nieto 9950bb4e 2014-01-24T20:23:17 diff: rename the file's 'oid' to 'id' In the same vein as the previous commits in this series.
Russell Belfer 10672e3e 2013-10-15T15:10:07 Diff API cleanup This lays groundwork for separating formatting options from diff creation options. This groups the formatting flags separately from the diff list creation flags and reorders the options. This also tweaks some APIs to further separate code that uses patches from code that just looks at git_diffs.
Russell Belfer 3ff1d123 2013-10-11T14:51:54 Rename diff objects and split patch.h This makes no functional change to diff but renames a couple of the objects and splits the new git_patch (formerly git_diff_patch) into a new header file.
Russell Belfer 4b11f25a 2013-09-11T16:38:33 Add ident filter This adds the ident filter (that knows how to replace $Id$) and tweaks the filter APIs and code so that git_filter_source objects actually have the updated OID of the object being filtered when it is a known value.
Russell Belfer a9f51e43 2013-09-11T22:00:36 Merge git_buf and git_buffer This makes the git_buf struct that was used internally into an externally available structure and eliminates the git_buffer. As part of that, some of the special cases that arose with the externally used git_buffer were blended into the git_buf, such as being careful about git_buf objects that may have a NULL ptr and allowing for bufs with a valid ptr and size but zero asize as a way of referring to externally owned data.
Russell Belfer 2a7d224f 2013-09-10T16:33:32 Extend public filter api with filter lists This moves the git_filter_list into the public API so that users can create, apply, and dispose of filter lists. This allows more granular application of filters to user data outside of libgit2 internals. This also converts all the internal usage of filters to the public APIs along with a few small tweaks to make it easier to use the public git_buffer stuff alongside the internal git_buf.
Russell Belfer 85d54812 2013-08-28T16:44:04 Create public filter object and use it This creates include/sys/filter.h with a basic definition of a git_filter and then converts the internal code to use it. There are related internal objects (git_filter_list) that we will want to publish at some point, but this is a first step.
Russell Belfer effdbeb3 2013-07-24T17:48:37 Make rename detection file size fix better The previous fix for checking file sizes with rename detection always loads the blob. In this version, if the odb backend can get the object header without loading the whole thing into memory, then we'll just use that, so that we can eliminate possible rename sources & targets without loading them.
Russell Belfer 39a1a662 2013-07-24T13:09:07 Don't unload diff data unless loaded
Russell Belfer 74ded024 2013-06-17T17:03:34 Add "as_path" parameters to blob and buffer diffs This adds parameters to the four functions that allow for blob-to- blob and blob-to-buffer differencing (either via callbacks or by making a git_diff_patch object). These parameters let you say that filename we should pretend the blob has while doing the diff. If you pass NULL, there should be no change from the existing behavior, which is to skip using attributes for file type checks and just look at content. With the parameters, you can plug into the new diff driver functionality and get binary or non-binary behavior, plus function context regular expressions, etc. This commit also fixes things so that the git_diff_delta that is generated by these functions will actually be populated with the data that we know about the blobs (or buffers) so you can use it appropriately. It also fixes a bug in generating patches from the git_diff_patch objects created via these functions. Lastly, there is one other behavior change that may matter. If there is no difference between the two blobs, these functions no longer generate any diff callbacks / patches unless you have passed in GIT_DIFF_INCLUDE_UNMODIFIED. This is pretty natural, but could potentially change the behavior of existing usage.
Russell Belfer 360f42f4 2013-06-12T14:18:09 Fix diff header naming issues This makes the git_diff_patch definition private to diff_patch.c and fixes a number of other header file naming inconsistencies to use `git_` prefixes on functions and structures that are shared between files.
Russell Belfer 5dc98298 2013-06-11T11:22:22 Implement regex pattern diff driver This implements the loading of regular expression pattern lists for diff drivers that search for function context in that way. This also changes the way that diff drivers update options and interface with xdiff APIs to make them a little more flexible.
Russell Belfer 114f5a6c 2013-06-10T10:10:39 Reorganize diff and add basic diff driver This is a significant reorganization of the diff code to break it into a set of more clearly distinct files and to document the new organization. Hopefully this will make the diff code easier to understand and to extend. This adds a new `git_diff_driver` object that looks of diff driver information from the attributes and the config so that things like function content in diff headers can be provided. The full driver spec is not implemented in the commit - this is focused on the reorganization of the code and putting the driver hooks in place. This also removes a few #includes from src/repository.h that were overbroad, but as a result required extra #includes in a variety of places since including src/repository.h no longer results in pulling in the whole world.