thodg/libgit2/src/diff_driver.h
Tag
-
Show log
Commit
-
Hash : d8896bda
Author :
Date : 2018-01-03T16:07:36
diff_generate: avoid excessive stats of .gitattribute files When generating a diff between two trees, for each file that is to be diffed we have to determine whether it shall be treated as text or as binary files. While git has heuristics to determine which kind of diff to generate, users can also that default behaviour by setting or unsetting the 'diff' attribute for specific files. Because of that, we have to query gitattributes in order to determine how to diff the current files. Instead of hitting the '.gitattributes' file every time we need to query an attribute, which can get expensive especially on networked file systems, we try to cache them instead. This works perfectly fine for every '.gitattributes' file that is found, but we hit cache invalidation problems when we determine that an attribuse file is _not_ existing. We do create an entry in the cache for missing '.gitattributes' files, but as soon as we hit that file again we invalidate it and stat it again to see if it has now appeared. In the case of diffing large trees with each other, this behaviour is very suboptimal. For each pair of files that is to be diffed, we will repeatedly query every directory component leading towards their respective location for an attributes file. This leads to thousands or even hundreds of thousands of wasted syscalls. The attributes cache already has a mechanism to help in that scenario in form of the `git_attr_session`. As long as the same attributes session is still active, we will not try to re-query the gitmodules files at all but simply retain our currently cached results. To fix our problem, we can create a session at the top-most level, which is the initialization of the `git_diff` structure, and use it in order to look up the correct diff driver. As the `git_diff` structure is used to generate patches for multiple files at once, this neatly solves our problem by retaining the session until patches for all files have been generated. The fix has been tested with linux.git by calling `git_diff_tree_to_tree` and `git_diff_to_buf` with v4.10^{tree} and v4.14^{tree}. | time | .gitattributes stats without fix | 33.201s | 844614 with fix | 30.327s | 4441 While execution only improved by roughly 10%, the stat(3) syscalls for .gitattributes files decreased by 99.5%. The benchmarks were quite simple with best-of-three timings on Linux ext4 systems. One can assume that for network based file systems the performance gain will be a lot larger due to a much higher latency.
-
src/diff_driver.h
-
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
/* * Copyright (C) the libgit2 contributors. All rights reserved. * * This file is part of libgit2, distributed under the GNU GPL v2 with * a Linking Exception. For full terms see the included COPYING file. */ #ifndef INCLUDE_diff_driver_h__ #define INCLUDE_diff_driver_h__ #include "common.h" #include "attr_file.h" #include "buffer.h" typedef struct git_diff_driver_registry git_diff_driver_registry; git_diff_driver_registry *git_diff_driver_registry_new(void); void git_diff_driver_registry_free(git_diff_driver_registry *); typedef struct git_diff_driver git_diff_driver; int git_diff_driver_lookup(git_diff_driver **, git_repository *, git_attr_session *attrsession, const char *); void git_diff_driver_free(git_diff_driver *); /* diff option flags to force off and on for this driver */ void git_diff_driver_update_options(uint32_t *option_flags, git_diff_driver *); /* returns -1 meaning "unknown", 0 meaning not binary, 1 meaning binary */ int git_diff_driver_content_is_binary( git_diff_driver *, const char *content, size_t content_len); typedef long (*git_diff_find_context_fn)( const char *, long, char *, long, void *); typedef int (*git_diff_find_context_line)( git_diff_driver *, git_buf *); typedef struct { git_diff_driver *driver; git_diff_find_context_line match_line; git_buf line; } git_diff_find_context_payload; void git_diff_find_context_init( git_diff_find_context_fn *findfn_out, git_diff_find_context_payload *payload_out, git_diff_driver *driver); void git_diff_find_context_clear(git_diff_find_context_payload *); #endif