src/diff_tform.c


Log

Author Commit Date CI Message
Russell Belfer f62c174d 2013-12-02T13:49:58 GIT_DIFF_FIND_REMOVE_UNMODIFIED sounds better
Russell Belfer 97ad85b8 2013-12-02T13:30:05 Add GIT_DIFF_FIND_DELETE_UNMODIFIED flag When doing copy detection, it is often necessary to include UNMODIFIED records in the git_diff so they are available as source records for GIT_DIFF_FIND_COPIES_FROM_UNMODIFIED. Yet in the final diff, often you will not want to have these UNMODIFIED records. This adds a flag which marks these UNMODIFIED records for deletion from the diff list so they will be removed after the rename detect phase is over.
Russell Belfer 2123a17f 2013-12-02T13:27:06 Fix bug making split deltas a COPIED targets When FIND_COPIES is used in combination with BREAK_REWRITES for rename detection, there was a bug where the split MODIFIED delta was only used as a target for RENAME records and not for COPIED records. This fixes that, converting the split into a pair of DELETED and COPIED deltas when that circumstance arises.
Russell Belfer e7c85120 2013-11-01T11:39:37 More tests and fixed for merging reversed diffs There were a lot more cases to deal with to make sure that our merged (i.e. workdir-to-tree-to-index) diffs were matching the output of core Git.
Russell Belfer 3940310e 2013-10-30T13:56:42 Fix some of the glaring errors in GIT_DIFF_REVERSE These changes fix the basic problem with GIT_DIFF_REVERSE being broken for text diffs. The reversed diff entries were getting added to the git_diff correctly, but some of the metadata was kept incorrectly in a way that prevented the text diffs from being generated correctly. Once I fixed that, it became clear that it was not possible to merge reversed diffs correctly. This has a first pass at fixing that problem. We probably need more tests to make sure that is really fixed thoroughly.
Russell Belfer 74a627f0 2013-10-21T09:07:19 Tweak to git_diff_delta structure for nfiles While the base git_diff_delta structure always contains two files, when we introduce conflict data, it will be helpful to have an indicator when an additional file is involved.
Russell Belfer 10672e3e 2013-10-15T15:10:07 Diff API cleanup This lays groundwork for separating formatting options from diff creation options. This groups the formatting flags separately from the diff list creation flags and reorders the options. This also tweaks some APIs to further separate code that uses patches from code that just looks at git_diffs.
Russell Belfer 3ff1d123 2013-10-11T14:51:54 Rename diff objects and split patch.h This makes no functional change to diff but renames a couple of the objects and splits the new git_patch (formerly git_diff_patch) into a new header file.
Vicent Martí 6208bd49 2013-09-03T12:29:18 Merge pull request #1804 from ethomson/rewrites Minor changes for rewrites
Edward Thomson 17c7fbf6 2013-08-21T14:07:53 Split rewrites, status doesn't return rewrites Ensure that we apply splits to rewrites, even if we're not interested in examining it closely for rename/copy detection. In keeping with core git, status should not display rewrites, it should simply show files as "modified".
Russell Belfer b6ac07b5 2013-08-22T14:45:10 Trying to fix Win32 warnings
Russell Belfer 7edb74d3 2013-08-04T14:06:13 Update rename src map for any split src When using a rename source that is actually a to-be-split record, we have to update the best-fit mapping data in both the case where the target is also a split record and the case where the target is a simple added record. Before this commit, we were only doing the update when the target was itself a split record (and even in that case, the test was slightly wrong).
Russell Belfer d730d3f4 2013-07-31T16:40:42 Major rename detection changes After doing further profiling, I found that a lot of time was being spent attempting to insert hashes into the file hash signature when using the rolling hash because the rolling hash approach generates a hash per byte of the file instead of one per run/line of data. To optimize this, I decided to convert back to a run-based file signature algorithm which would be more like core Git. After changing this, a number of the existing tests started to fail. In some cases, this appears to have been because the test was coded to be too specific to the particular results of the file similarity metric and in some cases there appear to have been bugs in the core rename detection code where only by the coincidence of the file similarity scoring were the expected results being generated. This renames all the variables in the core rename detection code to be more consistent and hopefully easier to follow which made it a bit easier to reason about the behavior of that code and fix the problems that I was seeing. I think it's in better shape now. There are a couple of tests now that attempt to stress test the rename detection code and they are quite slow. Most of the time is spent setting up the test data on disk and in the index. When we roll out performance improvements for index insertion, it should also speed up these tests I hope.
Russell Belfer a16e4172 2013-07-25T12:27:39 Fix rename detection to use actual blob size The size data in the index may not reflect the actual size of the blob data from the ODB when content filtering comes into play. This commit fixes rename detection to use the actual blob size when calculating data signatures instead of the value from the index. Because of a misunderstanding on my part, I first converted the git_index_add_bypath API to use the post-filtered blob data size in creating the index entry. I backed that change out, but I kept the overall refactoring of that routine and the new internal git_blob__create_from_paths API because it eliminates an extra stat() call from the code that adds a file to the index. The existing tests actually cover this code path, at least when running on Windows, so at this point I'm not adding new tests to cover the changes.
Russell Belfer effdbeb3 2013-07-24T17:48:37 Make rename detection file size fix better The previous fix for checking file sizes with rename detection always loads the blob. In this version, if the odb backend can get the object header without loading the whole thing into memory, then we'll just use that, so that we can eliminate possible rename sources & targets without loading them.
Russell Belfer a5140f4d 2013-07-24T17:11:49 Fix rename detection for tree-to-tree diffs The performance improvements I introduced for rename detection were not able to run successfully for tree-to-tree diffs because the blob size was not known early enough and so the file signature always had to be calculated nonetheless. This change separates loading blobs into memory from calculating the signature. I can't avoid having to load the large blobs into memory, but by moving it forward, I'm able to avoid the signature calculation if the blob won't come into play for renames.
Russell Belfer f5c4d022 2013-07-24T13:44:35 Fix incorrect comment
Russell Belfer 18e9efc4 2013-07-24T13:10:16 Don't check rename if file size difference is huge
Edward Thomson d55bed1a 2013-07-17T16:55:00 don't include ignored as rename candidates
nulltoken c4ac556e 2013-06-29T12:48:58 Fix compilation warnings
Russell Belfer e4acc3ba 2013-06-18T16:14:35 Fix rename looped reference issues This makes the diff rename tracking code more careful about the order in which it processes renames and more thorough in updating the mapping of correct renames when an earlier rename update alters the index of a later matched pair.
Russell Belfer a1683f28 2013-06-14T16:18:04 More tests and bug fixes for status with rename This changes the behavior of the status RENAMED flags so that they will be combined with the MODIFIED flags if appropriate. If a file is modified in the index and also renamed, then the status code will have both the GIT_STATUS_INDEX_MODIFIED and INDEX_RENAMED bits set. If it is renamed but the OID has not changed, then just the GIT_STATUS_INDEX_RENAMED bit will be set. Similarly, the flags GIT_STATUS_WT_MODIFIED and GIT_STATUS_WT_RENAMED can both be set independently of one another. This fixes a serious bug where the check for unmodified files that was done at data load time could end up erasing the RENAMED state of a file that was renamed with no changes. Lastly, this contains a bunch of new tests for status with renames, including tests where the only rename changes are case changes. The expected results of these tests have to vary by whether the platform uses a case sensitive filesystem or not, so the expected data covers those platform differences separately.
Russell Belfer 37f66e82 2013-06-12T15:21:21 Fix Windows warnings This fixes problems with missing function prototypes and 64-bit data issues on Windows.
Vicent Martí 88c401be 2013-06-12T14:54:32 Merge pull request #1643 from ethomson/rename_source Keep data about source of similarity
Edward Thomson 690bf41c 2013-06-10T15:16:44 keep source similarity in rename detection
Russell Belfer 114f5a6c 2013-06-10T10:10:39 Reorganize diff and add basic diff driver This is a significant reorganization of the diff code to break it into a set of more clearly distinct files and to document the new organization. Hopefully this will make the diff code easier to understand and to extend. This adds a new `git_diff_driver` object that looks of diff driver information from the attributes and the config so that things like function content in diff headers can be provided. The full driver spec is not implemented in the commit - this is focused on the reorganization of the code and putting the driver hooks in place. This also removes a few #includes from src/repository.h that were overbroad, but as a result required extra #includes in a variety of places since including src/repository.h no longer results in pulling in the whole world.
Russell Belfer 49f70f2c 2013-05-23T15:48:06 Fill out diff rename test coverage This extends the rename tests to make sure that every rename scenario in the inner loop of git_diff_find_similar is actually exercised. Also, fixes an incorrect assert that was in one of the clauses that was not previously being exercised.
Russell Belfer 67db583d 2013-05-23T15:06:07 More diff rename tests; better split swap handling This adds a couple more tests of different rename scenarios. Also, this fixes a problem with the case where you have two "split" deltas and the left half of one matches the right half of the other. That case was already being handled, but in the wrong order in a way that could result in bad output. Also, if the swap also happened to put the other two halves into the correct place (i.e. two files exchanged places with each other), then the second delta was left with the SPLIT flag set when it really should be cleared.
Russell Belfer c68b09dc 2013-05-23T11:52:34 Fix dereference of freed delta I was accidentally using a value that I had just freed. This moves the clearing of the delta internal flags into a better place.
Russell Belfer a21cbb12 2013-05-22T10:37:12 Significant rename detection rewrite This flips rename detection around so instead of creating a forward mapping from deltas to possible rename targets, instead it creates a reverse mapping, looking at possible targets and trying to find a source that they could have been renamed or copied from. This is important because each output can only have a single source, but a given source could map to multiple outputs (in the form of COPIED records). Additionally, this makes a couple of tweaks to the public rename detection APIs, mostly renaming a couple of options that control the behavior to make more sense and to be more like core Git. I walked through the tests looking at the exact results and updated the expectations based on what I saw. The new code is different from the old because it cannot give some nonsense results (like A was renamed to both B and C) which were part of the outputs previously.
Russell Belfer 9be5be47 2013-05-20T13:37:21 More git_diff_find_similar improvements - Add new GIT_DIFF_FIND_EXACT_MATCH_ONLY flag to do similarity matching without using the similarity metric (i.e. only compare the SHA). - Clean up the similarity measurement code to more rigorously distinguish between files that are not similar and files that are not comparable (previously, a 0 could either mean that the files could not be compared or that they were totally different) - When splitting a MODIFIED file into a DELETE/ADD pair, actually make a DELETED/UNTRACKED pair if the right side of the diff is from the working directory. This prevents an odd mix of ADDED and UNTRACKED files on workdir diffs.
Russell Belfer d958e37a 2013-05-17T17:21:45 Fix issues with git_diff_find_similar There are a number of bugs in the rename code that only were obvious when I started testing it against large old repos with more complex patterns. (The code to do that testing is not ready to merge with libgit2, but I do plan to add more thorough tests.) This contains a significant number of changes and also tweaks the public API slightly to make emulating core git easier. Most notably, this separates the GIT_DIFF_FIND_AND_BREAK_REWRITES flag into FIND_REWRITES (which adds a self-similarity score to every modified file) and BREAK_REWRITES (which splits the modified deltas into add/remove pairs in the diff list). When you do a raw output of core git, rewrites show up as M090 or such, not at A and D output, so I wanted to be able to emulate that. Publicly, this also changes the flags to be uint16_t since we don't need values out of that range. Internally, this contains significant changes from a number of small bug fixes (like using the wrong side of the diff to decide if the object could be found in the ODB vs the workdir) to larger issues about which files can and should be compared and how the various edge cases of similarity scores should be treated. Honestly, I don't think this is the last update that will have to be made to this code, but I think this moves us closer to correct behavior and I tried to document the code so it would be easier to follow..
Vicent Martí 71596200 2013-05-15T15:47:46 Merge pull request #1588 from arrbee/fixes-for-checkout-and-diff Bug fixes for checkout and diff
Russell Belfer 09fae31d 2013-05-15T14:58:26 Improve robustness of diff rename detection Under some strange circumstances, diffs can end up listing files that we can't actually open successfully. Instead of aborting the git_diff_find_similar, this makes it so that those files just won't be considered as valid rename/copy targets instead.
nulltoken 1fed6b07 2013-05-13T21:57:37 Fix trailing whitespaces
Edward Thomson 0462fba5 2013-04-30T14:56:41 renames!
Russell Belfer b7f167da 2013-04-29T13:52:12 Make git_oid_cmp public and add git_oid__cmp
Russell Belfer 8cfd54f0 2013-03-26T12:27:15 Fix Windows/Win32 warning
Edward Thomson aa408cbf 2013-03-11T11:18:00 handle small files in similarity metrics
Russell Belfer 0a008913 2013-02-22T10:21:02 Minor improvements to find_similar code This moves a couple of checks outside of the inner loop of the find_similar rename/copy detection phase that are only dependent on the "from" side of a detection. Also, this replaces the inefficient initialization of the options structure when a value is not provided explicitly by the user.
Russell Belfer f8275890 2013-02-22T10:19:50 Replace static data with configured metric Instead of creating three git_diff_similarity_metric statically for the various config options, just create the metric structure on demand and populate it, using the payload to specific the extra flags that should be passed to the hashsig. This removes a level of obfuscation from the code, I think.
Russell Belfer d4b747c1 2013-02-21T16:44:44 Add diff rename tests with partial similarity This adds some new tests that actually exercise the similarity metric between files to detect renames, copies, and split modified files that are too heavily modified. There is still more testing to do - these tests are just partially covering the cases. There is also one bug fix in this where a change set with only MODIFY being broken into ADD/DELETE (due to low self-similarity) without any additional RENAMED entries would end up not processing the split requests (because the num_rewrites counter got reset).
Russell Belfer 960a04dd 2013-02-21T12:40:33 Initial integration of similarity metric to diff This is the initial integration of the similarity metric into the `git_diff_find_similar()` code path. The existing tests all pass, but the new functionality isn't currently well tested. The integration does go through the pluggable metric interface, so it should be possible to drop in an alternative to the internal metric that libgit2 implements. This comes along with a behavior change for an existing interface; namely, passing two NULLs to git_diff_blobs (or passing NULLs to git_diff_blob_to_buffer) will now call the file_cb parameter zero times instead of one time. I know it's strange that that change is paired with this other change, but it emerged from some initialization changes that I ended up making.
Russell Belfer 71a3d27e 2013-02-08T10:06:47 Replace diff delta binary with flags Previously the git_diff_delta recorded if the delta was binary. This replaces that (with no net change in structure size) with a full set of flags. The flag values that were already in use for individual git_diff_file objects are reused for the delta flags, too (along with renaming those flags to make it clear that they are used more generally). This (a) makes things somewhat more consistent (because I was using a -1 value in the "boolean" binary field to indicate unset, whereas now I can just use the flags that are easier to understand), and (b) will make it easier for me to add some additional flags to the delta object in the future, such as marking the results of a copy/rename detection or other deltas that might want a special indicator. While making this change, I officially moved some of the flags that were internal only into the private diff header. This also allowed me to remove a gross hack in rename/copy detect code where I was overwriting the status field with an internal value.
Russell Belfer 9bc8be3d 2013-02-19T10:25:41 Refine pluggable similarity API This plugs in the three basic similarity strategies for handling whitespace via internal use of the pluggable API. In so doing, I realized that the use of git_buf in the hashsig API was not needed and actually just made it harder to use, so I tweaked that API as well. Note that the similarity metric is still not hooked up in the find_similarity code - this is just setting out the function that will be used.
Russell Belfer 5e5848eb 2013-02-14T17:25:10 Change similarity metric to sampled hashes This moves the similarity metric code out of buf_text and into a new file. Also, this implements a different approach to similarity measurement based on a Rabin-Karp rolling hash where we only keep the top 100 and bottom 100 hashes. In theory, that should be sufficient samples to given a fairly accurate measurement while limiting the amount of data we keep for file signatures no matter how large the file is.
Russell Belfer 99ba8f23 2013-01-22T15:27:08 wip: adding metric to diff
Philip Kelley 11d9f6b3 2013-01-27T14:17:07 Vector improvements and their fallout
Edward Thomson 359fc2d2 2013-01-08T17:07:25 update copyrights
Ben Straub c7231c45 2012-11-30T16:31:42 Deploy GITERR_CHECK_VERSION
Ben Straub ca901e7b 2012-11-29T15:16:19 Deploy GIT_DIFF_FIND_OPTIONS_INIT
Russell Belfer db106d01 2012-10-30T09:40:50 Move rename detection into new file This improves the naming for the rename related functionality moving it to be called `git_diff_find_similar()` and renaming all the associated constants, etc. to make more sense. I also moved the new code (plus the existing `git_diff_merge`) into a new file `diff_tform.c` where I can put new functions related to manipulating git diff lists. This also updates the implementation significantly from the last revision fixing some ordering issues (where break-rewrite needs to be handled prior to copy and rename detection) and improving config option handling.