libexec/got-read-pack/got-read-pack.c


Log

Author Commit Date CI Message
Stefan Sperling fae7e038 2022-05-07T11:50:56 run the search for deltas to reuse in got-read-pack This significantly speeds up the deltification step of packing by avoiding imsg traffic. gotadmin no longer requests individual raw deltas from got-read-pack to check whether it can reuse them. Instead, got-read-pack obtains a list of objects we want to pack, and hands back the list of all deltas in its pack file which can be reused. Messages are now batched such that imsg buffers are filled as much as possible. Another advantage is that deltas we are not going to reuse will no longer be written to the delta cache file, saving disk space. Before this patch, any raw delta candidate was written to the delta cache file by got-read-pack, and the decision whether to reuse the delta happened afterwards in the gotadmin process. Code for reading individual raw deltas is now unused and could be removed at some point. ok op@
Stefan Sperling 2d9e6abf 2022-05-04T13:43:24 store deltas in compressed form while packing, both in memory and cache file This reduces memory and disk space consumption during packing. with tweaks + memleak on error fix from op@ ok op@
Stefan Sperling d7b5a0e8 2022-04-20T14:00:12 inline struct got_object_id in struct got_object_qid Saves us from doing a malloc/free call for every item on the list. ok op@
Stefan Sperling 67fd6849 2022-02-13T00:10:25 reuse existing deltas when creating pack files tested by thomas, naddy, and myself
Stefan Sperling db696021 2022-01-04T16:11:52 avoid the creation of new temporary files whenever a packed object is read This speeds up the creation of pack files by about 30%.
Stefan Sperling c0df5966 2021-12-31T09:33:01 wrap overlong lines
Stefan Sperling 030daac8 2021-09-25T10:26:13 fix some integers that had a slightly wrong type; patch by Omar Polo
Stefan Sperling 50127d69 2021-09-25T10:24:15 add 'static' qualifier to local functions in got-read-pack; patch by Omar Polo
Stefan Sperling c3564dfa 2021-07-15T07:48:40 don't scan pack index offsets for large values if pack file is < 2GB This saves an iteration over the entire h->offsets array when opening a pack index which should not contain large offsets in the first place. ok millert@
Christian Weisgerber dbdddfee 2021-06-23T20:48:35 switch from SIMPLEQ to equivalent STAILQ macros The singly-linked tail queue macros were added to OpenBSD 6.9 and are more widely available on other systems. ok stsp
Stefan Sperling 40e3cb72 2021-06-22T19:01:44 fix raw object size sent by got-read-pack
Stefan Sperling 59d1e4a0 2021-03-10T22:49:22 implement raw object data access; this will be required for packing
Stefan Sperling 08578a35 2021-01-22T11:05:05 make close(2) failure checks consistent; check 'close() == -1' everywhere ok millert, naddy
Stefan Sperling 56b63ca4 2021-01-22T11:04:47 make fclose(3) failure checks consistent; check 'fclose() == EOF' everywhere ok millert, naddy
Christian Weisgerber 62d463ca 2020-10-20T22:43:59 indentation fixes
Christian Weisgerber 23c57b28 2020-09-11T19:44:18 Stop including <sys/syslimits.h> directly. POSIX says the limits defined there are available from <limits.h>, which almost all affected source files already included anyway. ok millert stsp
Christian Weisgerber 81a12da5 2020-09-09T19:35:32 do not rely on <zlib.h> to pull in <unistd.h> ok stsp
Stefan Sperling 00927983 2020-04-19T09:11:37 remove got_packidx_get_object_idx_sha1(); it is not actually needed
Stefan Sperling ded8fbb8 2020-04-19T08:57:07 include the relevant object ID in NO_OBJ error messages where possible
Stefan Sperling e70bf110 2020-03-22T17:20:18 move functions from privsep.c to the helpers which use them where possible
Stefan Sperling 61a7d79f 2020-02-29T08:05:11 remove GOT_ERR_NOT_ABSPATH and fix existing users to deal with absolute paths
Stefan Sperling 63f810e6 2020-02-29T07:54:55 when "bad path" errors occur, always show the path in question
Stefan Sperling ca6e02ac 2020-01-07T11:14:52 add support for first-parent history traversal to got-read-pack
Stefan Sperling 5aa81393 2020-01-06T19:38:23 add copyright year for files already touched in 2020
Stefan Sperling b87b4170 2020-01-06T14:10:36 rename to parsed_tree_entries_free for clarity (don't use with got_tree_object)
Stefan Sperling b64b1f95 2020-01-06T13:10:00 plug a memory leak; parsed tree entries were not freed
Stefan Sperling 3022d272 2019-11-14T17:12:32 reduce the amount of memcpy() and strdup() while parsing tree entries
Stefan Sperling ab2f42e7 2019-11-10T15:51:05 cache delta data buffers in an LRU cache
Stefan Sperling 42c69117 2019-11-10T15:51:05 stop storing decompressed delta data in struct got_delta; fetch it on demand
Stefan Sperling 12ce7a6c 2019-08-12T20:56:30 include <limits.h> instead of <sys/limits.h>; patch by Thomas Klausner
Stefan Sperling 79c99a64 2019-05-23T22:24:13 don't leak objects which can't be cached
Stefan Sperling 6869fb7f 2019-05-23T21:56:42 remove unused include in got-read-pack
Stefan Sperling cb5e38fd 2019-05-23T21:52:49 fix some leaks in got-read-pack
Stefan Sperling 704b89c4 2019-05-23T21:44:01 make got-read-pack actually use its object cache
Stefan Sperling 638f9024 2019-05-13T12:40:57 rename got_error_prefix_errno() to got_error_from_errno()
joshua stein 656b1f76 2019-05-11T14:26:12 while (1) -> for (;;)
joshua stein 230a42bd 2019-05-11T13:04:38 got_error_from_errno -> got_error_prefix_errno also add got_error_prefix_errno2 and got_error_prefix_errno3 which should hopefully all be merged into a single function with variadic args (but can't alloc mem)
Stefan Sperling 3a6ce05a 2019-02-11T12:32:53 check for errors from close(2)
Stefan Sperling fb43ecf1 2019-02-11T11:59:09 check for errors from fclose()
Stefan Sperling 0ae4af15 2019-02-01T22:33:39 add missing error check in got-raed-pack's tag_request()
Stefan Sperling 5d56da81 2019-01-13T17:15:53 happy new copyright year
Stefan Sperling 85a703fa 2019-01-13T17:04:03 got_pack_get_object_size() -> got_pack_get_max_delta_object_size()
Stefan Sperling ac544f8c 2019-01-13T15:58:39 pass smallish blobs directly via imsg buffer
Stefan Sperling ebc55e2d 2018-12-24T14:18:56 eliminate got_object_open() round-trip when opening blobs
Stefan Sperling 268f7291 2018-12-24T11:18:58 eliminate got_object_open() round-trip when opening tags
Stefan Sperling 13c729f7 2018-12-24T09:35:07 eliminate got_object_open() round-trip when opening trees
Stefan Sperling 6e72e6a3 2018-12-23T23:21:35 there's no point in embedding the object header in commits
Stefan Sperling 1785f84a 2018-12-23T23:15:42 eliminate got_object_open() round-trip when opening commits
Stefan Sperling f4a881ce 2018-11-17T16:04:05 more support for tag objects; new code is not yet reachable
Stefan Sperling 80d5f134 2018-11-11T12:59:22 don't be noisy on stderr when Ctrl-C is hit
Stefan Sperling 99437157 2018-11-11T12:52:59 abort checkout operations cleanly when Ctrl-C is hit
Stefan Sperling 41fa1437 2018-11-05T15:30:15 back out mini-commits; no significant performance difference
Stefan Sperling 710f3f4e 2018-11-05T15:26:18 make got-read-pack pre-seed the main process mini commit cache
Stefan Sperling 05e1230b 2018-11-05T12:42:03 rename got_commit_object_mini to got_mini_commit_object
Stefan Sperling 7762fe12 2018-11-05T11:46:30 add a mini-commit object for use by commit graph
Stefan Sperling 106807b4 2018-09-15T20:57:10 make object caching actually work in got-read-pack
Stefan Sperling 48d5fe42 2018-09-15T20:36:29 close object cache when got-read-pack exits
Stefan Sperling 56bef47a 2018-09-15T20:05:22 fix non-mmap read mode in got-read-pack
Stefan Sperling 2ff12563 2018-09-15T12:39:48 fix 'make PROFILE=1' build
Stefan Sperling 3840f4c9 2018-09-13T00:06:12 fix "rpath" pledge violation in got-read-pack
Stefan Sperling c59b3346 2018-09-11T12:29:46 stop sending deltas in imsg; cache deltas in got-read-pack
Stefan Sperling 55da3778 2018-09-10T18:39:31 read packed blobs with privsep
Stefan Sperling e7885405 2018-09-10T17:26:06 read packed trees with privsep
Stefan Sperling cfd633c2 2018-09-10T17:07:59 read packed commits with privsep
Stefan Sperling 876c234b 2018-09-10T12:30:47 start reading pack files with privsep; still WIP