lib/pack.c


Log

Author Commit Date CI Message
Stefan Sperling 2d9e6abf 2022-05-04T13:43:24 store deltas in compressed form while packing, both in memory and cache file This reduces memory and disk space consumption during packing. with tweaks + memleak on error fix from op@ ok op@
Stefan Sperling d7b5a0e8 2022-04-20T14:00:12 inline struct got_object_id in struct got_object_qid Saves us from doing a malloc/free call for every item on the list. ok op@
Stefan Sperling 36751b1b 2022-03-08T15:11:37 remove needless header inclusion; lib/pack.c does not need got_opentemp.h
Omar Polo 8934ea8b 2022-02-14T13:14:36 factorize imsg_clear calls after imsg_flush failures imsg_clear frees and closes resources allocated as part of enqueueing imsgs so it's a no-op after reads. discussed with and ok stsp@
Stefan Sperling 67fd6849 2022-02-13T00:10:25 reuse existing deltas when creating pack files tested by thomas, naddy, and myself
Stefan Sperling 0ee7a4dc 2022-01-26T10:33:21 fix double-free and double-close issues in error paths of got_packidx_open()
Stefan Sperling 3bfc2334 2022-01-11T15:37:06 fix file corruption regression in 'got checkout' caused by commit db696021 got-read-pack must rewind its files temporary files before reusing them. Problem reported by naddy ok tracey naddy
Stefan Sperling 3976db15 2022-01-10T14:46:30 add missing checks for reads beyond the mapped memory area of a pack file
Stefan Sperling 404bde06 2022-01-03T21:51:02 fix bogus "object not found" errors when object ID begins with 00 ok thomas_adam
Stefan Sperling c0df5966 2021-12-31T09:33:01 wrap overlong lines
Stefan Sperling e7ae0baf 2021-12-31T09:11:22 add O_CLOEXEC (close-on-exec) to openat(2) calls suggested by millert ok thomas_adam
Stefan Sperling c3564dfa 2021-07-15T07:48:40 don't scan pack index offsets for large values if pack file is < 2GB This saves an iteration over the entire h->offsets array when opening a pack index which should not contain large offsets in the first place. ok millert@
Stefan Sperling 1124fe40 2021-07-07T17:21:52 handle pack index files which lack a corresponding pack file ok millert
Stefan Sperling aea75d87 2021-07-06T07:57:00 make got_packidx_get_packfile_path() usable for callers who only have a path
Christian Weisgerber dbdddfee 2021-06-23T20:48:35 switch from SIMPLEQ to equivalent STAILQ macros The singly-linked tail queue macros were added to OpenBSD 6.9 and are more widely available on other systems. ok stsp
Stefan Sperling c4330eff 2021-06-22T19:11:54 expose got_pack_parse_ref_delta() for library-internal use This will be needed by a future 'gotadmin listpack' command.
Stefan Sperling 02828bfd 2021-06-22T19:05:05 expose got_packidx_get_object_offset() for library-internal use This will be needed by a future 'gotadmin listpack' command.
Stefan Sperling b4f37570 2021-06-19T16:32:39 mark got_pack_stop_privsep_child() static; it is only used inside pack.c
Tracey Emery cc2a8ef4 2021-06-19T16:12:52 fix a leaky ibuf. ok stsp
Stefan Sperling 509c9973 2021-04-10T13:11:17 add got_packidx_get_packfile_path() for library-internal use
Stefan Sperling ac62b712 2021-03-30T14:50:44 when reading a pack index byte-swap fields at compile-time where possible suggested by mpi@ ages ago
Stefan Sperling 08578a35 2021-01-22T11:05:05 make close(2) failure checks consistent; check 'close() == -1' everywhere ok millert, naddy
Yang Zhong 6d5a9006 2020-12-16T11:16:27 add fd field to got_repository, modify got_packidx_open to use fds These changes are intended to make got more compatible with FreeBSD's Capsicum. ok stsp
Stefan Sperling 5a17689a 2020-09-24T00:28:58 fix matching the object ID of the first object in a pack index An object ID which appears first in a pack index was wrongly ignored by got_packidx_match_id_str_prefix(). The result was a spurious "object not found" error when this object ID was passed on the command line. Reported by jrick on freenode who was lucky enough to this edge case!
Christian Weisgerber 78fb0967 2020-09-09T21:41:22 use the POSIX standard endian functions and explicitly include <endian.h> ok stsp
Christian Weisgerber 81a12da5 2020-09-09T19:35:32 do not rely on <zlib.h> to pull in <unistd.h> ok stsp
Stefan Sperling 00927983 2020-04-19T09:11:37 remove got_packidx_get_object_idx_sha1(); it is not actually needed
Stefan Sperling d582f26c 2020-03-18T16:13:48 write large objects to disk when resolving deltas; raise in-mem delta threshold
Stefan Sperling 4788f1ce 2020-03-18T16:13:46 extract large objects to a temporary file in got-index-pack
Stefan Sperling 2e5a6fad 2020-03-18T16:13:44 add mmap support to got-index-pack
Stefan Sperling 55fdd257 2020-03-18T16:13:43 do not buffer more data than necessary in got_inflate_to_mem_fd()
Stefan Sperling 1e87a3c3 2020-03-18T16:13:42 avoid re-reading the entirety of an object's data to calculate the CRC
Stefan Sperling 668a20f6 2020-03-18T16:13:41 rewritten got-index-pack; sorry about the monster commit
Stefan Sperling 3ab5e33c 2020-03-18T16:13:41 add optional 'consumed' output parameter to got_inflate_to_mem_fd()
Stefan Sperling afbfc7fb 2020-03-15T16:15:58 remove unneeded memory allocations from ref delta parser which were leaking
Stefan Sperling ca6e02ac 2020-01-07T11:14:52 add support for first-parent history traversal to got-read-pack
Stefan Sperling 5aa81393 2020-01-06T19:38:23 add copyright year for files already touched in 2020
Stefan Sperling e40b19ed 2020-01-06T16:16:51 fix reading ref deltas with GOT_PACK_NO_MMAP builds
Stefan Sperling 5f25cc85 2019-11-26T22:43:12 remove obj->path_packfile which is unused nowadays; saves a few free() calls
Stefan Sperling ab2f42e7 2019-11-10T15:51:05 cache delta data buffers in an LRU cache
Stefan Sperling 42c69117 2019-11-10T15:51:05 stop storing decompressed delta data in struct got_delta; fetch it on demand
Stefan Sperling 6dd1ece6 2019-11-10T11:32:03 when an object id string is bad, include the bad string in the error message
Stefan Sperling 5e6be232 2019-11-08T21:55:41 fix reading pack files larger than 2GB; ok tb@; problem found by mpi@
Stefan Sperling c6368c2e 2019-10-11T09:10:54 properly handle read errors in got_packidx_init_hdr (found by jj@ scan-build)
Stefan Sperling b1317e77 2019-09-22T10:51:10 plug error leak in got_packidx_init_hdr(); found by jj@ with scan-build
Hiltjo Posthuma 6772cf22 2019-08-27T19:01:04 got_packidx_open: free packidx object when open(2) fails
Stefan Sperling de30857e 2019-08-23T16:40:51 fix reading pack index files with large offsets (patch by Hiltjo Posthuma)
Stefan Sperling a0de39f3 2019-08-09T11:56:24 fix build on OpenBSD/sparc64 (gcc)
Stefan Sperling 0adc7bcc 2019-06-29T23:28:23 call got_object_id_queue_free() instead of implementing it inline
Stefan Sperling dd88155e 2019-06-29T23:26:36 disambiguate abbreviated object IDs based on object type
Stefan Sperling 4277420a 2019-06-29T12:58:30 speed up matching of abbreviated commit IDs in pack files
Stefan Sperling e09a504c 2019-06-28T20:59:14 introduce support for abbreviated object IDs
Stefan Sperling ff563a3d 2019-05-23T21:04:23 replace got_pack_get_packfile_size() with simple fstat()
Stefan Sperling 57377f07 2019-05-23T14:50:57 stat .idx file to determine its size, not the .pack file
Stefan Sperling 1a35c1bc 2019-05-22T11:48:27 fix double-frees in error path when resolving deltas fails
Stefan Sperling 638f9024 2019-05-13T12:40:57 rename got_error_prefix_errno() to got_error_from_errno()
joshua stein 230a42bd 2019-05-11T13:04:38 got_error_from_errno -> got_error_prefix_errno also add got_error_prefix_errno2 and got_error_prefix_errno3 which should hopefully all be merged into a single function with variadic args (but can't alloc mem)
Stefan Sperling 324d37e7 2019-05-11T09:14:30 make got path APIs available to library consumers
Stefan Sperling 0dd5271b 2019-05-10T19:57:25 plug fd leak in got_pack_stop_privsep_child()
Stefan Sperling a5b57ccf 2019-04-11T13:37:50 omit unused mode argument to open(2)
Stefan Sperling 3a11398b 2019-02-21T14:15:33 more specific mmap(2) failure checks
Stefan Sperling 5ea74b5a 2019-02-11T14:20:09 don't close input files in dump_delta_chain_to_file(); caller does it
Stefan Sperling 3a6ce05a 2019-02-11T12:32:53 check for errors from close(2)
Stefan Sperling fb43ecf1 2019-02-11T11:59:09 check for errors from fclose()
Stefan Sperling 5d56da81 2019-01-13T17:15:53 happy new copyright year
Stefan Sperling 85a703fa 2019-01-13T17:04:03 got_pack_get_object_size() -> got_pack_get_max_delta_object_size()
Stefan Sperling ac544f8c 2019-01-13T15:58:39 pass smallish blobs directly via imsg buffer
Stefan Sperling 15a94983 2018-12-23T14:48:47 remove struct got_object from public library API
Stefan Sperling 96732e0b 2018-11-11T15:05:43 fix error handling in got_pack_stop_privsep_child()
Stefan Sperling 34fca9c3 2018-11-11T11:10:11 bounds checks before memcpy in got_delta_apply_in_mem()
Stefan Sperling 106807b4 2018-09-15T20:57:10 make object caching actually work in got-read-pack
Stefan Sperling faaa1c0f 2018-09-15T20:05:47 handle errors from read(2) better while reading pack files
Stefan Sperling 3840f4c9 2018-09-13T00:06:12 fix "rpath" pledge violation in got-read-pack
Stefan Sperling c59b3346 2018-09-11T12:29:46 stop sending deltas in imsg; cache deltas in got-read-pack
Stefan Sperling 876c234b 2018-09-10T12:30:47 start reading pack files with privsep; still WIP
Stefan Sperling 40cf7208 2018-09-09T17:26:34 remove unused includes from pack.c
Stefan Sperling 7e212e3d 2018-09-09T17:24:46 make got_packfile_extract_object_to_mem() accept pack instead of repo
Stefan Sperling 24140570 2018-09-09T17:19:27 make got_packfile_extract_object() accept a pack instead of a repo
Stefan Sperling 2090a03d 2018-09-09T17:00:33 make got_packfile_open_object() accept a pack instead of a repo
Stefan Sperling 1510f469 2018-09-09T16:33:32 move pack caching code from pack.c to repository.c
Stefan Sperling 35c73765 2018-09-09T16:14:11 make open_packed_object() expect a pack instead of a repo
Stefan Sperling a98c62b1 2018-09-09T16:09:01 remove unused parameter of open_delta_object()
Stefan Sperling 2ce68b2f 2018-09-09T16:07:11 don't redundantly open pack file in dump_delta_chain_to_file()
Stefan Sperling 48095039 2018-09-09T16:03:39 don't open pack redundantly in dump_delta_chain_to_mem()
Stefan Sperling c8ecd499 2018-09-09T16:00:41 remove unused parameter from functions resolving packfile deltas
Stefan Sperling 4fb0d740 2018-09-09T15:57:26 remove unused parameter of open_packfile()
Stefan Sperling 4a9c75d9 2018-09-09T15:56:38 remove unused parameter of get_packfile_path()
Stefan Sperling 3413160a 2018-09-09T15:55:40 remove unused parameter of get_object_idx()
Stefan Sperling 817c5a18 2018-09-09T15:50:02 move packidx header initialization to a separate function
Stefan Sperling c336f889 2018-07-23T10:53:49 don't store copies of packfile paths in each delta
Stefan Sperling 673702af 2018-07-23T10:37:54 fix leak in dump_delta_chain_to_file()
Stefan Sperling ab3ad429 2018-07-23T10:34:33 fix leaks in resolve_offset_delta()
Stefan Sperling 3efa19e7 2018-07-13T10:51:34 fix more memleaks
Stefan Sperling e8f89a81 2018-07-13T10:38:31 only unmap map if non-NULL in got_pack_close()
Stefan Sperling 45202a8f 2018-07-11T20:57:44 fix reading ref deltas
Stefan Sperling 63581804 2018-07-09T22:23:00 rename zbuf to inflate
Stefan Sperling fc79a48d 2018-07-09T17:00:51 add a compile time switch to read pack index with read(2)
Stefan Sperling af9b7fee 2018-07-09T16:32:39 add compile-time switch which disables mmap for pack files
Stefan Sperling 57363308 2018-07-09T16:20:48 fall back to read() as intended if mapping packfile fails
Stefan Sperling d7464085 2018-07-09T15:30:16 read pack files with mmap