Commit 4851cbe40687c6a02698b998717d281912498d20
2024-08-26T12:14:20
djpeg/jpeg_crop_scanline(): Disallow crop vals < 0 Because the crop spec was parsed using unsigned 32-bit integers, negative numbers were interpreted as values ~= UINT_MAX (4,294,967,295). This had the following ramifications: - If the cropping region width was negative and the adjusted width + the adjusted left boundary was greater than 0, then the 32-bit unsigned integer bounds checks in djpeg and jpeg_crop_scanline() overflowed and failed to detect the out-of-bounds width, jpeg_crop_scanline() set cinfo->output_width to a value ~= UINT_MAX, and a buffer overrun and subsequent segfault occurred in the upsampling or color conversion routine. The segfault occurred in the body of jpeg_skip_scanlines() --> read_and_discard_scanlines() if the cropping region upper boundary was greater than 0 and the JPEG image used chrominance subsampling and in the body of jpeg_read_scanlines() otherwise. - If the cropping region width was negative and the adjusted width + the adjusted left boundary was 0, then a zero-width output image was generated. - If the cropping region left boundary was negative, then an output image with bogus data was generated. This commit modifies djpeg and jpeg_crop_scanline() so that the aforementioned bounds checks use 64-bit unsigned integers, thus guarding against overflow. It similarly modifies jpeg_skip_scanlines(). In the case of jpeg_skip_scanlines(), the issue was not reproducible with djpeg, but passing a negative number of lines to jpeg_skip_scanlines() caused a similar overflow if the number of lines + cinfo->output_scanline was greater than 0. That caused jpeg_skip_scanlines() to read past the end of the JPEG image, throw a warning ("Corrupt JPEG data: premature end of data segment"), and fail to return unless warnings were treated as fatal. Also, djpeg now parses the crop spec using signed integers and checks for negative values.