Commit 45cd2ded885f9a74bc1b7e25c07181802bad32ac
2022-11-28T21:02:42
12-bit: Prevent RGB-to-YCC table overrun/underrun cjpeg relies on the various file I/O modules to range-limit the input samples, but no range limiting is performed by the jpeg_write_scanlines() function itself. With 8-bit samples, that isn't a problem, because sample values > MAXJSAMPLE will overflow the data type and wrap around to 0. With 12-bit samples, however, it is possible to pass sample values < 0 or > 4095 to jpeg_write_scanlines(), which would cause the RGB-to-YCbCr color converter to underflow or overflow the RGB-to-YCbCr conversion tables. That issue has existed in libjpeg all along. This commit mitigates the issue by masking off all but the lowest 12 bits of each 12-bit input sample prior to using the input sample value to index the RGB-to-YCbCr conversion tables. Fixes #633