Commit 6d48aaacd21911d8484289b7396b880eb1f393e0
2025-10-22T10:30:53
TJ: Handle lossless/CS params w/ YUV enc/compress - If TJPARAM_LOSSLESS was set, then tj3EncodeYUV*8() called jpeg_enable_lossless() (via setCompDefaults()), which caused the underlying libjpeg API to silently disable subsampling and color conversion. This led to three issues: 1. Attempting to encode RGB pixels produced incorrect YCbCr or grayscale components, since color conversion did not occur. The same issue occurred if TJPARAM_COLORSPACE was explicitly set to TJCS_RGB. 2. Attempting to encode RGB pixels into a grayscale plane caused tj3EncodeYUVPlanes8() to overflow the caller's destination pointer array if the array was not big enough to accommodate three pointers. If called from tj3EncodeYUV8(), tj3EncodeYUVPlanes8() did not overflow the caller's destination pointer array, but a segfault occurred when it attempted to copy to the Cb and Cr pointers, which were NULL. The same issue occurred if TJPARAM_COLORSPACE was explicitly set to anything other than TJCS_GRAY. 3. Attempting to encode RGB pixels into subsampled YUV planes caused tj3EncodeYUV*8() to overflow the caller's buffer(s) if the buffer(s) were not big enough to accommodate 4:4:4 (non-subsampled) YUV planes. That would have been the case if the caller allocated its buffer(s) based on the return value of tj3YUVBufSize() or tj3YUVPlaneSize(). The same issue occurs if TJPARAM_SUBSAMP is explicitly set to TJSAMP_444. tj3EncodeYUV*8() now ignores TJPARAM_LOSSLESS and TJPARAM_COLORSPACE. - If TJPARAM_LOSSLESS was set, then attempting to compress a grayscale plane into a JPEG image caused tj3CompressFromYUVPlanes8() to overflow the caller's source pointer array if the array was not big enough to accommodate three pointers. If called from tj3CompressFromYUV8(), tj3CompressFromYUVPlanes8() did not overflow the caller's source pointer array, but a segfault occurred when it attempted to copy from the Cb and Cr pointers, which were NULL. This was similar to Issue 2 above. The same issue occurred if TJPARAM_COLORSPACE was explicitly set to anything other than TJCS_GRAY. tj3CompressFromYUV*8() now throws an error if TJPARAM_LOSSLESS is set, and it now ignores TJPARAM_COLORSPACE. These issues did not pose a security risk, since security exploits involve supported workflows that function normally except when supplied with malformed input data. It is documented that colorspace conversion, chrominance subsampling, and compression from planar YUV images are unavailable when TJPARAM_LOSSLESS is set. When TJPARAM_LOSSLESS is set, the library effectively sets TJPARAM_SUBSAMP to TJSAMP_444 and TJPARAM_COLORSPACE to TJCS_RGB, TJCS_GRAY, or TJCS_CMYK, depending on the pixel format of the source image. That behavior is strongly implied by the documentation of TJPARAM_LOSSLESS, although the documentation isn't specific about whether TJPARAM_LOSSLESS applies to tj3EncodeYUV*8(). In any case, setting TJPARAM_LOSSLESS before calling tj3CompressFromYUV*8() was never a supported or functional workflow, and setting TJPARAM_LOSSLESS before calling tj3EncodeYUV*8() was never a functional workflow. Thus, there should be no applications "in the wild" that use either workflow. Such applications would crash every time they attempted to encode to or compress from a YUV image. In other words, setting TJPARAM_LOSSLESS or TJPARAM_COLORSPACE required the caller to understand the ramifications of the loss of color conversion and/or subsampling, and failing to do so was essentially API abuse (albeit subtle API abuse, hence the desire to make the behavior more intuitive.) This commit also removes no-op code introduced by 6da05150efa9b7a190d05bf82295127c778d15ab. Since setCompDefaults() returns after calling jpeg_enable_lossless(), modifying the subsampling level locally had no effect. The libjpeg API already silently disables subsampling in jinit_c_master_control() if lossless compression is enabled, so it was not necessary for setCompDefaults() to handle that. Fixes #839