Commit 62590d428b8c059a4a469de28f36d199f9392192
2023-04-04T19:06:20
Decomp: Don't enable 2-pass color quant w/ RGB565 The 2-pass color quantization algorithm assumes 3-sample pixels. RGB565 is the only 3-component colorspace that doesn't have 3-sample pixels, so we need to treat it as a special case when determining whether to enable 2-pass color quantization. Otherwise, attempting to initialize 2-pass color quantization with an RGB565 output buffer could cause prescan_quantize() to read from uninitialized memory and subsequently underflow/overflow the histogram array. djpeg is supposed to fail gracefully if both -rgb565 and -colors are specified, because none of its destination managers (image writers) support color quantization with RGB565. However, prescan_quantize() was called before that could occur. It is possible but very unlikely that these issues could have been reproduced in applications other than djpeg. The issues involve the use of two features (12-bit precision and RGB565) that are incompatible, and they also involve the use of two rarely-used legacy features (RGB565 and color quantization) that don't make much sense when combined. Fixes #668 Fixes #671 Fixes #680