Hash :
78a36f6d
Author :
Date :
2022-11-15T17:01:17
Fix buffer overrun in 12-bit prog Huffman encoder Regression introduced by 16bd984557fa2c490be0b9665e2ea0d4274528a8 and 5b177b3cab5cfb661256c1e74df160158ec6c34e The pre-computed absolute values used in encode_mcu_AC_first() and encode_mcu_AC_refine() were stored in a JCOEF (signed short) array. When attempting to losslessly transform a specially-crafted malformed 12-bit JPEG image with a coefficient value of -32768 into a progressive 12-bit JPEG image, the progressive Huffman encoder attempted to store the absolute value of -32768 in the JCOEF array, thus overflowing the 16-bit signed data type. Therefore, at this point in the code: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/8c5e78ce292c1642057102eac42f12ab57964293/jcphuff.c#L889 the absolute value was read as -32768, which caused the test at https://github.com/libjpeg-turbo/libjpeg-turbo/blob/8c5e78ce292c1642057102eac42f12ab57964293/jcphuff.c#L896 to fail, falling through to https://github.com/libjpeg-turbo/libjpeg-turbo/blob/8c5e78ce292c1642057102eac42f12ab57964293/jcphuff.c#L908 with an overly large value of r (46) that, when shifted left four places, incremented, and passed to emit_symbol(), exceeded the maximum index (255) for the derived code tables. Fortunately, the buffer overrun was fully contained within phuff_entropy_encoder, so the issue did not generate a segfault or other user-visible errant behavior, but it did cause a UBSan failure that was detected by OSS-Fuzz. This commit introduces an unsigned JCOEF (UJCOEF) data type and uses it to store the absolute values of DCT coefficients computed by the AC_first_prepare() and AC_refine_prepare() methods. Note that the changes to the Arm Neon progressive Huffman encoder extensions cause signed 16-bit instructions to be replaced with equivalent unsigned 16-bit instructions, so the changes should be performance-neutral. Based on: https://github.com/mayeut/libjpeg-turbo/commit/bbf61c0382c4f8bd1f1cfc666467581496c2fb7c Closes #628
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
/*
* jchuff.h
*
* This file was part of the Independent JPEG Group's software:
* Copyright (C) 1991-1997, Thomas G. Lane.
* libjpeg-turbo Modifications:
* Copyright (C) 2022, D. R. Commander.
* For conditions of distribution and use, see the accompanying README.ijg
* file.
*
* This file contains declarations for Huffman entropy encoding routines
* that are shared between the sequential encoder (jchuff.c) and the
* progressive encoder (jcphuff.c). No other modules need to see these.
*/
/* The legal range of a DCT coefficient is
* -1024 .. +1023 for 8-bit data;
* -16384 .. +16383 for 12-bit data.
* Hence the magnitude should always fit in 10 or 14 bits respectively.
*/
#if BITS_IN_JSAMPLE == 8
#define MAX_COEF_BITS 10
#else
#define MAX_COEF_BITS 14
#endif
/* The progressive Huffman encoder uses an unsigned 16-bit data type to store
* absolute values of coefficients, because it is possible to inject a
* coefficient value of -32768 into the encoder by attempting to transform a
* malformed 12-bit JPEG image, and the absolute value of -32768 would overflow
* a signed 16-bit integer.
*/
typedef unsigned short UJCOEF;
/* Derived data constructed for each Huffman table */
typedef struct {
unsigned int ehufco[256]; /* code for each symbol */
char ehufsi[256]; /* length of code for each symbol */
/* If no code has been allocated for a symbol S, ehufsi[S] contains 0 */
} c_derived_tbl;
/* Expand a Huffman table definition into the derived format */
EXTERN(void) jpeg_make_c_derived_tbl(j_compress_ptr cinfo, boolean isDC,
int tblno, c_derived_tbl **pdtbl);
/* Generate an optimal table definition given the specified counts */
EXTERN(void) jpeg_gen_optimal_table(j_compress_ptr cinfo, JHUFF_TBL *htbl,
long freq[]);