Commit 1f3614f167d8658a9df88dfe351ef5e751fc0461
2025-10-08T10:42:18
TJ: Guard against reused JPEG dst buf w/0 buf size The libjpeg in-memory destination manager has always re-allocated the JPEG destination buffer if the specified buffer pointer is NULL or the specified buffer size is 0. TurboJPEG's destination manager inherited that behavior. Because of fe80ec22752cce224c55d7b429d46503634ef034, TurboJPEG's destination manager tries to reuse the most recent destination buffer if the same buffer pointer is specified. (The purpose of that is to enable repeated invocations of tj*Compress*() or tj*Transform() to automatically grow the destination buffer, as needed, with no intervention from the calling program.) However, because of the inherited code, TurboJPEG's destination manager also reallocated the destination buffer if the specified buffer size was 0. Thus, passing a previously-used JPEG destination buffer pointer to tj*Compress*() or tj*Transform() while specifying a destination buffer size of 0 confused the destination manager. It reallocated the destination buffer to 4096 bytes but reported the old destination buffer size to the libjpeg API. This caused a buffer overrun if the old destination buffer size was larger than 4096 bytes. The documentation for tj*Compress*() is contradictory on this matter. It states that the JPEG destination buffer size must be specified if the destination buffer pointer is non-NULL. However, it also states that, if the destination buffer is reused, the specified destination buffer size is ignored. The documentation for tj*Transform() does not specify the function's behavior if the destination buffer is reused. Thus, the behavior of the API is at best undefined if a calling application attempts to reuse a destination buffer while specifying a destination buffer size of 0. If that ever worked, it only worked in libjpeg-turbo 1.3.x and prior. This issue was exposed only through API abuse, and calling applications that abused the API in that manner would not have worked for the last 11 years. Thus, the issue did not represent a security threat. This commit merely hardens the API against such abuse, by modifying TurboJPEG's destination manager so that it refuses to re-allocate the JPEG destination buffer if the buffer pointer is reused and the specified buffer size is 0. That is consistent with the most permissive interpretation of the TurboJPEG API documentation. (The API already ignored the destination buffer size if the destination buffer pointer was reused and the specified buffer size was non-zero. It makes sense for it to do likewise if the specified buffer size is 0.) This commit also modifies TJUnitTest so that it verifies whether the API is hardened against the aforementioned abuse.