Commit bb1d540a807783a3db8b85bab2993d70b1330287
2024-09-05T12:00:04
TJ: Explicitly reject OOB lossless crop regions tj*Transform() relied upon the underlying transupp API to check the cropping region. However, transupp uses unsigned integers for the cropping region, whereas the tjregion structure uses signed integers. Thus, casting negative values from a tjregion structure produced very large unsigned values. In the case of the left and upper boundary, this was innocuous, because jtransform_request_workspace() rejected the values as being out of bounds. However, jtransform_request_workspace() did not always reject very large width and height values, because it supports expanding the destination image by specifying a cropping region larger than the source image. In certain cases, it allowed those values, and the libjpeg memory manager subsequently ran out of memory. NOTE: Prior to this commit, image expansion technically worked with tj*Transform() as long as the cropping width and height were valid and automatic JPEG buffer (re)allocation was used. However, that behavior is not a documented feature of the TurboJPEG API, nor do we have any way of testing it at the moment. Official support for image expansion can be added later, if there is sufficient demand for it. Similarly, this commit modifies tj3SetCroppingRegion() so that it explicitly checks for left boundary values exactly equal to the scaled image width and upper boundary values exactly equal to the scaled image height. If the specified cropping width or height was 0 (which is interpreted as {scaled image width} - {left boundary} or {scaled image height} - {upper boundary}), then such values caused a cropping width or height of 0 to be passed to the libjpeg API. In the case of the width, this was innocuous, because jpeg_crop_scanline() rejected the value. In the case of the height, however, this caused unexpected and hard-to-diagnose errors farther down the pipeline.