Android JPEG compression those things

Android JPEG compression those things

JPEG basics

JPEG (Joint Photographic Experts Group, Joint Photographic Experts Group) is a widely used lossy compression standard method .

For picture files compressed in the JPEG format, the most commonly used extension format is .jpg, and other commonly used extensions include .JPEG, .jpe, .jfif, and .jif.

JEPG coding principle

Although JEPG files can be encoded in various ways, the most common is to use JFIF encoding. The encoding process includes the following steps:

  • Color space conversion

    Convert the image from RGB to different color spaces of Y'CbCr . The Y'component represents the brightness of the pixel, and Cb and Cr represent the "color difference value" (divided into blue and red components). The Y'CbCr color space allows for greater compression without significant impact on the perceived image quality.

    About "Color Difference"

    The concept of "chromatic aberration" originated in the television industry. The earliest televisions were black and white. At that time, the transmission of television signals only needed to transmit the luminance signal, that is, the Y signal. After the advent of color television, people added two more to the Y signal. The color difference signal is used to transmit color information. The purpose of this is to be compatible with black and white TVs, because black and white TVs only need to process the Y signal in the signal.

    According to the principle of three primary colors, people find that the brightness contributed by the three colors of red, green and blue are different. Green has the largest "brightness" and blue is the darkest. Let the share of brightness contributed by red be KR, and the share contributed by blue as KB, then the brightness is

    According to experience, KR=0.299, KB=0.114, then

    The color difference between blue and red is defined as follows

    Finally, the mathematical formula for converting RGB to YCbCr is

  • Downsampling

    Compared with the color of the image (Cb and Cr components), the human eye is more sensitive to the brightness (Y' component) of the image in terms of image fineness.

    For the human eye, the change of brightness in the image is more easily perceivable, which is caused by the structure of the human eye. There are two types of photoreceptor cells on the retina, rod cells that can perceive changes in brightness, and cone cells that can perceive color. Since rod cells are much larger in number than cone cells, we are more likely to perceive light and dark details. For example, the picture below

    Keep only Y'component

    Only keep the Cb component

    Keep only the Cr component

    Using this feature, the Y'CbCr color space can be further down-sampled, that is, to reduce Cb and Cr , the spatial resolution of components .

    The down-sampling rate "4:4:4" means no down-sampling;

    The downsampling rate is "4:2:2", which means a reduction of 2 times in the horizontal direction

    The downsampling rate is "4:2:0", which means a reduction of 2 times in the horizontal and vertical directions (most commonly used)

    The downsampling rate is usually expressed as a three-part ratio

    j:a:b
    , If there is transparency, it is divided into four parts, which describes the number of luminance and chrominance samples in a conceptual area of j pixels wide and 2 pixels high.

    • j represents the horizontal sampling rate reference (the width of the concept area)
    • a represents the color difference sampling of the first line (Cr, Cb)
    • b represents the color difference sampling (Cr, Cb) change of the second line and the first line

  • Block partition

    After downsampling, each channel must be divided into 8x8 pixel blocks, and the minimum coding unit (MCU) depends on the downsampling used.

    If the downsampling rate is "4:4:4", the size of the minimum coding unit block is 8x8;

    If the downsampling rate is "4:2:2", the size of the minimum coding unit block is 16x8;

    If the downsampling rate is "4:2:0", the size of the minimum coding unit block is 16x16;

    If the channel data cannot be cut into blocks of integer multiples, it is usually filled with a solid color, such as black.

  • Discrete Cosine Transform

  • Quantify

    The human eye is good at seeing small brightness differences in relatively large areas, but cannot distinguish the exact intensity of high-frequency brightness changes well. This allows people to greatly reduce the amount of information in high-frequency components. Just divide each component in the frequency domain by the constant of that component, and then round (lossy operation) to the nearest integer.

    This step is irreversible

  • Use a lossless algorithm ( a variant of Huffman coding ) to further compress all 8 8 blocks of data.

JEPG compression effect

imageQuality ([1,100])Size (bytes)Compression ratio
Highest quality (100)814472.7:1
High quality (50)1467915:1
Medium quality (25)940723:1
Low quality (10)478746:1
Lowest quality (1)1523144:1

JEPG coding implementation

  • libjpeg

    The widely used C library is used to read and write JPEG image files.

  • libjpeg-turbo

    High-performance JEPG image decoder, using SIMD instructions to accelerate the compression and decompression of JEPG files on x86, x86-64, Arm and PowerPC systems, and progressive compression on x86, x86-64 systems .

    On x86 and x86-64 systems, the speed of libjpeg-turbo is 2-6 times that of libjpeg , and on other systems, it can also be much better than libjpeg.

Android image decoding

To display an image on Android, it is necessary to decode the image into a Bitmap object . Bitmap represents a collection of image pixels. The memory size of the pixel depends on the Bitmap configuration. Currently, Android supports the following configurations:

  • ALPHA_8

    Store only the transparency channel

  • ARGB_4444

    Each pixel uses 2 bytes of storage

  • ARGB_8888

    Each pixel uses 4 bytes of storage (default)

  • HARDWARE

    Special configuration, Bitmap data is stored in dedicated graphics memory (Native)

  • RGBA_F16

    Each pixel uses 8 bytes of storage

  • RGB_565

    Each pixel uses 2 bytes of storage, with only RGB channels.

Source code analysis

Usually we can call

BitmapFactory.decodeStream
The method is decoded from the image stream, the Java layer is just a simple entry, and the relevant implementations are all in the Native layer
BitmapFactory.doDecode
Method.

//frameworks/base/libs/hwui/jni/BitmapFactory.cpp 
static jobject doDecode (JNIEnv* env, std::unique_ptr<SkStreamRewindable> stream, jobject padding, jobject options, jlong  inBitmapHandle,jlong  colorSpaceHandle)  {
   //...  
}
Copy code

1. Initialize related parameters

  • sampleSize

    Sampling Rate

  • onlyDecodeSize

    Whether to decode only the size

  • prefCodeType

    Preferred color type

  • isHardware

    Whether to store in dedicated image memory

  • isMutable

    Variable

  • scale

    Zoom factor

  • requireUnpremultiplied

    Whether the color channel does not need to be "pre-multiplied" the transparent channel

  • javaBitmap

    Reusable Bitmap

2. Create a decoder

According to the decoded image format, create different decoder SkCodec.

Image formatSkCodec
JPEGSkJpegCodec
WebPSkWebpCodec
GifSkGifCodec
PNGSkPngCodec

SkCodec is responsible for the core implementation, and SkAndroidCodec is a wrapper class of SkCodec to provide some Android-specific APIs. Similarly, SkAndroidCodec creates different SkAndroidCodec based on the image format.

Image formatSkAndroidCodec
JPEG, PNG, GifSkSampledCodec
WebPSkAndroidCodecadapter

Three. Create a memory allocator

According to whether there is a reusable Bitmap and whether it needs to be scaled, a different memory allocator Allocator is used.

Four. Allocate pixel memory

transfer

SkBitmap.tryAllocPixels
The method attempts to allocate the required pixel memory. The following conditions may cause the allocation to fail.

  • Java Heap OOM
  • Native Heap OOM
  • The reusable Bitmap used is too small

Five. Perform decoding

transfer

SkAndroidCodec.getAndroidPixels
The method begins to perform the encoding operation.

SkCodec::Result SkAndroidCodec::getAndroidPixels ( const SkImageInfo& requestInfo,
         void * requestPixels, size_t requestRowBytes, const AndroidOptions* options)  {
   //... 
return  this -> onGetAndroidPixels (requestInfo,requestPixels,requestRowBytes,*options);
}
Copy code

SkAndroid.onGetAndroidPixels
There are two implementations of the method, namely SkSampledCodec and SkAndroidCodecadapter.

Here we take JPEG image decoding as an example. From the above, it uses SkSampledCodec and SkJpegCodec, and SkJpegCodec is the core implementation.

In addition to supporting the use of BitmapFactory for complete decoding, Android also supports the use of BitmapRegionDecoder for partial decoding, which is particularly useful when dealing with very large images.

Android JPEG compression

Android has always had a problem with image compression. For image files of the same size, the iOS display is always more delicate, that is, the compression effect is better. For a more detailed discussion on this issue, you can read this article: github .com/bither/bith...

In general, it is an open source 2D rendering engine Skia maintained by the bottom layer of Android. Skia relies on the libjpeg library to decode JPEG image files. The libjpeg compression parameter is called optimize_coding . This parameter is TRUE, which can bring A better compression effect will also consume more time.

Below 7.0, Google sets this value to FALSE in order to be compatible with devices with poor performance. For 7.0 and above, it has been set to TRUE.

Regarding optimize_coding to FALSE, for more discussion, please see groups.google.com/g/skia-disc...

Below 7.0: androidxref.com/6.0.1_r10/x...

7.0 and above: androidxref.com/7.0.0_r1/xr...

Therefore, the current mainstream approach is that in versions below 7.0, JPEG image file compression can be implemented based on libjpeg-turbo.

Source code analysis

You can call

Bitmap.compress
Method to perform image compression, optional configurations are:

  • format

    Compressed image formats include JPEG, PNG, WEBP.

  • quality

    Compression quality, optional values are 0-100.

Similarly, the Java layer only provides API entry, and the implementation is still in the Native layer

Bitmap.Bitmap_comperss()
method.

//framework/base/libs/hwui/jni/Bitmap.cpp 
static jboolean Bitmap_compress (JNIEnv* env, jobject clazz, jlong  bitmapHandle,jint format, jint quality, jobject jstream, jbyteArray jstorage)  {
}
Copy code

1. Create an encoder

Create different encoders according to the image format.

Image formatEncoder
JPEGSkJpegEncoder
PNGSkPngEnccoder
WebPSkWebpEncoder

2. Set coding parameters

Android JPEG decoding is dependent on libjpeg and libjpeg-turbo .

Before starting compression encoding, a series of parameters will be set.

  • Image size

  • Color type

    Commonly used color types are:

    JCS_EXT_BGRA,/* blue/green/red/alpha */
JCS_EXT_BGRA,/* blue/green/red/alpha */
Copy code

  • Downsampling rate

    Currently Android supports "4:2:0" (default), "4:2:2" and "4:4:4".

  • Best Huffman Code Table

    The default is true, which means to use the best Huffman coding table, although it will reduce the compression performance, but improve the compression efficiency.

    //Tells libjpeg-turbo to compute optimal Huffman coding tables 
//for the image. This improves compression at the cost of
//slower encode performance.
fCInfo.optimize_coding = TRUE;
Copy code

  • quality

    This parameter will affect the "quantization" step in JPEG encoding

Three. Perform coding

//external/skia/src/imagess/SkImageEncoder.cpp 
bool  SkEncoder::encodeRows ( int numRows)  {
	 //... 
  this -> onEncodeRows (numRows);
}
Copy code

JPEG image encoding is implemented by SkJpegEncoder.

//txternal/skia/src/images/SkJpegEncoder.cpp 
bool  SkJpegEncoder::onEncodeRows ( int numRows)  {
	 //... 
  for ( int i = 0 ; i <numRows; i++) {
     //Perform libjpeg-turbo encoding operation 
  	jpeg_write_scanlines (fEncoderMgr-> cinfo (), &jpegSrcRow, 1 );
  }
}
Copy code

Sampling algorithm

When adjusting the size of an image, it is necessary to reprocess the original image pixel data, which is called image sampling processing.

At present, Android supports two sampling algorithms, Nearest neighbor (proximity sampling) and Bilinear (bilinear sampling) by default .

  • Nearest neighbor

    Each pixel in the resampled raster gets the same value as the nearest pixel in the original raster. This processing time is the fastest, but it can also cause jagged images.

  • Bilinear (bilinear sampling)

    Each pixel in the resampled raster is the result of the weighted average of the 2x2 4 nearest pixels in the original raster.

In addition to the above two, there are better algorithms with the following effects:

  • Bicubic (bicubic sampling)

    Each pixel in the resampled raster is the result of the weighted value of the 4x4 16 nearest pixel values in the original raster, and the closer pixels will have a higher weight.

  • Lanczos

    High-order interpolation algorithm, it considers more surrounding pixels and retains the most image information.

  • Magic Kernel

    Fast and efficient, but can produce surprisingly clear and sharp results. More detailed introduction: www.johncostella.com/magic/.

Spectrum

Spectrum is Facebook's open source cross-platform image transcoding dependency library. Compared with the default jpeg-turbo that comes with the Android system, it has the following advantages:

  • JPEG encoding is based on mozjpeg . Compared with jpeg-turbo, it improves the compression rate, but also increases the compression processing time.
  • Support Bicubic (bicubic sampling) and Magic Kernel sampling algorithms.
  • The core is implemented using CPP, which can achieve consistent compression effects on both Android and iOS platforms.
  • Support more custom configuration, including chroma sampling mode and so on.

Benchmarks

Based on google/butteraugli to compare the quality difference between the original image and the compressed image , the smaller the value, the better.

Device information: Huawei P20 Pro, Android 10

A Compression quality 80

coreCompression qualityChroma sampling modeQuality differenceFile sizetime consumingCompression ratio
Original image-S444-8.7MB--
jpeg-turbo80S4442.9433522.5MB2255ms71%
mozjpeg80S4442.4862662.8MB3567ms67%
mozjpeg80S4202.493475 (-15%)2.3MB2703ms73%(+2%)

B Compression quality 75

coreCompression qualityChroma sampling modeQuality differenceFile sizetime consumingCompression ratio
Original image-S444-8.7MB--
jpeg-turbo75S4443.0758842.3MB2252ms73%
mozjpeg75S4442.6989832.4MB3188ms72%
mozjpeg75S4202.670076 (-13%)2MB2470ms77%(+4%)

C Compression quality 70

coreCompression qualityChroma sampling modeQuality differenceFile sizetime consumingCompression ratio
Original image-S444-8.7MB--
jpeg-turbo70S4442.7397942.1MB2230ms75%
mozjpeg70S4442.8385952.2MB3089ms74%
mozjpeg70S4202.810702(+2%)1.8MB2404ms79%(+4%)

D Compression quality 65

coreCompression qualityChroma sampling modeQuality differenceFile sizetime consumingCompression ratio
Original image-S444-8.7MB--
jpeg-turbo65S4443.7341051.9MB2227ms78%
mozjpeg65S4443.1777062MB2775ms77%
mozjpeg65S4203.251182(-12%)1.6MB2116ms81%(+3%)

E Compression quality 60

coreCompression qualityChroma sampling modeQuality differenceFile sizetime consumingCompression ratio
Original image-S444-8.7MB--
jpeg-turbo60S4444.5269811.8MB2189ms79%
mozjpeg60S4443.4863471.8MB2454ms79%
mozjpeg60S4203.479777 (-23%)1.5MB2035ms82%(+3%)

From the above data, we can see that compared with jpeg-turbo + S444 using mozjpeg + S420, the compression rate is improved by 3% on average, and the image quality is improved by 12%.

Reference link

www.cnblogs.com/Arvin-JIN/p...

zh.wikipedia.org/wiki/JPEG

scc.ustc.edu.cn/zlsc/sugon/...

www.robertstocker.co.uk/jpeg/jpeg_n...

libjpeg.sourceforge.net/

libjpeg-turbo.org/

github.com/bither/bith...

groups.google.com/g/skia-disc...

mp.weixin.qq.com/s/H9Tz1n4O2...

abraia.me/docs/image-...