Having a basic understanding of compression is critical when analyzing, and testifying about, video evidence. Let’s have a quick go of explaining it in simple terms, using a Dilbert comic strip as the foundation.

Firstly, what is compression? It’s a set of algorithms that function to reduce the file size of a video. If a 30 FPS video was a true compilation of 4k RAW photographs, a 5-minute video would be about 270 GB. Not practical. Compression algorithms implement spatial and temporal compression to dramatically reduce file size. Let’s toss Dilbert in here to help:

Courtesy of scott adams

Spatial compression is applied to a single frame (intraframe). Think of it like taking a big RAW photograph and converting it to a JPG, which is a much smaller file. Some detail is lost, but the image looks largely the same because of some brilliant tricks.

For instance, less important data is discarded. Things like fine shading, slight variations in color, or patterns. So, maybe Pointy-Haired Boss’ tie is checkered, and we’d see that in a RAW image, but the compression algorithm turned it all blue. Or maybe a RAW image would have shown the individual hairs in the back of that guy’s head, but to minimize data, the whole area was just called one color.

More technically speaking (skip this section if you just wanted Friday fun), the algorithm works like this:

  1. Convert all RGB values to YCbCr (Luminance, Chrominance blue, Chrominance red). Luminance is essentially brightness, Cb is how blue or yellow a pixel is, and Cr is how red or green a pixel is.

  2. Now we simplify with Chroma Subsampling. 4:2:0 is the most common form we’ll see in forensics. This means for a four-wide by two-tall block of pixels that all have a unique color, all (4) pixels will get their own luminance value in each row, but only two (2) pixels in the top row will get their own color, and no (0) pixels in the bottom row will have their own color. Instead, those colors will be copied from the row above. The figure below does a great job of showing how that works.

  3. A discrete cosine transformation [DCT] is applied. Here, macroblocks (blocks of pixels, say 16 by 16) are analyzed and details are summarized into a list of patterns that are based on cosine waves, some big and smooth and some small and more detailed.

  4. Coefficients created during the DCT phase are then divided by a number and rounded off during Quantization. This process is applied more heavily to colors, as humans are more sensitive to brightness, which is preserved with gentler quantization.

  5. The resulting quantized coefficients are then encoded into a compressed bitstream.

Credit: Tobia Montanari. No subsampling in top row, 4:4:4. Most common subsampling 4:2:0.


Temporal Compression is the other key method. Where spatial compression is applied within a single frame, this is applied between frames (interframe). A full keyframe (I-frame) is written with all fresh pixels, and then the algorithm looks for what changed and applies resources there. So neighboring frames (P or B-frames) will have pixels that remained unchanged from the I-Frame, that were simply moved within that original I-Frame, or were written anew.

Back to Dilbert, the desk and the heads in the foreground don’t change. Those pixels can be held constant throughout all three frames, saving a lot of space. Between panels 1 and 2, the pixels forming Pointy-Haired Boss’ arm can just be moved down. His eyes opened wider though, so we’d need some new data there.

Codecs (COder, DECoder), like H.264, H.265, or AV1, are responsible for the processes above. They're basically recipes for how to compress and decompress a video. When you watch a video, the player uses the codec to unpack the file, form the images, and play it back with the appropriate composition. What a trip.

There’s a lot to it, but hopefully this helps you understand the basics. For more on the topic, check out my podcast with guru Mark Crouch, his and Stephen Cash's class, or books like How Video Works.

Thanks for reading, keep exploring!

Lou Peck
Lightpoint | JS Forensics

P.S. Axiom Forensic has been acquired by Jeff Suway's group, JS Forensics. I'm honored to join his talented and innovative group. This does not impact Lightpoint. They're distinct entities as discussed here.