All about codecs: how video and sound are squeezed into a file.

A codec is a method for squeezing video or sound into far less space than it naturally takes, and for unpacking it again at the other end. The word is short for coder–decoder. H.264, H.265, AV1, MPEG-2 and DivX are video codecs; AAC, MP3 and Dolby Digital are audio codecs. Which one a file uses decides three things: how good it looks and sounds, how big it is, and which devices can play it. It is a separate matter from the file type. MKV, MP4 and AVI are containers, the boxes that codecs travel in.

What a codec is

Every codec has two halves. The encoder takes raw pictures or sound and produces a compact stream of data. The decoder takes that stream and rebuilds pictures or sound from it. Your camera, your capture software and every streaming service run encoders. Your phone, TV and media player run decoders.

What makes something “a codec” in the sense of H.264 is a published specification of what the compressed stream looks like and exactly how to decode it. That is deliberately all it specifies. How to produce a good stream is left open, so encoders compete, and they differ a great deal. This is why three words that sound alike mean different things:

So a file can be “an MKV” (container), holding “H.264” (codec), made by “x264” (encoder), and all three statements are true at once. It is also why two H.264 files of the same size can look quite different: a patient software encoder will find a better way to spend the same number of bits than a chip that has to keep up with a live camera.

Why video needs one: the arithmetic

A digital picture is a grid of pixels, and each pixel is three numbers: how much red, green and blue. Video is 25 or more of those grids every second. Multiply it out for high definition:

1,920 × 1,080 pixels × 3 bytes × 25 frames = 155 megabytes every second, or about 560 GB an hour.

A film in high definition from a streaming service arrives at roughly 5 megabits a second, which is a little over 2 GB an hour. That is about 250 times smaller, and on a decent screen most people are satisfied with the result. Nothing else in everyday computing is compressed anywhere near that hard. A ZIP file manages perhaps two to one on a document.

Standard definition, the size of everything that came off a tape, is smaller but tells the same story:

• Uncompressed as captured75 GB
• Lossless HuffYUV, FFV1about 30 GB
• DV camcorder tape13 GB
• MPEG-2 as on a DVDabout 2.2 GB
• H.264 2.5 Mbit/sabout 1.1 GB

The four tricks every video codec uses

From MPEG-1 in 1993 to AV1 today, every mainstream video codec is built from the same four ideas. Newer codecs do each one more cleverly; none has replaced them.

Trick 1: store the colour at lower detail

Your eye is far better at seeing fine detail in brightness than in colour. So the first thing a codec does is stop storing red, green and blue, and store instead one brightness value per pixel plus two colour values that say how far the pixel leans towards blue and towards red. Then it keeps the brightness at full resolution and the colour at a quarter of it: one colour for each 2 × 2 block of pixels.

This is called chroma subsampling, and the arrangement above is written 4:2:0. Nearly all video made for watching uses it: DVD, Blu-ray, broadcast, streaming, phones. Professional and editing formats often use 4:2:2 (colour halved across but not down) and occasionally 4:4:4 (no reduction). The idea is older than digital video. Analogue colour television gave colour less room than brightness for the same reason, and VHS gave it less still.

Trick 2: describe each frame as broad shapes plus fine detail, then drop detail nobody will miss

The codec cuts the frame into small square blocks. For each block it runs a piece of mathematics (a transform, usually the discrete cosine transform) that re-describes the block not as a grid of pixel values but as a recipe: this much flat colour, this much gentle gradient, this much fine texture. It is the same idea as the frequency bands in an MP3, applied to a picture.

Described that way, most blocks turn out to be mostly broad shapes with only a little fine texture. Now the encoder rounds the numbers off, coarsely for the fine texture and carefully for the broad shapes. Many of the fine-detail numbers round to zero and vanish. This step is called quantisation, and it is where the quality is actually lost. Turn the rounding up and the file shrinks and the picture softens; turn it up too far and the edges of the blocks themselves start to show.

This is exactly how a JPEG photograph works. A codec that did only tricks 1 and 2 on every frame would be a stack of JPEGs, and that codec exists: it is called Motion JPEG.

Trick 3: do not store what has not changed

This is the big one, and it is what separates video compression from picture compression. Consecutive frames are nearly identical. In a shot of someone talking, the wall behind them is the same in every frame for a minute.

So instead of storing each frame, the encoder predicts it from frames it has already stored. For every block it searches the earlier frame for the best match and writes down only two things: where the match was (“the same as the block 12 pixels to the left in the previous frame”), and the small difference between the match and the real thing. Because it can follow things as they move, this is called motion compensation. The differences are then put through trick 2.

The saving is enormous when the picture is predictable and small when it is not. A locked-off shot of a wall costs almost nothing. Rain, confetti, rippling water, film grain and the snowy noise of a worn VHS tape are the hard cases, because random detail cannot be predicted from anything. That is why a noisy tape makes a bigger file than a clean one at the same quality setting.

Trick 4: write common things briefly

What is left after the first three tricks is a long list of small numbers, many of them zero. The final step packs that list with a conventional lossless compressor, giving short codes to common values and long codes to rare ones. Nothing is lost here; it is the same family of technique that FLAC uses on audio, and we explain it in the audio formats lesson.

Put together, an encoder’s work on each block looks like this:

1. Predict the block from a neighbouring block or an earlier frame.
2. Subtract to get the part the prediction missed.
3. Transform that leftover into broad shapes and fine detail.
4. Round off the numbers. This is the lossy step.
5. Pack what remains as tightly as possible.

Take out step 4 and you have a lossless video codec: prediction and packing only, every pixel recovered exactly, files a third to a half the size of uncompressed.

I, P and B frames: why video behaves the way it does

Trick 3 means frames are not all stored the same way. There are three kinds:

• I-frames (intra) are complete pictures, compressed on their own like a JPEG. They are also called keyframes. They are large.
• P-frames (predicted) store only what changed from an earlier frame. Much smaller.
• B-frames (bidirectional) borrow from a frame before and a frame after. Smallest of all.

Almost every quirk of digital video follows from this arrangement:

• Skipping lands on particular spots. A player can only start decoding at a keyframe. When you drag the slider it either jumps to the nearest one or pauses while it decodes forward from it.
• You can only cut cleanly at a keyframe. Tools that trim a video without re-encoding it will snap your cut to the nearest one, because the frames after any other point depend on frames you just removed.
• Damage spreads. An error in one frame is inherited by every frame predicted from it, until the next keyframe wipes the slate. That is the smear of melting blocks trailing behind a moving object when a TV signal breaks up.
• Frames are stored out of order. A B-frame needs a later frame decoded first, so the file holds frames in decoding order, not viewing order. The container has to keep track of both, which is one of the things AVI was never designed to do.
• Editing codecs skip the whole idea. DV, Motion JPEG, ProRes and the lossless capture codecs make every frame a keyframe. The files are far bigger, but any frame can be reached, cut or replaced instantly, which is what an editor needs.

Bitrate: the quality dial

Bitrate is the number of bits the encoder may spend on each second of video, and it is the main thing that determines both file size and quality. File size is simply bitrate multiplied by duration.

It is not the same thing as resolution. Resolution is how many pixels there are; bitrate is how much information is available to describe them. A 4K stream starved of bits looks worse than a well-fed high-definition one, because the encoder has four times the pixels to describe and has to round all of them off harder.

There are three ways to tell an encoder how much to spend:

• Constant bitrate. The same number of bits every second, whether the scene needs them or not. Used for live broadcast and streaming, where the pipe has a fixed width.
• Average bitrate. A target for the whole file; the encoder moves bits from easy scenes to hard ones. Used when the result must fit a known size.
• Constant quality. You name a quality level and the encoder spends whatever each scene needs to reach it. This is the right choice for a file you are keeping, because you care about how it looks, not what it weighs. In the x264 and x265 encoders the setting is called CRF: lower numbers mean better quality, around 18 to 23 is the usual range, and every 6 steps roughly halves or doubles the file size.

When there are too few bits, a codec fails in recognisable ways:

• Blocking: visible squares, worst in fast movement and dark scenes.
• Banding: a smooth sky or a dim wall breaks into stripes.
• Mosquito noise: a shimmer around sharp edges and lettering.
• Smearing: fine texture such as grass, hair and skin is wiped smooth, so faces look waxy.

Lossless, lossy, and the two kinds in between

The rule from audio applies without change. Lossy compression is a one-way door. What the rounding step discarded is gone, and converting a lossy file to a “better” codec or a higher bitrate cannot bring it back; it only runs the rounding a second time on a picture that has already been through it. Every lossy re-encode is a photocopy of a photocopy. Keep the best version you have, and make every other copy from that one.

The video codecs you will actually meet

Most of these have two names, because two standards bodies (the ITU, which numbers things H.26x, and the MPEG group) developed them jointly and each published under its own title.

A word on “MPEG-4”, which causes more confusion than any other label. It is a family of standards, and people use the name for three different members: the DivX-era codec (Part 2), H.264 (Part 10), and the MP4 container (Part 14). “An MPEG-4 file” could mean any of them.

The pattern across the generations is consistent: each new codec gives about the same picture in roughly half the bits of the one before, and demands several times the computing power to do it.

• MPEG-2 1995100
• H.264 2003about 50
• H.265, VP9 2013about 25–30
• AV1 2018about 20–25

The audio codecs

Sound in a video file is compressed separately, by its own codec. The principles are covered in WAV, FLAC and MP3; these are the names you will see beside a video stream.

Why a file plays on one device and not another

Decoding video is heavy work, and three things decide whether a particular device can do it.

Whether there is a chip for it. Phones, TVs and laptops contain dedicated circuits that decode specific codecs. That is how a phone plays hours of video on a small battery. The catch is that the list of codecs is fixed when the chip is made. A television from the mid-2010s decodes H.264 and has never heard of AV1, and no update will change that. A computer can fall back to decoding in software on its main processor, with the fan running; a TV usually cannot.

Which corner of the codec the file uses. A codec standard is a big toolbox, divided into profiles (which tools are in use) and levels (how large and how fast the picture may be). A device supports some of them. H.264 at “High profile, level 4.1” covers high definition at up to 30 frames a second and plays on nearly everything; the same codec with 10-bit colour plays on very little. So “it is H.264” is not quite a guarantee, though with ordinary settings it is close to one.

Whether somebody paid for it. Most codecs are covered by patents, and a manufacturer pays royalties to include a decoder. H.264’s terms were clear and it went everywhere. H.265’s were split between several competing patent groups, which slowed its adoption, left it missing from some browsers and operating systems, and led directly to AV1: a codec designed by the companies who stream the most video, so that nobody would have to pay anyone.

Do I need to install a codec?

Almost certainly not. In the 2000s, Windows played only what it had a decoder installed for, and people collected decoders in “codec packs” that often fought with each other. That era is over. A free player such as VLC carries its own decoders for practically every codec in this article and needs nothing installed beside it.

The habit has left one lasting hazard. A website that tells you to download a codec, or a special player, before you can watch its video is a long-standing way of delivering malware. No legitimate video needs it. Close the page.

Which codec to use

• To share, or to play on anything: H.264 video with AAC sound, in an MP4. It is twenty years old and that is the point; every device made in that time plays it.
• To keep home video, including tape captures: H.264 at a generous bitrate, in an MKV. “Generous” means a constant-quality setting around CRF 18 to 20, or several megabits a second for standard definition. The file is a few gigabytes an hour and visually very close to the source.
• To capture footage you plan to restore or re-edit: a lossless codec such as FFV1 or HuffYUV, if you have roughly 30 GB an hour to give it. Clean-up tools keep improving, and a lossless capture can be put through them again later with nothing lost in between.
• For 4K, or when space is tight: H.265, provided you know the devices that need to play it can. AV1 where everything involved is recent.
• For editing: let the editing program use its intermediate codec, and export to H.264 at the end.
• Do not re-encode files you already have just to modernise them. Converting an old DivX or MPEG-2 file to H.265 makes it smaller and slightly worse. Storage is cheap; the quality does not come back.

Common questions