Image and video processing

Overview

Video is the application FPGAs were built for, whether or not anyone planned it that way. A camera produces a fixed stream of pixels at a fixed rate, every pixel needs the same arithmetic, and the deadline never moves. That is a dataflow problem, and dataflow is what an array of registers and multipliers does well. This hub is for engineers who need to get pixels in, transform them and get them out again, with the arithmetic to know whether a given device can keep up.

The work splits into three layers. Interfaces (MIPI CSI-2 and DSI, HDMI, DisplayPort, SDI) move pixels on and off the chip. A streaming pipeline, usually built on AXI4-Stream or the Altera equivalent, transforms them one line at a time. A memory layer, through a video DMA engine into DDR, holds whole frames when an operation needs random access or when input and output rates differ.

Key ideas

Interfaces are mostly serdes problems

MIPI CSI-2 (camera in) and DSI (display out) run over D-PHY or C-PHY lanes at up to 2.5 Gb/s per D-PHY lane in the common versions. Older FPGAs need an external bridge chip. Newer families such as Lattice CrossLink-NX, Efinix Titanium and AMD UltraScale+ and Versal have hard or near-hard D-PHY blocks. HDMI up to 1.4 uses TMDS at up to 340 MHz pixel clock, which many FPGA I/O banks can drive directly with a 10:1 serialiser. HDMI 2.0 and above, DisplayPort and 12G-SDI need true gigabit transceivers, so they are covered on /topics/high-speed-io/ as well as here. Whatever the physical layer, the first thing you build is a converter to the internal streaming format, so that everything downstream is interface-agnostic.

Line buffers, not frame buffers, where you can

Most ISP stages (demosaic, denoise, sharpen, scaling, lens correction) need only a few neighbouring lines. A 3x3 kernel on a 1,920 pixel wide image needs two lines of history, which at 24 bits per pixel is 92,160 bits, under 3 block RAMs of 36 Kb. The stage then runs at one pixel per clock with a fixed latency of a couple of lines. A 4K line (3,840 pixels) doubles that, and 8-bit raw Bayer halves it. Only operations that need the whole frame (rotation, temporal filtering, frame-rate conversion, most compression) justify a trip through external memory. Keeping the pipeline streaming is the single biggest decision in a video design, because it sets block RAM use, latency and DDR bandwidth at once.

The bandwidth arithmetic

Write the numbers down before choosing a part. Pixel rate equals width times height times frame rate, plus blanking (typically 10 to 25 percent). 1080p60 is 1,920 x 1,080 x 60 = 124 Mpixel/s active, 148.5 MHz with blanking. At 24 bits that is 3.0 Gb/s per stream. 4K60 is four times that, 594 MHz pixel clock, which forces 2 or 4 pixels per clock in the fabric because the logic will not close timing at 594 MHz. Every frame-buffer pass costs that rate twice (write and read). A single 32-bit DDR4-2400 channel offers 76.8 Gb/s peak and perhaps 60 to 70 percent of it in practice with mixed traffic, so two 4K60 streams through a buffer, plus a processor, is already tight.

Compression is usually bought, not written

JPEG encoders are small and plenty of open designs exist. H.264, H.265 and AV1 are different: a conforming encoder is tens of thousands of lines of RTL and years of tuning. AMD ships a hard VCU (H.264/H.265 up to 4K60) in some Zynq UltraScale+ MPSoC EV parts and Versal AI Edge, and Altera, Lattice and third parties sell soft encoder IP. Soft H.265 IP commonly needs hundreds of thousands of LUTs for 4K. If your product needs standard codecs, choose the device for its codec story first and its LUT count second.

Where to start

Video pipelines are mostly counters, FIFOs and block RAM wrapped around small arithmetic, so the beginner track covers the right foundations. Read /learn/beginner/04-registers-clocks-and-state/ for the pipeline register model, /learn/beginner/08-counters-timers-and-debouncing/ because a video timing generator is two counters, and /learn/beginner/09-memories-block-ram-and-fifos/ for the line buffer. A first project is a VGA or HDMI test-pattern generator: a timing generator, a pattern, and a serialiser, no memory at all. An intermediate track on streaming protocols and VDMA is coming.

Tools and vendors

The open-source world has usable HDMI TX cores for the TMDS range, Litex video components, and SpinalHDL and Amaranth video examples. There is no open-source H.264 encoder you would ship.

Common mistakes

• Choosing a device before doing the pixel-rate and DDR arithmetic above, then discovering 4K needs multi-pixel-per-clock everywhere.
• Treating tuser (start of frame) and tlast (end of line) as optional. Downstream IP recovers frame alignment from them, and a missing tuser makes the VDMA roll the image by a line.
• Running the whole pipeline at the pixel clock when a 2x or 4x narrower bus at a higher fabric clock would halve the resources.
• Forgetting blanking when budgeting bandwidth, and forgetting refresh and page misses when budgeting DDR.
• Buffering a single frame. Three buffers (write, read, spare) are the minimum for tear-free output when input and output are asynchronous.
• Ignoring the sensor side. Most time on a camera project goes into the I2C configuration of the sensor, not the FPGA pipeline.

Going deeper

• AMD UG934, AXI4-Stream Video IP and System Design Guide, the de facto definition of tuser and tlast use in video streams.
• AMD PG020, AXI Video Direct Memory Access, with the frame-buffer and genlock modes explained.
• MIPI Alliance CSI-2 specification page, with the public overview of lanes, virtual channels and data types.
• HDMI Forum and VESA for the display standards (full text needs membership).
• Lattice CrossLink-NX product page, a good example of a device built around hard MIPI.