FPGA Guide
Topic hub

AI and machine learning on FPGA

How neural networks run on FPGAs, the difference between dataflow and overlay accelerators, the main vendor and open-source flows, and when an FPGA beats a GPU or NPU.

Updated 13 Nov 2026 · quantisation, hls4ml, FINN, Vitis AI, DPU, FPGA AI Suite, VectorBlox, sensAI

Overview

An FPGA runs a neural network the same way it runs anything else: as fixed arithmetic fed by a fixed memory schedule. That makes it a poor fit for training, where the workload changes every few weeks, and a good fit for inference at the edge, where one model runs for years, the input is a sensor stream, and latency, power or determinism matter more than peak throughput. This hub is for engineers deciding whether a model belongs on an FPGA at all, and if so which of the two very different ways of building the accelerator to use.

The honest summary at the time of writing: for large language models and anything else dominated by dense matrix multiplication at batch size, GPUs and dedicated NPUs win on throughput per watt and on software maturity. FPGAs win on small, latency-bound models with odd data types, on models fused directly into a sensor or network pipeline, and on anything that must run on a device already chosen for other reasons.

Key ideas

Quantisation is the whole game

Floating point costs an FPGA dearly. A 32-bit multiply-add needs a DSP slice plus a lot of logic, while an 8-bit multiply can be packed two per DSP on most families and a 2-bit or binary network needs no DSP at all, only LUTs. Quantisation is therefore the first design step, not an optimisation. INT8 with per-channel scaling loses under one percent accuracy on most convolutional networks. Below 8 bits you need quantisation-aware training, which the FINN and hls4ml flows support through Brevitas and QKeras. The reward is proportional: halving the bit width roughly doubles the compute you can fit and halves the memory bandwidth, which is usually the real limit.

Dataflow versus overlay

There are two architectures, and they suit different problems.

A dataflow accelerator compiles the network into hardware. Every layer becomes its own block, sized to the layer’s compute, connected by FIFOs, all layers running at once. hls4ml and FINN build these. Latency is microseconds or below, throughput is fixed at design time, and the whole model’s weights usually live on chip. The cost is that one bitstream serves one model, and the model has to fit. This is the right shape for trigger-level physics, anomaly detection on a link, keyword spotting and small classifiers.

An overlay accelerator is a fixed processor-like engine (a systolic array or vector unit with its own instruction stream) that the tools program per model. AMD’s DPU under Vitis AI, the Altera FPGA AI Suite, Microchip VectorBlox and the Lattice sensAI CNN accelerator are all overlays. Any supported model runs without rebuilding the bitstream, weights stream from DDR, and the compiler handles scheduling. The cost is a fixed architecture that cannot exploit sparsity or unusual layer types and spends energy moving data through a general datapath. This is the right shape for camera analytics, where models change and resolution is high.

Dataflowconv18 PEconv232 PEconv332 PEdense4 PEFIFO between every layer. Weights in LUT RAM or block RAM. Latency fixed, a few microseconds.Each PE count is chosen so every layer takes the same time per input.Overlaysystolic arraye.g. 16 x 16 INT8 MACfixed at build timeschedulerinstructionstreamDDRweights,activationsCompiler maps any supportedmodel onto the same bitstream.Bandwidth-bound at high res.
Dataflow (one block per layer, weights on chip) versus overlay (one fixed engine, instructions and weights from DDR)

When the FPGA wins, and when it does not

An FPGA is the better choice when the model is small enough for dataflow and latency is the specification, when the network sits inside a data path that is already in the FPGA (a radar receiver, a camera ISP, a packet classifier), when the data type is unusual (4-bit, ternary, custom activations) or when the product needs a 10 to 15 year supply life. It is the wrong choice when the model is a transformer with billions of parameters, when the team has no hardware engineers, when the model will change monthly, or when a 10 USD NPU-equipped microcontroller meets the target. Power comparisons are often quoted at the vendor’s favourite operating point, so measure your own model on a development board before committing.

Where to start

The arithmetic underneath every accelerator is a multiply-accumulate in a pipeline, so start with /learn/beginner/03-combinational-logic-and-luts/ and /learn/beginner/04-registers-clocks-and-state/, then /learn/beginner/09-memories-block-ram-and-fifos/ for the weight and activation storage. The /topics/dsp/ hub covers fixed-point arithmetic, which is the same skill. For a first accelerator, run the hls4ml tutorial on a small Keras model and look at the generated HLS, or run a Vitis AI example on a Kria KV260. An intermediate track on HLS and accelerators is coming.

Tools and vendors

Flow Type Owner Targets Notes
hls4ml Dataflow, HLS Fast Machine Learning community (open source) AMD, Altera, Catapult Keras, PyTorch, ONNX in. Best for small MLPs and CNNs
FINN Dataflow, RTL/HLS AMD research (open source) AMD Brevitas quantised models, down to 1 bit
Vitis AI with DPU Overlay AMD Zynq UltraScale+, Versal, Alveo Largest model zoo, INT8
FPGA AI Suite Overlay Altera Agilex, Arria 10, Cyclone 10 GX OpenVINO front end
VectorBlox Overlay Microchip PolarFire, PolarFire SoC Open SDK, aimed at low power
sensAI Overlay and compiled Lattice iCE40 UltraPlus, CrossLink-NX, Certus-NX Milliwatt-class always-on tasks
Efinity with TinyML Compiled, RISC-V plus accelerators Efinix Trion, Titanium TensorFlow Lite micro based

Achronix sells Speedster7t parts with machine-learning processor blocks aimed at the data centre, covered on /topics/data-centre-and-acceleration/.

Common mistakes

  • Starting from a floating-point model and hoping the tools will sort out precision. Quantise first, then measure accuracy, then build.
  • Comparing TOPS figures. Vendor peak TOPS assume every MAC busy every cycle, which no real model achieves. Compare measured frames per second at your resolution.
  • Putting a dataflow design on a device without checking that the weights fit on chip.
  • Ignoring the pre- and post-processing. Resizing, normalisation and non-maximum suppression often take longer than the network when left on a soft processor.
  • Treating the accelerator as the product. Software integration, model updates and tooling licences dominate the lifetime cost.

Going deeper

Latest news on this topic

Recent papers

  • ConvLUT: Rethinking Convolutional Architectures for Ultra-Low-Latency LUT-Based Inference

    M. Andronic, O. Weng, D. Zuberi et al. · arXiv cs.LG · digest

    LUT-based neural networks, where each neuron becomes a lookup table that is unrolled directly into logic, have so far been limited to MLP-style models. This paper extends the idea to convolutions, with a LUT-tree that tames the receptive-field fan-in problem, a LUT-based pooling layer and a hardware-aware pruning schedule. On the MLPerf Tiny tasks (CIFAR-10, Visual Wake Words and keyword spotting) the fully unrolled designs run with latencies below about 50 ns and beat earlier LUT-based models on CIFAR-10 by at least 20 percentage points of accuracy. The models are still small and fully unrolled, so this targets trigger-style and sensor-front-end workloads rather than general vision.

  • PENSA: A Spatial Accelerator for BitNet b1.58 Inference on an HBM-equipped AMD Alveo FPGA

    K. B. Kalyanasundaram, I. S. Maksymov, P. H. W. Leong · arXiv cs.AR · digest

    Ternary LLMs replace multiplications in their linear layers with sign-controlled adds, which moves the bottleneck to feeding packed weights and reducing the results. PENSA runs the full BitNet b1.58 2B4T model on an Alveo U50, streaming TQ1_0-packed weights from eight HBM pseudo-channels into two matrix engines that each handle 1,280 ternary weights per cycle, while attention and vocabulary projection stay resident on the device. At 104.1 MHz it reaches 58.09 tokens per second at a context of 128 and 24.60 at 2048, which the authors put at 2.14 to 3.14 times a Xeon E-2356G baseline. The clock is low for an UltraScale+ part and the comparison is against a CPU rather than a GPU, so treat it as a feasibility result.

  • SkipLane: Write-Once, Conflict-Free Accumulation for Zero-Skipping CNN Engines on FPGAs

    A. Saboo, P. Verma · arXiv cs.AR · digest

    Skipping zero activations in a CNN engine saves multiplies but makes accumulation messy, because products from a compressed window land in different output columns. The authors' earlier design merged them with read-modify-write on a row buffer, which made sparse inputs slower than dense ones. SkipLane gives each kernel row its own lane with three partial-sum registers, retires finished columns into banks that cannot conflict, and defers all cross-lane summation to readout, so nothing is ever read back. Six engines meet 140 MHz on a PYNQ-Z2 in 21.1k LUTs and run LeNet-5 bit-exactly, with conv1 needing 1.94 times fewer cycles than a dense engine. On the narrow conv2 layer it is slower than dense, a reminder that sparsity schemes depend heavily on layer shape.