Skip to content

4Kp30 Multi-Sensor Camera with AI Inference Solution System Example Design for Agilex™ 5 Devices

The design is compatible with Altera® Quartus® Prime Pro Edition version 25.1 Linux.

Overview

The 4Kp30 Multi-Sensor Camera with AI Inference Solution System Example Design for Agilex™ 5 Devices demonstrates a practical glass-to-glass smart camera solution. The exclusive support for industry-standard MIPI (Mobile Industry Processor Interface) D-PHY and MIPI CSI-2 interface on Agilex™ 5 FPGAs, along with Altera®'s FPGA AI Suite, provides a powerful tool for smart camera product development.

An example of AI Detect
An example of AI Pose
Detect example POSE example

The MIPI interface supports up to 2.5Gbps per lane and up to 8x lanes per MIPI interface, enabling seamless data reception from multiple 4K image sensors to the FPGA fabric for further processing. Each MIPI CSI-2 IP instance converts pixel data to AXI4-Streaming outputs, enabling connectivity to other IP cores within Altera®'s Video and Vision Processing (VVP) Suite.

Altera®'s FPGA AI Suite is flexible and configurable for a variety of smart camera use cases. In the 4Kp30 Multi-Sensor Camera with AI Inference Solution System Example Design, the FPGA AI Suite IP is optimized to run the Ultralytics YOLOv8 nano detection and pose inference models.

The design is a hardware-software co-design. The hardware component comprises an Image Signal Processor (ISP), various VVP IPs, Altera®'s FPGA AI Suite IP (AI IP), Hard Processor Subsystem (HPS) and various connectivity IPs. The software stack is Linux based and runs on the HPS. The software runs compiled YOLOv8 nano models from the microSD card. The End User must license and download the models directly from Ultralytics.

The hardware includes a multi-sensor input video switch feeding into an Image Signal Processing (ISP) subsystem. The ISP is a video processing pipeline incorporating many VVP IP cores such that the raw sensor image data can be processed into RGB video data. The backend of the ISP pipeline feeds the AI pipeline which consists of many VVP IP cores to buffer, format, and scale the video suitable for AI processing. The backend of the AI pipeline drives the resulting 4Kp30 streaming video output data (complete with AI inference overlay) through an Altera® DisplayPort IP.

The software stack consists of an application software binary running on the Linux operating system with various layers of drivers. The backend part of the application software interrogates the hardware, dynamically discovers the IP components and configures them. The AI inference part of the application software schedules inference requests to Altera®'s FPGA AI Suite IP, and processes the inference results. The results are rendered as graphics in a frame buffer, which the hardware overlays on the video stream. Multiple feedback loops, in the application software, monitor the hardware and keep various hardware components in lockstep. Some of the notable feedback loops are Automatic White Balance (AWB), Auto Exposure (AE), and Adaptive Noise Reduction (ANR) algorithms, reading their relevant statistics and adjusting various coefficients and Look Up Tables (LUTs) in real time. The frontend of the software creates a web-based Graphical User Interface (GUI) and runs it over a web server.

The following diagram provides an overview of the interaction of the software running in the Hard Processor Subsystem (HPS) and hardware components running in the Programmable Logic parts of the device. (For more information on ARM HPS in Altera® Agilex™ Devices refer to the Other Documentation and References section.)


VVP_ISP_AI_ED_Top_level

High-Level Block Diagram of the Camera with AI Inference Solution System Example Design


Pre-requisites

Hardware Requirements


Agx5-MDK

Agilex™ 5 FPGA E-Series 065B Modular Development Kit



Software Requirements to run

  • Host PC with:
    • 8 GB of RAM (less if not rebuilding binaries).
    • Linux/Windows OS installed.
    • Serial terminal (such as GtkTerm or Minicom on Linux, and TeraTerm or PuTTY on Windows).
      • FTDI FT232R USB UART drivers (for a Windows host).
    • Tool to write images for removable USB drives or microSD cards such as Win32DiskImager on Windows or "dd" command on Linux.
    • Altera® Quartus® Prime Pro Edition version 25.1 Programmer and Tools.
    • Ethernet connection (either direct from Host PC to development board, or via a switch or router).
      • Note, you may need to disconnected/disabled VPN if it is installed on the Host PC.
    • Web browser.


Download and Compile the AI Models

The Altera® FPGA AI Suite IP in the Camera with AI Inference Solution System Example Design, is optimized to run both the ultralytics YOLOv8 nano detection and pose inference models, switching between them at runtime. The End User must go to ultralytics website to review and accept licensing and copyright information, before downloading the YOLOv8 nano models. The models must then be compiled for the FPGA AI Suite IP. This will only need to be done once:

  • Visit the ultralytics YOLO website.
  • Review and accept the licensing and copyright terms.
  • Download the YOLOv8 nano detection inference model yolov8n.pt
  • Download the YOLOv8 nano pose inference model yolov8n-pose.pt
  • Download the model_compiler to your <workspace> directory
  • Compile the models for FPGA AI Suite IP:

Note

The downloaded YOLOv8 nano models must be placed in the directory specified

mkdir -p <workspace>
cd <workspace>
git clone [https://github.com/altera-fpga/agilex-ed-camera-ai] .
cd yolo_cnn

echo "Download the YOLOv8 nano inference models from ultralytics website into directory yolo_cnn"
wget <url>

mkdir -p compile
cd compile
echo "This step can take some time to extract Altera® FPGA AI Suite"
cmake -G Ninja ..
echo "This step can take some time to generate a python virtual environment"
ninja
cd output
tree

.
├── generated_arch.arch
│   ├── yolov8n-pose_dla_m2m_compiled_640_384.bin
│   └── yolov8n_dla_m2m_compiled_640_384.bin
├── yolov8n-pose_categories.txt
└── yolov8n_categories.txt

1 directory, 4 files

The model_compiler generates the following YOLOv8 nano inference model binaries:

  • Detection generated_arch.arch/yolov8n_dla_m2m_compiled_640_384.bin
  • Pose generated_arch.arch/yolov8n-pose_dla_m2m_compiled_640_384.bin

Additionally, the model_compiler generates the following category identifier files: * Detection yolov8n_categories.txt * Pose yolov8n-pose_categories.txt


Getting Started - run with pre-built binaries

Follow the instructions provided in this section to run the Camera Solution System Example Design on the Agilex™ 5 FPGA E-Series 065B Modular Development Kit.

Download the pre-built Binaries

  • Download the pre-built Camera Solution System Example Design binaries for the Modular Development Kit:


Binaries

Source Link Description
QSPI top.core.jic Allows the Camera Solution System Example Design to be booted from the microSD card
microSD Card Image hps-first-vvp-isp-demo-image-agilex5_mk_a5e065bb32aes1.wic.gz The Camera Solution System Example Design


Programming

Setting Up the Development Kit

Warning

Handle ESD-sensitive equipment (boards, microSD cards, camera sensors, etc.) only when properly grounded and at an ESD-safe workstation

  • Configure the Agilex™ 5 FPGA E-Series 065B Modular Development Kit switches as shown:


board-1

Modular Development Kit - Default Switch Positions


FPGA configuration modes used by the Camera Solution System Example Design

The Camera Solution System Example Design can use one of the following FPGA configuration modes: JTAG or ASx4(QSPI).
The MSEL switch (S4) on the SOM board instructs the FPGA device on which configuration mode to use.

JTAG mode: Modular Development Kit SOM Board S4=OFF:OFF, or
ASx4 (QSPI) mode: Modular Development Kit SOM Board S4=ON:ON

The pre-built version of the design uses ASx4 (QSPI) mode, S4=ON:ON.


  • Make the required connections between the Host PC and the Modular Development Kit as shown in the following diagram:


ed-conn

Host PC and Modular Development Kit Connections diagram


  • Connect micro USB cable between the Modular Development Kit Carrier Board (J35) and the Host PC. This will be used for QSPI programming / FGPA configuration over JTAG. Look at what serial ports are enumerated on your Host computer. There should be a series of four.
  • Connect micro USB cable between the Modular Development Kit SOM Board (J2, HSP_UART) and the Host PC. This will be used to access HPS serial console. Look at what ports are enumerated on your Host computer. There should be a series of four. Use the 3rd one in the list as the HPS serial port.
  • Connect an RJ45 cable between the ethernet port on the Modular Development Kit SOM Board (J6, ETH 1G HPS) and make sure it is on the same network as your Host PC. You can check the eth0 IP address after boot using the Linux ip a command.


Agx-MDK-Conn

Modular Development Kit Connector Locations


Burn the microSD Card Image

  • Either use your own or download the pre-built <name>.wic.gz image.

  • If required, extract <name>.wic image from the compressed download file.

    • On Linux, use the dd utility:
    tar -xzf `<name>.wic.gz`
    • On Windows, use the 7-Zip program (or similar):
      • Right click <name>.wic.gz file, and select "Extract All..."

  • Write the <name>.wic image to the microSD card using a USB writer:

    • On Linux, use the dd utility:
    # Determine the device associated with the SD card on the host computer.
cat /proc/partitions
# This will return for example /dev/sd<x>
# Use dd to write the image in the corresponding device
sudo dd if=<name>.wic of=/dev/sd<x> bs=1M
# Flush the changes to the microSD card
sync

    • On Windows, use the Win32DiskImager program (or similar):

      • Click browse icon and select "*.*" filter:

      disk-imager-browse

      Navigate to your download and select <name>.wic in the "Disk Imager" tool


      • Write the image (note your Device may be different to that shown):

      disk-imager

      Write the microSD Card using the "Disk Imager" tool

  • Turn off the Modular Development Kit and insert the microSD card in the microSD card slot located on the Modular Development Kit SOM Board.


Copying the Compiled AI Models to the microSD Card

The compiled models must be copied onto the microSD card for the Application Software to use at runtime:

  • scp and ssh the compiled models directly to the Development Kit (using its <ip address>):

    • Power up the Modular Development Kit (if not already powered) and set up the serial terminal emulator (minicom, TeraTerm, PuTTY, etc.):
      • Select the correct COMx port. (The Modular Development Kit presents 4 serial COM ports over a single connection and the Linux system uses the 3rd port in order). Set the port configuration as follows:
        • 115200 baud rate, 8 Data bits, 1 Stop bit, CRC and Hardware flow control disabled.
      • The Linux OS will boot.

      • Take note of the Modular Development Kit IP address.

        • The IP address can also be found using the terminal by logging in as root (no password required) and querying the Ethernet controller:
        root
ifconfig
        • eth0 provides the IPv4 or IPv6 address to connect to.

    • Using a Linux terminal (or Windows equivalent like PowerShell) on your Host, copy the files from the output directory to the Development Kit:

      cd compile/output
scp -r * root@<ip address>:
      * Ensure sdcard has stored the files 
      ssh root@<ip address>
sync
ls *_categories.txt *.arch
      Outputs: 
      yolov8n-pose_categories.txt yolov8n_categories.txt

generated_arch.arch:
yolov8n-pose_dla_m2m_compiled_640_384.bin yolov8n_dla_m2m_compiled_640_384.bin
      * The Development Kit can be powered down, and restarted to load the models.


Program the QSPI Flash Memory

This should only need to be done once. To program the QSPI flash memory:

  • Ensure the Modular Development Kit is powered off. Set MSEL=JTAG by setting the S4 dip switch on the Modular Development SOM Board to OFF-OFF.
    • This prevents any bootloader from starting and leaves the JTAG chain in a default state.
  • Power up the Modular Development Kit.
  • Either use your own or download the pre-built JIC image, and write it to the QSPI Flash memory using either the command:
    quartus_pgm -c 1 -m jtag -o "pvi;top.core.jic"

  • or, optionally using the Quartus® Programmer GUI:

    • Launch the Quartus® Programmer and Configure the "Hardware Setup..." settings as following:

hw-setup-set

Programmer - GUI Hardware Settings


  • Click "Auto Detect", select the device A5EC065BB32AR0, and press "Change File.."

programmer-agx5

Programmer - After "Auto Detect"


Select your top.core.jic file. The MT25QU02G device should be shown (see below). Check the "Program/Configure" box and press the "Start" button. Wait until the programming has been completed (which can take several minutes).

programmer-agx5-2

Programming the QSPI Flash with the JIC file


  • Power down the Modular Development Kit. Set MSEL=ASX4 (QSPI) by setting the S4 dip switch on the Modular Development SOM Board to ON-ON.
    • This starts the HPS bootloader and FPGA configuration from the microSD Card after power up.

Running

Setting Up the Camera Solution

Warning

Handle ESD-sensitive equipment (boards, microSD Cards, Camera sensors, etc.) only when properly grounded and at an ESD-safe workstation

  • Make the required connections between the Host PC and the Agilex™ 5 FPGA E-Series 065B Modular Development Kit as detailed in the Setting Up the Modular Development Kit section.
  • Connect the Framos cable(s) between the Framos Camera Module(s) and the MIPI connector(s) on the Modular Development Kit Carrier Board taking care to align the cable(s) correctly with the connector(s) (pin 1 to pin 1). When using a single camera module, either MIPI connector can be used.


board-mipi

Modular Development Kit Carrier Board MIPI Connector Locations


mipi-ribbon

Modular Development Kit Carrier Board with MIPI Framos Flex Cable Connected


camera-ribbon

Framos Camera with Flex Cable Connected


  • If using the optional Framos GMSL3 solution (and the Camera Solution System Example Design supports it):
    • Connect the Framos FFA-GMSL-SER-V2A Serializer module back-to-back to the Framos Camera module.
    • Using the Framos 150mm flex-cable connect the Framos FFA-GMSL-DES-V2A Deserializer module to the MIPI0 connector on the Modular Development Kit Carrier Board taking care to align the cable correctly with the connector (pin 1 to pin 1).
      • Note the System Example Design only supports one GMSL3 link on the MIPI0 port. However, a second Framos Camera module can be connected directly to the MIPI1 port using a Framos 150mm flex-cable.
    • Connect the serializer module to the deserializer module using the GMSL3 5m coax cable.
    • Connect the power supply to the deserializer module.
      • Note the GMSL3 deserializer module must be powered up before the Modular Development Kit.
    • Power up the Modular Development Kit and ensure the deserializer modules Lock LED is illuminated green.


GMSL

GMSL Connections


  • Connect the Modular Development Kit Carrier Board DisplayPort Tx connector to the Monitor using a suitable cable (and the adapter if you are using an HDMI cable).


full-system

Modular Development Kit with Connections


Connecting with a Web Browser

  • Power up the Modular Development Kit (if not already powered) and set up the serial terminal emulator (minicom, TeraTerm, PuTTY, etc.):
    • Select the correct COMx port. (The Modular Development Kit presents 4 serial COM ports over a single connection and the Linux system uses the 3rd port in order). Set the port configuration as follows:
      • 115200 baud rate, 8 Data bits, 1 Stop bit, CRC and Hardware flow control disabled.
  • The Linux OS will boot and the Camera Solution System Example Design Software Application should run automatically.
  • A few seconds after Linux boots, the Software Application will detect the attached Monitor and the ISP processed output will be displayed using the best supported format.

  • Take note of the Modular Development Kit's IP address.

    • The IP address can also be found using the terminal by logging in as root (no password required) and querying the Ethernet controller:

      root
ifconfig
    • eth0 provides the IPv4 or IPv6 address to connect to.

ed-conn

An Example ifconfig Output for a DHCP Network


ed-conn

An Example ifconfig Output for a Network with no DHCP support or is using a direct connection


  • Connect your web browser to the boards IP address so you can interact with the Camera Solution System Example Design using the GUI.
    • To connect using IPv6 in the example address shown above, you would use http://[fe80::a8bb:ccff:fe55:6688] (note the square brackets)
    • To connect using IPv4 for the DHCP example shown above, you would use http://192.168.0.1


ed-conn

An Example Web Browser URL for an IPv6 Address


ed-conn

An Example Web Browser URL for an IPv4 address


  • During connection, you will see the Altera® splash screen, after which you will be presented with the Web GUI.


ed-conn

An Example Camera Solution System Example Design GUI


Sources

The sources listed in this table are the most current and highly recommended for Altera® Quartus® Prime Pro Edition version 25.1 Linux builds. Users are advised to utilize the updated versions of these building blocks in production environments. Please note that this is a demonstration design and is not suitable for production or final deployment.

Camera with AI Inference Solution System Example Design Source Repository

Component Location Branch
Assets Release Tag https://github.com/altera-fpga/agilex-ed-camera-ai/releases/tag/rel-25.1 rel-25.1
Repository https://github.com/altera-fpga/agilex-ed-camera-ai rel-25.1


With the available source, you can create and modify, build and compile, and execute the Camera Solution System Example Design. There are 3 recommended flows that you can explore:

Note

The free OpenCore Plus feature allows you to evaluate licensed IP cores in simulation and hardware before purchase. OpenCore Plus evaluation supports the following two operation modes:
Untethered — run the design containing the licensed IP for a limited time.
Tethered — run the design containing the licensed IP for a longer time or indefinitely. This operation requires a JTAG connection between your Development Kit and the Host computer.
Note: All video IP cores that use OpenCore Plus time out simultaneously when any one video IP core in the design times out.

The free FPGA AI Suite license operates independently to the OCP license and is typically limited to 100k inferences (around 56 minutes using a 30FPS inference model). Note that the OCP licensed IP will continue to run even if the FPGA AI Suite license expires.


Flows

Recommended User Flows

User Flow Description User Flow 1 User Flow 2 User Flow 3
Pre-requisites Hardware Requirements. ✓ ✓ ✓
Software Requirements to build. ✗ ✓ ✓
License Requirements to build. ✗ ✓ ✓
Software Requirements to run. ✓ ✓ ✓
Download and Compile the AI Models. ✓ ✓ ✓
Download the Pre-built Binaries. ✓ ✗ ✗
HW-Compilation Creating and Building the Design based on the SOF Modular Design Toolkit (MDT) Flow ✗ ✓ ✗
Creating and Building the Design based on the RBF Modular Design Toolkit (MDT) Flow ✗ ✗ ✓
SW-Compilation Create microSD card image (.wic.gz) using YOCTO/KAS
NOTE: use KAS_MACHINE=agilex5_mk_a5e065bb32aes1 and kas/agilex_camera_ff.yml configuration		 ✗ ✓ ✗
Create microSD card image (.wic.gz) using YOCTO/KAS
NOTE: use KAS_MACHINE=agilex5_mk_a5e065bb32aes1 and kas/agilex_camera.yml configuration		 ✗ ✗ ✓
Programming Setting Up the Development Kit. ✓ ✓ ✓
Burn the microSD card image. ✓ ✓ ✓
Copying the Compiled AI Models to the microSD Card. ✓ ✓ ✓
Program the QSPI Flash Memory. ✓ ✗ ✓
Running Setting Up the Camera Solution. ✓ ✓ ✓
Program the FPGA SOF. ✗ ✓ ✗
Connecting with a Web Browser. ✓ ✓ ✓


Resources

Documentation

References

Other Repositories Used

Component Location Branch
Modular Design Toolkit https://github.com/altera-fpga/modular-design-toolkit rel-25.1
Linux https://github.com/altera-opensource/linux-socfpga socfpga-6.6.22-lts
Arm Trusted Firmware https://github.com/ARM-software/arm-trusted-firmware socfpga_v2.11.0
U-Boot https://github.com/altera-opensource/u-boot-socfpga v2024.01
Yocto Project: poky https://git.yoctoproject.org/poky scarthgap

Other Documentation and References


Notices & Disclaimers

Altera® Corporation technologies may require enabled hardware, software or service activation. No product or component can be absolutely secure. Performance varies by use, configuration and other factors. Your costs and results may vary. You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Altera or Intel products described herein. You agree to grant Altera Corporation a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document, with the sole exception that you may publish an unmodified copy. You may create software implementations based on this document and in compliance with the foregoing that are intended to execute on the Altera or Intel product(s) referenced in this document. No rights are granted to create modifications or derivatives of this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Altera disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. You are responsible for safety of the overall system, including compliance with applicable safety-related requirements or standards. © Altera Corporation. Altera, the Altera logo, and other Altera marks are trademarks of Altera Corporation. Other names and brands may be claimed as the property of others.

OpenCL* and the OpenCL* logo are trademarks of Apple Inc. used by permission of the Khronos Group™.


Last update: January 23, 2026
Created: December 10, 2025
Ask in the Forum