4Kp30 Multi-Sensor Camera with AI Inference Solution System Example Design for Agilex™ 5 Devices - Web Graphical User Interface Functional Description¶
Running the Graphical User Interface (GUI)¶
The application software of the design has a web-server GUI that is used to
control and demonstrate various hardware and software features of the Camera
Solution System Example Design. Follow the
instructions to
run the GUI on a web browser. The GUI is optimized for displaying the full
screen on a 1920x1080 resolution screen. Press F11 in your browser to go full
screen.
The Software Application is flexible in discovering available hardware
components and only drawing the components present in the design on the GUI.
Therefore, some parts of the GUI elements can be missing or different across
different camera solution designs. Also note that there may be slight
variations in the GUI compared to the example images shown in this section.
Connecting to the Demo Application Web GUI¶
Take note of the Modular Development Kit IP address. This will be displayed on the HPS console serial port when the demo application starts. Alternatively, use the following command to list IP addresses:
Use a web browser of your choice (such as Chrome) on any device connected to
the same network as the Modular Development Kit. If connecting directly without
a switch or router, the self-assigned IPv6 address can be used, or assign a
static IPv4 Address.
Descriptions of the Tabs¶
This section summarizes the GUI tabs in the Camera Solution System Example Design.
Input Config Tab¶
Input Config Tab
The Input Config Tab controls two separate parts of the application:
- The Input Stage which corresponds to selecting which input to use and the parameters for those input sources.
- The Control Loops, corresponding to the Auto White Balance and Auto Exposure
functions.
This section summarizes the GUI controls in the Input Config Tab:
- Input Select
- Test Pattern Generator
- Camera Control
- Sensor Profile
- Histogram
- Auto Exposure
- Auto White Balance
Input Select¶
Input Select
The Input Source UI controls which input is selected:
- Input TPG corresponds to the Test Pattern Generator (TPG). The TPG RGB output image is passed through a Remosaic IP to convert it to a Color Filter Array (CFA) format image (also known as Bayer) that is consistent with the camera input. It can be used to verify the system without a sensor module.
- Camera N, where N corresponds to the sensor module or modules connected to the board. The number of sensor modules visible in the dropdown menu depends on the camera solution design and the number of sensor modules physically connected to the development board.
Test Pattern Generator¶
Test Pattern Generator
The Test Pattern Generator UI allows you to control the Input Test Pattern Generator operation mode:
- SMPTE standard color bars.
- Solid Color. Use the Color Select dropdown box to set the color from the list (including standard SMPTE color bar colors, and a mid-level gray).
- Rainbow. Cycles every few seconds the color selection from the Solid Color mode.
Camera Control¶
Camera Control - FSM-IMX678 Sensor
The camera control provides manual exposure controls for the corresponding
imaging sensor. The UI presents a camera control tile for each sensor module
connected to the board. The model field shows the model of the optical sensor
module detected.
The Analog Gain slider adjusts the light sensitivity of the sensor, allowing
the image to be brightened or dimmed. However, increasing exposure via analog
gain has a tradeoff of reducing signal-to-noise ratio resulting in a grainier
image.
The Frame Rate slider adjusts the sensor frame rate. Reducing the frame rate
gives more time to accumulate time, while increasing it reduces motion blurring
artifacts. For flickering light sources, we recommend setting the frame rate to
an integer multiple or reciprocal integer multiple of the electrical grid
frequency. For example, for 50 Hz grid frequency you may set the frame rate to
50, 25, 12.5 or 6.25 Hz to mitigate flickering.
The Shutter Speed slider controls the light integration time of the sensor,
allowing the image to be brightened or dimmed. The maximum shutter speed is
inversely proportional to the frame rate, where a lower frame rate gives more
time for the sensor to integrate more light.
We recommend the following flow for manually adjusting the total exposure of a scene:
- Minimize the analog gain.
- If the image is flickering adjust frame rate to 50 or 60 Hz depending on the power line frequency.
- Adjust shutter speed for optimum exposure.
- If the image is underexposed even at maximum shutter speed, do one or both of
the following, depending on your motion artifact vs. noise preference, until
satisfied:
- Reduce the frame rate, halving it if flickering, and further increase the shutter speed. This will reduce the responsiveness of the video and increase motion artifacts without increasing graininess.
- Increase the analog gain. This will increase the graininess of the video without changing responsiveness.
Sensor Profile¶
Sensor Profile
The Import Camera Profile button loads a sensor profile (a JSON format pre-calibrated sensor settings file) into the application. The application comes with a default calibration profile preloaded on the microSD card image for the IMX678 as featured in the Camera Solution System Example Design. Therefore, is not necessary to load a new profile in normal operation. The IMX678 profile was produced using the Calibration Tab in the UI, as well as some offline processes for best-in-class color reproduction. Note that using the calibration tab requires expert knowledge and is therefore hidden in the UI by default.
Histogram¶
Histogram Statistics
The Histogram Statistics displays the real-time output of the Histogram Statistics IP. It is intended to be used as a visual aid while adjusting exposure controls.
Auto Exposure¶
Auto Exposure Controls
The Auto Exposure function built into the camera solution demonstration, uses
data from the Histogram Statistics IP to control the sensor analog gain and
shutter speed. When using the ROI, the algorithm will prioritize the region
specified when calculating the exposure.
Auto Exposure - Region of Interest Editor
The Region of Interest (ROI) editor allows you to select a region of the image to use for the Auto Exposure function. Drag the region with a mouse stretch and drag the window freely, or enter starting coordinates and the size of the region manually. Highlighting ROI makes the region visible on the ISP output as an overlay. Clicking the circular arrow on the tile resets the configuration to its default values.
Auto White Balance¶
The Auto White Balance UI allows control of the Automatic White Balance (AWB) function built into the camera solution demonstration. It uses the calibration settings loaded via the Sensor Profile UI.
Auto White Balance Controls
The White Balance control has five modes of operation:
- Disabled. This mode turns off the white balance function by bypassing the White Balance Correction and Black Level Correction IPs, as well as unlocking the controls of those IPs in the UI in the ISP Pipeline Tab. You may then enable those IPs and manually change their coefficients. This mode also disables the Color Correction Matrix function by programming it to unity.
- Choose Temperature. This mode lets you enter a color temperature freely using the Temperature (K) slider. The temperature must be set correctly otherwise incorrect coefficients will be used causing the image to look incorrect. As a guide, scenes generally require a color temperature ranging from 4000 to 6500 Kelvin.
- Choose Lighting. This mode operates similarly to the Choose Temperature mode. However, in this mode Temperature (K) slider is disabled. Instead, you can choose the lighting conditions from the Lighting dropdown menu. Note that the real temperature varies among the family of light sources.
- Automatic. Automatic mode turns off all user input and instead uses the White Balance Statistics IP pre- and post- White Balance Correction to attempt to guess the scene's color temperature.
- Custom Preset (Spot). This is a one-shot automatic white balance mode which runs the AWB function for 2 seconds before locking in the temperature. Clicking the Snapshot button or changing the ROI, resets the mode and triggers another 2 seconds one-shot. This mode is useful where the scene is too colorful. You can position a gray card or position the camera to focus on a gray or white object in the scene, and using the ROI to focus in on just that area, snapshot the temperature to keep it constant for the entire scene.
The Tint Adjustment slider is unlocked across all modes. This control enables
you to adjust the magenta-green color balance when the light source illuminating
the scene is not an ideal black light source following the black body radiation
curve (Planckian locus).
The CCM strength slider controls the color vibrancy. A unity value of 1.0 gives
the most accurate color reproduction following the sensor profile. If the light
source is not ideal, then a value of 1.0 might over-saturate some colors, which
requires you to lower it until satisfied. Conversely, you may increase the color
vibrancy for artistic purposes.
Automatic and Custom Preset (Spot) modes unlock the ROI editor. The selected
region defines the area over which the White Balance Statistics IP performs its
calculations. Resizing or moving the active region automatically highlights the
region on the ISP output as an overlay for 2 seconds. For mixed scenes with
plenty of gray content, a full-screen statistics gives decent performance. For
mixed scenes, it is beneficial to localize the region of interest onto gray
parts of the image as described above.
Clicking the circular arrow on the tile resets the configuration to its default values.
ISP Pipeline Tab¶
ISP Pipeline Tab
The ISP Pipeline tab contains the controls corresponding to the core ISP functions and are summarized as follows:
- Defective Pixel
- Adaptive Noise Reduction
- Black Level Correction
- Vignette Correction
- White Balance Correction
- Demosaic
- Unsharp Mask Filter
- Color Correction Matrix
Defective Pixel Correction¶
Defective Pixel Correction
The Defective Pixel Correction UI provides controls for bypassing the Defective Pixel Correction (DPC) IP functionality, and when enabled setting the DPC strength. The default strength is Very Weak which provides a good balance between mitigating defective pixels, and preserving details and specular highlights. Increasing the strength mitigates more defective pixels with a tradeoff of softening the image.
Adaptive Noise Reduction¶
Adaptive Noise Reduction
The Adaptive Noise Reduction UI provides controls for bypassing or setting the strength of the denoising algorithm of the Adaptive Noise Reduction (ANR) IP. A higher strength provides better denoising with a tradeoff of softening the image.
Clicking the circular arrow on the tile resets the ANR configuration to its default values.
Black Level Correction¶
Black Level Correction
The Black Level Correction UI provides access to low level Black Level Correction (BLC) IP controls. This tile is locked, displaying live values for most of the white balance modes in the Auto White Balance Control as the settings are controlled by the Application software. Note that disabling Auto White Balance to enable manual BLC controls is not implemented by default. This is because the SW App does not configure the imaging sensor and the MIPI connectivity IPs to pass the Optical Black Region (OBR) required for processing the black level statistics in real time. Instead, it uses offline calibrated values.
Vignette Correction¶
Vignette Correction Controls
Vignette Correction (VC) is used to compensate for non-uniform intensity across
the image caused by the uneven light gathering limitations of the sensor and
optics. The most common non-uniformity is where the center of the lens gathers
more light compared to the outer regions. The VC IP corrects uniformity using a
runtime mesh of pre-calibrated coefficients and interpolating coefficients for
any given pixel.
The Vignette Correction UI controls the Vignette Correction (VC) IP and
provides controls for bypassing as well as radial and lens corona correction.
Inverting the functionality allows adding vignetting to the image for artistic
purposes or showcasing the effect. The Radial Vignette slider simulates radial
vignetting as commonly seen on fisheye lenses. Likewise, the Lens Corona
simulates the effects of a strong light source on an imperfect lens, showing as
a bright spot in the center. The Application software translates the controls
into a mesh of coefficients that the VC IP uses.
Clicking the circular arrow on the tile resets the configuration to its default values.
White Balance Correction¶
White Balance Correction Controls
The White Balance Correction UI provides access to low level White Balance
Correction (WBC) IP controls. This tile is locked, displaying live values for
most of the white balance modes in the
Auto White Balance Control as the settings are
controlled by the Application software. Disable Auto White Balance to enable
manual WBC controls.
Clicking the circular arrow on the tile resets the configuration to its default values.
Demosaic¶
Demosaic Controls
The bypass control is the only option in the Demosaic UI controls. In bypass mode the Demosaic IP generates an RGB monochrome output image by simply replicating the single color of a given input CFA pixel to all 3 color planes to generate the given output RGB pixel.
Unsharp Mask Filter¶
Unsharp Mask Filter Controls
The Unsharp Mask Filter UI controls the strength of the sharpening applied to
the image. Positive strengths will sharpen the image, while negative strengths
will soften the image.
Clicking the circular arrow on the tile will reset the configuration of the IP to its default values.
Color Correction Matrix¶
Color Correction Matrix Control
The Color Correction Matrix (CCM) UI is an artistic control interface for tuning the CCM (which is implemented by the Color Space Converter (CSC) IP). The white balance function of the application software calculates a base CCM targeting an accurate representation of the scene. The application software calculates the final CCM based on the control settings on the Color Correction Matrix UI:
- Hue, Saturation, and Value (Brightness) controls enable transformations in the HSV color space.
- Enabling color temperature post-processing enables further control for the color temperature and tint on top of the values set by the white balance controls.
- Red, Green, Blue Channel Strength sliders enable transformations in the RGB color space.
- Contrast adjustment and raising image floor change the dynamic range of the image.
Clicking the circular arrow on the tile of the UI resets the functionality of these controls to their default values.
Output Config Tab¶
Output Config Tab
The Output Config tab contains UI controls for IP Cores typically located after
the ISP core pipeline, right up to and including the output.
This section summarizes the GUI controls in the Output Config Tab:
3D LUT¶
3D LUT Control
The 3D LUT IP can be used for color space conversion, chroma keying, dynamic range conversion (standard to high and high to standard), and artistic effects (sepia, hue rotation, color volume adjustment, etc.).
The 3D LUT IP can support dual buffer mode to allow 2 LUT tables to be loaded and switched between. However, for resource utilization reasons, the Camera Solution System Example Designs only single buffer mode is supported and so the LUT 2 section of the UI is grayed out.
Note that 3D LUT files are not supplied with the Camera Solution System Example Design.
Tone Mapping Operator¶
The Tone Mapping Operator (TMO) UI has controls to bypass the tone mapping and sliders to adjust the threshold and level. The threshold slider controls the strength of the tone mapping. The level slider controls the alpha blending percentage between fully processed TMO output and its original input. The tone mapping feature can use an ROI which can be adjusted from a pop-up panel.
TMO Controls
TMO ROI Editor
1D LUT¶
1D LUT Control
The 1D LUT UI allows you to apply a transfer function to the output video image to match the video sink. You may bypass the functionality or use it in one of two modes:
- Enable Gamma Layer calculates industry standard OETF, EOTF and OOTF LUTs from the options provided. The default transfer function is an OETF Gamma Curve compliant with BT.709 standard.
- Enable Custom Curve allows you to draw a custom transfer function curve using the Custom Curve Editor popup window. The background histogram allows you to guide shaping the transfer function, and you have the option of using the full screen or ROI histograms. You adjust the ROI using the UI pane provided in the bottom left corner of the popup window. The number of segments allows you to control more points on the transfer function window. Resetting the curve changes the curve to a unity transfer function.
Clicking the circular arrow on the title of the UI resets the functionality of these controls to their default values.
Custom Curve Editor
Logo¶
Logo Control
The Logo UI controls the position and opacity of the Altera® logo on the screen. The UI also lets you adjust the location of the logo which can be positioned in each corner or configured to bounce across the screen. The opacity slider determines how transparent the Logo is, where 1 is fully opaque and 0 is fully transparent. The Logo also has a configurable screensaver function to protect screen burn-in where applicable. If you enable the screensaver, the screen output will drop to a black background with the logo bouncing across the screen. Interact with the GUI on your web browser to get out of the screensaver.
Frame Writer¶
Frame Writer Controls
The Frame Writer UI allows you to take screenshots of the raw sensor input
image or the final processed output image prior to the final Gamma 1D LUT.
Clicking Raw Camera Snapshot or ISP Output Snapshot, captures a single image
from the input or output of the video pipeline, respectively. The Download
Image button allows you to download a 48-bit MSB-aligned RGB TIFF image to the
device running the Web browser. Just as in Demosaic bypass, the application
software replicates CFA data to the RGB channels when you select raw capture.
Note that the output flickers while the image capture is in progress and the GUI becomes unresponsive for a moment. A popup notification indicates that the application software has finished the capture process, and that the image is ready for download.
Output Source¶
Output Source Control
The Output Source UI controls which output is selected:
- Select the ISP option to output the fully processed final image.
- Select the SMPTE Bars option to output an unprocessed SMPTE color bar image. This option allows you to test the video connection to the video sink.
Pipeline Statistics Tab¶
Pipeline Statistics Tab
The Pipeline Statistics Tab contains UI controls to help visualize the
statistics IPs within the ISP pipeline.
This section summarizes the GUI controls in the Pipeline Statistics Tab:
Black Level Statistics¶
Black level Statistics
The Black Level Statistics UI visualizes the black level from the white balance algorithm. The channels are color-coded so that they match the color channels of the CFA pattern. Note that the BLS IP remains in bypass in normal operation and is only used during calibration. The application software shows these calibrated statistics used by the white balance algorithm.
White Balance Statistics¶
White Balance Statistics
The White Balance Statistics UI shows a color ratio representation of a
captured scene from the camera input. The scene is split into 7x7 zones and
clicking on a grid square will select and visualize the ratios below.
It is important to note that the color grid does not capture luminosity, only
color ratios. Therefore, results in some zones may appear unusual. For example,
dark gray and light gray sections will be indistinguishable.
The UI works on a captured scene. You can use the "Update" button to force a scene capture update.
Histogram Statistics¶
Histogram Statistics
The Histogram Statistics UI shows a luminosity histogram of the current scene.
The Auto Exposure algorithm uses the histogram statistics in the background as
part of the brightness control function.
The histogram automatically updates at a frequency of 1Hz, which you may
disable using the Auto Update checkbox. Read Stats button triggers a manual
update of the histogram.
The Histogram Statistics IP calculates two histograms, one for the entire frame
and the other for an ROI. The display mode dropdown box allows you to display
either histogram statistics separately, or overlays them on the UI.
Finally, the "Edit ROI" button opens a popup window for adjusting the ROI.
AI Runtime Tab¶
AI Runtime Tab
The AI Runtime Tab contains UI controls to select AI network and results processing parameters.
This section summarizes the GUI controls in the AI Runtime Tab:
AI Runtime¶
AI Runtime
The AI Runtime UI allows you to select the active AI network and modify its parameters.
The Network drop box allows selection between the AI networks on the microSD card:
- YOLO 8n Detect. This network is the YOLO v8 nano detection model with an input image size of 640x384. This option performs bounding box estimation and classification from the eighty items in the COCO dataset. This model has an inference rate of 30 FPS.
- YOLO 8n Pose. This network is the YOLO v8 nano pose estimation model with an input image size of 640x384. This option performs bounding box estimation for the ‘person’ category with seventeen skeletal key-point estimations. This model has an inference rate of 30 FPS.
The Detection Threshold slider controls the confidence threshold, where a valid detection requires the candidate’s confidence value to be greater than the threshold. The confidence level ranges from 0.0 for no confidence to 1.0 for extremely high confidence.
The Intersection Over Union (IOU) Threshold slider allows the filtering of overlapping results. The application software calculates an IOU value for each pair of overlapping result candidates. The IOU is the area of intersection of the bounding boxes divided by the area of union of the bounding boxes. An IOU value of 1.0 means the bounding boxes are identical in size and position while a value of 0.0 means the boxes do not overlap. Values between zero and one provide a measure of similarity. The application software discards the result candidate with the lowest confidence when a pair of result candidates have an IOU value exceeding the IOU threshold. This algorithm is known as non-maximal suppression and allows decluttering of the results.
The Keypoint Threshold slider is only applicable to pose networks and is disabled for detection networks. The keypoint threshold uses confidence values of each of the skeletal keypoints and the application software only draws the pose-lines (bones) between linked keypoints when both keypoint confidence values are above the keypoint threshold.
The Display Results checkbox controls whether the application software renders the results over the video output. Disabling rendering is useful for decluttering the image during scene setup and fine-tuning the ISP components.
Created: October 6, 2025