Loading your First Example

Loading your First Example#

Goals#

Understand the system architecture of an x86 Ryzen AI processor
Introduction to the NPU architecture and tiles
Introduction to the Riallto npu Python package
Launch an example of a video processing application on the NPU for the first time by calling it from Python

References#

Introduction#

These notebooks will walk you through the key architectural features of the Ryzen AI NPU using a series of video processing examples.

This is a reminder that this material can be viewed on the Riallto webpages.

If you have a Ryzen AI laptop, you should install Riallto and open this material as Jupyter notebooks from within the Riallto framework. You will be able to execute the code in the cells below on the Ryzen AI NPU in your laptop.

System Architecture#

These examples assume you are using a laptop with a Ryzen 7040 “Phoenix” APU with the Ryzen AI NPU and an integrated webcam, or compatible hardware. The typical architecture of the system for the examples is shown below.

Ryzen 7040 connected to system memory, webcam and display

The AMD Ryzen 7040 “Phoenix” family of laptop and desktop processors has an AMD Zen 4 CPU (which is an x86 CPU), and Ryzen AI NPU integrated into the same chip, as indicated in the image.

The APU is connected to system memory. System memory is used as the interface between the NPU and the rest of the system. For example, the webcam, controlled by the CPU, will place input data into system memory where the NPU can read it. The NPU will write its results to system memory where it can then be accessed by the CPU and sent to the display.

The role of the Ryzen AI NPU is to offload computationally intensive AI workloads from the main x86 CPU or GPU (integrated GPU or discrete GPU) to the NPU. The NPU runs these workloads more power efficiently than the CPU or GPU.

Video Pipeline#

The CPU controls the webcam and display that are used as the data source and sink for the application. The CPU will configure the webcam which will write input data to system memory. The NPU will read the webcam data and write its results back to system memory where the CPU can access the results for the display.

This diagram shows a simplified representation of the capture-to-display video processing pipeline of your laptop.

Simplified Ryzen AI NPU video processing pipeline

This is the same video pipeline that Microsoft Windows Studio Effects uses for automatic framing, gaze adjustment and background effects.

Ryzen AI NPU Architecture#

The Ryzen AI NPU is any array of compute, memory and interface tiles. There are 20 compute tiles arranged in a 4x5 grid of rows and columns. There are 5 memory tiles and 4 interfaces tiles. Interface tiles connect the NPU to system memory. You will notice an asymmetry in the number of interface tiles and memory tiles.

Tiles are connected to each other through a network of streaming interfaces.

Ryzen AI NPU organization

Compute tiles are where the program or subprograms of a software application are run. Memory and interface tiles move data into and out of the NPU and marshal data movement within the array. Each of the tiles, their features, and interfaces will be introduced in more detail as we progress through these notebooks.

Color Threshold Example#

We selected the color threshold as it is the simplest application to start exploring the capabilities of the NPU.

The first example we will use is a color thresholding application. The algorithm works on each pixel individually. The threshold function \(f(x)\) compares each pixel \(x\) against a threshold value, \(tv\), where \(0 \leq tv \leq 255\), such that:

\[\begin{split} f(x) = \begin{cases} 0, & \text{if } x \leq t \\ v, & \text{if } x > t \end{cases} \end{split}\]

In the example, you will be able to set the threshold value \(tv\) and see the effect it has on the output video.

The threshold algorithm is applied to each of the red, blue and green color channels in the original image separately. Pixel values in each channel that are below the threshold become black. The remaining pixels become red, green or blue respectively. Note that we assign minimum and maximum color values. The outputs of the three color channels are re-combined to create the final image.

The image below shows an example of the operation of the thresholding function. Using the Original image shown on the left, the image is split into its three color channels. Each channel is processed by the threshold function with a given threshold value. The three channels will be recombined to form the processed output image. The output image will depend on the values of the threshold values used for each channel.

Example of the color threshold algorithm

Riallto `npu` Python Package and the video library#

To make the NPU more accessible and easier to work with, as part of Riallto we have developed a Python package called npu. In addition to other functionality that we will encounter later that will be used to build custom NPU applications, the Riallto npu package provides several useful utilities for checking the status of the NPU. One of these utilities is nputop (modelled after the Linux top command) which shows applications currently running on the NPU. Check the utilization of the NPU by running the following cell:

from npu import nputop

nputop()

The output will report the current state of the NPU in your system. For example, if Microsoft Studio Effects are loaded, you will see this in the nputop report.

The Riallto npu package, includes a library of video processing example applications in the videoapps module in npu.utils. Explore the list of available applications by running the following cell:

from npu.utils import videoapps

videoapps()

['ColorDetectVideoProcessing',
 'ColorThresholdVideoProcessing',
 'DenoiseDPVideoProcessing',
 'DenoiseTPVideoProcessing',
 'EdgeDetectVideoProcessing',
 'ScaledColorThresholdVideoProcessing',
 'VideoApplication']

We will start by running the Color Threshold application (ColorThresholdVideoProcessing) and explore more these applications in later notebooks.

Run the Color Threshold Application#

The code to run the color threshold example is shown in the next code cell.

This will load the application into the Ryzen AI NPU, and start the stream of data from your laptop webcam. The processed video output will be displayed below the code cell.

The thresholding example has three widgets. If you run this notebook on a Ryzen AI laptop, you can use each of these widgets to control the threshold value for each of the red, blue and green channels and see the effect on the video stream. Experiment with the three widgets to see how different threshold levels on each channel effects the output.

from npu.lib import ColorThresholdVideoProcessing

app = ColorThresholdVideoProcessing()
app.start()

You should now see the application you just loaded in the nputop report:

nputop()

After you have explored the application, click the Stop button under the video to finish the video stream and release the NPU and webcam.

Next Steps#

In the next notebook we will see how the color threshold application can be mapped to the NPU, and explore the properties of the NPU tiles.