Run a TensorFlow Lite model on the i.MX 8M Plus NPU

This article refers to Platform v3.4.0. The current Platform version is v3.4.0.

Overview

The ADLINK MXA-200 and the Schneider Electric Harmony BX1 are built on the NXP i.MX 8M Plus processor, which includes a Neural Processing Unit (NPU): a dedicated accelerator that runs neural network inference much faster than the processor's CPU cores.

Barbara publishes a base image with everything a TensorFlow Lite application needs to use the NPU, so your application only has to bring its model and its code. In this tutorial, you will:

Write a docker-compose.yaml file that starts from the base image and gives the container access to the NPU.
Place the model and the application code in the ./persist folder of the application.
Upload the application to the App Library, deploy it to an Edge Node, and follow the inference results in the logs.
Optionally, extend the base image with the Python packages your application needs.

How the base image works

The base image is bdr.barbara.tech/public/npu-imx8mp:tflite2.9.1, a public ARM64 image based on Ubuntu 22.04:

Component	Details
TensorFlow Lite	Version 2.9.1, with the `tflite_runtime` Python module for Python 3.10
VX delegate	`/usr/lib/libvx_delegate.so`, the TensorFlow Lite delegate that runs models on the NPU. Its path is also in the `NPU_DELEGATE` environment variable
NPU drivers	NXP's user-space drivers for the NPU (OpenVX, TIM-VX, and their dependencies)
Python packages	`numpy` and Pillow
Examples	A MobileNet v1 model, its labels, and a test image under `/opt/npu/examples`

The image also sets the environment variables that route TensorFlow Lite graphs to the NPU, so your code only has to load the VX delegate.

The NPU device, /dev/galcore, belongs to the Edge Node's operating system, and your Compose file passes it to the container. The container accesses it as root, which is the default user of the base image.

Prepare the application

The application is a Python script that classifies the images in a folder every 30 seconds and prints the result. Organize the files in this structure:

npu-classifier/
├── docker-compose.yaml
└── persist/
    ├── app.py
    ├── model/
    │   ├── model.tflite
    │   └── labels.txt
    └── images/
        └── grace_hopper.bmp

The model, the code, and the input images live in ./persist. When Barbara installs the application, the agent copies the contents of ./persist into the workload's persistent folder, and the container reads them from there. Your application does not need its own image: it runs directly on the base image.

npu-classifier.zip

Download the complete example, with the MobileNet v1 model and a test image: npu-classifier.zip.

1) Write the Compose file

The Compose file runs the application on the base image, passes the NPU device to the container, and mounts the persistent folder:

docker-compose.yaml
services:
  npu-classifier:
    image: bdr.barbara.tech/public/npu-imx8mp:tflite2.9.1
    command: ["python3", "/persist/app.py"]
    devices:
      - "/dev/galcore:/dev/galcore"
    volumes:
      - "./persist/:/persist/"
    environment:
      - VIV_VX_CACHE_BINARY_GRAPH_DIR=/persist/npu-cache
      - INTERVAL=30
    restart: unless-stopped

What each setting does:

devices: gives the container access to the NPU. Without it, the VX delegate cannot open the device and the application fails at startup.
volumes: mounts the persistent folder at /persist, where the application finds its code, its model, and its images.
command: runs the application from the persistent folder.
VIV_VX_CACHE_BINARY_GRAPH_DIR: stores the model compiled for the NPU in the persistent folder (see the next step).
restart: restarts the application if it stops. Remember that the always policy is not accepted on Barbara nodes.

2) Write the application

The application loads the model with the VX delegate, so TensorFlow Lite runs it on the NPU:

persist/app.py
"""Classify the images in /persist/images on the NPU, every INTERVAL seconds."""
import glob
import os
import time

import numpy as np
from PIL import Image
from tflite_runtime.interpreter import Interpreter, load_delegate

MODEL_PATH = os.environ.get("MODEL_PATH", "/persist/model/model.tflite")
LABELS_PATH = os.environ.get("LABELS_PATH", "/persist/model/labels.txt")
IMAGES_DIR = os.environ.get("IMAGES_DIR", "/persist/images")
INTERVAL = int(os.environ.get("INTERVAL", "60"))

# The NPU driver keeps the compiled model here, so it is compiled only once.
os.makedirs(os.environ["VIV_VX_CACHE_BINARY_GRAPH_DIR"], exist_ok=True)

interpreter = Interpreter(
    model_path=MODEL_PATH,
    experimental_delegates=[load_delegate(os.environ["NPU_DELEGATE"])],
)
interpreter.allocate_tensors()
inp = interpreter.get_input_details()[0]
out = interpreter.get_output_details()[0]
_, height, width, _ = inp["shape"]
with open(LABELS_PATH) as f:
    labels = [line.strip() for line in f]


def classify(path):
    image = Image.open(path).convert("RGB").resize((width, height))
    data = np.expand_dims(np.asarray(image), 0).astype(inp["dtype"])
    interpreter.set_tensor(inp["index"], data)
    start = time.monotonic()
    interpreter.invoke()
    elapsed_ms = (time.monotonic() - start) * 1000
    scores = interpreter.get_tensor(out["index"])[0].astype(np.float32)
    scale, zero_point = out["quantization"]
    if scale:
        scores = (scores - zero_point) * scale
    best = int(np.argmax(scores))
    print("%s: %s (%.2f) in %.1f ms" % (os.path.basename(path), labels[best], scores[best], elapsed_ms), flush=True)


while True:
    for path in sorted(glob.glob(os.path.join(IMAGES_DIR, "*"))):
        classify(path)
    time.sleep(INTERVAL)

The first inference takes several seconds, because the NPU driver compiles the model for the NPU. The driver stores the compiled model in VIV_VX_CACHE_BINARY_GRAPH_DIR, which points to the persistent folder, so the model is compiled only once: after a restart, the driver loads it from the cache.

To use your own model, replace model.tflite and labels.txt. If your model expects a different input (for example, a floating-point model that needs normalized values), adapt the classify function.

3) Create the ZIP archive

From the npu-classifier folder, package the Compose file and the persist folder:

zip -r ../npu-classifier.zip docker-compose.yaml persist

Deploy the application

Upload and deploy the ZIP archive like any other Docker application, as described in Package a Docker app for Barbara:

In the App Library, add a Docker application and upload npu-classifier.zip. Under Platform Compatibility, keep only ARM64/V8: the base image is built for that architecture.
On the Node Details page of an MXA-200 or Harmony BX1 node, add the application.
Enable Logs on the workload card.

The Barbara agent downloads the base image from bdr.barbara.tech, so the Edge Node needs access to the Docker registry. See Domains and ports.

Once the application starts, the logs show the VX delegate's startup messages, followed by one line per image. The first one includes the compilation of the model:

grace_hopper.bmp: military uniform (0.87) in 5291.0 ms
grace_hopper.bmp: military uniform (0.87) in 2.8 ms

After a restart, the first inference loads the compiled model from the cache and takes a fraction of a second.

For reference, the example model (MobileNet v1, quantized) takes approximately 3 ms per inference on the NPU, and approximately 46 ms on the CPU cores of the same node, more than 15 times slower.

Update the model

To deploy a new model, add a new version of the application with the new model.tflite (and labels.txt, if the classes change) in ./persist/model.

When the agent installs the new version, it copies the contents of ./persist from the ZIP archive into the persistent folder: files with the same name are replaced, and the rest are kept. Your images and the compiled models in the cache stay in place, and the driver compiles the new model on its first inference.

Add your own dependencies

If your application needs Python packages or system libraries that the base image does not include, add a Dockerfile next to the Compose file that starts from the base image and installs them:

Dockerfile
FROM bdr.barbara.tech/public/npu-imx8mp:tflite2.9.1

RUN pip3 install --no-cache-dir paho-mqtt==2.1.0

Then replace image: with build: in the Compose file, and keep the rest unchanged:

docker-compose.yaml
services:
  npu-classifier:
    build: .
    command: ["python3", "/persist/app.py"]

The Barbara agent builds the image on the Edge Node when it installs each version of the application, so the node also needs access to the package index your Dockerfile uses.

warning

Keep the model and the code in ./persist, mounted as a volume. Do not add them to the image with COPY persist/...: on the node, the build starts after the agent has moved ./persist to the persistent folder, so the build cannot find those files.

Summary

You built a Docker application that runs a TensorFlow Lite model on the NPU of an i.MX 8M Plus Edge Node: the Compose file starts from the Barbara NPU base image and declares the NPU device, and the model and the code travel in the ./persist folder of the application.

The base image carries the TensorFlow Lite runtime and the NPU drivers, so you deploy and update AI models on the NPU like any other application, from Barbara Panel and without building or maintaining the NPU stack yourself.

To automate the loop from a trained model to the edge, continue with the Python SDK.

Overview​

How the base image works​

Prepare the application​

1) Write the Compose file​

2) Write the application​

3) Create the ZIP archive​

Deploy the application​

Update the model​

Add your own dependencies​

Summary​