Person detection

This application demonstrates real-time person detection from a camera stream using model inference on the Axon NPU.

Application overview

The Person Detection application captures images from an Arducam Mega camera module and processes them through a neural network model running on the Axon NPU. The camera captures an image with a resolution of 128×128 pixels. This image is later padded with neutral gray color to match the model input size of 128×160 pixels. The application converts the camera input to the model format, runs inference, and post-processes the output to extract bounding boxes with confidence scores.

The application uses the person_det model from MCUnet. It uses three detection heads with different spatial scales to identify objects at various sizes. Detected bounding boxes are post-processed using Non-Maximum Suppression (NMS) to remove overlapping detections and filter low-confidence predictions.

The application provides visual feedback through LEDs for image capture and current detection status.

The application continuously captures frames at regular intervals, processes them through the model, and logs any detected objects along with their bounding box coordinates and confidence scores.

Requirements

The application supports the following development kit:

Hardware platforms

PCA

Board name

Board target

nRF54LM20 DK

PCA10184

nrf54lm20dk

nrf54lm20dk/nrf54lm20b/cpuapp

The application also requires an SPI camera and was tested with Arducam Mega 5MP (B401) camera. Configure the development kit using Board Configurator to provide 3.3V to power the camera.

Pin mapping

See the following table for the camera-to-DK pin mapping:

Description

Arducam Mega Pin

nRF54LM20 DK Pin

Power supply (3.3V)

VCC

VDD:IO

Ground

GND

GND

Chip select

CS

P1.7

SPI MOSI

MOSI

P1.6

SPI MISO

MISO

P1.5

SPI Clock

SCK

P1.4

For detailed pin configuration, refer to the device tree overlay boards/nrf54lm20dk_nrf54lm20b_cpuapp.overlay file.

User interface

This section describes the user interface of the application.

LEDs

LED0 (capture LED):

Toggles (changes state between on and off) each time a frame is captured from the camera and processed.

LED1 (detection LED):

Turns on when persons are detected in the frame. Turns off when no detections are found.

Configuration

See Configuring and building for information about how to permanently or temporarily change the configuration.

Configuration options

The following application-specific Kconfig options are used in this application:

CONFIG_SCORE_THRESHOLD

(int) Model score threshold, in per mille

The confidence score threshold for each person detection. Detections with confidence scores below this threshold are filtered out.

CONFIG_IOU_THRESHOLD

(int) Boxes IoU threshold, in per mille

The Intersection-over-Union threshold for Non-Maximum Suppression (NMS). Used to eliminate overlapping bounding boxes from multiple detections of the same person.

CONFIG_ARDUCAM_MEGA

(bool)

[Experimental]

Building and running

This application can be found under applications/person_detection in the Edge AI Add-on folder structure.

To build the application, follow the instructions in Building an application for your preferred building environment. See also Programming for programming steps and Testing and optimization for general information about testing and debugging in the nRF Connect SDK.

Testing

After programming the application to your development kit, complete the following steps to test it:

  1. Connect the kit to the computer using a USB cable. The kit is assigned a serial port. Serial ports are referred to as COM ports on Windows, /dev/ttyACM devices on Linux, and /dev/tty devices on macOS. To list Nordic Semiconductor devices connected to your computer together with their serial ports, open a terminal and run the nrfutil device list command. Alternatively, check your operating system’s device manager or its equivalent.

  2. Connect to the kit with a terminal emulator (for example, the Serial Terminal app). See Testing and optimization for the required settings and steps.

  3. Observe LED0 toggling on each frame capture.

  4. Place a person in front of the camera module.

  5. Observe LED1 changing state when detections occur.

  6. Check the serial output for bounding box coordinates and confidence scores.

Application output

The application shows the following output:

[00:00:01.070,782] <inf> main: Person detection start
[00:00:01.211,020] <inf> main: No detections
[00:00:01.721,870] <inf> main: No detections
[00:00:02.200,188] <inf> main: Bounding box 0: head s16, [59.1, 58.9, 121.1, 125.6] score 0.403
[00:00:02.713,248] <inf> main: Bounding box 0: head s32, [37.5, 43.8, 131.0, 124.6] score 0.554

Dependencies

This application uses the following Edge AI Add-on library:

This application uses the following Zephyr libraries: