Wakeword and Keyword Spotting

This application demonstrates wakeword detection and keyword spotting from a digital microphone stream using nRF Edge AI Library and Axon-based inference. It also features integrated model observability, allowing you to monitor and analyze inference metrics during operation.

Application overview 

The application samples single-channel 16 kHz audio from a PDM microphone and feeds it to nRF Edge AI models. The wakeword phrase used by the bundled model is “Okay Nordic”. In wakeword detection stage, model output is postprocessed with a predictions history window. Parameters used for postprocessing are prediction probability threshold, predictions history length and number of predictions above threshold in predictions history. When a wakeword is detected, the application switches to keyword detection stage.

The bundled keyword spotting model supports the following keywords:

Go
Stop
Up
Down
Yes
No
On
Off
Right
Left

In the keyword spotting stage, the application smooths class probabilities using an exponential moving average and reports a detection when class-specific criteria are met. After period without any keywords spotted the application switches back to wakeword stage.

You can also configure the application to stay in one of these stages using the application-specific Kconfig options.

Observability 

The application collects runtime observability metrics using the nRF Edge AI Observability Library. Enable the CONFIG_MODELS_OBSERVABILITY Kconfig option to wire metrics into the bundled models. Each model stage owns its observability context and Memfault transport binding in the src/ww/wakeword.c and src/kws/kws.c files.

The wakeword stage only registers the Probability distribution metric. The keyword spotting stage registers the Probability distribution and Transition matrix metrics.

Metrics are collected every 24 hours (see the CONFIG_NRF_EDGEAI_OBSV_MEMFAULT_AUTO_COLLECT Kconfig option) and sent to the Memfault using Custom Data Recording registered by the nRF Edge AI Observability Library.

Replacing models 

You can replace the bundled models using the Text to Wake Word Detection feature of the Nordic Edge AI Lab or one of ready-to-use models.

To replace the model used in wakeword detection stage complete the following steps:

Replace the files inside src/ww/nrf_edgeai_generated directory with files downloaded from Nordic Edge AI Lab.
In ww_init(), set the ww_model pointer using the model getter from the generated files.
Adjust the Kconfig options to tune wakeword detection postprocessing to your model.

To replace the model used in keyword spotting stage complete the following steps:

Replace the files inside src/kws/nrf_edgeai_generated directory with files downloaded from Nordic Edge AI Lab.
In kws_init(), set the kws_model pointer using the model getter from the generated files.
Update the keyword_detection_ctxs array in the src/kws/kws.c file with keyword labels from src/kws/nrf_edgeai_generated/nrf_edgeai_user_model_labels.h file and thresholds for keyword spotting.
When using the observability feature, set the CONFIG_NRF_EDGEAI_OBSV_MAX_CLASSES Kconfig option to number of keywords spotted plus 2 for auxiliary classes.
Adjust the Kconfig options to tune keyword spotting postprocessing to selected model.

Requirements 

The application supports the following development kit:

Hardware platforms	PCA	Board name	Board target
nRF54LM20 DK	PCA10184	nrf54lm20dk	`nrf54lm20dk/nrf54lm20b/cpuapp`

The application also requires PDM digital microphone connected to the pins specified in the boards/nrf54lm20dk_nrf54lm20b_cpuapp.overlay file. The application is expecting audio data to be provided on left channel.

Pin mapping 

The application was tested with an Adafruit PDM MEMS Microphone module. It can be powered from 1.8V VDD:IO supply. The following table show how to connect this module to the DK:

Adafurit DMIC	nRF54LM20 DK
`3V`	`VDD:IO`
`GND`	`GND`
`SEL`	`GND`
`CLK`	`P1.4`
`DAT`	`P1.5`

The SEL pin is responsible for selecting audio channel and connecting it to ground selects left channel.

To use other microphones, adapt configuration parameters of PDM in the src/dmic.c file.

User interface 

LED0:

Behavior depends on the selected application mode:

Lit when keyword spotting stage is active in wakeword-gated keyword spotting mode.
Blinks for one second when wakeword is detected in wakeword only mode.
Stays off in keyword spotting only mode.

LED1:

Blinks for one second on each keyword spotted.

UART30 (VCOM0):

Prints runtime state messages:

Waiting for wakeword
Wakeword detected
Waiting for keywords
Keyword spotted: <name>
Keyword spotting window timeout

Configuration 

See Configuring and building for information about how to permanently or temporarily change the configuration.

Configuration options 

The following application-specific Kconfig options are used in this application:

CONFIG_APP_MODE_WW_GATED_KWS

(bool) Wakeword-gated keyword spotting mode

Run keyword spotting model only after wakeword was detected. After configurable duration without any keyword spotted switch back to wakeword detection.

CONFIG_APP_MODE_WW_ONLY

(bool) Wakeword detection only mode

Run wakeword detection continuously without switching to keyword spotting.

CONFIG_APP_MODE_KWS_ONLY

(bool) Keyword spotting only mode

Run keyword spotting continuously without the need to detect wakeword first.

CONFIG_WW_PROBABILITY_THRESHOLD

(int) Wakeword probability threshold

Probability threshold for wakeword detection model. The value is expressed in units of 1/1000.

CONFIG_WW_HISTORY_SIZE

(int) Size of wakeword predictions history

Size of history of previous results from wakeword detection model. Each entry corresponds to 30 milliseconds of past audio.

CONFIG_WW_COUNT_THRESHOLD

(int) Threshold for wakeword detection from predictions history

Number of predictions above threshold in predictions history required to confirm wakeword detection.

CONFIG_KWS_PERIOD_MS

(int) Length of keyword spotting window in wakeword-gated mode

CONFIG_KWS_EMA_ALPHA

(int) Exponential moving average coefficient for class probability

Coefficient for new entries in exponential moving average calculation for average class probability. The value is expressed in units of 1/1000.

CONFIG_MODELS_OBSERVABILITY

(bool) Edge AI observability metrics

Wire observability into the bundled models. Each model file owns its observability context and Memfault transport binding.

CONFIG_MODELS_OBSERVABILITY_MDS

(bool)

Additional configuration 

Check and configure the following library options that are used by the application:

CONFIG_MEMFAULT_NCS_PROJECT_KEY - A key to your Memfault project
CONFIG_NRF_EDGEAI_OBSV_MAX_CLASSES - Number of classes the observed model predicts
CONFIG_NRF_EDGEAI_OBSV_MEMFAULT_AUTO_COLLECT_INTERVAL_SEC - Interval of automatic metrics collection. For testing, use a shorter interval and enable server debug mode for the device in the Memfault dashboard.

Build types 

For build type overview and guidance on how to run them, see Custom build types and Providing CMake options. The application supports the following build types:

Wakeword and Keyword Spotting build types
Build type	File name	Description
Debug (default)	`prj.conf`	Debug version of the application with logging and assertions enabled.
Release	`prj_release.conf`	Release version of the application with logging disabled and compiler optimizations.

Configuration files 

The application provides predefined observability.conf configuration file for enabling observability. This file enables the CONFIG_MODELS_OBSERVABILITY Kconfig option, Bluetooth LE, Memfault Diagnostic Service and required dependencies to provide observability for bundled models.

Check Providing CMake options and use EXTRA_CONF_FILE variable to include this configuration file.

Note

Set the CONFIG_MEMFAULT_NCS_PROJECT_KEY Kconfig option to your Memfault project key to successfully send metrics to Memfault.

Building and running 

This application can be found under applications/ww_kws in the Edge AI Add-on folder structure.

To build the application, follow the instructions in Building an application for your preferred building environment. See also Programming for programming steps and Testing and optimization for general information about testing and debugging in the nRF Connect SDK.

Testing 

After programming the application to your development kit, complete the following steps to test it:

Connect the kit to the computer using a USB cable. The kit is assigned a serial port. Serial ports are referred to as COM ports on Windows, /dev/ttyACM devices on Linux, and /dev/tty devices on macOS. To list Nordic Semiconductor devices connected to your computer together with their serial ports, open a terminal and run the nrfutil device list command. Alternatively, check your operating system’s device manager or its equivalent.
Connect to the kit with a terminal emulator (for example, the Serial Terminal app). See Testing and optimization for the required settings and steps. The application is using both serial ports.
Open one terminal for Zephyr logs and one for control output from UART30.
Say the wakeword phrase “Okay Nordic”.
Observe LED0 light up.
Say one of the bundled model keywords (for example, “Yes” or “No”).
Observe that LED1 blinks for one second.
Stop speaking and wait for timeout.

If you have enabled the model observability, also complete the following steps:

Connect gateway to your device. Ready-to-use gateways are nRF Connect Device Manager or Memfault WebBluetooth Client. Check Peripheral MDS sample for details.
Wait for a duration set by the CONFIG_NRF_EDGEAI_OBSV_MEMFAULT_AUTO_COLLECT_INTERVAL_SEC Kconfig option.
In a web browser, navigate to Memfault.
In the left-hand menu, go to CDR Payloads.
Select a CDR payload from your test device with “edgeai_observability” as reason. You can filter the list of CDR payloads by device name and time.
Download the payload for metrics analysis.
Decode it on a host PC using scripts/decode_edgeai_obsv_cdr/decode_edgeai_obsv_cdr.py script in binary mode:
```
./scripts/decode_edgeai_obsv_cdr/decode_edgeai_obsv_cdr.py --binary --file <downloaded>.bin
```
See Decoding CDR payloads for installation and other input modes. Payload format is defined in the lib/nrf_edgeai_obsv/obsv.cddl file.

Application output 

The application shows the following output from UART30 (VCOM0):

Waiting for wakeword
Wakeword detected
Waiting for keywords
Keyword spotted: Yes
Keyword spotting window timeout

Dependencies 

This application uses the following Edge AI Add-on library:

This application uses the following Zephyr libraries:

Logging
Audio (DMIC)
UART Driver

API documentation 

Wakeword and Keyword Spotting application