Model lifecycle and observability
Model observability closes the loop between a neural network model running on a device in the field and the tools you used to create it. This page provides an overview of model observability, the production issues it can help you investigate, and where it fits in the Edge AI model lifecycle — from model creation through deployment, field monitoring, analysis, and remediation.
To integrate observability into your application, see nRF Edge AI Observability Library.
Overview
Deployed Edge AI models behave differently in production than in controlled environments. Class distributions shift, transitions between predicted labels change over time, and confidence scores drift as conditions change. Model observability gives you a structured way to capture these statistics on-device and send them off-device for analysis, so you can tell whether a deployed model is still performing as expected.
The Edge AI Add-on implements model observability through the nRF Edge AI Observability Library. The library works with any inference engine that exposes a class-probability vector, an input-feature vector, or both — depending on which built-in or custom metrics you enable — including the nRF Edge AI Library, Axon NPU, and Edge Impulse deployments.
Note
Model observability is designed primarily around nRF Edge AI Library, which provides the most seamless integration. To get the most out of model observability, consider using nRF Edge AI Library for your models.
Use cases
Collected observability data enables the following:
Model quality monitoring — Tracks if prediction confidence and class frequencies stay within expected bounds after deployment.
Dataset collection guidance — Identifies which classes are under-represented or confused in the field and targets data collection accordingly.
Retraining triggers — Detects distribution shift early and decides when a model update is needed before accuracy degrades noticeably.
A/B testing — Compares metric snapshots from devices running different model versions in production conditions.
Hardware and environment health signals — Helps investigate whether a shift is model drift, a degrading sensor, or an unexpected deployment environment.
Software regression detection — Correlates a sudden metric shift with an application update timeline, or rules out post-processing defects when the product misbehaves but metrics stay within bounds.
The Edge AI model lifecycle
A model does not stop evolving once it is deployed. It moves through a cycle of creation, integration, deployment, observation, and analysis. Observability is what tells you when to start the cycle again.
The following diagram shows this lifecycle at a glance as a continuous cycle.
The Edge AI model lifecycle is a continuous cycle.
The following diagram adds remediation paths (steps 6a–6c) for cases where analysis points outside the model.
Model observability in the context of the software and product lifecycle.
The numbered stages in the diagrams map to the following sections, including analysis and troubleshooting paths when issues are detected in production.
Model creation and training
Create and train a model using one of the following tools, depending on your target hardware and workflow preferences:
See Solution overview if you are unsure which tool fits your use case.
Application integration
Integrate the trained model into your nRF Connect SDK application using one of the following guides:
nRF Edge AI Library integration for models trained with the Nordic Edge AI Lab.
Axon inference integration for models compiled directly for the Axon NPU driver.
Edge Impulse integration for models trained with Edge Impulse studio.
While integrating the model, configure the nRF Edge AI Observability Library metrics you want to collect in production. See nRF Edge AI Observability Library for Kconfig options, initialization, and API usage.
Note
nRF Edge AI Library does not currently expose the extracted feature vector through its API, so input-feature metrics are not yet available for models integrated through it. Support for capturing this data from nRF Edge AI Library is planned for a future release.
Deployment
Build and flash your application to your target devices, then ship it as part of your product.
See Wakeword and Keyword Spotting for a reference application that integrates model observability through the CONFIG_MODELS_OBSERVABILITY Kconfig option.
Later deployments — whether they carry a retrained model, an application defect fix, or a hardware-related configuration change — reach devices already in the field through your product’s device firmware update (DFU) mechanism. See Bootloaders and DFU for the DFU mechanisms available in nRF Connect SDK.
If your application uses Memfault as observability transport, its OTA release management can stage an update to a cohort of devices so you can confirm it performs as expected using the same observability metrics before releasing it to the rest of the fleet. See Memfault Over-the-Air Updates for details.
Note
Today, a model is built into the application binary, so updating it means shipping a full application update. Support for updating only the model, without re-flashing the rest of the application, is planned.
Observing production behavior
Once the product is in the field, the nRF Edge AI Observability Library accumulates metric snapshots from every inference and periodically uploads them off-device for analysis. The library ships with a Memfault transport that stages snapshots as a Memfault Custom Data Recording (CDR) uploaded to nRF Cloud, or you can implement a custom transport over any connectivity solution your product already uses.
Analyzing collected data
Analyze uploaded metric snapshots to determine whether the deployed model still meets your quality bar, and if not, whether the cause is the model, an application defect, the hardware, or the deployment environment.
Note
The Nordic Edge AI Lab does not yet provide a way to analyze metric snapshots, but this is planned for future releases. For now, download and decode raw payloads with the decode_edgeai_obsv_cdr script and feed them into your own analysis pipeline.
See Built-in metrics for the full list of built-in metrics and what each one indicates.
A gradual trend over weeks or months is typical of drift — whether caused by the environment, changing usage patterns, or the model becoming stale.
A sudden, sustained shift that starts right after an application update points to a pre-processing regression.
Visible product misbehavior after an update, with metrics still within bounds, points to a post-processing regression (see Fix pre-processing or post-processing defects).
See Diagnosing production issues for the detailed patterns behind each of these cases.
Addressing the root cause
Once analysis identifies what is behind an observed issue, address it depending on where the root cause lies. If the model itself is at fault, retraining closes the loop back into the model lifecycle. If the root cause is application code or the hardware and deployment environment, the fix happens outside that loop.
Retrain or update the model
When analysis indicates that a model update is needed, retrain or fine-tune the model using the same tool you used to create it, guided by the dataset gaps and confusion patterns that observability data revealed. Then repeat application integration for the updated model and deploy it to devices in the field (see Deployment).
A model update does not always require full retraining. Sometimes observability data instead points to a configuration issue — for example, a stale anomaly threshold or a change in feature scaling percentiles — that can be corrected without changing the model itself.
Fix pre-processing or post-processing defects
When a software regression — rather than model or hardware drift — is behind an observed issue, fix the corresponding code path:
Pre-processing — for example, correct an error in sample windowing that no longer matches what the model was trained on.
Post-processing — for example, correct a decision threshold applied to the wrong class index.
Deploy the corrected application to devices in the field (see Deployment).
Revise hardware or the deployment environment
Some issues that observability data reveals cannot be fixed by changing software alone. A failing sensor, an enclosure that distorts the signal, or a deployment environment consistently different from what the product was designed for are hardware or product design problems.
Depending on the root cause, the fix might be replacing or recalibrating the affected sensor, revising the product’s mechanical or electrical design, or updating installation guidance for the deployment environment. Once you revise your design and introduce improvements, continue observing the product to confirm that the metrics return to expected levels.
Diagnosing production issues
Not every quality issue that observability data reveals traces back to the model itself. The following patterns are paths to explore when metrics move outside expected bounds. Which one applies depends on your application, sensors, and deployment context.
Model drift
Gradual shifts in class distributions, confidence scores, or streak lengths over weeks or months can suggest that the model no longer matches real-world conditions. Examples to look for include a rising prediction switching rate, a probability distribution that drifts away from the training-time baseline, or a class streak distribution dominated by short, flickering streaks instead of stable detections.
Hardware and deployment environment
A sustained drop in confidence or a rise in ambiguous predictions can also come from degrading or poor-quality hardware — for example, a microphone that no longer picks up enough signal — or from a deployment environment the model was not trained for — for example, a location with more background noise than the training data. If investigation points here rather than at the model, the fix may lie in revising the hardware or the product’s environment, not retraining (see Revise hardware or the deployment environment).
Input-feature metrics, such as the mel energy descriptor and mel spectral descriptor, observe the raw feature vector fed into the model rather than its output. A sustained drop in mean energy across many devices can suggest a sensor issue rather than the model, while a shift in spectral band ratios can indicate a noisier acoustic environment. If the shift appears on all devices at once, right after an update, pre-processing code is a more likely place to look than the sensors themselves.
Application pipeline issues
Because metrics are computed on the probability and feature vectors at inference time, a pre-processing bug can distort them in ways that look like model drift — typically as a sudden change that coincides with a firmware update rather than a slow trend. This includes defects that basic input validation would not catch, for example an accelerometer’s full-scale range shifting from 2g to 16g without the application accounting for it.
A post-processing issue downstream of inference — for example, a decision threshold applied to the wrong class index — does not distort these metrics directly, but can surface indirectly when the product visibly misbehaves in the field while the metrics stay within expected bounds (see Fix pre-processing or post-processing defects).
Real-world usage patterns
Observability data also reflects how people actually use the product, which does not always match the assumptions made while collecting training data. For example, a wearable held or worn at unexpected angles can show up as class confusion (see Transition matrix) or a higher switching rate. Likewise, a keyword-spotting model can show low decision confidence or repeatedly fail to reach a confident, held detection for a command, consistent with real-world phrasing or accents the model was not trained on.
Depending on the case, one path forward is to retrain with data that better reflects real usage; another is to adjust the product itself (for example, clearer user guidance or ergonomics) rather than the model.
Next steps
nRF Edge AI Observability Library — Library reference, built-in metrics, Kconfig, and integration steps.
Wakeword and Keyword Spotting — Reference application with option to enable model observability.
Integrations — Model integration guides for Nordic Edge AI Lab, Axon, and Edge Impulse.
decode_edgeai_obsv_cdr script — Host-side payload decoder.