.. _edge_impulse_integration: Edge Impulse integration ######################## .. contents:: :local: :depth: 2 `Edge Impulse`_ is a development platform that can be used to enable `embedded machine learning`_ on |NCS| devices. You can use this platform to collect data from sensors, train a machine learning model, and then deploy it to your Nordic Semiconductor device. .. note:: If you are new to the |EAI| and |EI| and need a simpler, automated workflow with minimal configuration, consider following the :ref:`quick_start_edge_impulse` guide. The steps on this page describe the tested integration flow used by the :ref:`hello_ei_sample` sample. Integration prerequisites ************************* Before you start working with |EI|, ensure you have: * Completed :ref:`setting up Edge Impulse steps ` to prepare your |EI| account, tooling, and the |EAI| environment. * An |EAI|-based application. Use the :ref:`hello_ei_sample` sample as a starting point, as it already includes the build-system integration described on this page. Solution architecture ********************* The |EI| integration consists of three main components: * Edge Impulse SDK - A C++ library that provides the inference engine and digital signal processing (DSP) functions required to run machine learning models on embedded devices. The SDK is integrated into the build system as a Zephyr module through the west manifest file of the |EAI|, and is pulled in automatically when the ``CONFIG_EDGE_IMPULSE_SDK`` Kconfig option is enabled. Depending on the generated package, a deployment archive can also include an SDK copy, which is not required by this integration flow. * Deployed machine learning model - A package generated by |EIS| that contains your trained model's parameters and generated source files. It is provided as a :file:`zip` archive. The archive structure depends on the deployment format (:guilabel:`Zephyr library` or :guilabel:`Nordic Axon NPU Library`). * Application code - Your |NCS| application that collects sensor data, feeds it to the inference engine through the Edge Impulse SDK API, and processes the classification results. The typical workflow involves collecting sensor samples, running inference using a sliding window approach, and interpreting the classification results to make decisions or trigger actions (see :ref:`hello_ei_sample` sample). Integration overview ******************** To integrate the |EI| machine learning model into an |EAI| application, first prepare and deploy it for your embedded device. This model is prepared using the `Edge Impulse studio`_ external web tool. It relies on sensor data that can be provided by different sources, for example, data forwarder. .. note:: To collect data, use either a development board supported directly by |EI|, or your mobile phone. As an alternative, use or modify the :ref:`data_forwarder_sample` sample to forward data from a sensor connected to any board available in the |NCS|. Enable the ``CONFIG_DATA_FWD_PROTO_ASCII_MODE`` Kconfig option when using the `Edge Impulse's data forwarder`_ CLI. You can also use :ref:`data_forwarder_host_tool` to visualize and save the streamed sensor data to your local machine. The deployed machine learning model archive is then referenced from your |EAI| application, and the build system unpacks and compiles it automatically. When starting from the :ref:`hello_ei_sample` sample, you do not need to unpack the archive or write any CMake code manually, deployed machine learning model is referenced through the ``CONFIG_EDGE_IMPULSE_MODEL_PATH`` Kconfig option. Integration steps ***************** To generate the model archive, add it to the build system, and verify the result, complete the following steps: 1. :ref:`ug_edge_impulse_adding_preparing` #. :ref:`ug_edge_impulse_adding_building` #. :ref:`ug_edge_impulse_adding_verifying` .. _ug_edge_impulse_adding_preparing: .. rst-class:: numbered-step Preparing the machine learning model ==================================== In this step, you will prepare the machine learning model and obtain a :file:`zip` archive that is required as input for the next step. 1. Prepare your machine learning model. Use `Edge Impulse studio`_ and follow one of the tutorials described in the `Edge Impulse getting started guide`_. For example, to test the solution, complete the `Continuous motion recognition tutorial`_. You will: a. Collect data from sensors and upload them to the |EIS|. #. Design your machine learning model (an *impulse*). #. Deploy your machine learning model. You can obtain the deployment archive in one of the following ways: .. tabs:: .. group-tab:: Using |EIS| web interface Use the |EIS| web interface and deploy in one of the formats supported by this integration: * :guilabel:`Zephyr library` for CPU inference. * :guilabel:`Nordic Axon NPU Library` for Axon NPU inference. To generate the archive: a. Go to the :guilabel:`Deployment` tab. #. Select one of the supported deployment formats. #. Click :guilabel:`Build`. The application generates a :file:`.zip` archive with model files that you will use in the build step. .. figure:: ../../integrations/images/ei_deploy_cpu.png :scale: 50 % :alt: CPU model deployment in |EIS| dashboard CPU model deployment in |EIS| dashboard .. figure:: ../../integrations/images/ei_deploy_axon.png :scale: 50 % :alt: Axon NPU model deployment in |EIS| dashboard Axon NPU model deployment in |EIS| dashboard .. group-tab:: Using |EI| west extensions .. note:: This method currently does not allow deploying Axon NPU models. For Axon NPU deployment, use the |EIS| web interface. |EI| provides west command extensions that let you build and deploy the machine learning model from |EIS| using the command line instead of the web interface. The commands are already configured in the |EAI| west manifest and are ready to use. a. Find the two parameters that are required by the commands: * ``Project ID`` - You can find it in the :guilabel:`Project info` panel under :guilabel:`Dashboard`. .. figure:: ../../integrations/images/ei_project_id.png :scale: 50 % :alt: Project ID in |EIS| dashboard Project ID in |EIS| dashboard * ``API key`` - You can find it under the :guilabel:`Keys` tab in the |EI| project dashboard. .. figure:: ../../integrations/images/ei_api_key.png :scale: 50 % :alt: API key under the Keys tab in |EIS| API key under the Keys tab in |EIS| #. Build the machine learning model by running the following command: .. code-block:: console west ei-build -p -k #. Deploy the machine learning model by running the following command: .. code-block:: console west ei-deploy -p -k As a result, the :file:`.zip` file is downloaded to your working directory. You will use this archive in the build step. For more details on how to use these commands, see the `Automated Deployment with West Commands`_ documentation. .. note:: Not every model can be compiled for the Axon NPU. The Axon NPU has hardware limits (such as maximum filter counts and buffer sizes) that some model architectures may exceed. If the build fails with an error such as ``NRF_AXON_COMPILER_RESULT_INVALID_FILTER_COUNT`` or a similar compile-layer error, the model is not compatible with the current Axon NPU compiler and you might need to: * Adjust the model architecture (for example, reduce filter counts or layer dimensions) and rebuild it in |EIS|. * Fall back to CPU deployment. .. _ug_edge_impulse_adding_building: .. _ug_edge_impulse_adding_building_cpu: .. _ug_edge_impulse_adding_building_axon_npu: .. rst-class:: numbered-step Building the application ======================== To build the application with the integrated |EI| model, you need a :file:`.zip` archive with the deployed model, obtained in the :ref:`ug_edge_impulse_adding_preparing` step. The instructions in this section cover the tested integration path based on the :ref:`hello_ei_sample` sample. The same build-system integration is used for both CPU and Axon NPU deployment formats. The :ref:`hello_ei_sample` sample integrates an |EI| model archive through the ``CONFIG_EDGE_IMPULSE_MODEL_PATH`` Kconfig option and CMake's ``FetchContent``. At configuration time, CMake reads ``CONFIG_EDGE_IMPULSE_MODEL_PATH``, unpacks the obtained :file:`.zip` archive into the build tree, and adds it to the build. The integration step is the same for both formats (CPU archive and Axon NPU archive): set the ``CONFIG_EDGE_IMPULSE_MODEL_PATH`` Kconfig option to point to your archive and build. .. tabs:: .. group-tab:: CPU The CPU deployment archive includes: * CMake and west manifest files (:file:`CMakeLists.txt`, :file:`west.yml`, and :file:`zephyr/module.yml`) used to integrate the Edge Impulse SDK as a Zephyr module. * A :file:`model-parameters/` directory with generated headers describing the model. * A :file:`tflite-model/` directory with the generated TFLite model source files. The exact file names inside :file:`tflite-model/` depend on your model and deployment settings. 1. Place the model archive into your application directory. You can keep the auto-generated archive name, or rename it to a simpler name such as :file:`.zip`. Do not unpack the archive, as ``FetchContent`` extracts it automatically at configuration time. #. Set the ``CONFIG_EDGE_IMPULSE_MODEL_PATH`` Kconfig option to the path of the archive. If the path is not absolute, it is resolved relative to the application source directory. For example, your application's :file:`prj.conf` will contain the following line: .. code-block:: cfg CONFIG_EDGE_IMPULSE_MODEL_PATH=".zip" Keep in mind that some tools might interpret the ``#`` character in the auto-generated name as a comment. If you encounter this, rename the archive to :file:`.zip`. #. Enable the following Kconfig options in your application's :file:`prj.conf`: .. code-block:: cfg CONFIG_CPP=y CONFIG_STD_CPP11=y CONFIG_REQUIRES_FULL_LIBCPP=y CONFIG_EDGE_IMPULSE_SDK=y The ``CONFIG_FPU`` Kconfig option is implied by default if floating point unit (FPU) is supported by the hardware. Using FPU speeds up calculations. #. Disable the ``CONFIG_FP16`` Kconfig option in your application's :file:`prj.conf`. The |EI| library is not compatible with half-precision floating point support introduced in Zephyr. #. Build the application: .. code-block:: console west build -b CMake unpacks the archive into the build directory and compiles the model sources together with the application. .. group-tab:: Axon NPU The Axon NPU archive includes: * Its own copy of the Edge Impulse SDK (:file:`edge-impulse-sdk/`). * An Axon-specific :file:`nordic-axon-model/` directory with generated headers. * A :file:`conf_overlay.conf` file used to configure applicable Kconfig options. * A raw :file:`trained.tflite` model file. For Axon NPU integration, the :ref:`hello_ei_sample` sample's :file:`CMakeLists.txt` file automatically detects when the ``CONFIG_NRF_AXON`` Kconfig option is enabled. Once detected, it wires up the Axon-specific include paths and the model-name compile definition, so you do not need to manually modify CMake. 1. Place the model archive into your application directory. You can keep the auto-generated archive name or rename it to :file:`.zip`. Do not unpack the archive, as ``FetchContent`` extracts it automatically at configuration time. #. Set the ``CONFIG_EDGE_IMPULSE_MODEL_PATH`` Kconfig option to the path of the archive, relative to the application source directory or absolute. For example, in your :file:`prj.conf` file it should look as follows: .. code-block:: cfg CONFIG_EDGE_IMPULSE_MODEL_PATH=".zip" #. Enable Axon in your device tree by adding the following lines to a board overlay file: .. code-block:: dts &axon { status = "okay"; }; #. Enable the following Kconfig options in your application's :file:`prj.conf`: .. code-block:: cfg CONFIG_CPP=y CONFIG_STD_CPP11=y CONFIG_REQUIRES_FULL_LIBCPP=y CONFIG_EDGE_IMPULSE_SDK=y CONFIG_NRF_AXON=y #. Disable the ``CONFIG_FP16`` Kconfig option in your application's :file:`prj.conf`. #. Set the ``CONFIG_NRF_AXON_MODEL_NAME`` Kconfig option to the name of your model. You can find the model name in one of the following locations inside the archive: * The :file:`conf_overlay.conf` file at the root of the archive: .. code-block:: console CONFIG_NRF_AXON_MODEL_NAME="ei_model_nrf_accel_sim_54" * The names of the Axon model header files in :file:`nordic-axon-model/`: .. code-block:: console nrf_axon_model_ei_model_nrf_accel_sim_54_.h nrf_axon_model_ei_model_nrf_accel_sim_54_layers_.h nrf_axon_model_ei_model_nrf_accel_sim_54_test_vectors_.h This means that the model name is ``ei_model_nrf_accel_sim_54``. Add this to your application's :file:`prj.conf` file. .. note:: The :file:`conf_overlay.conf` file in the archive contains predefined Kconfig values for your model, including ``CONFIG_NRF_AXON_MODEL_NAME``. If you prefer, you can include this file as a Kconfig fragment in your build system instead of manually copying the values to :file:`prj.conf`. #. Set the ``CONFIG_NRF_AXON_INTERLAYER_BUFFER_SIZE`` and ``CONFIG_NRF_AXON_PSUM_BUFFER_SIZE`` Kconfig options to fit your model's requirements. .. note:: If the buffers are too small, the inference fails with an error indicating insufficient buffer size and the required size, for example: .. code-block:: console init model ei_model_nrf_accel_sim_54 failed! interlayer buffer too small! Allocated 0, need 36 ERR: nrf_axon_nn_model_validate failed! E: Classification failed with error code: -36 You can find the required buffer sizes in the following places: * In the console output from testing the model in |EIS|, when deploying the model: .. code-block:: console ------------------------------------------------------------------------------- Model EI_MODEL_NRF_ACCEL_SIM_54 results (variant : ei_model_nrf_accel_sim_54) ------------------------------------------------------------------------------- Memory Usage (in bytes) model_const_buffer_size: 1024 interlayer_buffer_size: 36 psum_buffer_size: 0 cmd_buffer_size: 736 * In the :file:`nrf_axon_model__.h` file, for example: .. code-block:: c #define NRF_AXON_MODEL_EI_MODEL_NRF_ACCEL_SIM_54_MAX_IL_BUFFER_USED 36 #define NRF_AXON_MODEL_EI_MODEL_NRF_ACCEL_SIM_54_MAX_PSUM_BUFFER_USED 0 #. Build the application: .. code-block:: console west build -b .. _ug_edge_impulse_adding_verifying: .. rst-class:: numbered-step Verifying the integration ========================= After building the application, flash it to your device and observe the output to confirm that inference runs. 1. Flash the application: .. code-block:: console west flash #. Observe the device output. If you encounter problems, consider the following: * Ensure that the deployment archive was generated with a supported deployment format: * :guilabel:`Zephyr library` for CPU. * :guilabel:`Nordic Axon NPU Library` for Axon NPU. |EI| also provides other deployment formats (for example, :guilabel:`WebAssembly` and :guilabel:`Arduino`) that produce different archive content and are not supported by this integration. If the downloaded archive has an unexpected structure after unpacking, regenerate it using one of the supported formats listed above. * Ensure that the ``CONFIG_EDGE_IMPULSE_MODEL_PATH`` Kconfig option points to an existing :file:`.zip` file. If the path is not absolute, it is resolved relative to the application source directory. A missing or misnamed file typically produces a CMake error from ``FetchContent`` indicating that the resource is not available. * Ensure that all required Kconfig options are enabled in the :file:`prj.conf` file. * For Axon NPU models, check the model name and buffer size Kconfig values. .. _ug_edge_impulse_building: Applications and samples ************************ The following samples demonstrate the |EI| integration in the |EAI|: * :ref:`hello_ei_sample` sample - Demonstrates the deployment of models in |EI| and usage of the inference engine provided by |EI| SDK. * :ref:`data_forwarder_sample` sample - Demonstrates how you can send sensor data to |EIS| using `Edge Impulse's data forwarder`_.