SDK testing procedures

This following page provides an overview on the various levels of testing within the Amazon Sidewalk SDK. Specifically, it is designed to show what tests are available and to provide detailed instructions on how to run them.

Testing procedures

Testing levels are crucial for a structured approach to quality assurance, enabling you to conduct targeted testing at each stage of development. This helps to reduce defects and ensures the reliability of your software.

The following are the most commonly recognized levels of testing in software development:

  • Unit testing - Focuses on individual functions to ensure they work correctly in isolation.

  • Integration testing - Tests the interaction between integrated components to identify issues arising from their interactions.

  • System testing - Validates the complete and integrated software system against specified requirements.

  • Acceptance testing - Confirms that the software meets the business needs and is ready for deployment, often involving end-user validation.

Note

It is important to distinguish between integration tests and functional tests, as both involve multiple components interacting. In this document, the term integration is specifically used to describe tests that verify the components, but not necessarily the entire system.

For system testing and acceptance testing, you can use Amazon Sidewalk sample variants, such as:

  • CLI - An application to test against the Amazon Sidewalk test specification.

  • Sensor monitoring - An example of a sensor device use case, with release configuration support.

This document primarily addresses Unit and Integration testing provided in the sdk-sidewalk repository.

The following table provides a comparison of Unit and Integration testing in the Amazon Sidewalk SDK:

Testing Characteristics

Test Attributes

Unit

Integration

Scope

One source file

One module

Platform

  • unit_testing (PC)

  • native_sim (PC)

  • nRF development boards

Frameworks

  • Ztest

  • Twister

  • FFF

  • Ztest

  • Twister

Allows mocking

Yes

No

Unit testing

Unit tests are designed to verify individual functions or methods in isolation, ensuring they work as expected. To run the unit test, complete the following steps:

  1. Go to the unit test folder:

    cd sidewalk/tests/unit/<test_name>

  2. Build the test application by running the following command:

    west build --no-sysbuild -b unit_testing

  3. Run the test application by running the following command:

    ./build/testbinary

Integration testing

Integration tests are designed to ensure that multiple components function together as expected. The objectives of those tests are as follows:

  • Ensuring that various files and functions operate as one module, for example, crypto, timer, and storage. Those tests typically use the Sidewalk Platform Abstraction Layer (PAL) API, and have a dedicated Kconfig option that enables or disables the module.

  • Validating that the module performs correctly on the supported SoCs. This type of testing is occasionally referred to as unit test on hardware.

To run the integration test, complete the following steps based on the environment you are using:

  1. Go to the integration test folder:

    cd sidewalk/tests/integration/<test_name>

  2. Build the test application by running the following command:

    west build -b native_sim

  3. Run the test application by running the following command:

    ./build/<test_name>/zephyr/zephyr.exe

West command testing

The west extension commands provided by the add-on are tested on the host, without an nRF development board. The tests are located in the sidewalk/scripts/west_commands/tests directory and use the unittest framework, with the mocked build output, device JSON files and tools in the test_data subdirectory.

In CI, they run whenever a west command or the provisioning tooling changes.

For example, the tests of the west sid provision command create a build directory with the mfg_storage partition, and verify that the generated manufacturing HEX file starts at the partition address and contains the provisioning data from the device JSON file.

The tests replace nRF Util with a stub, so that no nRF development board is needed. Each test prepends a directory holding an executable named nrfutil to the PATH of the tested command, which makes the command call the stub instead of the real tool. The stub is the test_data/nrfutil.py script, which reports a connected J-Link device instead of listing the real ones, and keeps a copy of the HEX file instead of programming it.

To run the tests, complete the following steps:

  1. Install the Python dependencies by running the following command:

    pip install west -r sidewalk/tools/provision/requirements.txt

    You can skip this step if the nRF Connect SDK Python environment is already active, as it provides both west and the script dependencies.

  2. Run the tests from the workspace directory by running the following command:

    python3 -m unittest discover --start-directory sidewalk/scripts/west_commands/tests