Building a Robust Testing Framework for Embedded Development: The Journey of Pykiso

What was our target?

Our goal was to design and implement a comprehensive integration test framework tailored for embedded developers. Specifically, we aimed to:

• Create an integration test framework that’s suitable for the unique needs of embedded development.
• Ensure that the framework has a familiar look and feel for embedded developers, making adoption easier.
• Support the shift-left testing paradigm by enabling testing earlier in the development cycle, ultimately leading to higher code quality and faster delivery times.

What we built

After extensive research and development, we created a solution with the following core components:

• Modular Hardware Setup: We defined a modular approach, allowing users to describe a hardware setup and define how to interact with it in a structured way. This approach supports different configurations and customizations, making it adaptable to various embedded systems.
• Robust Components with Advanced Logging: Each component is designed to be highly reliable and includes an advanced logging mechanism. This feature helps developers quickly identify and troubleshoot any potential issues in the interactions with the hardware.
• Advanced Testing Framework Based on Python unittest: Although based on Python, our testing framework is designed to feel as familiar as traditional C/C++ embedded testing frameworks, easing the learning curve for embedded developers who are new to Python. For example, we will have a typical test design approach composed with a test-suite definition with a setup and teardown and per test-case, the possibility to define a setup, run and teardown function. Through ids, the developer would be able to define the order of execution of its test.

Concretely, it looks like:

1. We load a configuration file describing the HW interaction
2. We generate the associated software components
3. We collect the tests to execute
4. We execute the tests
5. We generate the reports

What we achieved

Our testing framework, Pykiso, has already shown success in continuous testing across multiple levels, from component to system tests, in high-demand environments, such as Bosch eBike. The framework is now integral of the verification and validation strategy of embedded systems across various stages of development.

What we missed

Despite our achievements, there are areas we feel we could have improved:

• Increased visibility for the Open-Source project: The project hasn’t gained as much traction or community contribution as we hoped, limiting its growth potential.
• Collaborations with other Open-Source projects: We missed opportunities to connect with other projects that could benefit from or contribute to Pykiso, such as zephyr or FreeRTOS.
• TestApp for embedded system validation: We did not complete a dedicated TestApp for running on embedded devices, which would allow developers to validate their software components directly on the target system.

Why did we miss these targets?

Several factors contributed to these gaps:

• Highly opinionated framework: The framework’s design is intentionally opinionated, which helps enforce certain standards but has led to a high learning curve. This decision has resulted in strong reactions among users. Some highly appreciate its rigor, while others found it very challenging.
• Partial mismatch with target user needs: Many embedded software developers prefer to continue using familiar tools. For some, transitioning to Python, even though the framework reminded them on how they develop in C/C++, they still needed to learn a little bit how Python works to be able to use it. On another side, the developers that embraced Python, realized pretty earlier that the framework did not allow the flexibility that Python usually enable.

What do we need to do?

Our focus now is on learning from feedback to close these gaps and broaden the framework’s appeal. Our strategy includes the following key actions:

1. Separate the Testing Framework from Test-Setup creation
   • Decoupling these elements will allow users to define and manage HW interactions in a more Pythonic way. See pykiso-design-overview
   • This separation will also make Pykiso more accessible for exploratory testing and support various interfaces for different testing needs. A user would be able to manage the interaction with its hardware independent of any other framework. If he wants the "embedded feel", he can use our flavored testing-framework, else, he could integrate the pykiso-hw-setup capability to any other framework. We could expose the setup via different communication channels so it can be controlled from even 3rd party solutions remotely.
     
2. Develop auxiliary tools and connector acceptance tests
   • By providing tools for creating minimal Pykiso modules, we’ll enable developers to quickly implement compliant auxiliaries and connectors, speeding up validation and reducing friction for new contributors. Auxiliaries need to fulfil specific APIs and behavior as defined in pykiso-how-to-auxiliary which can be at first difficult to do. These tools should enable contributors to quickly see what is missing or may not work as expected.
     
3. Foster collaboration with Open-Source projects
   • We aim to connect with projects like Zephyr and other popular embedded frameworks to help them leverage Pykiso in testing their hardware integration. This will enable mutual growth, enhance testing capabilities, and attract more contributors. For example, zephyr-ztest and unity focus more on unittests on the target embedded system. By making pykiso remotely usable, we could extend these unittests to also validate the communication layer they use.
4. Refine framework to align with user needs
   • Based on the feedback, we’ll streamline aspects of the framework to reduce the learning curve and appeal more broadly to embedded developers while maintaining the core functionalities and standards. User feedback will be analyzed and a roadmap will be created according to the github issue voting solution.
     

Moving forward

As a maintainer, I am confident that Pykiso has a promising future. With ongoing improvements and new partnerships, we hope to create a tool that not only supports rigorous testing in embedded development but also fosters a thriving community around open-source innovation.

If you’re interested in using Pykiso or supporting its development, leave us a star on GitHub and feel free to get in touch! Together, we can make a difference in embedded systems testing.

---