This blog originally ran on the Antmicro website. For more Zephyr development tips and articles, please visit their blog.

Solving problems that require real-time calculations and precise control typically calls for using an RTOS. While we have been working with a wide variety of RTOSs for various applications (like Contiki-NG for IoT, RTEMS for space applications, eCos for satellite equipment, FreeRTOS in many other fields etc), Antmicro’s RTOS of choice these days has been Zephyr, a Linux-Foundation driven, well-structured, vendor-independent and scalable real-time OS. We’ve ported and adapted Zephyr to many platforms, encouraged its use as a standard on RISC-V, promoted it in less standard contexts like FPGA devices.

So if you have a single device with a real-time requirement, typically it’s not that hard to decide how to approach the problem – just use Zephyr!

The problem starts when there are multiple heterogeneous devices that have to communicate in a standardised and robust way, performing a complex operation involving a network protocol while never leaving the “real-time” world. Scenarios like this are typical in the aerospace, automotive, robotics industries, and increasingly those industries are looking to reuse technologies known from the commercial/consumer market to leverage the massive scale offered by omnipresent, commodity tech.

For your everyday use case, the easiest way to connect multiple devices is of course Ethernet, but plain old Ethernet does not list real-time capabilities in its dictionary – how then can it be used in a real-time use case?

The set of standards that define Time Sensitive Networking is the answer to that problem. Leveraging the physical and logical foundations of Ethernet and extends it to cover real-time use cases by defining different aspects of time sensitive communication: clock synchronization, traffic shaping, scheduling, fault tolerance etc.

TSN seems then like a good fit, and sure enough, open source support for TSN is widely available in Linux. In the RTOS world however, there has previously not existed a proper implementation of TSN, readily available and tested on real hardware platforms, well, not until Zephyr 1.13!

Initial work: towards a TSN implementation in Zephyr

As a member of the Zephyr Project, Antmicro is always excited to add new functionalities to the OS, especially in fields that open it up for adoption in new use cases. Here, we were happy to work with another Zephyr project member, Intel on getting gPTP support added to Zephyr. “gPTP” stands for “generic Precision Time Protocol” and is responsible for clock synchronization. When we joined the project it was already in progress, but far from being finished. We implemented the missing state machines and fixed various bugs in the existing code.

The initial target was making Zephyr’s clocks synchronize with external Grand Masters.

Our focus was getting it to work on Microchip SAM E70 Xplained. At that time, the platform already had a Zephyr port (including the Ethernet driver), but it lacked drivers for the PTP clock.

The TSN support was also tested on the NXP FRDM K64F development platform which also has a Zephyr port and hardware timestamping support.

After initial support was done and merged, we proceeded with configuring Zephyr nodes as Grand Masters, as well as ensuring operational Zephyr-to-Zephyr clock synchronization.

Qav: an important part of TSN

PTP is only a part of Time Sensitive Networking. Another important part of TSN is queue management.

The platform of our choice (SAM E70 Xplained) has multiple hardware queues built into its MAC controller, which allowed us to use the same platform to extend Zephyr’s TSN capabilities.

Antmicro implemented support for credit-based shaper algorithms in Zephyr, which are described in the 802.1Qav standard.

The work in that area required us to design an API to manage the Qav-capable Ethernet queues. Through this API/management interface, we made it possible to set and read various parameters, like idle slope, delta bandwidth, traffic class, etc.

Additionally, some status parameters were implemented. These are now shown in the regular networking shell in Zephyr for the supported network interfaces.

Running a basic Zephyr gPTP sample application

To find out how to run the basic gPTP sample, please refer to Zephyr’s official documentation.

More general info about the gPTP stack implementation can be found in a dedicated chapter.

Testing in Renode Cloud Environment

A network stack plays a critical role in an operating system like Zephyr. It is also constantly under very heavy development by various parties. Our work on the TSN/gPTP support was heavily influenced by all the changes in the networking subsystem. As can be expected in large development campaigns, these completely unrelated things would break our implementation repeatedly.

The reason for that was lack of more sophisticated testing of the setup. Sure, there were multiple unit tests which directly tested our stack implementation, but Time Sensitive Networking can be broken by seemingly minor changes in other parts of the networking stack.

Obviously, network protocol testing is difficult. You can either use synthetic tests that easily get outdated and don’t really reflect real life scenarios or you can create complicated physical network setups connected to a CI system – which is costly, difficult to maintain and creates only a single, static configuration.

A much better, more scalable solution is to use simulation. With Antmicro’s Renode open source simulation framework you can create script-defined complex configurations, allowing you to verify virtually every scenario imaginable.

In Renode’s 1.7 release, Antmicro added support for the SAM E70 platform, along with Ethernet with gPTP capabilities. With these new features we were able to create a CI setup testing upstream Zephyr in a virtual environment.

And thanks to an integration with the Robot Framework, it’s very easy to create new test cases in Renode. That’s why, for Zephyr, we decided to create a suite of tests verifying a range of aspects of a single application.

This lines up perfectly with the introduction of Renode Cloud Environment – a new CI system introduced by Antmicro, that you’ll be able to read about more on our blog soon. Here is a sneak peek of the TSN testing setup running in RCE.

First, Renode verifies if the board sends a PTP packet, which means that the PTP stack is started properly. Next, we analyze its reception and the proper reaction from the recipient’s OS. We analyze whether the compile-time configuration of the PTP stack is properly reflected in its runtime, and the highest level properties: whether the correct Grand Master node has been selected and whether the slave nodes are properly synchronizing their clocks.

The whole testing setup can be easily recreated with upstream Renode and Zephyr. For instructions, please refer to the TSN testing tutorial.

Building a TSN system?

If you’d like to use TSN is your system, and feel that an RTOS like Zephyr is a good fit for your needs, be sure to reach out to us at contact@antmicro.com – we’d be happy to help you apply TSN on existing and new hardware, and perhaps in simulation, for real-world use cases!

Written by Ivo Clarysse, CTO at Blue Clover Devices

This blog previously ran on the Blue Clover Devices website. You can view the website here.

IoT OS Landscape

These days there’s no lack of operating systems to choose from for embedded systems; Wikipedia counts about 100 of them. The Eclipse survey still shows Linux leading the pack, with Windows, FreeRTOS and Mbed OS being widely used as well.

For devices that have the necessary resources, full-blown operating systems like Linux (Android) or Windows dominate the field, but for more constrained devices, there’s a wide range of systems being used.

The Eclipse IoT Developer Survey 2019 shows more use of actual operating systems in IoT device firmware, as opposed to bare-metal programming or building on top of a minimal kernel.

Linux is widely used for IoT applications, but requires at least a Cortex-A MCU (or equivalent), and is not the preferred choice for more limited systems, such as Cortex-M.

FreeRTOS is quite popular in the embedded world and gets more support after the acquisition by Amazon in 2017. However, FreeRTOS is a bare operating system.  Everything else such as drivers, file systems, crypto modules, network stacks, middleware, and a bootloader must be added from other sources.

ARM’s Mbed OS has the out-of-the-box integration with ARM’s Pelion Device Management going for it, making it a great choice to learn about IoT device provisioning, connection and management through LwM2M. Provided by ARM, it obviously does not support popular IoT platforms like ESP32 or RISC-V.

Mynewt has everything you could wish for in an operating system for resource constrained IoT devices, but the BSP support is fairly limited.

Zephyr originated from the Virtuoso DSP operating system which initially got rebranded as “Rocket” kernel, following its acquisition by Wind River Systems, but became Zephyr in 2016 when it became a Linux Foundation hosted Collaborative Project. Major sponsors and contributors of this open source collaborative effort include Intel, Nordic Semiconductor, NXP, SiFive, Synopsys and TI.

Like many other operating systems, Zephyr provides:

• Secure bootloader (MCU Boot)
• Kernel
• Network stacks
• File systems (NFFS, LittleFS, NVS)
• Middleware (including the MCUmgr OTA mechanism and LwM2M client functionality)
• Device drivers

Zephyr Licensing

Zephyr is mostly licensed under the Apache 2.0 license, but drivers from Nordic and NXP are licensed under the permissible BSD-Clause-3 version, although some of the build tooling is GPLv2.

Example Project

An example Zephyr firmware project can be found on https://github.com/bdevices/ly10-zephyr-fw

This project is a demonstration firmware for our nRF52-based “LY10” demo board, which we ship with our Production Line Tool.

Clone

Use the following Terminal command to obtain a local copy of this example project.

git clone https://github.com/bcdevices/ly10-zephyr-fw.git

Building

To simplify building the firmware, we’ve setup this project with Docker-based build scripts, avoiding the need to install anything else than Docker itself.

To build:

make docker

The resulting ly10-zephyr-fw-VERSION.hex file can be programmed through the west tool (if you’ve installed the Zephyr tooling locally), or use target-specific Vendor tools (nRF Connect Programmer).

Source Code

The demo firmware source code is very simple, contained in the src/app folder:

• main.c: Main controller
• app_ble.c: BLE adapter
• app_buzzer.c: Control PWM-driven piezo buzzer
• app_ledstrip.c: Control APA102 LED strip
• app_sensor.c: Read out SHT3XD sensor

Board Definitions

Building for one of the target boards directly supported by zephyr is a matter of specifying the correct board name.

Since the LY10 is a custom board that is not defined in the upstream Zephyr project, we’re defining a custom board in boards/arm/ly10demo:

• board.cmake : cmake build file, specifying programming adapters to use.
• Kconfig : system-wide Kconfig settings affected by the LY10 board
• Kconfig.board : board-specific Kconfig settings
• Kconfig.defconfig : Kconfig defaults when building for an LY10 board
• ly10demo_defconfig : devicetree defaults when building for an LY10 board
• ly10demo.dts : board devicetree source
• ly10demo.yaml : YAML definition

References

• Zephyr RTOS Project, Linux Foundation hosted Collaboration Project
• Zephyr 2.0.0 Release Notes, Zephyr Project (September 2019)
• IoT Developer Survey 2019, Eclipse Foundation (April 2019)
• Zephyr Now Supports OMA LwM2M Protocol! What Can it Do for Me?, Zephyr Project (March 2018)
• Mynewt MCUmgr

About the Author

Ivo Clarysse is CTO at Blue Clover Devices. At heart, he is a software engineer with extensive experience working on embedded systems software and linux device drivers.

Amy Occhialino is the Chair of the Zephyr Project Governing Board and Director of Software Engineering at Intel. She has more than 20 years experience in technology and has recently shared insight into the social change in open source software at several conferences. She recently gave the keynote at The Consortium for Computing Sciences in Colleges and the Society of Women Engineers. In this blog, Amy shares some of that insight from her talks. 

You may know Intel only as a hardware company, and in many ways this is true.  Intel’s core business is semiconductor design and manufacturing. What may be news to you is that Intel has spent close to two decades working in the open source software community, collaborating on projects that enhance Intel Architecture and advocating for the beauty, elegance, and possibilities that exist within open source software development. 

Intel software engineers are key contributors to the Linux kernel, are Linux kernel maintainers, and hold many software standards and leadership positions within open source projects and communities.  Open source software development is an enormous commitment and investment at Intel. In fact, we went from sponsoring 12 open source projects to 200 spanning cloud, edge, and device growth over the last decade. The Zephyr Project is a great example of this. 

Leading the future will require a wide range of perspectives, backgrounds, and ideas to effectively solve the world’s toughest challenges. We, as a technology community, are bound to fail if we do not demonstrate a commitment and passion to increasing and achieving full representative diversity within the open source software industry.  

Our success will be threatened in three ways:

• Market Failure:  Without diversity-fueled creativity, innovation is stifled.  The same perspective and thoughts are generated and reinforced, preventing new solutions from emerging.

• Customer Failure:  We will lose customers because we don’t listen to them, engage with them, understand them, and learn from them in a full perspective of ways.

• Talent Failure:  We will lose top talent because individuals with diverse backgrounds feel out of place in our culture and environment.

The Zephyr Project is committed to achieving an inclusive and diverse open source environment. We encourage any and all developers interested in the RTOS landscape to join the conversation and share your interests and talents with the community. Change doesn’t happen all at once, it’s incremental, one person doing one thing every day. 

For me, I personally increase diversity in my own teams through my hiring practices, I have spent a decade creating support systems through my leadership of women’s groups, and I actively advocate for social change within Intel and my tech community.  I am proud that Zephyr is part of this as an open source project with a high amount of diversity 

Open source software gives us the potential conditions, but we must actively engage with it, and monitor it, for it to be what we want it to be.

If you’re new to Zephyr RTOS, please see our Getting Started Guide and check out our Contributor Guide. Or, you can join the conversation and ask questions on our Slack channel or Mailing List and follow #zephyrproject on IRC.

This blog originally ran on the Antmicro website. For more Zephyr development tips and articles, please visit their blog.

Antmicro’s open source simulation framework, Renode, was built to enable simulating real-life scenarios – which have a tendency to be complex and require hybrid approaches.

That’s why, besides other things, the Renode 1.7.1 release has introduced an integration layer for Verilator, a well known, fast and open source HDL simulator, which lets you use hardware implementations written in Verilog within a Renode simulation.

When you are working on ASIC or FPGA IP written in an HDL, forming a part of a bigger system with unknowns both in the hardware and software, many things can go wrong on multiple levels. That’s why ultimately it’s best to test it within the scope of the full system, with drivers and test software, in a real-world use case. Simulating complete platforms with CPUs and all peripherals using actual HDL simulation, however, can be too slow for effective software development (and sometimes downright impossible, e.g. when access to the entire SoC’s HDL is not available). Renode models will give you better speed and flexibility to experiment with your architectural choices (as in the security IP development example of our partner Dover Microsystems) than HDL, but there might still be scenarios where you could quickly try to directly use complex peripherals you already have in HDL form before going on to model them in Renode. For these use cases Antmicro has enabled the option of co-simulating HDL in Renode using Verilator. Co-simulating means you’re only ‘verilating’ one part of the system, and may in turn expect a much faster development experience than with trying to perform an HDL simulation of the whole system.

In the 1.7.1 release of Renode you will find a demo which includes a ‘verilated’ UARTLite model connected to a RISC-V platform via the AXI4-Lite bus running Zephyr.

Integration layer overview

The integration layer was implemented as a plugin for Renode and consists of two parts: C# classes which manage the Verilator simulation process, and an integration library written in C++ that allows you to turn your Verilog hardware models into a Renode ‘verilated’ peripheral.

The ‘verilated’ peripheral is compiled separately and the resulting binary is started by Renode. The interprocess communication is based on sockets.

How to make your own ‘verilated’ peripheral

An example ‘verilated’ UARTLite model is available on Antmicro’s GitHub.

To make your own ‘verilated’ peripheral, in the main cpp file of your verilated model you need to include C++ headers applicable to the bus you are connecting to and the type of external interfaces you want to integrate with Renode – e.g. UART’s rx/tx signals. These headers can be found in the integration library.

// uart.h and axilite.h can be found in Renode's VerilatorPlugin
#include "src/peripherals/uart.h"
#include "src/buses/axilite.h"

Next, you will need to define a function that will call your model’s eval function, and provide it as a callback to the integration library struct, along with bus and peripheral signals.

void eval() {
#if VM_TRACE
  main_time++;
  tfp->dump(main_time);
#endif
  top->eval();
}

void Init() {
  AxiLite* bus = new AxiLite();

  //==========================================
  // Init bus signals
  //==========================================
  bus->clk = &top->clk;
  bus->rst = &top->rst;
  bus->awaddr = (unsigned long *)&top->awaddr;
  bus->awvalid = &top->awvalid;
  bus->awready = &top->awready;
  bus->wdata = (unsigned long *)&top->wdata;
  bus->wstrb = &top->wstrb;
  bus->wvalid = &top->wvalid;
  bus->wready = &top->wready;
  bus->bresp = &top->bresp;
  bus->bvalid = &top->bvalid;
  bus->bready = &top->bready;
  bus->araddr = (unsigned long *)&top->araddr;
  bus->arvalid = &top->arvalid;
  bus->arready = &top->arready;
  bus->rdata = (unsigned long *)&top->rdata;
  bus->rresp = &top->rresp;
  bus->rvalid = &top->rvalid;
  bus->rready = &top->rready;

  //==========================================
  // Init eval function
  //==========================================
  bus->evaluateModel = &eval;

  //==========================================
  // Init peripheral
  //==========================================
  uart = new UART(bus, &top->txd, &top->rxd,
  prescaler);
}

As part of the last step, in the main function, you have to call simulate, providing it with port numbers, which are passed as the first two command-line arguments of the resulting binary.

Init();
uart->simulate(atoi(argv[1]), atoi(argv[2]));

Now you can compile your project with Verilator:

verilator -cc top.v --exe -CFLAGS "-Wpedantic -Wall -I$(INTEGRATION_DIR)" sim_main.cpp $(INTEGRATION_DIR)/src/renode.cpp $(INTEGRATION_DIR)/src/buses/axilite.cpp $(INTEGRATION_DIR)/src/peripherals/uart.cpp

make -j 4 -C obj_dir -f Vtop.mk

The resulting simulation can be attached to the Renode platform and used in a .repl file as a ‘verilated’ peripheral.

uart: Verilated.VerilatedUART @ sysbus <0x70000000, +0x100>
  simulationFilePath: @verilated_simulation_file_path
  frequency: 100000000

When you load such a platform in Renode and run a sample application, this is the output you’ll see. Keep in mind that the UART window displays data printed by the verilated peripheral.

You can also enable signal trace dumping by setting the VERILATOR_TRACE=1 variable in your shell. The resulting trace is written into a vcd file and can be viewed in e.g. GTKWave viewer.

Renode’s powerful co-simulation capabilities

Whether you are working on a new hardware block or you want to reuse the HDL code you have, Renode’s co-simulation capabilities allow you to test your IP in a broader context than just usual hardware simulation, connecting it to entire RISC-V, ARM or other SoCs even without writing any model.

You can use Renode’s powerful tracing and logging mechanisms to observe your peripheral’s behavior when used by an operating system of your choice, in an environment of your choice – be it a full-blown Linux-capable multi-core system or a small RTOS-ready SoC, or even a mix of those options.

Want to debug your driver via GDB but your target FPGA does not have a debugger connector? Or maybe it is just too small to contain the whole SoC you’d like to run? Perhaps you’d like to run a Python script to create a nice graph on each peripheral access? Renode has got you covered with all these features available out of the box.

If this sounds interesting, you can start using Renode’s co-simulation capabilities today or let us know about your use case directly so that we can potentially help you improve your simulation-driven workflow – all you need to do is get back to us at contact@renode.io.

If you’re new to Zephyr RTOS, please see our Getting Started Guide and check out our Contributor Guide. Or, you can join the conversation and ask questions on our Slack channel or Mailing List and follow #zephyrproject on IRC.