Becoming champions of Europe, Africa and the Middle-East using software built at LJZ

To aid the Vrije Universiteit (VU) Amsterdam in offering all Computer Science bachelor students a fitting thesis project, an autonomous systems engineering software- and hardware framework was developed. The software development was led by LJZ and did not only reduce professor workload, but also gained much attention after winning gold at NXP cup, the internationally renowned autonomous racing competition.

After three years of research, development and competition in the NXP cup together with the VU-team consisting of Hugo van Wezenbeek, Eduardo Lira, Max Gallup (Gallup Engineering) and Elias Groot (LJZ Digital Solutions B.V.), we started in September 2023 with the wish to (a) build a software- and hardware framework so flexible and reliable that 20 Bachelor students could use it to visualize their thesis research with "real-world backed up evidence" and to (b) show the mere power and innovation of this framework by competing in the NXP cup again.

These two goals were not always in line with each other, and required us to carefully weigh student interests and competition interests when making design decisions. An RC-scale car had to be built to use for research and racing. It had to be lightweight (for quick acceleration and high top speeds) but also extensible, so that students could hook in new sensors, actuators and other electronics that they want to use for their research. The software stack should be modular (to enable students to build their own robotics pipeline where they can mix and match existing and self-built components) but the communication overhead should be minimal so that image processing, steering, accelerating and braking calculations can happen at high speeds. With a strict deadline in sight (both the thesis period and the NXP cup commence in April), we got to work quickly.

Hardware and electronics

The hardware team consisting of Max Gallup (Gallup Engineering) and Hugo van Wezenbeek presented the first ideas, sketches and prototypes soon after the project started. The entire car was built from the ground up, with all parts designed and prototyped at the VU. The team cooperated with the VU Fine-Mechanical Instrumentation department, who went above and beyond to make the car come to life.

The electronics, from PCB design to assembly, were also in the good hands of Niels Althuisius. Working closely together with the hardware and software team, he managed to design an incredibly well thought out, capable and extensible power delivery board and circuit board. Without it, all hardware and software would be useless. Once more, it showed how fortunate the VU is to have all this experience in-house, especially for a non-technical university.

In no-time, the hardware and electronics team transformed their barebone models to a fuly assembled prototype. A big breakthrough for us. It is satisfying to watch a plan come together, even if it is about hardware ;)

Powering all processing: the Debix model A

To power the embedded development done at LJZ, a capable, efficient and reliable board had to be chosen. We at LJZ have been in close contact with the team at Debix and their model A single-board computer was selected to be used as the brain of the operation. With its credit-card size and excellent power usage, it made development and testing of the car a lot easier.

It is safe to say that, before deciding on the board, we tested the model A extensively. From dropping nuts and bolts on an unprotected PCB (yeah, we know) to crashing a car in the wall full-speed (it happens!), the Debix board always survived. A testament to its durability and saving us valuable time in the development cycle.

Developing a capable software framework

Together with Eduardo Lira and Max Gallup, LJZ's Elias Groot was responsible for the software design and development. Elias led the design and development decisions and helped building the foundations of the educational documentation for students.

The software stack for autonomous systems should be thought of as a "pipeline": flowing from sensor inputs (camera frames, lidar readings, etc.) to actuators (motors, brakes, lights, etc.) with image processing and actuation decision algorithms in between. This way, the car can react autonomously to changes in the environment, as noticed by its sensors.

As mentioned earlier, the pipeline should be extensible: so that students can conveniently replace components or add new ones as they see fit, but its extensibility should not impose a significant communication overhead, which would hinder racing performance. Go (Golang) was used to develop individual components of the pipeline. Go is a popular choice in the world of embedded development because of its speed, developer experience and rich ecosystem of libraries.

It was envisioned that students could "bring any language they want" to build a component in. Suppose that they wanted to replace the processing algorithm with a neural network, they will probably want to use Python or Swift. If they wanted to replace the Actuator drivers, they might prefer C, C++ or Rust (if they are really cool). As such, the team did not want to force them to use Go, or any specific language for that matter. ZeroMQ, a language-agnostic and highly efficient multi-protocol inter-process message library was incorporated to take care of communication between the components in the pipeline. Protobuf was used to encode messages to a standardized binary format, with encoding and decoding libraries available for virtually any programming language.

Implementing service discovery

To allow students to plug in components into the pipeline as they see fit, a central service discovery engine was developed. It keeps track of all running processes on the car, takes care of automatic retries and dependency resolution. Taking away this burden from the students, they reported that the service discovery helped them greatly in quickly running, understanding and A/B-testing their components.

Because the service discovery tooling was taken into account during the software design phase, it has been tightly integrated into the entire software stack. This allowed us at LJZ to develop very extensive tooling. With a fully custom and tightly integrated debugger our software allows students to visualize their pipeline (see above) as well as to livestream, record and playback component data such as camera frames, distance sensor charts, acceleration and speed charts and to even tune the car realtime, while it is driving to see the immediate effects of the applied changes. The debugger was written in Svelte and uses WebRTC connections to power realtime and bi-directional streaming and parameter tuning.

Below you can see a still frame of this debugger. The car is (autonomously) steering into a corner and the charts on the right side reflect the steering decisions for the left and right wheel. The camera view reflects what the car sees in realtime.

Distributing educational software smoothly

All software components available to students were subject to frequent change and rapid iteration during the development phase. Together with Max Gallup (Gallup Engineering), a robust continuous integration and delivery pipeline was set up, enabling automatic publication of all 40 (and counting) packages, with semantic versioning based on conventional commit messages. Packages were shared through Dockerhub and the custom developed software center, which allowed for automatic and seemless updates for students. Documentation and hands-on guides were provided to make the barrier to entry for students even lower.

Below you find an example of the documentation for the "Imaging" module that students can use and enable in their pipeline. There are several ways to integrate this module, all highlighted here.

Becoming champions of Europe, Africa and the Middle East

In April 2024, the qualifications for the international autonomous racing competition organized by NXP semiconductors, the NXP cup EMEA commenced. 150 teams competed and the LJZ-backed VU team finished first place in the local qualification rounds in Eindhoven, the Netherlands.

Five weeks later, we traveled to Hamburg, Germany for the final competition, racing against the 20 best teams of Europe, Africa and the Middle-East. After a breath-taking competition. The VU car, powered by software developed at LJZ, finished first place, being 2.5 seconds faster than the silver medalists and finishing in one attempt. You can read more here and here.

Results in education

Besides winning the NXP cup, the software and hardware framework was used successfully by 13 students during the first run of the Autonomous Systems Engineering (ASE) bachelor project course from April - July 2024. LJZ has been co-leading the course creation, organization, daily supervision and technical support. Supervised by Dr. Natalia Silvis, without whom this all would not have been possible, and in great cooperation with the talented Hugo van Wezenbeek, Eduardo Lira and Max Gallup, the ASE course will return in 2025, with an even further improved software stack and daily supervision from LJZ again.