BeaglePilot: cyclictests with different kernels

By / March 16, 2023

Ensuring the proper functioning of autopilot technology is crucial for the safety and success of drone operations. One of the key factors that determine the efficiency of an autopilot system is its latency.

This post presents the tests and conclusions reach over the work in the last days. I spent time compiling 4 different kernels flavours (all of them v3.8.13):

  1. vanilla kernel
  2. vanilla kernel with PREEMPT option
  3. RT_PREEMPT patches
  4. Xenomai patches

In order to generate computational tasks the stress linux command has been used. This command generates about 100% of computational load which is great for testing the different kernels reaction.

The tests have been done using cyclictests. In a nutshell, cyclictests is a Real-Time Benchmark that measures the latency of response to a stimulus. This latency can vary due to several factors:

  • additional external interrupts
  • processor emerging from sleep states
  • cache migration of data used by woken process
  • block on sleeping lock

Several factors such as external interrupts, processor sleep states, and cache migration of data used by woken process were observed to influence the latency levels of the different kernels. The findings of the experiment can be used to optimize the performance of autopilot systems and ensure the safe and efficient operation of drones.

The pseudocode of this algorithm is presented below:

clock_gettime((&now))

next = now + par->interval

while (!shutdown) {

  clock_nanosleep((&next))

  clock_gettime((&now))

  diff = calcdiff(now, next)

  # update stat-> min, max, total latency, cycles

  # update the histogram data

  next += interval

}

The command used for the tests is:

./cyclictest -l100000 -m -n -a0 -t1 -p99 -i400 -h1000 -q

These 4 kernels have been evaluated resulting in the following histogram (the x-axis unit is us):

The histogram shows that the vanilla kernel is the one who obtains the minimum latencies. Following this one (on minimum latencies) we have the vanilla with the PREEMPT option, the the Xenomai and finally the RT_PREEMPT one.

However the for our application (running an autopilot in a Linux-based computer) the most interesting thing is the Maximum time. We need to make sure that the sensors are sampled at a specific rate no matter how expensive the computational tasks are of we want our drones to remain in the air. The following table pictures the minimum, average and maximum timings measured with cyclictests:

 Kernel 

 Min (us)

 Avg (us)

Max (us)

vanilla

 14

 19

193

PREEMPT

 16

 21

 68

RT_PREEMPT

 20

 27

 91

Xenomai

 15

23

630

One of the mentors of BeaglePilot, Andrew Tridgell, pointed out in a talk the timing needs. In short:

    • 10 ns:
      • SPI bus transitions
    • 1 us:
      • PWM (done already at PRUs)
      • PPMSUM (expected at PRUs)
    • 1 ms
      • Gyros
      • Accels
    • 20 ms
      • Barometer
      • Compass
      • Airspeed
      • Sonnar
    • 200 ms
      • GPS

The 10 ns and 1us region is solved by hardware but we should definitely meet the 1 ms. For this objective and according to the results presented the right choices that assure a minimum maximum latency will be:

  • vanilla kernel with PREEMPT option
  • RT_PREEMPT kernel

The code used to plot the results is:

#!/bin/bash

PLOT1=vanilla.data

PLOT2=PREEMPT.data

PLOT3=RT_PREEMPT.data

PLOT4=xenomai.data

echo ‘plot “‘$PLOT1‘” with steps ls 1 title “vanilla”,\

“‘$PLOT2‘” with steps ls 2 title “PREEMPT”, \

“‘$PLOT3‘” with steps ls 3 title “RT_PREEMPT”, \

“‘$PLOT4‘” with steps ls 4 title “Xenomai”‘ | gnuplot -persist

Sources:

  • https://rt.wiki.kernel.org/index.php/Cyclictest
  • http://www.stresslinux.org/sl/
  • http://elinux.org/images/0/01/Elc2013_rowand.pdf
Scroll to Top