BeaglePilot: cyclictests with different kernels

For an autopilot to run properly it’s highly important the latencies. If not met then the robot will probably fall into the ground (in the best case).

This post presents the tests and conclusions reach over the work in the last days. I spent time compiling 4 different kernels flavours (all of them v3.8.13):

  1. vanilla kernel
  2. vanilla kernel with PREEMPT option
  3. RT_PREEMPT patches
  4. Xenomai patches

In order to generate computational tasks the stress linux command has been used. This command generates about 100% of computational load which is great for testing the different kernels reaction.

The tests have been done using cyclictests. In a nutshell, cyclictests is a Real-Time Benchmark that measures the latency of response to a stimulus. This latency can vary due to several factors:

  • additional external interrupts
  • processor emerging from sleep states
  • cache migration of data used by woken process
  • block on sleeping lock

The pseudocode of this algorithm is presented below:

 

clock_gettime((&now))

 

next = now + par->interval

 

while (!shutdown) {

 

  clock_nanosleep((&next))

 

  clock_gettime((&now))

 

  diff = calcdiff(now, next)

 

  # update stat-> min, max, total latency, cycles

 

  # update the histogram data

 

  next += interval

 

}

The command used for the tests is:

./cyclictest -l100000 -m -n -a0 -t1 -p99 -i400 -h1000 -q

These 4 kernels have been evaluated resulting in the following histogram (the x-axis unit is us):

The histogram shows that the vanilla kernel is the one who obtains the minimum latencies. Following this one (on minimum latencies) we have the vanilla with the PREEMPT option, the the Xenomai and finally the RT_PREEMPT one.

However the for our application (running an autopilot in a Linux-based computer) the most interesting thing is the Maximum time. We need to make sure that the sensors are sampled at a specific rate no matter how expensive the computational tasks are of we want our drones to remain in the air. The following table pictures the minimum, average and maximum timings measured with cyclictests:

 Kernel 

 Min (us)

 Avg (us)

Max (us)

vanilla

 14

 19

193

PREEMPT

 16

 21

 68

RT_PREEMPT

 20

 27

 91

Xenomai

 15

23

630

One of the mentors of BeaglePilot, Andrew Tridgell, pointed out in a talk the timing needs. In short:

    • 10 ns:
      • SPI bus transitions
    • 1 us:
      • PWM (done already at PRUs)
      • PPMSUM (expected at PRUs)
    • 1 ms
      • Gyros
      • Accels
    • 20 ms
      • Barometer
      • Compass
      • Airspeed
      • Sonnar
    • 200 ms
      • GPS

The 10 ns and 1us region is solved by hardware but we should definitely meet the 1 ms. For this objective and according to the results presented the right choices that assure a minimum maximum latency will be:

  • vanilla kernel with PREEMPT option
  • RT_PREEMPT kernel

The code used to plot the results is:

 

#!/bin/bash

   
 

PLOT1=vanilla.data

 

PLOT2=PREEMPT.data

 

PLOT3=RT_PREEMPT.data

 

PLOT4=xenomai.data

   
 

echo ‘plot “‘$PLOT1‘” with steps ls 1 title “vanilla”,\

 

“‘$PLOT2‘” with steps ls 2 title “PREEMPT”, \

 

“‘$PLOT3‘” with steps ls 3 title “RT_PREEMPT”, \

 

“‘$PLOT4‘” with steps ls 4 title “Xenomai”‘ | gnuplot -persist

Sources:

  • https://rt.wiki.kernel.org/index.php/Cyclictest
  • http://www.stresslinux.org/sl/
  • http://elinux.org/images/0/01/Elc2013_rowand.pdf