L'Hexapod: All the problems of multi-threading without the threads

One of the problems of moving from the simple three byte SSC style control system to a system where we can do clever things with multiple servos at once is that the data required to do these clever things is bigger than the data required to do the simple things. See here for the details of the new control data structure. The problem with data that is bigger is reading or writing it atomically (that is, ensuring that our multi-byte data structures are in a consistent state at all times). Why do we need to worry about atomic writes? Interrupts. My day job is writing multi-threaded C++ server code. I’m used to having to worry about protecting sections data from concurrent access by multiple threads of execution. I’m used to using locks to make sure that multi-byte structures, are updated together, atomically. I’m used to seeing the problems that can occur when this isn’t done correctly. So, the first thing that worried me about changing the serial protocol to something more complex was that I had two threads of execution accessing the data structures that would be required and that suddenly these data structures were more than single byte values.

The first thread of execution is the serial protocol code that runs in the main loop of the servo controller program. This will need to read and write the new data structures. The second thread of execution is the code that runs as part of the PWM setup routine from the Timer1 interrupt. This can start running when the serial code is at any point in its execution, it literally interrupts. The problem is that the PWM code may interrupt whilst the serial code is in the middle of an update to the configuration data for a servo and the PWM code may then read data that is half the old data and half the new. We don’t have that problem at the moment as the control data is a single byte and the single byte can be written and read atomically so there’s no way that the interrupting code can try and read data whilst it’s being updated.

I expect a picture or two might help…

The existing servo control data consists of a single byte per servo and that byte can be written to by the serial protocol code (in blue) and read from by the PWM setup code (in red). Arrows into the box are writes to the data and arrows out are reads from it.

The problem is that the ’target position’, ‘step size’ and ‘step when’ values should only be used when all of them are either ’new’ or ‘old’ but due to how the timer code that reads these values can interrupt the serial protocol code that writes them we have a bit of a problem. Or we do unless we design the code to carefully avoid the problems.

The standard solution to this kind of thing in my day job is to use some form of lock to protect the data access. The update thread would lock the lock before it starts updating and unlock it after it was done. The reading thread would lock the lock before it starts reading and unlock it when it was done. If the lock was locked when a thread wanted to lock it then the thread would wait until the lock has been unlocked. This explicit synchronisation is common and useful but it’s not possible for us here. The situation we have here is subtly different to the synchronisation issues that I face from day to day. Here only the PWM code can interrupt the serial code. Normally, when dealing with real threads, any thread can interrupt any other thread, we’re lucky in that we can guarantee that the serial code will never interrupt the PWM code. The most obvious fix for this kind of problem would be to turn off interrupts for the critical period of time where we are updating the control values. This would work but, right now at least, I don’t want to delay the PWM code at all and turning off interrupts would do that, albeit only slightly. The approach that we will take instead is to make sure that we can update the shared data in such a way that the code that accesses it responds to changes in single byte values, rather than requiring the complete structure to be consistent at all times. This is harder to verify as you have to work harder to understand the code and the consequences than if you were simply checking that there was a cli, sei pair around each set of atomic updates; for now though this is the approach we’ll take.

The simplest of the new serial protocol commands that we’re adding to the servo controller is the stop command; but even this simple command needs some care to implement in the presence of interrupts. A stop could essentially set the target position to the actual position. However if we were to do that then we might find that the servo oscillates rather than stopping smoothly. The problem is that to update the target position to the current actual position we need to read the actual position and then write the value to the target position. If the PWM code interrupts the serial code after the read but before the write then by the time the serial code comes to write the new target position the actual position has changed. The next time the PWM code decides to step the target towards the actual it will move the servo again. The larger the ‘step size’ the bigger the potential oscillation.

It’s far better to implement the stop command as follows. First we update the ‘step when’ value to 0. We can structure the PWM code so that this causes it to stop trying to move the actual position towards the target position. If ‘step when’ is zero then the PWM code should just use the value of ‘actual position’ for the servo’s position, no matter what the other control values say. Once the atomic update of the ‘step when’ value is complete the serial code can safely read the actual position as it will not be changed by the PWM code. It can then update the target position. However, since the a zero value for ‘step when’ effectively means that the servo remains static in its current position we could leave the values for ’target position’ and ‘step size’ unchanged during a stop.

Using this ‘atomic stop’ method we can now implement everything else in a similar style. Before updating either the ’target position’ or the ‘step size’ we should always update the ‘step when’ value to 0. Once we have completed our updates we can then update the ‘step when’ value to something else if need be.