Inertial measurement units (IMUs) are widely used to provide a steady stream of multi-axis position information from accelerometers, gyroscopes, and other sensors. With the many degrees of freedom (DOF) all generating data, the merged data streams from these devices can keep system processors in constant wake mode and tax them as they sift through the raw IMU data to extract useful gesture and system location information. What designers need is a way to offload this sifting function from the main processor. Machine learning may be the answer.
After a brief overview of IMU use, this article introduces the 6DOF LSM6DSO from STMicroelectronics. It then uses this device to show how the addition and integration of machine learning and decision tree processing into IMUs can offload real-time position and movement processing from the host application processor and how these features can be used in real applications.
A quick IMU review
IMUs integrate a number of motion sensors into one device and can provide high accuracy positioning information. They can be used for a variety of applications including consumer (mobile phones), medical (imaging), industrial (robotics), and military (head tracking). They react to the motion of the sensor and incorporate one or more of the following motion sensor types:
- Gyroscope sensors measure angular position changes, usually expressed in degrees per second. Integrating angular rate over time results in a measured angle of travel that can be used to track changes in orientation. Gyroscopes track relative movement independently from gravity, so errors from sensor bias or integration result in a position error called “drift,” which can be compensated for with software.
- Accelerometers measure linear acceleration, including acceleration components caused by device motion and acceleration due to gravity. The acceleration unit of measurement is g, where 1 g = the earth’s gravitational force = 9.8 meters/second2. Accelerometers are available with one, two, or three axes, which define an X, Y, Z coordinate system.
- Magnetic sensors measure magnetic field strength, typically in units of microTeslas (µT) or Gauss (100 µT = 1 Gauss). The most common magnetic sensor used for mobile electronics is a three-axis Hall effect magnetometer. By computing the angle of the detected earth’s magnetic field, and comparing that measured angle to gravity as measured by an accelerometer, it is possible to measure a device’s heading with respect to magnetic north with high accuracy.
Motion tracking using IMUs employs sensor fusion to derive a single, high accuracy estimate of relative device orientation and position from a known starting point and orientation. Sensor fusion usually employs software to combine the IMU’s various motion sensor outputs using complex mathematical algorithms developed either by the IMU manufacturer or the application developer.
Position calculations using sensor fusion can produce the following measurements:
- Gravity – specifically the earth’s gravity, which excludes the acceleration caused by the motion being experienced by the device. An accelerometer measures the gravity vector when the IMU is stationary. When the IMU is in motion, the gravity measurement requires fusing data from an accelerometer and a gyroscope and subtracting out the acceleration caused by motion.
- Linear acceleration – equivalent to the acceleration of the device as measured by the accelerometer, but with the gravity vector subtracted using software. IMU linear acceleration can be used to measure movement in three-dimensional space.
- Orientation (attitude) – the set of Euler angles including yaw (azimuth), pitch, and roll, as measured in units of degrees.
- Rotation vector – derived from a combination of data from accelerometer, gyroscope, and magnetometer sensors. The rotation vector represents a rotation angle around a specified axis.
Sources of IMU error
Gyroscopes sense orientation through angular velocity changes, but they tend to drift over time because they only sense changes and have no fixed frame of reference. Adding accelerometer data to the gyroscope data allows software to minimize gyroscope bias for a more accurate location estimate. Accelerometers sense changes in direction with respect to gravity, and that data can be used to orient a gyroscope.
Accelerometers are more accurate for static (as opposed to dynamic) calculations. Gyroscopes are better at detecting orientation when the system is already in motion. Accelerometers react quickly, so accelerometer jitter and noise produce accumulated error when that data is used alone. Additionally, accelerometers tend to distort accelerations due to external forces such as gravitational forces, which also accumulate in the system as noise. Filtering this data improves accuracy.
Combining a gyroscope’s short-term accuracy with an accelerometer’s long-term accuracy results in more precise orientation readings by relying on each sensor’s strengths to cancel or at least reduce the other sensor’s weaknesses. The two sensor types complement each other to help reduce errors, but there are other ways by which errors are reduced.
Fused filtering needed to reduce error
IMU software uses filtering to minimize positioning error from IMU data. Several filtering methods for fusing sensor data are available, each with varying degrees of complexity. A complementary filter combines a high pass gyroscope filter and low pass accelerometer filter. High frequency noise in the accelerometer data is therefore filtered out in the short term and smoothed by the gyroscope data.
The computational horsepower needed to perform all of this sensor processing, filtering, and fusing consumes energy, which can be a problem in battery-powered systems, especially when the IMU information is not needed as a continuous stream. For many embedded applications, significant power savings can be realized if the IMU can generate an interrupt that awakens the host processor from sleep mode so that it can initiate processing or take some action as a result of the interrupt. To enable this capability, some IMU vendors are starting to incorporate processing and decision making features in their IMUs.
Let the IMU do the thinking
The 6DOF LSM6DSO from STMicroelectronics is one such IMU. It incorporates three microelectromechanical systems (MEMS) gyroscopes and three MEMS accelerometers and can detect orientation changes and gestures without oversight or assistance from a host processor, all using on board processing. The IMU consumes 0.55 milliamps (mA) running in its highest performance mode.
In this mode, the LSM6DSO can continuously monitor its own attitude and movement in space and can generate an interrupt upon a prearranged condition that awakens the host processor to perform additional processing on the sensor stream. Using a low-power IMU that can always remain operational is beneficial because it lets the host processor sleep, awakening it only when necessary. This is a tried and trusted means of saving energy in battery-powered systems.
In addition to its gyroscope and accelerometer sensors, the LSM6DSO IMU contains a signal conditioning and filter block, a finite state machine (FSM) that can run as many as 16 programs —all sharing a common, configurable output data rate—and a machine learning core. Used together, these resources can generate event detection interrupts for the following conditions:
- Free fall
- Wakeup
- 6DOF orientation
- Single-click and double-click sensing
- Activity/inactivity recognition
- Stationary/motion detection
The signal conditioning block applies conversion factors stored in its sensitivity registers to scale the raw sensor data. It then converts the raw IMU sensor data stream into a 16-bit, half-precision floating point (HFP) byte format that the FSM can understand. The IMU’s MEMS sensors (the accelerometers and gyroscopes) along with the two analog-to-digital converters (ADCs) and four filter blocks are shown in Figure 1. The filter blocks are used to convert the analog MEMS sensor signals into filtered digital data streams.
The programmable FSM consists of a configuration block and sixteen program blocks. The FSM’s configuration block configures and controls the entire FSM. Each of the FSM’s sixteen program blocks consists of an input selector block and a code block (Figure 2). Both of these blocks are controlled by values written to registers within the IMU.
The Input Selector block routes the selected input data from one of the IMU’s internal sensors or from an external sensor connected to the IMU’s sensor hub to the code block. The IMU’s sensor hub can accommodate as many as four additional external sensors such as magnetometers, which are connected to the IMU via an I2C port.
The FSM’s code block contains one program for the state machine. The program block’s data section’s fixed portion consists of six bytes that define the number of thresholds, hysteresis, mask, and timer settings for the program. The program block’s variable data section holds the actual threshold, hysteresis, mask, and timer settings for each program as defined by the values stored in the fixed part of the data section.
The fixed portion of the data section also defines the size of the variable portion of the code block’s memory footprint, a programmable reset vector, and a program counter. Because these are all 8-bit values, each FSM program is limited to 256 bytes.
The program block’s instruction section contains the actual FSM program. Program instructions include opcodes for checking sensor inputs against thresholds, checking for zero crossings, and checking timer values for timeout comparisons. The opcodes specify the condition needed to pass from the current FSM state to the next. In addition, there are command opcodes for selecting thresholds and masks stored in the program’s variable data section; for setting the IMU’s sensor hub multiplex selector to connect to one of the four possible external sensors; and for asserting an interrupt.
Each FSM program can generate an interrupt and can modify the contents of a corresponding register value based on the selected input signal. These register values are used to pass data from the IMU to the host processor.
It’s useful to think of the FSM as a microprocessor minus the arithmetic logic unit. The FSM can make selections, perform comparisons, and can make decisions about its next state based on those comparisons. It does not compute values other than the Boolean results from the comparisons.
The FSM is not a microprocessor. It can make comparisons and perform simple changes to the program flow based on these comparisons. Because it’s a simple machine, it’s programmed directly with the FSM opcodes. There is no high level language compiler for the FSM, but the programs are generally so simple that no compiler is needed.
Using the FSM
The LSM6DSO IMU’s FSM can be programmed to generate interrupt signals activated by predefined motion patterns. The FSM can run as many as 16 simultaneous, independent programs to detect motion. Each FSM program consists of a sequence of if-then-else steps and uses the sensor streams from the LSM6DSO’s accelerometers and gyroscopes as inputs. If any of the FSM programs detects a match to its preprogrammed pattern, the FSM can generate an interrupt to the host processor.
Each of the sixteen possible FSM programs contains three memory sections for fixed data, variable data, and instructions. A single FSM program block diagram is shown in Figure 3.
The structure of a single program in a code block consists of three sections in a block of memory:
- A fixed data section, which has the same size for all the FSM programs
- A variable data section, which can vary in size
- An instruction section, which contains conditions and commands
Programming each FSM code block involves loading the three memory sections with programing values that determine the FSM’s behavior. STMicroelectronics provides an FSM programming tool within its downloadable Unico evaluation development software and development environment. STMicroelectronics has also included several sample FSM programs with the Unico development tools as an aid in learning how to program the FSM. These sample programs demonstrate several IMU based interrupt scenarios including:
- A basic pedometer
- System in free fall
- Simple motion detection
- System has been picked up
- System has been shaken
- System has stopped moving (stationary)
- Wrist tilt
The sample FSM program examples demonstrate the use of the various FSM features. Any of these sample programs can be installed into an IMU demo platform like the STEVAL-MKI109V3 eMotion STM32 eval board, which has a 28-pin socket that accepts the LSM6DSO STEVAL-MKI197V1 IMU adapter board. Programming the STEVAL-MKI109V2 board with one of the example programs requires only a few clicks in the Unico development environment.
However, there’s a bit more to the LSM6DSO than meets the eye.
The machine learning core
The LSM6DSO IMU also incorporates a more sophisticated and programmable pattern matching engine called the machine learning core. This can identify classes of movement using the multiple sensor data streams from the internal IMU sensors and any attached external sensors. Recognizable event classes include stationary (no movement), walking, jogging, biking, and driving. Classification takes the form of decision trees within the machine learning core.
The machine learning core consists of three blocks: the sensor data block, the computation block, and the decision tree (Figure 4). The machine learning core’s sensor data block aggregates data streams from the IMU’s internal accelerometers and gyroscopes and from any external sensors attached to the IMU through the I2C interface. The computation block can filter the sensor data using predefined filtering parameters, and compute windowed statistics including the mean, variance, peak-to-peak amplitude, minimum, maximum, and zero crossing for the sensor data. The decision tree compares the computed sensor data statistics against thresholds to classify the input data.
As with the LSM6DSO’s FSM, a dedicated tool in the Unico development environment is used to program the IMU’s machine learning core.
The finite state machine and the machine learning core can also be used in conjunction with a host processor to implement more sophisticated position tracking algorithms. The downloadable STMicroelectronics X-CUBE-MEMS1 software pack for the company’s STM32Cube development system includes the following example software routines:
- Activity recognition – Provides information on the type of activity being performed by the user including holding still, walking, fast walking, jogging, biking, or driving. This algorithm might typically be used in a mobile phone or some sort of wearable device.
- Motion duration detection – When combined with pedometer data, motion duration detection can be used to determine the number of seconds that a user is active. This algorithm might typically be used in a wearable device for fitness or health tracking.
- Vibration or motion intensity detection – Provides information about the intensity of user motion and can distinguish motion intensity in a range from 0 (still) to 10 (sprinting). This algorithm might typically be used in a mobile phone or some sort of wearable fitness device.
- Carrying position recognition – Provides information about how the user is carrying a device and can distinguish among the following positions: on a desk, in a hand, near the head, in a shirt pocket, in a trouser pocket, in a jacket pocket, and held in a swinging arm. This algorithm might typically be used in a mobile phone or some other sort of carried device for activity related context detection.
Conclusion
The need to keep a host processor running to maintain a position fix and to detect movement and gestures from IMU data can be a difficult goal to achieve with battery-powered embedded designs because of the host processor’s relatively high power consumption. However, a new generation of low-power IMUs with sufficient on board processing to perform machine learning can solve this problem by allowing the host processor to sleep in a low current mode until it’s needed.
