Over time, sensors have morphed from simple analog and mechanical constructs to chip-based digital devices that connect to a machine to monitor the machine’s health as well as environmental conditions. Similarly, sensor fusion — multiple types of sensors working together to solve a problem — has combined the threads of many other technologies to create something very new and exciting.
The idea of using a computational device to sort out the data from multiple sensors and combine information to draw a conclusion has been around since at least the 1950s. But it was exceptionally difficult to do. Around 1960, several mathematicians developed sets of algorithms in an effort to have a machine draw a conclusion based on input from multiple sensors. These filters also removed meaningless data from noise or other sources. Of course, it wasn’t long before the military decided that this technology would be useful in their applications. Being able to process inputs from multiple sources and compare it with stored data would allow the military to better track and identify potential airborne targets and even compute the certainty of the results. With better computers and sensors, the technology was advancing, but there were still complex and expensive problems to solve.
Potential Applications
When the microprocessor first became available, people described it as a solution in search of problems. The same case can be made for sensor fusion. If you have the power and intelligence to monitor multiple sensors, analyze the data in real time, and either provide a simple direction or control an action, heretofore-unthought-of applications can be almost limitless. The following examples just scratch the surface:
- Health monitoring — including healthy athletics, patient monitoring, and research
- Monitoring the elderly — wellness monitoring to reduce the burden of staffing
- Automotive, transportation systems — monitoring and controlling efficiency and safety functions
- Public safety — identification of potential hazardous conditions with much greater accuracy than simple fire and security systems
- Entertainment — gaming, including controllers and virtual reality headsets
- Weather — intelligent weather forecasting stations, that not only warn of changing conditions, but control systems to prepare for a storm (for example, closing storm shutters, closing valves, etc.)
- HVAC/Air Quality — intelligent control of room temperature, humidity, air quality, system maintenance, etc.
While all of these types of capabilities have existed in some form for many years, the ability of a system to observe multiple sensors and come to an intelligent conclusion, and even initiate action, is revolutionary.
A Convergence of Technologies Fortunately, as they have done in other areas of electronics, a number of IC manufacturers have taken on the task of doing the heavy lifting. With off-the-shelf sensor fusion and sensor hub chips, it is now possible to efficiently interface to a variety of digital sensors, as well as other pathways. The burden of creating your own algorithms has been eliminated.
While their terminology does vary a bit, a number of IC manufacturers have either adapted existing lines of products or created entirely new ones to tackle sensor fusion tasks. The processing is done by a specialized controller chip, which may be identified as an MCU, a sensor hub or a sensor fusion processor. We are already seeing this technology applied in the consumer market in smart phones, activity monitors and other devices.
The latest generation of smart phones from Apple, Samsung, and others contain powerful and diverse sensing capabilities, even without the need for external interfacing. These include a three-axis magnetometer, a three-axis accelerometer, and a three-axis gyroscope. This combined capability is often referred to as 9-DoF, nine degrees of freedom.
For the most part, these functions are “always on” in a cell phone. If the processing of data from these sensors was managed by the phone’s central microcontroller, battery life would be significantly shortened. Instead, highly efficient dedicated MCU chips process the data as sensor hubs, using a fraction of the power. The NXP ARM M3 series of MCUs is one example. According to Chipworks, a product teardown specialist, as reported by EETimes, Apple uses a customized version of the NXP chip to monitor its sensors in the iPhone 5S. “The M7 controls functions from a variety of discrete sensors including a gyroscope, an accelerometer, and a compass.” Samsung takes on the same task with a microcontroller from Atmel, the Core 8-bit AVR MCU.
With such powerful on-board sensing technology, apps are appearing that take advantage of the 9-DoF cell phone hardware to provide health and activity monitoring, or to function in concert with GPS and external data to provide even more information for the user. Now, add to those already-diverse sensor inputs data from an external device that communicates via Bluetooth and the capabilities seem limitless. The goal of the chip manufacturers is to make it practical for engineers to design systems that provide real-time sensor data which can be used to provide the desired contextual awareness with minimal power consumption and maximum battery life. Beyond smart phones, highly optimized solutions can address such applications as tablets, Ultrabooks, IoT-enabled devices, gaming, healthcare, environmental monitoring, and wearable computing.
Development boards are available that allow design engineers to easily get their feet wet in this technology. One such example is the ATAVRSBIN2 by Atmel. Atmel has embraced sensor fusion with a wide variety of products, which they call “the Complete Sensor Ecosystem.” Atmel identified that the simultaneous analysis and fusion of data from different sensors and sensor types was not a task it could handle solo. To get past these complexities, the company partnered with a number of leading sensor manufacturers and sensor fusion specialists to provide a complete, easy-to-implement Sensor Hub Solution.
A current trend combines an MCU with three or more MEMS sensors in a single package. One example is STMicroelectronics’ LIS331EB, which combines a high-precision three-axis, digital accelerometer with a microcontroller in a single 3 x 3 x 1 mm-package. The microcontroller is an ultra-low-power ARM Cortex-M0, with 64-Kbyte Flash, 128-Kbyte RAM, embedded timers, 2x I²C (master/slave) and SPI (master/slave). The LIS331EB can also internally process data sensed by external sensors (for a total of nine), such as for gyroscope, magnetometer, and pressure sensors. Functioning as a sensor hub, it fuses together all inputs with the iNEMO Engine software. STMicroelectronics’ iNEMO engine sensor fusion software suite applies a set of adaptive prediction and filtering algorithms to make sense of (or fuse) the complex information coming from multiple sensors.
Freescale also offers a product line of devices that combine MCUs and sensors in a single package. Their FXLC95000 Xtrinsic Motion-Sensing Platform integrates a MEMS accelerometer and a 32-bit ColdFire MCU. Similar to the STMicroelectronics device, the FXLC95000 can simultaneously manage data from internal and external sensors. Freescale was the first company to market an MCU with a sensing hub embedded that is also programmable for customer-specific applications and algorithms. Up to 16 sensor inputs can be managed by a single device, allowing calibration, compensation and sensor functions to be offloaded from the application processor. It functions with either Freescale or third-party drivers.
Other manufacturers who are well-entrenched include Bosch, Fairchild, Honeywell, MicroChip, and TI.
Fusion Meets the Cloud
While quite a lot of functionality can be achieved at a local level, interaction with the Cloud is where the fun really begins. Remote sensor data can be processed by a sensor fusion device and sent to the Cloud for recording, further analysis, or even to order an action.
For example, an unattended pump operating in a remote location is always at some risk of failing. A few years ago, a remote sensor may have been in place to identify if it were running hot or had even failed. Now, the same pump can also be monitored for vibration, exhaust chemistry, bearing noise, and the external conditions around it. A predetermined program could empower the sensor fusion controller to shut down the pump or even cycle its operation until a technician can arrive. The system would also know in advance whether it is likely that the entire pump must be replaced or just a component. Here, a sensor fusion solution could eliminate downtime as well as costly emergency service calls, and even collect data to analyze how well the pump is working overtime. The same general idea applies to monitoring an aircraft engine in flight, a building elevator, or just about anything mechanical.
Another application of the Cloud is for the sensor fusion to take place there, instead of on site. With open-source sensor fusion software available, individual sensor data can be transmitted to a server, where the processing would take place.
Conclusion
Sensor fusion is a technology that has come of age, and at just the right time to take advantage of developments in sensors, wireless communication, and other technologies. Once out of reach of all but the most advanced government labs, the technology is now available off-the-shelf, at prices that even fit into the BOM budget for many consumer products.
Now closely linked to mobile technology and the rapid development of lower-cost digital sensors, sensor fusion is poised for explosive growth. For the design engineer, it is a good time to apply some creative thinking and to start experimenting!
Morrie Goldman is a veteran electronics industry marketer and technical writer based in the Chicago area. A former editor and writer of electronic text content for a major educational company, he has also authored numerous articles for trade publications and websites. He holds an Advanced Class amateur radio license and has had military electronics training.
