Consumer devices are becoming increasingly capable, featuring various sensors useful for monitoring fitness and wellbeing. We introduced sleep sensing in the Nest Hub a few years ago. This feature makes use of radar technology known as Soli to analyze sleep patterns[1] when the device is placed close to the bed. More recently, we demonstrated that the Soli radar platform’s fully contactless frequency modulated continuous wave (FMCW) radar technology can monitor vital signs like heart rate and breathing rate during sleep and meditation. In “UWB Radar-based heart rate monitoring: A transfer learning approach,” we present new research demonstrating that radar-based heart rate measurement can be performed with ultra-wideband (UWB) technology, which is already found in a lot of mobile phones. Despite its widespread use for features like secure vehicle unlocking and precise item location, UWB’s potential for radar sensing has largely been overlooked. We show how this already-existing hardware can be used to monitor vital signs like heart rate (HR).
Contents
Radar sensors available on consumer devices
The radar systems that have shown the most promise for vital sign measurement from consumer devices include millimeter wave frequency-modulated continuous wave (mm-wave FMCW) and impulse-radio ultra-wideband (IR-UWB) radar systems. FMCW technology was used by Google in the past to detect sleep, movement, and gestures on the Soli radar platform. This meant that we already had extensive datasets, studies, and machine learning algorithms trained for these tasks, including for heart rate monitoring using FMCW radar.
Meanwhile UWB — a multipurpose technology that has grown in popularity and is increasingly available on many current mobile phone models and other consumer devices, also offers radar capabilities. The radar capabilities of UWB have been thus far largely untapped, with current UWB applications leaning more on non-radar uses like localization and tracking, vehicle unlock features, or data transfer.
overcoming the obstacle presented by touchless sensing It is difficult to use radar to detect HR without making any contact because the much larger movements caused by breathing and general body motion easily obscure the minute movements of the chest wall caused by the heartbeat. The distinctive nature of the radar signal comes into play at this point. Its spatial resolution works in three dimensions, using both distance and direction to focus its measurement. This allows the radar to define a precise “measurement zone” around a person’s torso. As a result, it is able to isolate reflections coming from the chest area while ignoring movements or stationary background objects outside of this zone. It simultaneously samples the signal at a rate of up to 200Hz, which is fast enough to capture the heartbeat’s subtle and rapid motion. We developed a new method that makes optimal use of these unique 2-dimensional spatio-temporal properties of the radar signal to achieve highly accurate heart rate measurement.
Bridging the gap between radar types
We investigated if we could transfer the features learned from FMCW radar — where we had the benefit of large existing datasets and studies — to the UWB radar. The physical principles upon which the two radar systems operate are completely distinct. While UWB transmits extremely brief pulses with durations ranging from a few hundred picoseconds to a few nanoseconds, Mm-wave FMCW transmits a continuous sinusoidal wave whose frequency increases linearly with time. Our study is the first to show that learned features can be transferred between radar types for vital sign measurement. We chose heart rate as an initial task, for both its high potential utility and level of challenge.
Creating a brand-new radar-based deep learning model for heart rate To accomplish this task, we developed a novel deep learning framework designed to model the complex spatial-temporal relationships in radar signals for HR estimation. The architecture first uses a 2D ResNet to process the input data, in which one axis represents time and the other represents the spatial measurements. This initial stage is designed to extract features from the fine-grained spatio-temporal patterns created by chest wall movements.
Following this step, the model collapses the spatial dimension via average pooling. The resulting feature set is then fed into a 1D ResNet, which is designed to analyze the signal exclusively along the temporal dimension. This second stage identifies the longer-range, periodic patterns characteristic of a heartbeat from the features extracted in the first stage.
For heart rate measurement, the model achieves a mean absolute error (MAE) of 0.85 when trained with our FMCW dataset. This finding represents a substantial gain over prior state-of-the-art results on this dataset, halving the previous error rate.
using ultra-wideband radar to transfer learned features We then ran a study that collected UWB radar data, along with electrocardiogram (ECG) and photoplethysmogram (PPG) data as our ground truth for heart rate, using a setup that placed the UWB radar sensor in positions where users typically hold their phone, i.e., on a table in front of them or on their lap. The UWB radar dataset was much smaller than the FMCW dataset, which contained 980 hours of data, with 37.3 hours. As the UWB radar configuration was close to what is feasible on a mobile phone, with a much lower bandwidth, its range resolution was far lower than the FMCW dataset.
To ensure that our model was optimized to transfer to the UWB dataset, we retrained it after performing additional pre-processing steps to modify the mm-wave FMCW radar data to better resemble the target IR-UWB data, effectively lowering its range resolution. We then fine-tuned this model on the IR-UWB dataset, achieving an MAE of 4.1 bpm and mean absolute percentage error (MAPE) of 6.3%, a 25% reduction over the baseline error rate. Our baseline for performance on UWB radar was 5.4 bpm MAE and 8.4% MAPE, achieved by selecting the best model trained from scratch on our UWB dataset. With transfer learning, we enabled the UWB radar to meet the Consumer Technology Association standards for heart rate measurement for consumer devices: an accuracy of up to 5 bpm MAE and 10% MAPE.
Ensuring accuracy in different scenarios
To make sure our model is both accurate and reliable, we analyzed its performance across the various scenarios and user conditions captured in each dataset. In situations that were adequately represented, we discovered that performance on heart rate measurement is consistent for both kinds of radar. For example, on the FMCW radar, which collected data during overnight sleep sessions, the performance is maintained across various sleep positions and even when a person is moving between positions. For UWB radar, both tested device positions relative to the user—on a table in front of them or in their lap—are equally accurate for measuring heart rate. For more details on this subgroup analysis and other results, see the full research paper.
The big picture: Everyday health monitoring
Heart rate measurement is useful for a range of health, fitness, and wellness applications, offering fundamental insight into an individual’s cardiovascular status and physiological responses across various health conditions. This demonstration of heart rate measurement could be a step towards using mobile devices to measure even more complex and subtle health signals from the heart and large blood vessels.
While wearable devices like fitness bands and rings have popularized continuous monitoring of health and fitness, the ability to measure heart rate in a contactless manner with consumer-device–grade radar sensors allows the benefits of this technology to reach a much wider audience of smartphone users. For this study, we focused on heart rate while sleeping (for FMCW) and on a setup where the radar sensor was in positions where the phone is usually held during use (UWB). As technology develops, continuous monitoring may become available in a variety of everyday settings and seamlessly integrate with a user’s daily activities. What this means for future devices
This work moves us closer to enabling contactless heart rate measurement using consumer devices, especially as ultra-wideband (UWB) technology becomes more prevalent in mobile phones. Although our study did not include direct testing using mobile phones in a real-world setting, this research establishes the crucial groundwork for such future applications.
A core finding of this work is the demonstration that a model trained on one type of radar (FMCW) can be successfully adapted for another (UWB) to measure heart rate. This method of transfer learning represents a significant advancement. It suggests a more efficient path for future research and development, where the foundational knowledge from existing, large datasets can be leveraged for new devices. Instead of starting from scratch with extensive data collection for each new piece of hardware, this method allows for a more streamlined process, accelerating the timeline for bringing such features to consumer devices.
