Prompto

The headset, which the user is wearing (in the image above), provides chat-based guidance in applying a memory technique to help her remember daily tasks and events better. It suggests to her to practice the technique when she is more calm and open to such memory training, and assists her in keeping track of her tasks.

Overview

Current memory training tools focus on providing prompts/notifications to start training sessions at timings when we are available. The Prompto project explores the idea of opportunistically providing prompts when users are cognitively available: during moments of lowered cognitive load or calm and idle moments. At these moments, we are possibly more receptive to learning and remembering. Prompto uses physiological (biosignal) signals of skin conductance/electrodermal activity (EDA) and heart-rate variability (HRV) to detect these moments.

Take a look at the overview video below for more details.

Role

I led this research:

• Developed the initial concept.
• Implemented Prompto’s real-time cognitive load detection system (version 1 rule-based and version 2 machine learning)
• Implemented Prompto’s chat interface on Android and iOS.
• Designed and conducted studies to evaluate if prompts at lowered cognitive load would improve users’ receptivity to the prompts.
• Work resulted in 3 publications and 3 recognitions.

Assistance in development: Vipula Dissanayake (Android), Shardul Sapkota (Android, iOS) Assistance in study analysis: Rebecca Mathews

Process

Concept

The project builds on a prior project, ProspecFit, that involved digitally facilitating a memory technique called the ‘when-then’ technique which has been known to strengthen memory in older adults (age 60 or above).

Prompto extends this work by presenting the technique in a chat-based natural-language interface and designing this work for older adults. Natural-language interaction, cognitive-awareness and physiological understanding of users and could create effective conversational memory tutors which relate to and understand us better.

Figure 1 below shows the interaction flow for Prompto:

1. Physiological readings are monitored for appropriate user states, which triggers prompts from the Prompto app
2. Prompt is displayed as a notification and an audible notification sound
3. User goes through memory training session for verbalisation and visualisation steps of the “when-then” technique.

Development of Chat Interface

A DialogFlow agent was created through the online Google DialogFlow Console to manage the chat between Prompto and the user. The conversational flow underwent several iterations to address issues from think-aloud testing with older adults.

The chat interface was first implemented on an Android app using Android Studio (Java) and subsequently implemented on iOS using Xcode (Swift). The app had a voice interface and a text-based chat interface for users who were uncomfortable with speech interaction or required silent interaction.

Figure 2 above shows the conversation flow. “Intents” represent user intentions that are matched based on users’ speech. They are shown in the square brackets.

1. Top Row of Figure 2: When a calm event is detected, the DialogFlow agent received an Event Request from the app (a feature in the DialogFlow API), invoking the “System Initiated Practice’’ intent. Prompto visually and audibly prompted to start a practice session or try again later.
2. Bottom Row of Figure 2: Users could initiate a practice session at any time. When the user opens the app, Prompto starts with the “Welcome” intent that presents the choice to start practice.
3. For both cases above, once users chose to start practice, Prompto guided the user through the “Verbalise” step (first step of the when-then technique) where the user was prompted to think about a task they intended to do, and related situation cues of event, time or location. It provided a summary of user responses, then encouraged them to verbalise the “when-then” sentence and corrected the sentence when needed.
4. The user was then led to the second step, to “Visualise” oneself performing the task with the situation cues in mind.
5. After that, the session was complete and the user could have more practice, learn more or leave the conservation.

Development of Cognitive Load Detector: Version 1 Rule-Based

Background: Biosignals and Cognitive Load

Electrodermal activity (EDA) refers to electrical changes on the skin which are measured through variations in skin conductivity due to reaction of sweat glands to external stimuli such as when you are surprised or stressed.

Heart rate variability (HRV) reflects changes in the time intervals between consecutive heartbeats called inter-beat-intervals (IBI) and is derived from photoplethysmography (PPG) signals which measure blood volume pulse. PPG sensors are common in fitness trackers and smartwatches for heart-rate readings and can be identified from the blinking green lights on the back of the watchface. The root mean square of successive differences between normal heartbeats (RMSSD) has been a commonly-used time-domain estimate of HRV.

EDA and HRV have been known as indicators for cognitive load. Calmness (lowered cognitive load) could be indicated from a lowered EDA and increased HRV (RMSSD).

Implementation

The Empatica E4 wristband was used as it had EDA and PPG sensors, and Bluetooth streaming capability. Raw EDA readings (4 Hz, microSiemens) from the E4 wristband were directly used by Prompto. The wristband automatically processed PPG signals to provide IBI values at 1 Hz. The data stream link was implemented on the Android and iOS apps using the E4 link SDKs.

Signal Processing

I implemented a function in the Android and iOS apps to calculate RMSSD values from the IBI data based on the following equation: $$RMSSD = \sqrt{\frac{1}{N-1}\sum_{n=1}^{N-1}(IBI_{n+1} - IBI_n)^2}$$ This equation has been used by other HRV processing tools like Kubios HRV.

Algorithm

The “lowered cognitive load detector” algorithm was developed based on analysis of biosignals from 21 older adults in a lab setting and 7 older adults for 7 days. For every IBI reading received (about every second), RMSSD was calculated from the previous 1 minute of IBI data (N = 60). Baseline EDA and RMSSD values were measured using the apps in a calibration sequence which asked users to sit and relax for 5 minutes. To detect a lowered cognitive load state, the average EDA and RMSSD values were calculated over 1 minute intervals.

If the following conditions were met:

60% of Baseline EDA value < Average EDA < 140% of Baseline EDA value, 80% of Baseline RMSSD value < Average RMSSD

A lowered cognitive load state was detected, an Event Request would be sent to the DialogFlow agent and would issue a prompt via the apps to start a training session.

Development of Cognitive Load Detector: Version 2 Machine Learning

I wanted to develop a more robust detector that did not require frequent baseline readings. The previous version used a rule-based algorithm. This version used a machine learning approach.

There were also EDA and HRV measures which were not taken into account in the previous version, such as the phasic EDA and frequency-domain HRV components, which have been found in previous work to reflect changes in cognitive load. This version incorporated these additional measures.

Additional Background on EDA and HRV

EDA signals can usually be split into tonic and phasic components. Tonic components are related to the skin conductance levels which slowly vary over time and phasic components are related to the sudden peaks in skin conductance values. The Empatica website explains these well. The previous detector mainly estimated the tonic component of EDA. In this version, both tonic and phasic components were used.

HRV can be split into time-domain and frequency-domain components. The previous version used RMSSD (time-domain). In this version, we include other time-domain components such as heart-rate (HR) and standard deviation of inter-beat intervals (SDNN). The frequency-domain components included were power of low frequency (LF, 0.04 to 0.15 Hz) and high frequency (HF, 0.15–0.4 Hz) bands, as well as the LF/HF ratio. More information on HRV can be found on the Kubios HRV website.

Implementation

Data Collection

Biosignal data was collected using E4 wristbands and smartphones (Android) from 5 older adults over 14 days (approx. 5 to 12 hours per day). Labels were collected through responses from hourly prompts which asked them to rate how calm they were (7-point scale) and if it was a good moment to start a memory training session (Responses: Yes / No / Maybe Later). “Calmness” rating of 5 to 7 were labelled as “low” cognitive load and rating of 1 to 3 were labelled as “high” cognitive load.

Signal Processing and Feature Extraction

The EDA and IBI signals were also processed through the data collection Android app (Java) to extract features for the classifier. A sliding window of 2 minutes (120 seconds, 50% overlap) was used.

The raw EDA data was filtered to remove noise using an exponential moving average filter. Tonic components of skin-conductance level and gradient were calculated from further normalising and filtering the signal with a low-pass Fast Fourier Transform (FFT) filter. Phasic components of peak amplitudes and number of peaks were calculated from filtering the EDA data with a high-pass Butterworth filter and a z-score peak detection algorithm.

The IBI data was used to calculate HRV-related components. The signal was first filtered to remove invalid IBI values (values which differ more than 20% from the previous value). Time-domain components of HR, RMSSD and SDNN were calculated based on established formulae. Frequency-domain components of LF and HF power were calculated using the Lomb-Scargle periodogram.

Overview of features and their descriptions:

Real-Time Classifier

A two-class (low/high cognitive load) classification system was built through Weka. Based on evaluation of classifier algorithm performances, I selected the Random Forest (RF) algorithm as it achieved the highest classification accuracies for the general model (82.14%) and person-specific model (highest 94.92%).

The trained RF general model was then deployed on an Android (Java) app which classified incoming processed EDA and IBI signals in real-time. Further implementation and data collection with more users are still in progress. More details on the system and evaluation are under preparation for a journal submission.

Results

Receptivity Study

Details on the design of the chat interface and Version 1 of the cognitive load detector were published in IMWUT December 2020 (see paper below titled: Prompto). The paper also contributes with an investigation of the effects of cognitive load on older adult’s receptivity to prompts.

In our controlled study with 21 users:

• Users were engaged in more memory training in low cognitive load (63%) than in high cognitive load (28.5%).
• Users were significantly more responsive to prompts in low cognitive load (90.5%) than in high cognitive load (48.8%).

In our in-the-wild deployment with 7 users for 7 days:

• Users’ response rate to the prompts was 67.9%.
• Majority of users appreciated the concept and found Prompto helpful

Ubicomp Conference 2021 Video Presentation:

Concept and Research Direction

The initial concept and design of Prompto was introduced as a paper in Augmented Humans 2019 (see paper titled: Prospero). I presented this research in the CHI 2020 Doctoral Consortium as part of my Ph.D. dissertation (see paper titled: Biosignal-Sensitive Memory Improvement and Support Systems).

Cognitive Load Detector

A manuscript detailing development of the real-time cognitive load classifier is in preparation for submission to a journal.

Awards

The project received three recognitions:

• Gold Pin (best in category) at The Best Design Awards 2020 by Designers Institute of New Zealand, an annual showcase of design excellence.
• Honorable Mention for Fast Company 2020 World Changing Ideas Award
• Female Founders Prize by UniServices at Velocity Innovation Challenge 2019