Predicting (Drops in) Vigilance using Machine Learning

Rosalie E. Lucas

6540384

June 26, 2021

Supervisor: Leon Kenemans

Abstract

The prediction of (drops in) vigilance can be useful in the prevention of car crashes. Previous

work showed that temperature data, demographic data and subjective measurements can be

useful in these predictions. With machine learning this study explores these different features

as possible predictors of vigilance in the Brief Stimulus Reaction Time task (BSRT). Three

FLIR camera may be useful predictors of vigilance, especially the forehead temperature.

However, the accuracy scores of the models were just slightly better than the naïve model. So,

further research should explore these features further and confirm these findings.

Introduction

In 2020 the Netherlands counted 620 fatal car crashes (Centraal Bureau voor de

Ponjaert-Kristoffersen, 2001). This can be explained by the fact that attention can be challenged

during routine tasks like driving (O'Connell et al., 2009). The question that therefore arises is

how these accidents can be prevented. Can people be alerted when their attention drops? To

answer this, more has to be known about the type of attention that is relevant for driving, named

vigilance. Vigilance is a state of readiness, manifested in the extent to which a person can detect

and respond at random time intervals to small changes occurring in the environment (Schmidt

et al., 2009). In other words, vigilance is the ability to pay attention, for a certain period of

time, to a task (Schmidt et al., 2009).

for the prediction of an outcome variable (Bzdok et al., 2018) because it learns from known

properties of the training data to give accurate predictions for unknown test data (Shanker,

2018). While classical statistics can also make predictions, machine learning is not bound by

the restrictions of these methods. Machine learning makes less assumptions and therefore is

able to detect the presence of complicated nonlinear interactions, such as double interactions,

of possible predictors, machine learning can also be useful to assess the relative importance of

different features, even in settings where the number of input variables exceeds the number of

participants (Bzdok et al., 2018).

Machine learning techniques also have a downside. A reason to be cautious with

machine learning is the so-called black box problem: the user has limited to no insight into the

That is why this research uses the Random Forest (RF) algorithm from Breiman (2001). A RF

is a classification algorithm based on randomised decision trees that together build a forest

(Breiman, 2001). These decision trees are easier to understand and give some insight in the

choices that the computer makes to classify each trial.

Another reason to use a RF is that Menze et al. (2009) found the RF to be the best

algorithm for feature selection. This is because it classifies based on different features. A RF

randomly selects features and uses these to split the decision tree based on their Gini impurity

(Menze et al., 2009). Gini impurity indicates how well a potential split is in classifying the data

with the Gini importance, this algorithm can thus be used to gain insight in the usefulness of

different features. The features found this way can subsequently be investigated in more depth.

Background

To determine which features are important to consider, more must be known about

previous research into predictors for vigilance. The focus will be on four groups: EEG,

Also, delta and theta activities have a high correlation with poor task performance and fatigue

(Li et al., 2014). Other studies have found a similar predictive relationship by using machine

learning algorithms, such as extreme machine learning and support vector machines (Shi & Lu,

2013; Li et al., 2020). Though EEG is a very powerful predictor of vigilance, it is not a very

practical one. EEG caps are not comfortable to wear and are quite fragile (Li et al., 2014). In

car for a long time. However, new research focusses on more wearable EEG systems (Li et al.,

2014).

Temperature. Fortunately, some research has been conducted into other predictors for

vigilance. Romeijn and Van Someren (2011) have conducted a vigilance experiment and found

skin temperature to be a possible predictor of vigilance. The fluctuations in skin temperature

can be described according to the Circadian Clock Mechanism or the Homeostatic Hourglass

(Te Lindert & Van Someren, 2018), implying skin temperature is related to the sleep-wake

cycle. A higher skin temperature results in feeling sleepier, having higher reaction times,

Skin temperature can be measured at different locations. These can be divided into

distal and proximal locations. Distal locations fluctuate more in temperature whereas proximal

locations are more consistent. Romeijn and Van Someren found the gradient between a distal

location and proximal location to be a good predictor of vigilance. Mainly the gradient finger-

chest and the gradient finger-wrist, were strong predictors of vigilance. Also, the gradient

regulation of vigilance (Romeijn et al., 2012). Furthermore, skin temperature measurements

seem to be applicable in everyday life (Romeijn & Van Someren). This means they could be

useful predictors for vigilance. This is precisely the topic of this research.

Subjective Measures. Other predictors could be found in the ability of people to

evaluate their own level of vigilance. For instance, there is a strong relationship between the

2009). This was investigated during a long drive. The relationship was absent towards the end

of the drive.

Degree of sleep deprivation is another possible measurement. The Pittsburgh Sleep

Quality Index (PSQI) assesses peoples’ sleep quality by calculating a person’s PSQI score

(Buysse et al., 1989). Being sleep deprived results in longer reaction times (Blatter et al., 2009).

When sleep quality is bad, a person does not sleep well and is effectively sleep deprived.

Finally, Blatter et al., (2009) found that reaction times increased the longer participants

were awake. The Morningness Eveningness Questionnaire (MEQ) assesses if a person is a

opposite could be true for an evening person who usually wakes up later and in combination

with an afternoon experiment has shorter reaction times.

Demographic measures. Lastly, demographic data could provide useful predictors.

Blatter et al. (2006) found that gender and age can influence reaction times. They found women

to react more slowly, but also more accurately than men. Furthermore, they saw that age had

Research Questions

In this research the focus was on the different temperatures, PSQI scores, MEQ scores,

gender and age as potential features to predict drops in vigilance. Some of these predictors have

been previously researched, while other predictors were explored for the first time in this study.

This has led to the following research questions: 1) Which feature is the best predictor of (drops

predicting vigilance? Three different RF models with different feature combinations were

compared to answer these questions.

Expectations

Based on previous literature the best temperature related features should be the chest

temperature and the finger-chest gradient (Romeijn & Van Someren, 2011). The pinna-mastoid

gradient could also be higher up in the hierarchy (Romeijn et al., 2012). For the other predictors

it is not possible to say something about the outcome yet.

Methods

In total 33 healthy participants participated in this study: 14 males and 19 females aged

19 - 59 years (M=26.15, SD=10.03). Participants were recruited within the social bubble of the

research leaders due to the COVID-19 pandemic. They joined on voluntary bases and were

rewarded with student credit or eight euros per hour (total of 20 euros for 2.5 hours). None of

the participants had any known history of sleep-related disorders. Because their vigilance level

or lenses) vision. They did not wear any hair products or make-up such that EEG electrodes

could be correctly placed.

Tasks.

BSRT. In this research the focus lies on the Brief Stimulus Reaction Time task. This is

a vigilance assessment task based on the task created by Romeijn and Van Someren (2011).

for the EEG signals. Participants have to focus on a fixation-cross (+ sign) and respond as fast

as possible when it changed into a fixation-cross with a shorter vertical line by pressing the

space bar with the index finger of their dominant hand. In the original paper participants had

to respond when the cross changed into a hyphen (- sign). The background is grey, and the

signs are black. The target presentation was only one frame (16.67ms), which is lower than the

original paper (25ms). Furthermore, this task had a staircase procedure. This resulted in the

change of difficulty of the task by altering the size of the vertical line of the fixation cross. The

most difficult condition of the task was a 10 pixels difference between the vertical line and the

trial will become more difficult. This is a 2-up-1-down procedure. The task comprised 144

stimuli with a 4 to 14 second interval and took around 20 minutes to complete.

Methods and materials. (Switched with procedure.)

iButtons. Skin temperature was measured using iButtons (type DS1922L,

Maxim/Dallas, USA). The iButtons sample skin temperature with a .0625 ºC resolution at 2-

sites: chest, forehead and mastoid. From this data three relative distal to proximal gradients

were calculated: finger minus chest (DPG

), nose minus forehead (DPG

), and

pinna minus mastoid (DPG

FLIR. Skin temperature was also measured using a FLIR camera (FLIR E53, FLIR

Systems Inc., Wilsonville USA). The camera samples skin temperature with a 240 x 180 pixels

frequency is 30 hertz. The camera was placed in front of the participant and focusses on the

head of the participants. One relative distal to proximal gradient was calculated: nose minus

forehead (DPG

Questionnaires. The PSQI and MEQ were both assessed in Dutch due to this being

the native language of most participants. The PSQI has been recoded according to the method

described by Buysse et al. (1988). This is done by calculating the scores for every component

of the PSQI and calculating the total score. However, for some participants some components

were missing. So, the total score calculated is the total score for all components divided by

intermediate person (total score between 42 and 58) and evening person (total score below

41).

Procedure.

All experiments were conducted in the lab environment of Utrecht University. This

study is part of a bigger study. The procedure described was the same for all participants, but

questionnaire. Next, participants were equipped with an EEG cap, face electrodes and iButtons.

During each session participants were seated before a 60 Hz screen in a dimly lit room at an

environmental temperature of 20.68 ºC (SD=0.56). Participants had to place their head in a

chin rest so that their movement was limited and the FLIR camera could be focused on their

head.

open. Next, they had to close their eyes for 2.5 minutes. Then participants were asked how alert

and awake the felt using a 100-likert scale. After that they were presented with the BSRT task

(Romeijn & Van Someren, 2011).

Data-Analysis.

All data was processed and analysed using Python 3.9 and software PyCharm

Professional 2021.1.1 (JetBrains, 2021). For the current study three different kinds of models

were trained multiple times on 500 different bootstrapped samples. One kind with only

temperature features, a second kind with only the subjective and demographic features and a

The third model does perform with 95% certainty better than the benchmark. This

because a difference of zero or lower was not present in the confidence interval (CI=[.0029,

1.00]).

Figure 5.

Accuracy Differences Between the Third Model with Temperature, Subjective and

confidence interval (CI= [-.0087, 1.00]).

Gini Importance. The Gini Importance of the features of the first, second and third

model were compared with the second benchmark created by the naïve model. These were

evaluated with a two-sided Confidence interval of 95% to see whether the features performed

better or worse than the benchmark. Figures 6, 7 and 8 display the differences between the Gini

importance and the benchmark of a feature. The 2.5 % lowest and highest values were removed,

thus creating a 95% Confidence interval. To test whether the feature performs better or worse

than the benchmark, it must be determined whether zero is included in the interval or not.

Figure 6.

The Confidence Intervals of the Differences Between Gini Importance of the Features of Model

1 and the Second Benchmark

Note. The dashed line marks the 0.0 difference between the GI of the feature and the second

benchmark.

Figure 6 displays the Gini hierarchy of the first model. There were 12 features in this

Figure 7.

The Confidence Intervals of the Differences Between Gini Importance of the Features of Model

2 and the Second Benchmark

Figure 7 displays the Gini hierarchy of the third model. There were 16 features in this

model, so the benchmark was set at .0625. The FLIR measured features performed better than

the benchmark with 95% certainty. From the subjective and demographic features performed

the participants PSQI score, Gender and Age worse than the benchmark.

The iButton measured temperatures and the participants MEQ type contained 0.0 in

with machine learning. The algorithm random forest (RF) was used due to its usefulness in

feature selection (Menze et al., 2009). Three different kind models were trained once and

finetuned. Then 500 bootstrapped samples were created, and the models were trained again on

these samples with the optimized hyperparameters. To evaluate these models a naïve model

was created that classifies all trials as hits (1.0). This was done by comparing the accuracy

data was very overfitted and therefore not useful for comparison. After adding the finetuned

hyperparameters they performed slightly better or the same as the naïve model. Below results

are discussed for each model separately.

The first model used only temperature related features. It had an accuracy of 100% for

the train data and 73.8% for the test data. The high accuracy for the train data shows that the

model overfitted the data. Therefore, the parameters of the algorithm had to be finetuned. After

finetuning, the model performed significantly better than the naïve model with a 95% accuracy.

The highest-ranking features are now the forehead temperature measured with the FLIR

Because the first model scores slightly better than the naïve model, conclusions can be

drawn from its Gini hierarchy. This model implicates that the temperature of the forehead

measured with the FLIR camera can be seen as the best predictor of (drops in) vigilance for

this task because it performs significantly better than the benchmark. The other features that

were measured or calculated with the FLIR camera, ranked relatively high but statistically

The second model used the subjective and demographic features. The model did not

show any overfitting but was still finetuned the same way as the first and third model. The

participants MEQ type performed significantly better than the benchmark. The participants

PSQI score and gender performed significantly worse than the benchmark. Statistically nothing

can be said about the age of the participant. The second model did not significantly perform

data on their own have no contribution to the prediction of vigilance for this task.

The third model used the temperature related features and the subjective and

demographic features. The accuracy of this model was 100% for the train data and 73.8% for

the test data. This model also has a high accuracy score for the train data. Therefore, the model

overfitted the data. The finetuning of this model was the same as for the first and second model.

After finetuning, the model performed significantly better than the naïve model with a 95%

accuracy. The highest-ranking features were measured with the FLIR camera. These were the

forehead temperature, nose temperature and the DPG

conclusions can be drawn from the Gini hierarchy of this model. The FLIR measured features

performed significantly better than the benchmark. The participants PSQI score, Gender and

Age performed significantly worse than the benchmark. Even though the nose temperature of

the iButton ranked high, statistically nothing can be said about its relative performance to the

benchmark. The same goes for the other features measured with the iButtons. Also, the third

further exploration while age, gender and PSQI score do not show any potential for this task.

However, some placements of features in the hierarchy are inconsistent with previous

literature. Romeijn and van Someren (2011) found the chest temperature to be a better predictor

than the DPG

hierarchies, it even scores significantly worse than the benchmark. In addition, the chest

iButton measured features, this DPG ranks relatively high. This is consistent with the findings

Romeijn and Van Someren which showed that this DPG is a strong predictor of vigilance.

However, its confidence interval still includes zero so nothing can be said statistically about its

performance relative to the benchmark.

Even though the first and third model perform slightly better than the naive model, it is

debatable how relevant these models are. The reason for performing just slightly better could

be found in the difference between participants. There can be different fluctuations in skin

temperature between participants. Romeijn and Van Someren (2011) already noted that these

Furthermore, this study added a staircase procedure to the BSRT task created by

Romeijn and Van Someren (2011). The staircase procedure made sure that the amount of hits

for each participant was consistent. However, it could have resulted in a confounding factor:

when a participant became less vigilant, instead of producing more misses the task became

easier resulting in more hits again. On the other hand, when a participant was vigilant a miss

and gender.

Furthermore, the finetuned hyperparameters were fixed for all the 500 samples. To be

conclusive these parameters had to be optimized for each bootstrapped sample to get optimal

performance. However, due to the lack of time this procedure was only done for one model

and those values were chosen to be the hyperparameters. Further research could add this

Lastly, due to the many different experimental leaders some things went wrong during

the data collection. Because there was not much time for the leaders to get used to the protocol,

some files were not (correctly) saved. Also, the processing of the FLIR data took a long time

so more participants had to be excluded from the study because their FLIR data was

unavailable. This limited the amount of data that could be used for analysing and therefore

reduced power.

Due to these limitations more research should be done about the exploration of different

features in the BSRT without the staircase. Also, other tasks like the CTET (O'Connell et al.,

tasks (O'Connell et al., 2009) which creates a more evenly distributed division between hits

and misses.

Also, more has to be known about the difference between individuals in fluctuations of

skin temperature. As mentioned before, participants can differ in their fluctuations in skin

temperature but also in their overall mean temperature. By training classification models for

Further research should also look at the kind of predictive relationship between the

features and output. While a RF is very useful for feature selection, it cannot say much about

this relationship. Different machine learning algorithms can be used to determine the specific

relationship and classical statistics should confirm these relationships.

In conclusion, the research questions of the current study can be answered. The best

camera. The hierarchies of the first and third finetuned models are the best indicators of the

usefulness of predictors. However, these models’ accuracies do not differ much from the naïve

model meaning that all included features in aggregate have little predictive power for drops in

vigilance. So, the conclusions that are drawn have to be confirmed with further research.

References

Blatter, K., Graw, P., Münch, M., Knoblauch, V., Wirz-Justice, A., & Cajochen, C. (2006).

Gender and age differences in psychomotor vigilance performance under differential

sleep pressure conditions. Behavioural Brain Research, 168(2), 312–317.

https://doi.org/10.1016/j.bbr.2005.11.018

Pittsburgh sleep quality index: A new instrument for psychiatric practice and research.

Psychiatry Research, 28(2), 193–213. https://doi.org/10.1016/0165-1781(89)90047-4

Bzdok, D., Altman, N., & Krzywinski, M. (2018). Statistics versus machine learning. Nature

Methods, 15(4), 233–234. https://doi.org/10.1038/nmeth.4642

Centraal Bureau voor de Statistiek. (2020, October 15). Hoeveel mensen komen om in het

mensen-komen-om-in-het-verkeer-

Chang, S., Cohen, T., & Ostdiek, B. (2018). What is the machine learning? Physical Review

D, 97(5), 056009-1 – 056009-6. https://doi.org/10.1103/PhysRevD.97.056009

De Raedt, R., & Ponjaert-Kristoffersen, I. (2001). Predicting at-fault car accidents of older

drivers. Accident Analysis & Prevention, 33(6), 809–819.

https://doi.org/10.1016/S0001-4575(00)00095-6

Fraiwan, L., Lweesy, K., Khasawneh, N., Wenz, H., & Dickhaus, H. (2012). Automated sleep

stage identification system based on time–frequency analysis of a single EEG channel