Research Paper

Roufiaa91

DynamicdriverfatiguedetectionusinghiddenMarkovmodelinrealdrivingcondition.pdf

Home >English homework help >Research Paper

Expert Systems With Applications 63 (2016) 397–411

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Dynamic driver fatigue detection using hidden Markov model in real

driving condition

Rongrong Fu a , Hong Wang b , ∗, Wenbo Zhao c

a College of Electrical Engineering, Yanshan University, Qinhuangdao, 066004, China b Department of Mechanical Engineering and Automation, Northeastern University, Shenyang 110819, China c Department of Mechanical and Industrial Engineering, Louisiana State University, Baton Rouge, LA, 70803, USA.

a r t i c l e i n f o

Article history:

Received 4 February 2016

Revised 6 April 2016

Accepted 23 June 2016

Available online 24 June 2016

Keywords:

Driver fatigue

Physiological signals

Context information

Hidden Markov Model

a b s t r a c t

Driver’s states in successive time slices are not independent, especially, fatigue is one of a cognitive state

that is developing over time. Meanwhile, driver fatigue is also influenced by some corresponding contex-

tual information at a certain time. In such case, classifying driving state at each time slice separately from

it in before and after time slices obviously has less meaning. Therefore, a dynamic fatigue detection model

based on Hidden Markov Model (HMM) is proposed in this paper. Driver fatigue can be estimated by this

model in a probabilistic way using various physiological and contextual information. Electroencephalo-

gram (EEG), Electromyogram (EMG), and respiration signals were simultaneously recorded by wearable

sensors and sent to computer by Bluetooth during the real driving. From these physiological information,

fatigue likelihood can be achieved using kernel distribution estimate at different time sections. Contex-

tual information offered by specific environmental factors were used as prior of fatigue. As time proceeds,

the posterior of fatigue can be gotten dynamically by this HMM-based fatigue recognition method. Based

on the results of the method in this paper, it shows that it provides an effective way in detecting driver

fatigue.

(

. Introduction

Driver fatigue is a constant occupational hazard for drivers,

hich is the major cause of road accidents and has implications

or road safety ( Jap, Lal, Fischer, & Bekiaris, 2009; Pylkkonen et al.,

015 ). According to statistics, highway traffic accidents accounted

or the total number of accidents in 11.09% ( Li, Liu, Yuan, & Liu,

010 ). Besides driver fatigue, many other reasons can also cause

raffic accidents, such as unsafe lane change maneuvers ( Hou,

dara, & Sun, 2015; You et al., 2015 ), overloading, illegal parking,

llegal overtaking, and driving over-speed. The driver fatigue is the

argest contributor to the highway traffic accidents, which has been

stimated to be involved in 2%–23% of all crashes ( Yang, Mao, Tije-

ina, & Pilutti, 2009 ). Due to the difficulty of assessment the exact

umber of fatigue-related collision, these numbers are still conser-

ative estimation.

The highway with the wide and flat pavement, few spatial ref-

rences, and high traffic speed provides monotonous driving en-

ironment. All vehicles follow their respective lanes, moving in

∗ Corresponding author. E-mail addresses: [email protected] (R. Fu), [email protected] (H. Wang),

[email protected] (W. Zhao).

ttp://dx.doi.org/10.1016/j.eswa.2016.06.042

rderly fashion on the highway with high speed. Long duration

f driving in this monotonous traffic environment require drivers’

ustained attention for long periods ( Ting, Hwang, Doong, & Jeng,

008 ). It is inevitably accompanied by a decrease in alertness and

esults in performance decrements and a higher risk of accidents

Eugene, Carolyn, Kayla, & John, 2015 ). Moreover, great decrement

f driver performance can be markedly influenced by two phys-

ological factors – circadian rhythm and sleep quality ( Ferguson

t al., 2012; Sahayadhas, Sundaraj, Murugappan, & Palaniappan,

015 ). Great proportion of fatigue-related accidents occur between

he hours of 2–6 a.m. and 2–4 p.m. approximately ( Williamson &

riswell, 2011 ). During these two time periods, driver’s body easily

et into natural drowsiness, which increase the chance of crashes.

eanwhile, sleep quality plays a critical role in driver’s behavior.

leep deprivation can cause essentially degradation of all aspects

f functions, including cognitive processes, attention and focusing,

igilance, physical coordination, judgment, awareness and deci-

ion making, communication, and numerous other parameters ( Al-

ultan, Al-Bayatti, & Zedan, 2013; Ji, Lan, & Looney, 2006 ). Many

revious studies noted that sleep deprivation almost have the same

azardous effects as drunk driving ( Williamson & Feyer, 20 0 0 ).

Over the past several decades, the fatigue detecting technol-

gy has been the widespread hope in the prevention of fatigue

elated accidents ( Shen, Li, Ong, Shao, & Wilder, 2008 ). Up to now,

398 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

researchers have developed several different types of fatigue detec-

tion technologies. According to the features used for fatigue recog-

nition, there are four main categories technologies based on differ-

ent features as contextual features, physiological features, driver’s

performances, and the combination of aforementioned features.

1) Contextual features technologies: from different point of

views, this kind of methods include,

(i) driver-related: personality, sleep quality, circadian

rhythm, and physical condition.

(ii) vehicle-related: noise, seating comfort degree, and tem-

perature;

(iii) road-related: monotony of road, density of vehicles, and

the number of lanes.

Questionnaire is always used for collecting such contextual fea-

tures. Context-based technologies are perhaps the easiest

method by which the extent of driver fatigue can be inves-

tigated from these features by some statistical methods.

2) Physiological measures: driver fatigue may be presented on

some physiological features, such as features from EEG, EMG,

ECG (electrocardiogram), respiration, and many other phys-

iological signals ( Khushaba, Sarath, Sara, & Gamini, 2011;

Sun and Xiong, 2014 ). EEG features can reflect the ongoing

brain activity and give abundant information on human cog-

nitive states directly. Rogado et al. developed a driver fatigue

recognition system based on Heart rate variability (HRV)

from Electrocardiograph (ECG) during driving period because

ECG is also found that it contains lots of fatigue relevant

information. EMG features were used by Hostens et al. un-

der high level monotonous driving condition ( Hostens & Ra-

mon, 2005 ). Besides these physiological signals, respiration

can contribute some fatigue related information. The meth-

ods based on physiological signals have been regarded as the

most accurate and objective fatigue recognition method.

3) Performance-related methods: fatigue can result in some

typical fatigue-related behaviors, such as reaction time, eye

blinking frequency, eye-closure rate, throttle/brake input,

steering angle, vehicle speed, lane deviation ( Son, Yoo, Kim,

& Sohn, 2015 ), gear changes ( Yang, 2007 ), head nodding, and

grasping position of driver’s hand on steering wheel ( Di Stasi

et al., 2012; Minin, Benedetto, Pedrotti, Re, & Tesauri, 2012 ).

Based on imaging processing or other measurement meth-

ods, the changes of these different f eatures can be moni-

tored to infer driver fatigue. The main drawback of these

methods is that their accuracy depends on the individual

characteristics of the vehicle and its driver ( Jo, Lee, Kang,

Kim, & Kim, 2014 ).

4) Methods based on combination of aforementioned features:

these integrated methods can take the advantages of the

three previous methods, meanwhile, try to avoid the disad-

vantages of them.

The previous three methods focus only on a certain aspect,

therefore they may lead to inaccurate results easily. First, in the

technologies based on contextual features, drivers can evaluate

their efficiency decline during driving. Self-feedback plays an im-

portant role in subjective measurement and may be affected by

subject’s will and consciousness ( Declerck, Boone, & Brabander,

2006 ). On the one hand drivers easily overestimate their driving

abilities, and on the other hand they also incline to underestimate

the risk of accidents. Sleepiness is such a powerful biological sig-

nal that it can happen in an uncontrolled and spontaneous way

( Ji et al., 2006 ). Most of us have experience that sometimes we

felt so drowsy that we fell asleep suddenly even when we were

driving ( Morris, Pilcher, & Switzer, 2015 ). In fact, in National Sleep

oundation’s 2005 Sleep in America poll, 60% of adult drivers –

bout 168 million people – admit they have driven a vehicle while

eeling drowsy in the past year, and more than one-third, (37%

r 103 million people), have actually fallen asleep while driving.

ational Highway Traffic Safety Administration conservatively es-

imates 10 0,0 0 0 police-reported crashes being the direct result of

river fatigue each year. This results in an estimated 1550 deaths,

1,0 0 0 injuries, and $12.5 billion in monetary losses. Therefore,

ethods only relay on drivers’ self-report cannot always reflect

eal objectivity. Second, studies of performance-based techniques

annot prove that these abnormal behaviors are exactly relevant to

river’s drowsiness state. Vehicle type, driver experience, driving

onditions ( Ueno, Kaneda, & Tsukino, 1994 ), and some other factors

an also result in those abnormal behaviors. Third, some validation

riteria used for fatigue recognition were based on the image pro-

essing techniques. Although there is great value for fatigue detec-

ion, many of these features were reported that they may vary in

ifferent driving conditions ( Lin et al., 2006 ). There are still some

oments when a driver still looks awake with wide open eyes but

oes not process any information ( Renner & Mehring, 1997 ).

Therefore, fusing as many features as possible is a better way

o get an accurate inference ( Chen & Meer, 2005 ) and make fa-

igue recognition more reliable. Physiological features, as men-

ioned above, contribute significantly to fatigue recognition be-

ause a person usually has little control over them, which makes

hey could provide reliable and objective source of information

o determine person’s fatigue ( Conati, 2012 ). Methods based on

used physiological features are perhaps the most accurate, valid

nd logical method ( Ji, Zhu, & Lan, 2004 ). Ji et al. developed fa-

igue detection model fusing contextual and physiological features

y a static Bayesian network ( Ji et al. 2004 ). However, in driver’s

atigue recognition, the dynamic character of features should be

onsidered. Therefore, Li and Ji proposed a dynamical fatigue de-

ection model based on dynamic Bayesian network in their further

tudy ( Li and Ji. 2005 ). The physiological features were utilized

y Ji et al. are all based on driver’s face expression information

otten by image processing technology. Many studies had shown

hat face expression features work well in detecting driver’s fa-

igue. However, on one side these visual features are vulnerable to

ights and it increases the difficulty in accurate recognition, on the

ther side the face image data have much larger size than physio-

ogical signals, this requires the algorithm has higher computation

peed. Based on these studies, Yang et al. involved EEG and ECG

n constructing a probabilistic driver’s fatigue detection model by

ynamic Bayesian network to enhance the reliability of fatigue de-

ection ( Yang, Lin, & Bhattacharya, 2010 ). These studies created a

ind of new thoughts in the driver’s fatigue detection research.

Based on this kind of new thought given in previous studies,

e constructed a dynamic fatigue detection model based on Hid-

en Markov Model. We developed this approach in the following

ays: (i) the previous studies were based on the simulated driv-

ng condition, and we built the fatigue inferring method under the

eal driving condition, which makes these kind of method more

eliable. (ii) to avoid the limitation of visual features, we utilized

hree physiological features from EEG, EMG and respiration signals.

nd these data acquisition were collected in wireless way. This is

ne of the differences from Yang’ work. (iii) we combined static

ayesian and dynamic Bayesian (HMM) to estimate the driver’s fa-

igue at initial time and following time periods. And we analyzed

he posteriors of fatigue by local and global contexts involving in

MM. (iv) by feature fusion, we can infer the fatigue states more

eliable and make the fatigue detection model as the 3-layer HMM.

his makes the fatigue inferring by combining more information

nd meanwhile the model is still with simple structure.

The purpose of this research is to establish an objective, re-

iable, and real-time model to detect and monitor driver fatigue,

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 399

Fig. 1. Real driving in highway. EEG, EMG, and respiration signals were transmitted via a cordless Bluetooth connection to a laptop simultaneously. The driver’s freedom of

movement is not restricted during driving. And the drive route from Shenyang to Dandong is extracted from Google maps.

(

hich is the major obstacle of fatigue detecting technology ac-

ording to National Transportation safety improvements. Therefore,

eveloping a driver fatigue recognition model based on the com-

ination of multiple physiological features and some contextual

atigue-related information becomes important. In this paper, a dy-

amic fatigue detection model based on Hidden Markov Model

HMM) is presented. With the combined information both from

hysiological and contextual aspect, driver fatigue can be inferred

ver time by this model in a probabilistic way. EEG, EMG and res-

iration signals can be obtained simultaneously and wirelessly by

hree wearable sensors. Likelihood and prior of fatigue can be in-

erred from physiological and contextual information. As time pro-

eeds, the posterior of fatigue can be gotten dynamically by this

MM-based fatigue recognition method.

. Experiments and data

.1. Participants and driving task

Twelve professional long-distance bus drivers (all males) with

he age of 41.3 ± 2.2 took part in the 3.5 h real highway driving. articipants performed in compliance with the following instruc-

ions to be included into the study. They were required to keep the

egular sleeping hours for two days prior to the experiment, which

nsured they have more than 7 h continuous sleeping time. They

ere asked to refrain from consuming alcohol, caffeinated drinks,

ea or drowsiness causing medications approximately 12 h before

he study. In addition, the drivers were generally in good physical

nd mental health, and they had no physical barriers to ensure safe

riving and complete the driving task successfully.

In the experiment stage, real driving route shown as in Fig. 1

as from Shenyang (41.78 ° N, 123.43 ° E) to Dandong (40.12 ° N, 24.38 ° E) for bus drivers. These two cities both belong to Liaoning, hina. Dandong is 254 km from Shenyang, the bus covered this dis-

ance in 3.5 h. The initial driving was approximately 45 min of driv-

ng in Shenyang urban district. After that, most driving time was

pent on monotonous highway driving, which was last about 2.5 h.

ollowing this highway driving, about 15 min was cost on the driv-

ng in Dandong urban district. According to bus schedule of the bus

ompany, we selected the one with departing time at 1:00 p.m., it

overed one of sleepy peak of circadian rhythm. Meanwhile, there

s a lot of differences between urban driving and highway driving.

ighway driving involved the participants with fewer road stimuli

han urban driving. This driving environment can also contribute

ifferent levels of fatigue. During the whole driving, drivers used

utomatic shift and were asked to refrain from turning on the ra-

io or using other in-car devices. Participants were also instructed

o avoid unnecessary movements in order to reduce artifacts in the

hysiological data recording ( Schmidt, Schrauf, Simon, Buchner, &

incses, 2011 ). Driver state was regarded as alert, mild fatigue, and

atigue. The happening time of these three states were informed

y drivers themselves.

.2. Data collection and preprocessing

During the whole 3.5 h driving, 13 data sections were recorded

ith about 15 min interval, and the fatigue levels among alert,

ild fatigue, and fatigue were reported by driver after each data

ecording sections. Simultaneous physiological measurements were

ecorded during the driving sessions. The modular and portable

iofeedback 20 0 0 x-pert system were using in recording phys-

ological parameters. EEG, EMG, and respiration signals can be

btained simultaneously and wirelessly by three wearable sen-

ors. Considering the need of practical application of this research

nd the results of previous research, the EEG signals were col-

ected from the O1 and O2 electrodes, which place at the very

ack of the brain (occipital lobe) and collects and interprets visual

mages ( Kayvan & Robert, 2006 ). Meanwhile, the collected elec-

rodes which are selected in this paper are consistent with many

ther previous researches on neurophysiology of mental fatigue

Alloway, Ogilvie, & Shapiro, 2006; Cajochen, Brunner, Krauchi,

raw, & Wirz-Justice, 1995; Cantero & Atienza, 20 0 0; Lin et al.

010; Stampi, Stone, & Michimori, 1995 ). The location and connec-

ion of them are described as follow:

• EEG module: Driver’s fatigue can result in the significant

change in the occipital region, followed this, we collected EEG

from O1 and O2. According to the 10–20 international stan-

dard of electrode placement. Sensor recorded EEG signal non-

invasively from the brain skin surface place at O1 and O2. The

reference electrode was placed on the mastoid bone behind the

left ear.

400 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

Table 1

Confusion Matrix.

Condition positive Condition negative

Actual positive a (true positives) b (false positives)

Actual negative c (false negatives) d (true negatives)

True positive fraction (TPF) is also called sensitivity, which can

be computed by a/(a + c). And false positive fraction (FPF) equals 1 minus specificity, it can be obtained by b/(b + d).

(

�

• EMG module: Considering the drivers’ comfort, driving safety,

and signal quality, EMG was collected from nape of drivers’

neck in differential input. • Respiration module: The module using for achieving driver’s

abdominal respiration signal is a kind of belt-like recording de-

vice. Respiration strap can attach the module over the clothing.

Sensor signals were filtered, amplified, digitalized and trans-

mitted via a cordless Bluetooth connection to an external laptop,

which is shown as Fig. 1 . The driver’s freedom of movement is not

restricted.

Specially, these physiological potentials were amplified by a dif-

ference amplifier with very high input resistance ( > 2 G ohm) and

digitalized using a 24 bit processor with a sampling rate of 200 Hz.

In order to minimize common-mode interference, the reference

channel was provided only with a driven right leg circuit. This cir-

cuit generated a floating mass that increased the common-mode

rejection ratio and provided necessary reference potential for data

recording.

Since it is inevitable that the recorded data were disturbed by

noise, like the artifacts induced by some driver’s movements, in-

cluding changing the shift, acceleration, throttle, and brake move-

ment during driving. Some preprocessing methods can help us re-

alize the artifacts de-nosing such as wavelet transformation and

empirical mode decomposition, however considering the dynamic

requirement of this driver fatigue detection model, the digital fil-

ter, that provides a much simpler way, is preferred to be used here.

And some features from the filtered signals can be given as follow-

ing.

• The raw EEG signals were divided into the usual frequency

bands by means of a Fast Fourier transformation with the buffer

width of 256. The FFT provided a spectral analysis of the real

and imaginary parts of the recorded and digitalized EEG po-

tential. Three fatigue related sub-band EEG waves with differ-

ent typical frequency peak rages were achieved by five order

Butterworth band-pass filter. They were theta (4–8 Hz), alpha

(8–13 Hz), and beta (13–30 Hz). The processed EEG feature was

computed from power spectrum of EEG data in these three

bands as shown in Eq. (1) ,

P ( θ + α) / ( α+ β ) = P θ + P α P α + P β (1)

where P α , P β and P θ are the mean power of α, β and θ respec- tively. For example P α =

∑ f i × P i /

∑ P i , in which f i is frequency in

α band and P i is the absolute power obtained using FFT in α band.

• By using the band-pass filter as given in previous EEG pro-

cessing, filtered EMG with frequency band of 25–500 Hz were

achieved. Root mean square of filtered EMG was used as EMG

feature. • Respiration feature was the mean frequency power of respira-

tion signal, it can be got by ‘pwelch’ function in Matlab, and

represents the feature of signals in the frequency way.

3. Feature optimization

The purpose of feature optimization was finding the appropri-

ate weights for combining the three different features into one op-

timized feature. Feature optimization can realize two things:

(1) Make the feature in different classes as separate as possible,

based on this optimized feature, we can get the best result

in identification of fatigue.

(2) Fuse the original three features into an optimized one fea-

ture that can be also regarded as feature reduction.

This fatigue indicator is worthy of investigation so that the op-

timal feature can be determined. In the current paper, the Relative

perating Characteristic (ROC) curves were used to identify the op-

imal indicator of driver fatigue.

.1. Relative operating characteristic curves

The ROC curves are built from a learning set of experimental

ecords where all segments have been labeled previously. These

urves can give the relationship between the correct classified

ecords and the incorrect classified ones, which are named as

robability of detection and probability of false alarms respectively

DuchÊne & Lamotte, 2001 ).

Generally speaking, ROC curves are the entire set of possible

rue and false positive fractions attained by dichotomizing a con-

inuous test result with different thresholds ( Bertozzi, Broggi, Fas-

ioli, Graf, & Meinecke,2004 ). That is, ROC curve plots true positive

ates against false positive rates as threshold varies ( Tang, Du, &

u, 2010 ). Statistic data are reported as confusion matrix as shown

n Table 1.

The values of TPF and FPF are needed when constructing the

OC curve. Combined the values of TPF and FPF obtained by differ-

nt thresholds, the ROC can be plotted by using FPF value as hor-

zontal axis and TPF value as vertical axis, that means the ROC is

he set as { (F P F (i ) , T P F (i )) , i ∈ (−∞ , ∞ ) } ( Liu & Zhao, 2012 ), which s with the monotone increasing trend from 0 to 1. The area under

OC curve (AUC) is an important statistic, and for these areas, the

arger the better.

.2. Features fusing method

As we discussed in data collection and preprocessing part, three

eatures were achieved from EEG, EMG and respiration signals.

e would like to fuse them into one dimension feature by using

q. (2) .

f ( λ1 , λ2 , λ3 ) . = −λ1 x 1 + λ2 x 2 + λ3 x 3

s.t.

3 ∑ i =1

λi = 1

0 ≤ λi ≤ 1 (2) here x 1, x 2 , x 3 represent respiration, EMG, and EEG feature re-

pectively. From off-line analysis, amplitudes of EEG and EMG fea-

ures get larger over time, however, respiration feature get smaller

n its amplitude as time goes on. Therefore, the ‘minus’ sign was

iven to make these three features give similar trend. About the

xtreme situations, when λ1 = λ2 = 0 , and λ3 = 1 , the fused fea- ure represents EEG signal; when λ1 = λ3 = 0 , and λ2 = 1 , the used feature represents EMG signal; λ2 = λ3 = 0 ,and λ1 = 1 , the used feature represents respiration signal.

Due to the characters of visualization and efficiency of ROC

urves, the higher AUC value gives more separable feature. To min-

mize misclassification risk and maximize features’ separability, se-

ect the weights combination for which AUC value got by fused

eature and corresponding class label is maximum, therefore the

ptimize objection stated formally as Eq. (3) .

∗ = arg max [ AUC ( f (�) , cl assl abl es ) ] (3)

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 401

Fig. 2. Optimum solution by cross-validation. Our strategy was to find an appro-

priate pair of parameters which can give the highest AUC value. As shown in this

figure, the optimum AUC value can be get when λ1 = 0 . 6 , λ2 = 0 . 3 and λ3 = 0 . 1 . Due to constraint conditions, λ1 and λ2 cannot larger than 0.5 at the same time.

So the points on right side of this λ1 − λ2 plant have no value, here we gave their values as 0.5 to make this figure clear.

Fig. 3. ROC plot of the optimized feature. The curve with the bulge shape and near

to the upper left hand corner of the figure shows that optimized feature can give

the highest AUC value of 0.936.

Fig. 4. Graphical model of a first-order Hidden Markov Model, in which the dis-

tribution P( y t | y t−1 ) of particular y t is conditioned on the value of y t−1 . This is a 3-layer HMM model of latent variables y i with each context c i and observation x i conditioned on the state of corresponding latent variable. It is a typical context-

state-observation structure.

(

here � = [ λ1 , λ2 , λ3 ] is the weights vector, and f ( �) represents he fused feature with different weights. With the help of the

onstraint condition given by Eq. (2) , the summation of all three

eights is 1, this turn out to be a two-parameter numerical mini-

ization problem. Some sample based method should be used to

stimate the appropriate values for λ1 and λ2 . Our approach to odel selection is to choose values of these weights. It can min-

mize the cross validated estimate of misclassification risk jointly.

n special, we built a parameters plane with different pair of val-

es of λ1 and λ2 , they are both with the range from 0 to 1, and e sampled them with interval of 0.1. The misclassification risk at

very point on this points’ grid plane can be evaluated.

In the λ1 and λ2 grids plane, the best estimated values of these wo parameters can be obtained from choosing the point with the

ighest AUC value, and weight λ3 value can be got by using 1 mi- us λ1 and λ2 values. The optimal weights combinations can be ot from the first and last data sections, as shown in Fig. 2 . It is a

ood way to evaluate the performance of different fused features

nd identify the optimized feature from constructing the experi-

ental ROC curves.

As optimum weights can be found by using cross-validation

ethod, we can achieve the optimized feature from fusing three

riginal features by using Eqs. (2) and (3) . ROC plot of the opti-

ized feature can be shown in Fig. 3.

. HMM-based fatigue modeling

As we shown before, driver states extrapolation is a compli-

ated task. Driver fatigue is accumulated with time. In consecu-

ive time slices, driver’s states are indeed highly correlated rather

han independent, in this case the independent and identically dis-

ributed assumption will be a poor one. These un-i.d.d assumption

ata can arise through contexts and measurement of time series.

e would like to estimate the current state in a time series by

iven the previous state. Moreover, we know in practice a driver’s

tate remain stable relatively. Therefore, knowing whether or not

he fatigue occur in this time is important to predict future state.

e consider the recent state is able to provide more information

han more historical one. So a dynamic fatigue detection model

ased on the first-order HMM can be used in predicting future val-

es. Graphical model (see Fig. 4. ) is used here to illustrate our fa-

igue detection model.

The joint distribution of first-order HMM can be expressed by

4 ), in which the initial probability of hidden state is represented

y P ( y 1 ).

(C, Y, X ) = P ( y 1 ) T ∏

t=2 P ( y t | y t−1 )

T ∏ t=1

P ( Y t | C t ) T ∏

t=1 P ( X t | Y t ) (4)

For fatigue modeling, we wish to assess driver fatigue by using

his HMM structure, as shown in Fig. 4 . In this case, fatigue is ob-

iously the target hypothesis variable that we intent to infer. This

odel is used to assess users’ fatigue state from both contextual

nformation and observations dynamically. We need to define the

layers in this model structure as follow:

The first layer is context. It provides information of some spe-

ific contextual factors, which could lead to fatigue. The probabil-

ties of P ( y t | c t ) can specify how the contextual information influ-

nce the driver’s state; The second layer is hidden state we wish

402 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

Table 2

Prior Probabilities for Contextual Information.

Contexts State values Probabilities

Sleep quality (SQ) Poor 0 .3498 Eq. A.2 in Appendix

Normal 0 .6502

Driving condition (DC) Poor 0 .1840 Eq. A.3 in Appendix

Normal 0 .8160

Circadian rhythm (CR) Drowsy 0 .1930 Eq. A.4 in Appendix

Active 0 .8070

The probabilities in Table 2 were obtained by the law of total probability based on

the conditional probabilities given by Ji. et al. 2006 . They represented the prob-

abilities of these three contexts considering all different conditions and variables

that can influence the contexts. So they provide the prior probability information,

and they are also of the general meaning.

to be able to infer. Its binary values are alert and fatigue; The third

layer is observation. The observation layer in here is the optimized

physiological feature as we discussed in Section 3.2 . The change

of optimized feature in different time slices is caused by the state

changing. In other words, this layer can give the symptom of fa-

tigue.

4.1. Kernel density estimation

Kernel density estimation (KDE) is a nonparametric method

used to estimate the probability density function of a random vari-

able in � d . The region � d is assumed to be a small d -dimensional hypercube centered at the point x . Therefore the total number of

data points inside this hypercube can be defined as ( 5 ),

K = N ∑

i =1 k

( x − x i

) (5)

where k ( u ) is defined as the kernel function, h is represented as

the length of an edge of that hypercube. In our study, we utilized

the Gaussian kernel function as the smoothing window, it can be

shown that the optimal value of h is,

h = (

4 ̂ σ 5

3 n

) 1 5

≈ 1 . 06 ̂ σ n −1 / 5 (6)

where ˆ σ is the sample standard deviation and n is the number of samples. And we used two methods to estimate the sample stan-

dard deviation, one is ‘std’ function in matlab, and the other is ‘iqr’

function. We chose the smaller one between the two standard de-

viations gotten by these two method as the optimal value of ˆ σ . We use V d as the volume of this hypercube in d dimensions,

the probability density at each point x i can be estimated by the

contribution of each sample x i as ( 7 ).

p(x ) = 1 N

N ∑ i =1

V d k

( x − x i

) (7)

Gaussian kernel ( Bishop, 2007 ) was chosen in our case. The

density is estimated by placing a Gaussian over each data sam-

ple and then adding up the contributions over the whole data set,

finally dividing by N to normalize the density. So with Gaussian

kernel density, ( 7 ) becomes as ( 8 ).

p(x ) = 1 N

N ∑ i =1

(2 π h 2 ) 1 / 2

exp

{ − ‖ x − x i ‖

2 h 2

} (8)

where h represents the standard deviation of the Gaussian kernel

function as we defined in Eq. (5) .

4.2. Contextual information

Driver fatigue has a very complicated generation process, many

factors can influence it in the interacting way. In order to capture

the dynamic aspect of fatigue, three contextual information that

can result in fatigue will be discussed as follow:

4.2.1. Sleep quality

Sleep quality is a significant contextual factor, which has a di-

rect influence to the driver fatigue. The driver’s sleep quality is re-

lated to some quantities as sleep time, sleep environment, sleep

condition, and so on. In which, sleep environment including light,

noise, heat, humidity contributes a lot for sleep quality.

4.2.2. Driving condition

Different driving conditions can also affect fatigue. Obviously,

the tedious/monotony of the road and the density of cars have

strong relations with driving environment.

.2.3. Circadian rhythm

Circadian rhythm is a key contextual component in driver’s fa-

igue recognition. It is regarded as an important consideration in

etecting the driver fatigue. There are two peaks of sleep each day

s we discussed in Introduction part.

Therefore, sleep quality, work condition, and circadian rhythm

re considered as factors mainly influencing the driver’s alertness.

In our fatigue detection model, the prior probabilities these

hree contextual information are tabulated in Table 2 . Ji et al. sum-

arized the conditional probabilities of some contexts, prior prob-

bilities of each context can be obtained using the law of total

robability. Due to many contributors can affect the sleep quality,

here are a lot of probabilities information need to compute it us-

ng the law of total probability. The computation details of proba-

ilities of sleep quality, driving condition and circadian rhythm will

e shown in the latter Appendix part.

.3. Decision making

Before giving more detailed analyses, firstly let us consider how

he probabilities work in making decision. Our goal is to decide

lass membership for each new input physiological signal feature,

t can be represent as x in the following of this section. We are

nterested in the probabilities of the two classes given this fea-

ure, which are represented by p ( C k |x ). Using Bayes’ theorem, these

robabilities can be expressed in the form of ( 9 ).

p( C k | x ) = p(x | C k ) p( C k ) p(x )

= p(x | C k ) p( C k ) 2 ∑

k =1 p(x | C k ) p( C k )

(9)

here p ( C k ) can be interpreted as the prior probability for each

lass C k , and p ( C k |x ) as the corresponding posterior probability. The

rior probability p ( C k ) represents the probability that driver’s state

ithout considering the physiological feature, just according to the

ontext information. Similarly, p ( C k |x ) is the corresponding poste-

ior probability using Bayes’ theorem considering the information

ontained in physiological feature. If our aim is to minimize the

hance of assigning x to the wrong class, then intuitively we would

hoose the class with higher posterior probability ( Scharcanski, De

liveira, Cavalcanti, & Yari, 2011 ).

In order to solving this decision problem, there are three steps

an be used in practical application,

1) Likelihood estimation

For each class, the class conditional densities p ( x | C k ), also

can be called as likelihood, should be determined at first.

From the physiological feature, fatigue likelihood of each

class can be achieved individually using kernel distribution

estimate at different time instants.

2) Prior inferring

At each time slides, the prior probabilities p ( C k ) of each class

should be estimated separately.

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 403

Fig. 5. Tree diagram for the conditional probabilities of fatigue state given the

three contexts including sleep quality, driving condition, and circadian rhythm.

And the values of these conditional probabilities are extracted from Ji et

al. 2006 . In our case, we only have fatigue and alert state, therefore the

p(fatigue|SQ,DC,CR) + p(alert|SQ,DC,CR) = 1 . The probabilities of alert given SQ, DC, and CR can be achieved according to this, thus they were ignored in Fig. 5.

Fig. 6. Tree diagram for the current conditional probabilities of fatigue state given

that value at the previous time t-1 and the three contexts. The numbers represent

the conditional probability for the current fatigue when the different events of SQ,

DC, CR, and the fatigue before the current fatigue take place simultaneously, and

their values are extracted from ( Yang et al., 2010 ). In here, we also ignore the cur-

rent conditional probabilities due to the same reason of Fig. 5.

3) Posterior calculation

The posterior class probabilities p ( C k |x ) can be obtained using

ayes’ theorem as in ( 9 ).

Having found these class posterior probabilities, we use the

aximum posterior ( Shiang & Van, 2009 ) decision theory to de-

ermine which of the two classes to assign to the new feature x .

nd these fatigue posterior probabilities provide a quantification

f uncertainty in this way.

.4. Validation and results

As we discussed above, our HMM-based fatigue model pro-

ides mathematically coherent for aggregating uncertain physio-

ogical information with the relevant contextual information. Here

e consider three cases under different conditions.

Case 1: Assessing fatigue based only on physiological informa-

ion.

The training data is same to the data which were used to get

he optimized feature, as shown in Section 3.2 , which are the

rst and last sections of the collected thirteen sections during the

hole driving experiment. We assume that the optimized feature

f the first section represents alert performance. And driver was in

atigue state during the last section. So they provide two hypothe-

es C 1 -alert and C 2 - fatigue. Using KDE, we can get the probabili-

ies given these two hypotheses individually. They are represented

y p ( x|C 1 ) and p ( x|C 2 ), dividing by p ( x|C 1 ) + p ( x|C 2 ) according to ( 9 ), hus the posterior probabilities without considering contextual in-

ormation can be obtained.

Case 2: Assessing driver’s fatigue based on physiological infor-

ation and global context prior probabilities.

In this case, the prior probabilities of context were used com-

ining with physiological information. Considering all different

onditions, the values of prior probability are shown in Table 2 ,

hich are of the general meaning and will be constant value in dif-

erent time slices, these prior probabilities are mentioned as global

ontext prior probabilities.

We can dynamically get the class decision slice by slice using

he three steps (likelihood-prior-posterior) as we given in Section

.3 . Before using this model to assess driver’s fatigue, the initial

robability of fatigue can be obtained by the conditional proba-

ilities of fatigue given the three contexts using ( 10 ) with static

ayesian method. The initial probability of inferring the fatigue

tate for the given contextual information are reported in Fig. 5.

p( f | c 1 ) = 2 ∑ i

2 ∑ j

2 ∑ l

p( f | S Q i , D C j , C R l ) p(S Q i ) p(D C j ) p(C R l ) (10)

here f represents fatigue, similarly a is used to represent the alert

tate in the following part. In ( 10 ), c means context, and i, j, l = 1,2 epresent different values of sleep quality, driving condition, and

ircadian rhythm, thus the probability of fatigue given context at

ime = t 1 can be computed by ( 10 ), also the probability of alert can e achieve here. In static Bayes’ view, this condition probability can

e interpreted as the prior probability under some context at that

ime without considering physiological feature. Next based on the

iscussion of Case 1, we already know how to compute the likeli-

ood. Therefore according to ( 9 ), the probability of fatigue given

ptimized feature at time = t 1 , represented by o 1 , is obtained as ( f|o 1 ) = 0.3642 and p ( a|o 1 ) = 0.6358.

Since we already have the initial values of each state by static

ayesian method, next we would like to compute the conditional

atigue probabilities given different f eatures and contexts over

ime. In this paper, a first-order HMM model was used to model

his dynamic relationship between these interconnected neighbor-

ng time sections. This means that we suppose that the current

atigue probability value is only influenced by that value in the

revious time. According to this assumption, the fatigue probabil-

ty at time = t in the different context conditions and correspond- ng fatigue probability at time = t −1 can be shown in Fig. 6 . There- ore, the transitional probabilities between states in two consecu-

ive time slices are specified accordingly.

Corresponding to the dynamic relationship as shown in

ig. 6 and the prior probabilities of context as in Table 2 , these

ynamic prior conditional fatigue probabilities given contexts over

ime can be calculated as ( 11 ).

p( f t | c t ) = 2 ∑ i

2 ∑ j

2 ∑ l

p( f t | S Q i , D C j , C R l , f t−1 ) p(S Q i )

× p(D C j ) p(C R l ) p( f | c t−1 ) (11) Similar to the computation of initial probabilities, the dynamic

atigue posteriors can be obtained by following the Bayes’ theorem

n ( 9 ).

404 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

Fig. 7. The experimental protocol. During the 3.5 h driving, 13 data sections were

recorded, each one lasts 3 min. As the discussion of changing in contextual factors

of circadian rhythm and driving condition, we can see that in the first three and

the last one section, drivers worked in active circadian rhythm and normal driving

condition, the fourth section provided the active circadian rhythm and monotonous

driving condition. From the fifth to twelfth sections, they all gave the poor driving

condition and drowsy circadian rhythm.

Fig. 8. The posterior probabilities of fatigue in different time sections. Left sub-

figure is the results of Case 1, which assessed the driver fatigue posterior prob-

abilities based only on physiological information. Middle sub-figure was assessing

driver’s fatigue based on physiological information and global context prior proba-

bilities as described in Case 2. The right sub-figure gives the posterior probabilities

of driver fatigue in the third case.

Case 3: Assessing driver’s fatigue based on physiological infor-

mation and local context prior probabilities.

In this case, we viewed context information in a more detail

way, they also change over time rather than regarding them as the

general constant prior probabilities as they were used in the sec-

ond case. Specifically, more information of our experiments were

considered in this case.

First, let us remind our experiment in the intuitional way. As

shown in Fig. 7 , our experiments happened from 1:00–4:30 p.m.,

this driving covered one of peeks of sleep which from 2:0 0–4:0 0

p.m. approximately. So the contextual information contributed by

circadian rhythm should be adjusted as the time goes on. More-

over, the driving condition also changed as urban-highway-urban

during the whole experiment. Considering these two changing

context factors, we assess the driver fatigue using our model.

Since the driver was required to have good sleep quality, the

experiment time at the beginning is 1:00 p.m., and the driving

started from urban, we can learn that the driver drove in active

circadian and normal driving condition at the beginning. In this

special condition, the initial fatigue probability can be obtained ac-

cording to Fig. 5 . Thus, the initial fatigue probability given by nor-

mal sleep quality, normal driving condition, and active circadian

rhythm should be set as 0.05. As for the dynamic fatigue detec-

tion, based on the condition probabilities in Fig. 6 , the transition

probabilities from time t −1 to time t used in this case are given as follow:

p ( f t |SQ = normal, DC = normal, CR = active, f t-1 = f ) = 0.81 p ( f t |SQ = normal, DC = poor, CR = active, f t-1 = f ) = 0.82 p ( f t |SQ = normal, DC = poor, CR = drowsy, f t-1 = f ) = 0.87 Meanwhile, when previous state change as alert, the corre-

sponding transition probabilities can be obtained as well. Using

( 11 ), the dynamic prior conditional fatigue probabilities can be cal-

culated section by section. By now, the estimated fatigue probabil-

ities can be computed followed ( 9 ).

In the following, the experimental results of these three cases

are shown as Fig. 8 . For each case, we computed the posterior

probabilities of driver fatigue over time. We collected 13 data seg-

ments that means we have 13 different time sections, thus in dif-

ferent time section the driver fatigue can be estimated in a proba-

bility way.

As shown in Fig. 8 , the posterior probabilities of fatigue increase

ignificantly with driving time, which gives the consistent conclu-

ion with the driver self-reported fatigue states as we given in

ection 2.1 . In the fatigue of case 3, the eighth value much higher

han the previous seven ones, and besides this section, the values

n two more consecutive sections are all larger than 0.5. At this

oint, driver was in mild fatigue state. And notably, another jump

appened from the tenth point to the eleventh one, which trans-

ates into the fatigue state has come.

Moreover, from Fig. 8 we can see that the second and third

ases give more realistic estimation than the first case. This exper-

mental result clearly demonstrate that the effect of contextual in-

ormation is also a significant cause of fatigue and risk of accidents.

specially, the posterior probabilities of fatigue in case 3, which

ynamic combined the physiological feature and context, clearly

evealed the decrease in alertness and increase in fatigue. This can

e evidenced by the progressive increase in probability of fatigue,

articularly during the last 2–3 session of the 3.5 h driving, the fa-

igue probabilities are much higher than those in the first period

f 5 sessions.

For the second and third case, they give very similar trend

n here, the obvious differences occur at the change of the

riving condition and circadian rhythm. In our experiment, the

onotonous highway driving and drowsy driving time account for

ore proportion as shown in Fig. 7 . So the context change didn’t

ive significant difference, even though from the difference we can

till learn that case 3 gives better description in dynamic estima-

ion than case 2. However, the real driving involves more complex

ontext, if we break the time limit and make the driving task per-

ormed at different time duration of day and night, dynamic con-

ideration of context respect the common sense whereas making

ational coherent inferences.

. Discussion

As the results shown in Section 4.4 , the probabilities of fatigue

an be recognized over time by using the HMM-based dynamic fa-

igue detection model. In building this model, the various physio-

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 405

Fig. 9. Optimum solution by cross-validation of all 12 subjects. Each sub-figures is for each subject with the number, for the explanations of this figure, please see the Fig. 2.

ogical (EEG, EMG, and respiration signals) and contextual informa-

ion were used in this paper. Due to the portable data collection, it

akes the application of this model in real driving condition more

easible. Thus, this work focused on realizing the dynamic predic-

ion of the probabilities of driver fatigue from real driving condi-

ions, including urban driving and monotonous highway which is

he most accident-prone situation. The long-haul highway driving

s repetitive and often predictable and does not necessarily require

ubstantial sensory awareness. Straight, uneventful and long roads

re well-known factors which can increase the risk of driver fa-

igue ( Ting et al., 2008 ).

We would like to compare fusion feature with individual fea-

ures from the aspect of separability evaluated by AUC of all twelve

ubjects as shown in Table 3.

The results by all the 12 subjects in this research were given

n Table 3 . And each AUC value is obtained by 10-fold cross valida-

ion to verify the stabilization of result and improve the generaliza-

ion of this accuracy evaluation. From the AUC values in this table,

e marked the largest AUC value among each subject with”∗”, and

406 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

Fig. 9. Continued

Table 3

Separability Comparison among all features of all subjects.

Subjects/features #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12

RESP 0 .925 0 .766 0 .597 0 .954 0 .649 0 .819 0 .931 0 .855 0 .856 0 .742 0 .728 0 .884

EMG 0 .739 0 .619 0 .592 0 .703 0 .731 0 .657 0 .562 0 .539 0 .612 0 .611 0 .531 0 .802

EEG 0 .506 0 .613 0 .594 0 .516 0 .724 0 .816 0 .807 0 .508 0 .723 0 .808 0 .593 0 .527

Fusion 0 .954 ∗ 0 .772 ∗ 0 .655 ∗ 0 .960 ∗ 0 .755 ∗ 0 .825 ∗ 0 .941 ∗ 0 .861 ∗ 0 .893 ∗ 0 .858 ∗ 0 .734 ∗ 0 .892 ∗

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 407

Fig. 10. 3D bars figures for posteriors of fatigue of all 12 subjects during driving by Fusion feature. (a) posteriors of fatigue given by Fusion feature with global contextual

information, (b) posteriors of fatigue given by Fusion feature with local contextual information.

e can see that fusion feature gave larger AUC value than other

ndividual features for every subjects. We would like to give the

d AUC distribution plots (x- λ1 ,y- λ2 ,z-AUC) to show how the opti- um solution were gotten by cross-validation for all these 12 sub-

ects, as shown in Fig. 9 . And AUC values of fusion feature in each

ub-figure in Fig. 9 are in last line in Table 3 . These can make the

esults given by fusion feature convincing.

We discussed the dynamic detection of driver fatigue by our

MM based model in three different cases. The first case, that

nly uses the physiological measures as predictor without using

he prior and context knowledge about these variables and the tar-

et states, are lack of the ability to handle uncertainty, complexity,

nd ambiguity involved in data. In building fatigue state assess-

ent models and committing classification tasks, such methods

sed to infer driver fatigue by the information from physiological

eature is not sufficient, therefore this method must be integrated

ith high-level models of the user and the environment ( Li & Ji,

005 ). So it make us to consider other cases, which can fuse the

hysiological and contextual information, with global or local prior

nowledge, to infer driver fatigue dynamically as the graphical net-

ork shown in Fig. 4 . As we verified the need of fusion, we put the

osterior probabilities of fatigue of all 12 subjects with fusion fea-

ure under the local and global contexts in the following 3D bars

gures as shown in Fig. 10.

Posteriors of fatigue by three individual physiological signals as

espiration, EMG and EEG are given in Fig. 11 , including both the

ocal and global contextual information. Thus, we provide six 3D

ars figures to show posteriors of fatigue in Fig. 11.

From the results given by individual feature and contexts, we

an see that the fatigue posteriors estimate by EEG feature and

ontexts gave the smallest individual differences among all sub-

ects, however these posteriors gave low estimation results. And

mong these three individual features, respiration feature gave

he best estimation of fatigue, however, the individual differences

mong subjects were large. By comparing the same feature under

ifferent contexts, we can find that local contexts can reflect more

etails of changing in fatigue posteriors. In the case of the require-

ent of precise recognition is not very high, we can still use the

lobal context to give fatigue estimation.

Karolinska Sleepiness Scale (KSS) was used as the subjective ba-

is for our estimation. KSS = 9 represents that driver is extremely leepy fighting sleep, however this driving state is too danger

o obtain under real driving condition, in our experiment, when

rivers finish the driving task, KSS value equals to 7, this means

rivers fall in to the drowsy state at that time. We used the driver

SS-report to roughly classify the driver state as alert, mild fa-

igue, and fatigue, we obtained three levels of driver state, KSS = 1– gave the first level; KSS = 3–5 fall into the second level; and SS = 5–7 respond to level three. The estimation results shown that rst seven data collection sessions were in level 1, the divers were

n alert state; the eighth to tenth data collection sessions were

n level 2 mild fatigue; the eleventh to thirteenth sessions were

n level 3 fatigue. Our estimation results essentially in agreement

ith KSS, and the early onset of driver fatigue happened around

fth to seventh session, the thresholds of mild fatigue and fatigue

re set to the posterior probabilities of fatigue as 0.5 and 0.8.

Although many previous papers also used this subjective report

s true labels of fatigue, researchers still need the authority data

et with truth fatigue labels. Without this kind of well accepted

ata set, there are a series of problems need us face. For the re-

earchers of driver fatigue or other human states, they use differ-

nt data set to train their algorithms. They may get very great re-

ult by using some certain data set, but there is no way to com-

are two studies. For building the fatigue detection model, many

ifferent data were used. Even same data were used to construct

he model, there are still many other factors can influence the esti-

ation results, such as pre-processing, feature selection, classifier

lgorithm and contextual information. The results can be compa-

able only when same processing methods are used for all these

teps. So this big challenge is formulating a well acceptable stan-

ards for the ground truth value of driver state.

. Conclusion

Driver fatigue results from a very complicated mechanism, and

any factors affect fatigue in interacting way ( Ji et al., 2006 ). Con-

idering driver fatigue is one of the accumulated value evolving

ver time, so our research purpose is to build a fatigue detection

odel to infer the probabilities of driver fatigue in a dynamical

anner. In this paper, a dynamic fatigue detection model based

n Hidden Markov Model (HMM) is proposed, which integrate the

used physiological feature from various physiological sources and

ontext knowledge to estimate the driver fatigue. These experi-

ents involving three different cases demonstrate the feasibility

nd effectiveness of the proposed model in making rational coher-

nt inferences.

408 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

Fig. 11. 3D bars figures for posteriors of fatigue of all 12 subjects during driving by individual features. (a) posteriors of fatigue by respiration-global contexts, (b) posteriors

of fatigue by respiration-local contexts, (c) posteriors of fatigue by EMG-global contexts, (d) posteriors of fatigue by EMG-local contexts, (e) posteriors of fatigue by EEG-global

contexts, (f) posteriors of fatigue by EEG-local contexts.

Viewed from an economic or ergonomic aspect, the current

study made the following three contributions,

1) Under real driving condition, the current study supplies

drivers with a relative reliable evaluation of their ability

to drive by fusing two aspect information, physiological

and contextual knowledge to assess probabilities of fatigue,

which provide a quantification of uncertainty. Such a model

respect more common sense of fatigue.

2) Propose an easy understanding way of physiological fusion.

This can help to get the optimized feature, which maximizes

the separable property between alert and fatigue state.

3) We combined static Bayesian and dynamic Bayesian (HMM)

to estimate the driver’s fatigue at initial time and following

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 409

(

Fig. A.1. Tree diagram for the conditional probabilities of sleep environment.

Fig. A.2. Tree diagram for the conditional probabilities of sleep quality.

(

time periods. And we analyzed the posteriors of fatigue by

local and global contexts involving in HMM.

It should be noted that although this study has produced

romising results for detection of driver fatigue automatically,

here still exists challenge that need to be aware of. For the pur-

ose of realizing the dynamic driver fatigue detection in this pa-

er, only three physiological signals and several simple feature pa-

ameters of them were analyzed. In future, as the acceleration of

he speed of online operation can be obtained, more physiological

ignals and more complex feature parameters can be used in this

odel to detect the driver fatigue. Meanwhile, the practical appli-

ability of fatigue assessment methods will be greatly enhanced

hrough the development of a range of approximate inference al-

orithms. Similarly, models based on other kernels can also pro-

ide significant impact on both algorithms and applications. Fur-

hermore, in future real driving studies, the time limitation should

e broken and the driving task should be performed at different

ime duration of day and night. In a long term perspective, re-

earch should focus on the on-line alarm and assistance counter-

easure including both the in-vehicle and environmental aspects

o reduce the adverse effects of driver fatigue, which are of the

ilder angle and practical meaning.

Finally, the impairment of driver alertness can be translated

nto the increasing probabilities of fatigue over time by using

his fatigue detection model. Based on the experiment result, the

hysiological, which includes EEG, EMG, respiration, and contex-

ual knowledge contributed a lot of information in driver fatigue.

herefore, this fatigue detection model can detect driver sleepy

hile driving, it provides the feasibility and effectiveness in mak-

ng the rational coherent inferences.

cknowledgment

This work were supported by the National Natural Science

oundation of China (Grant No. 51405073 ), Research and develop-

ent plan of Science and Technology of Heibei Province, China

Project No. 152177180 ), Research and development plan of Sci-

nce and Technology of Qinhuangdao City, China (Project No.

01502A037 ).We would like to thank all the participants for their

ooperation, members of our research team for their valuable sug-

estions and support in data collection.

ppendix

The computation of prior probabilities for contextual knowl-

dge: sleep quality, driving condition and circadian rhythm. The

onditional probabilities of these three contextual factors are given

y the following tree diagrams. With these marginal probabilities,

he joint conditional probabilities of driving condition and circa-

ian rhythm can be given as follow:

1) Computation of sleep quality (SQ). For the probability of sleep

quality, we should first compute probability of sleep environ-

ment by tree diagram in Fig. A.1 .

The probability of poor sleep environment can be obtained by

using Eq. (A.1) .

p(SE = poor) = 2 ∑

i =1

2 ∑ j=1

2 ∑ k =1

2 ∑ l=1

p ( SE = poor| humidit y i , hea t j ,

nois e k , ligh t l ) p(humidit y i ) p(hea t j ) p(nois e k )

p(ligh t l ) = 0 . 29 (A.1) With the probability of sleep environment, the probability

of poor sleep quality can be obtained by using Fig A.2 and

Eq. (A.2) .

p(SQ = poor) = 2 ∑

i =1

2 ∑ j=1

2 ∑ k =1

2 ∑ l=1

p ( SQ = poor| nappin g i ,

P S j , S E k , S T l )

p(nappin g i ) p(P S j ) p(S E k ) p(S T l )

= 0 . 3498 (A.2) where PS represents pre-sleep state and ST means sleep time.

2) Computation of driving condition (DC). For the probability of

poor driving condition, by Fig. A.3 we have,

p(DC = poor) = 2 ∑

i =1 p(DC = poor| D E i ) p(D E i )

= 0 . 2 × 0 . 72 + 0 . 8 × 0 . 05 = 0 . 1840 (A.3) where DE is short for driving environment, and i = 1,2 repre- sents driving in highway or urban condition.

410 R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411

Fig. A.3. Tree diagram for the conditional probabilities of driving condition, e.g.,

given highway driving environment, poor driving condition probability is 72%.

Fig. A.4. Tree diagram for the conditional probabilities of circadian rhythm, for ex-

ample, given drowsy peak time, the probability of circadian rhythm in drowsy is

60%.

(

3) Computation of driving condition (CR). For the probability of

drowsy circadian rhythm, by Fig. A.4 we have:

p(CR = drowsy ) = 2 ∑

i =1 p(CR = drowsy | t im e i ) p(t im e i )

= 0 . 26 × 0 . 6 + 0 . 74 × 0 . 05 = 0 . 193 0 (A.4) where i = 1,2 represents driving happens in the drowsy time or ac- tive time.

References

Al-Sultan, S. , Al-Bayatti, A. H. , & Zedan, H. (2013). Context-aware driver behavior detection system in intelligent transportation systems. IEEE Transactions on Ve-

hicular Technology, 62 (9), 4264–4275 . Alloway, C. E. , Ogilvie, R. D. , & Shapiro, C. M. (2006). The alpha attenuation test:

assessing excessive daytime sleepiness in narcolepsy-cataplexy. Sleep, 10 (2),

129–131 . Bertozzi, M. , Broggi, A. , Fascioli, A. , Graf, T. , & Meinecke, M. M. (2004). Pedestrian

detection for driver assistance using multiresolution infrared vision. IEEE Trans- actions on Vehicular Technology, 53 (6), 1666–1678 .

Bishop, C. M. (2007). Probability distributions, in pattern recognition and machine learning (pp. 122–123). Cambridge .

Cajochen, C. , Brunner, D. P. , Krauchi, K , Graw, P , & Wirz-Justice, A. (1995). Power

density in theta/alpha frequencies of the waking eeg progressively increases during sustained wakefulness. Sleep, 18 (10), 890–894 .

Cantero, J. L. , & Atienza, M. (20 0 0). Spectral and topographic microstructure of brain alpha activity during drowsiness at sleep onset and REM sleep. Journal of Psy-

chophysiology, 14 (3), 151–158 . Chen, H. , & Meer, P. (2005). Robust fusion of uncertain information. IEEE Transac-

tions on Systems, Man, and Cybernetics, Part B: Cybernetics, 35 (3), 578–586 .

Conati, C. (2012). Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence, 16 (7), 555–575 .

Di Stasi, LL , Renner, R , Catena, A. , Cañas, J. J. , Velichkovsky, B. M. , & Pan- nasch, S. (2012). Towards a driver fatigue test based on the saccadic main se-

quence: A partial validation by subjective report data. Transportation Research Part C: Emerging Technologies, 21 (1), 122–133 .

Declerck, C. H. , Boone, C. , & Brabander, B. D. (2006). On feeling in control: a biolog-

ical theory for individual differences in control perception. Brain and Cognition, 62 (2), 143–176 .

DuchÊne, J. , & Lamotte, T. (2001). Surface electromyography analysis in long-term recordings: application to head rest comfort in cars. Ergonomics, 44 (3), 313–327 .

Eugene, A , Carolyn, C , Kayla, J , & John, R (2015). Real-time driver drowsiness feed- back improves driver alertness and self-reported driving performance. Accident

Analysis and Prevention, 81 , 8–13 . Ferguson, S. A. , Paech, G. M. , Sargent, C. , Darwent, D. , Kennaway, D. J. , &

Roach, G. D. (2012). The influence of circadian time and sleep dose on subjective

fatigue ratings. Accident Analysis and Prevention, 45 (Suppl(1)), 50–54 . Hostens, I. , & Ramon, H. (2005). Assessment of muscle fatigue in low level

monotonous task performance during car driving. Journal of Electromyography and Kinesiology Official Journal of the International Society of Electrophysiological

Kinesiology, 15 (3), 266–274 .

ou, Y. , Edara, P. , & Sun, C. (2015). Situation assessment and decision making for lane change assistance using ensemble learning methods. Expert Systems with

Applications, 42 (8), 3875–3882 . ap, B. T. , Lal, S. , Fischer, P. , & Bekiaris, E. (2009). Using EEG spectral components to

assess algorithms for detecting fatigue. Expert Systems with Applications, 36 (2), 2352–2359 .

i, Q. , Lan, P. , & Looney, C. (2006). A probabilistic framework for modeling and re- al-time monitoring human fatigue. IEEE Transactions on Systems Man and Cyber-

netics - Part A Systems and Humans, 36 (5), 862–875 .

i, Q. , Zhu, Z. , & Lan, P. (2004). Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Transactions on Vehicular Technology, 53 (4), 1052–1068 .

o, J. , Lee, S. J. , Kang, R. P. , Kim, I. J. , & Kim, J. (2014). Detecting driver drowsiness us- ing feature-level fusion and user-specific classification. Expert Systems with Ap-

plications, 41 (4), 1139–1152 . ayvan, N , & Robert, S (2006). Biomedical signal and image processing . London New

York: Taylor&Francis Group Boca Raton .

hushaba, R. N. , Sarath, K. , Sara, L. , & Gamini, D. (2011). Driver drowsiness classifica- tion using fuzzy wavelet-packet-based feature-extraction algorithm. IEEE trans-

actions on bio-medical engineering, 58 (1), 121–131 . i, D. H. , Liu, Q. , Yuan, W , & Liu, H. X. (2010). Relationship between fatigue driv-

ing and traffic accident. Journal of traffic and transportation engineering, 10 (2), 104–109 .

in, C. T. , Wu, R. C. , Liang, S. F. , Chao, W. H. , Chen, Y. J. , & Jung, T. P. (2006).

Eeg-based drowsiness estimation for safety driving using independent compo- nent analysis. IEEE Transactions on Circuits and Systems I Regular Papers, 52 (12),

2726–2738 . in, C. T. , Chang, C. J. , Lin, B. S. , Hung, S. H. , Chao, C. F. , & Wang, I. J. (2010). A re-

al-time wireless brain-computer interface system for drowsiness detection. IEEE Transactions on Biomedical Circuits & Systems, 4 (4), 214–222 .

Liu, X. , & Zhao, Y. (2012). Semi-empirical likelihood inference for the ROC

curve with missing data. Journal of Statistical Planning and Inference, 142 (12), 3123–3133 .

i, X. , & Ji, Q. (2005). Active affective state detection and user assistance with dy- namic Bayesian networks. IEEE Transactions on Systems Man & Cybernetics Part

A Systems & Humans, 35 (1), 93–105 . orris, D. M. , Pilcher, J. J. , & Switzer, F. S. (2015). Lane heading difference: an inno-

vative model for drowsy driving detection using retrospective analysis around

curves. Accident; Analysis and Prevention, 80 , 117–124 . inin, L. , Benedetto, S. , Pedrotti, M. , Re, A. , & Tesauri, F. (2012). Measuring the ef-

fects of visual demand on lateral deviation: a comparison among driver’s per- formance indicators. Applied Ergonomics, 43 (3), 4 86–4 92 .

Pylkkonen, M. , Sihvola, M. , Hyvarinen, H. K. , Puttonen, S. , Hublin, C. , & Salli- nen, M. (2015). Sleepiness, sleep, and use of sleepiness countermeasures in

shift-working long-haul truck drivers. Accident Analysis and Prevention, 80 ,

201–210 . enner, G. , & Mehring, S. (1997). Lane departure and drowsiness-two major acci-

dent causes-one safety system. In Mobility for everyone. 4th world congress on intelligent transport systems (pp. 21–24) .

ahayadhas, A. , Sundaraj, K. , Murugappan, M. , & Palaniappan, R. (2015). Physiolog- ical signal based detection of driver hypovigilance using higher order spectra.

Expert Systems with Applications, 42 (22), 8669–8677 . Schmidt, E. A. , Schrauf, M. , Simon, M. , Buchner, A. , & Kincses, W. E. (2011). The

short-term effect of verbally assessing drivers’ state on vigilance indices during

monotonous daytime driving. Transportation Research Part F: Traffic Psychology and Behaviour, 14 (3), 251–260 .

Scharcanski, J. , De Oliveira, A. B. , Cavalcanti, P. G. , & Yari, Y. (2011). A particle-filter- ing approach for vehicular tracking adaptive to occlusions. IEEE Transactions on

Vehicular Technology, 60 (2), 381–389 . hen, K. Q. , Li, X. P. , Ong, C. J. , Shao, S. Y. , & Wilder, E. (2008). EEG-based mental fa-

tigue measurement using multi-class support vector machines with confidence

estimate. Clinical Neurophysiology, 119 (7), 1524–1533 . hiang, H. P. , & Van, d. S. M. (2009). Distributed resource management in multihop

cognitive radio networks for delay-sensitive transmission. IEEE Transactions on Vehicular Technology, 58 (2), 941–953 .

on, J. , Yoo, H. , Kim, S. , & Sohn, K. (2015). Real-time illumination invariant lane detection for lane departure warning system. Expert Systems with Applications,

42 (4), 1816–1824 .

tampi, C. , Stone, P. , & Michimori, A. (1995). A new quantitative method for assess- ing sleepiness: the alpha attenuation test. Work and Stress, 9 (2), 368–376 .

un, Y. , & Xiong, B. Y. (2014). An innovative nonintrusive driver assistance system for vital signal monitoring. IEEE Journal of Biomedical & Health Informatics, 18 (6),

1932–1939 . ang, L. , Du, P. , & Wu, C. (2010). Compare diagnostic tests using transformation-in-

variant smoothed roc curves. Journal of Statistical Planning & Inference, 140 (11),

3540–3551 . ing, P. H. , Hwang, J. R. , Doong, J. L. , & Jeng, M. C. (2008). Driver fatigue and highway

driving: a simulator study. Physiology and Behavior, 94 (3), 448–453 . eno, H. , Kaneda, M. , & Tsukino, M. (1994). Development of drowsiness detection

system. In Vehicle navigation and information systems conference, proceedings. IEEE (pp. 15–20) .

illiamson, A. M. , & Feyer, A. M. (20 0 0). Moderate sleep deprivation produces im-

pairments in cognitive and motor performance equivalent to legally prescribed levels of alcohol intoxication. Occupational & Environmental Medicine, 57 (10),

649–655 .

R. Fu et al. / Expert Systems With Applications 63 (2016) 397–411 411

illiamson, A. M. , & Friswell, R. (2011). Investigating the relative effects of sleep deprivation and time of day on fatigue and performance. Accident Analysis and

Prevention, 43 , 690–697 . ang, J. H. , Mao, Z. H. , Tijerina, L. , & Pilutti, T. (2009). Detection of driver fatigue

caused by sleep deprivation. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 39 (4), 694–705 .

ang, J. H. (2007). Analysis and detection of driver fatigue caused by sleep deprivation . Massachusetts Institute of Technology .

ang, G. , Lin, Y. , & Bhattacharya, P. (2010). A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Information Sci-

ences, 180 (10), 1942–1954 . ou, F. , Zhang, R. , Guo, L. , Wang, H. , Wen, H. , & Xu, J. (2015). Trajectory planning

and tracking control for autonomous lane change maneuver based on the co- operative vehicle infrastructure system. Expert Systems with Applications, 42 (14),

5932–5946 .