Research Paper

Roufiaa91

DistinguishingmentalattentionstatesofhumansviaanEEG-basedpassiveBCIusingmachinelearningmethods.pdf

Home >English homework help >Research Paper

Expert Systems With Applications 134 (2019) 153–166

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Distinguishing mental attention states of humans via an EEG-based

passive BCI using machine learning methods

Çi ̆gdem İnan Acı a , ∗, Murat Kaya a , Yuriy Mishchenko b

a Mersin University, Department of Computer Engineering, Mersin, 33343, Turkey b Izmir University of Economics, Department of Biomedical Engineering, İzmir, 35330, Turkey

a r t i c l e i n f o

Article history:

Received 30 January 2019

Revised 13 May 2019

Accepted 30 May 2019

Available online 30 May 2019

Keywords:

EEG

BCI

Mental state detection

Drowsiness detection

Support vector machine

Passive control task

a b s t r a c t

Recent advances in technology bring about novel operating environments where the role of human par-

ticipants is reduced to passive observation. While opening new frontiers in productivity and lifestyle,

such environments also create hazards related to the inability of human individuals to maintain focus

and concentration during passive control tasks. A passive brain-computer interface for monitoring mental

attention states of human individuals (focused, unfocused, and drowsy) by using electroencephalographic

(EEG) brain activity imaging and machine learning data analysis methods is developed in this work. An

EEG data processing pipeline and a machine learning mental state detection algorithm using the Sup-

port Vector Machine (SVM) method were designed and compared with k-Nearest Neighbor and Adaptive

Neuro-Fuzzy System methods. To collect 25 h of EEG data from 5 participants, a classic EEG headset

was modified. We found that the changes in EEG activity in frontal and parietal lobes occurring at 1–

5 Hz and 10–15 Hz frequency bands were associated with the changes in individuals’ attention state. We

demonstrated the ability to use such changes to identify individuals’ attention state with 96.70% (best)

and 91.72% (avg.) accuracy in experimental settings using a version of continuous performance task with

SVM-based mental state detector. The findings help guide the design of future systems for monitoring

the state of human individuals by means of EEG brain activity data.

. Introduction

Recent advances in automation and robotics bring about new

perating environments where the role of human participants is

ncreasingly reduced to passive observation. While opening radi-

ally new venues for improvements in productivity and lifestyle,

uch operating environments also create new hazards related to

he inability of human operators to maintain concentration on pas-

ive control tasks. One of the best approaches for monitoring the

tate of human individuals in such circumstances can be expected

o be Brain-computer interfaces (BCIs). In this section, the use of

CIs in detection of the driver’s mental attention is reviewed, and

he motivation of this study is emphasized.

∗ Corresponding author. E-mail addresses: [email protected] (Ç. ̇I. Acı), [email protected] (M.

aya), [email protected] (Y. Mishchenko).

ttps://doi.org/10.1016/j.eswa.2019.05.057

.1. Literature review

BCIs establish a direct communication channel between a brain

nd a computer or external device ( Shangkai, Yijun, Xiaorong, &

o, 2014 ). BCIs offer a novel communication paradigm between

uman beings and computers that bypass conventional interaction

hannels such as keyboard input or speech. Non-invasive BCI,

amely those that use electroencephalography (EEG), magne-

oencephalography, or magnetic resonance imaging to observe

rain activity, are of special interest for addressing the problem

f observing the mental state of human individuals by directly

onitoring their brain activity, thus avoiding the pitfalls of less

irect methods ( Myrden & Chau, 2017 ). The use of EEG in this

egard is of special interest given the established nature of EEG

echnology, the relative ease of use of modern EEG headsets, as

ell as the small size, cost, portability, and reliability of existing

odern EEG solutions ( Alirezaei & Sardouie, 2017; Aricò et al.,

016; Resalat & Saba, 2015 ).

Several studies in the past made use of EEG for fatigue de-

ection in the context of car driving. For instance, Hsieh, Liang,

o, Lin, and Lin (2006) developed a system that estimates a

154 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

vehicle’s positioning the lane by using the driver’s EEG signals in

the context of fatigue detection of car drivers. Yeo, Li, Shen, and

Wilder-Smith (2009) reported a connection between alerted and

drowsy states of car drivers and the changes in the EEG beta and

alpha rhythms. They subsequently proposed a system to automat-

ically detect the onset of fatigue of car drivers as well as to test

such a system in simulated and real driving conditions. Mardi, Ash-

tiani, and Mikaili (2011) studied the association between drowsi-

ness and certain chaotic features of EEG signals for a similar pur-

pose of fatigue detection. Simon et al. (2011) studied the possibility

of sleepiness detection of car drivers again by means of EEG alpha

spindles. They demonstrated that changes in the parameters of al-

pha spindles in the EEG signals can be associated with the increase

of the drivers’ fatigue. Hashemi, Saba, and Resalat (2014) developed

a Steady State Visually Evoked Potential EEG BCI for detecting the

onset of drowsiness of car drivers.

A few studies reported applications of EEG BCI for the discov-

ery of subjects’ mental states in settings different from car driv-

ing: Borghini, Astolfi, Vecchiato, and Mattia (2014) examined the

correlation between mental workload and EEG signals under dif-

ferent conditions such as those of airplane pilots or car drivers.

Arico et al. (2017) developed a passive EEG BCI for the assessment

of mental workload of air traffic controllers. A study by Myrden

and Chau (2017) described a passive EEG BCI for detecting changes

in fatigue, frustration, and attention states during mental workload

tasks including mental arithmetic, anagram solution, and short-

term memory recall.

The data generated by the studies of determining the men-

tal states of humans using EEG recordings led researchers to esti-

mate the mental state by using these datasets and machine learn-

ing methods: Li et al. (2011) classified three attention levels by a

k-Nearest Neighbor (kNN) classifier based on the Self-Assessment

Manikin mode. Subjects were given several mental tasks to under-

take and asked to report on their attention level during the tasks

using a set of attention classifications. The average accuracy rate

is shown to reach 57.03% after seven sessions of EEG training. In

the study of Liu, Chiang, and Chu (2013) , whether students were

attentive or inattentive during instruction was determined by ob-

serving their EEG signals. A Support Vector Machine (SVM) clas-

sifier was used to calculate and analyze these features to identify

the combination of features that best indicated whether students

were attentive. Their method provided a classification accuracy of

up to 76.82%. Lee et al. (2014) developed a driver monitoring sys-

tem that classifies driving mental fatigue condition by analyzing

EEG and respiration signals of a driver in the time and frequency

domains. Test results revealed that the combined use of the EEG

and respiration signals resulted in 98.6% recognition accuracy. Al-

though the result is promising, Lee et al.’s system needs the respi-

ration data of the driver as well as EEG signals. Ke et al. (2014) car-

ried out two experiments where all subjects were instructed to

perform tasks with three different attention levels (i.e. attention,

no attention and rest). Then, statistical analyses and classification

with SVM were performed. They obtained results with the accu-

racies of 76.19% and 85.24% in recognition of three levels of at-

tention for the two experiments, respectively. Wang, Jung, and Lin

(2015) developed a countermeasure to track drivers’ focus of at-

tention and engagement of operators in dual (multi)-tasking condi-

tions using SVM. The system achieved 84.6 ± 5.8% and 86.2 ± 5.4% classification accuracies in detecting the participants’ focus of at-

tentions on math and driving tasks, respectively. In another study

performed by Djamal, Pangestu, and Dewi (2016) , attention and

inattention states were recognized using wavelet filter and SVM.

They experimented with four subjects and achieved an accuracy of

77–83%. In the study of Nuamah and Seong (2018) , task engage-

ment indices of the five cognitive tasks were used as inputs to

SVM. The average classification accuracy across the six participants

as 93.33 ± 8.16%. They achieved very good results due to the dif- erences in cognitive task demand between six different tasks.

.2. Motivation of the study

In this paper, we studied the problem of detecting mental state

hanges in human individuals who need to remain dormant or

assive while also having to maintain a continuous significant level

f concentration or attention. An example of such a scenario can be

upervising automated processes or systems. Another example can

e controlling robotic vehicles or drones or security monitoring.

et another example can be long-term monitoring of aircraft pilots

hile under the control of the autopilot. In all these cases, non-

nterfering supervision of processes is desired, while also alertness

nd a quick reaction is required of the involved individuals.

With this study, we aimed to demonstrate that it is possible

o identify pure mental states such as engaged and focused atten-

ion versus detached and unfocused monitoring from EEG data and

ried to develop a machine learning-based system for solving such

task. Previous studies have used extra data (e.g., task engage-

ent index or respiration data) to support EEG signals or limited

he number of mental states detected to two to achieve high ac-

uracy. We tried to detect the differentiation of engaged / focused,

etached / unfocused, and drowsing mental states experimentally

n a version of continuous performance tasks with 96.70% (best)

nd 91.72% (avg.) accuracies by using only the EEG data. The ap-

roach to the subjects’ state detection used in this study is gener-

lly accepted and therefore can easily be generalized in the future

elated to the development of subject state monitoring systems in

ifferent settings such as patients’ state monitoring in hospitals, as

ell as helping improve safety mechanisms of modern automated

nd robotic systems.

We emphasize that, although the paper makes use of well-

stablished signal processing and data analysis techniques, the

opic of application of such techniques for the identification of

ental states of the brain’s activity, including “pure” mental states

hat have not been externally manifested in a clear way, is novel.

he identification of the passively disengaged state is of high rel-

vance to control/passive supervision processes and is, as far as is

nown, the first study in the literature. Similarly, the identification

f attention states of passive subjects is also the first study of that

ind in the literature – previous works focused on monitoring the

tates of car drivers and similar operators who are continuously ac-

ive by the nature of that setup. Such settings are considerably dif-

erent from identifying drowsiness or detachment of subjects who

re permanently passive. The general approach used to solve this

roblem here, namely using samples of EEG data obtained for self-

dentified mental states of subjects and machine learning, is highly

eneral and can be applied to the investigation of other mental

tates, including the pure mental states, which is also a novel point

f view in the literature.

. Material and method

.1. Experimental procedure

In this study, an original dataset that comprised a total of 25-

our EEG recordings collected from 5 participants engaged in a

ow-intensity control task was used. The task consisted of control-

ing a computer-simulated train using the “Microsoft Train Simu-

ator” program. Each experiment consisted of the participants con-

rolling a train for 35 to 55 min over a primarily featureless route

n the above-mentioned computer simulation program.

The 3 mental states examined in this study included the

ocused but passive attention, the unfocused or detached but

wake state and the drowsing state. The first, “focused” state, was

Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166 155

Fig. 1. The organization of the continuous performance task based on a virtual passive control task used in this work.

nderstood to correspond to passively supervising the train while

aintaining focus and concentration. Active engagement was not

mplied in this state. In fact, most of the task did not involve

ctive intervention into the travel on the part of the participants.

onetheless, continuous concentration and focus were required.

The second state was that of disengaged supervision, detached

ut awake, where the participants did not explicitly drowse, yet

topped paying attention to the developments on the screen and

eing alert. One can interpret this as a dangerous state that should

e detected and produce an alert, should it be realized. On the

ther hand, such a state may not be manifested clearly via any ex-

ernal cues and may be difficult to detect, such as by using video

onitoring. We can call such a state “pure”, in that it is performed

entally but without necessarily any clear external signatures. The

ethodology for the discrimination of such states is among the

ain objectives of this study.

The third state was that of explicit drowsing. As described

n the introduction, drowsing was previously associated with in-

reased alpha-band EEG activity, and its detection by the EEG data

as already discussed in the literature. At the same time, drowsi-

ess can also be detected via non-EEG means, such as by monitor-

ng eyelids in a video or by heart rate monitoring.

In the context of detecting the above-mentioned 3 mental

tates, the participants simulated those states during each experi-

ent, following instructions of the experiment’s supervisor, which

hey received at the beginning of each experiment. Participants

ontrolled a simulated passenger train over a primarily featureless

oute for a duration of 35 to 55 min. Specifically, during the first

0 min of each experiment, the participants were engaged in fo-

used control of the simulated train, paying close attention to the

imulator’s controls, and following the developments on the screen

n detail. During the second 10 min of the experiments, the partici-

ants stopped following the simulator and became de-focused. The

articipants did not provide any control inputs during that time

nd stopped paying attention to the developments on the com-

uter screen; however, they were not allowed to close their eyes

r drowse. Finally, during the third 10 min of the experiments, the

articipants were allowed to relax freely, close their eyes and doze

ff, as desired ( Fig. 1 ).

Specific settings of the train control simulation experiments

ncluded using the “Acela Express” modern locomotive and a 40-

inute segment of the “Amtrak-Philadelphia” route in the above-

entioned train simulator program. The segment of the route

hosen for simulations was flat and featureless so that the partic-

pants had to provide little control inputs during the simulations

ith the exception of the initial and the final 5-minute segments

f the route, where a higher level of involvement was necessary.

he participants were instructed to maintain the speed of the sim-

lated train at a steady 40 mph in all experiments. The controls

onsisted of the throttle adjustment (controlling the travel speed)

nd the brake application (to produce a rapid deceleration). The

ontrols were enacted via keys on a standard computer keyboard.

Each participant took part in 7 experiments, performing at most

ne experiment per day. The first 2 experiments were used for ha-

ituation, and the last 5 trials were used for collecting the data. All

xperiments were conducted between the evening hours of 7 pm

nd 9 pm to facilitate the participants entering the drowsy state

uring the 3rd phase of the trials. The participants were moni-

ored by the experiment’s supervisor and recorded on a video to

nsure that the experiments complied with the above-stated struc-

ure and that no significant disruptions, such as moving or talking,

ook place.

As well as the raw data of the experiments is available to the

esearchers on the Kaggle website ( Acı, Kaya, & Mishchenko, 2019 ),

n actual data snippet is given in Table 1 .

.2. EEG headset modification

The EEG data was acquired using a modified Epoc EEG head-

et and its classic wet electrodes. The Epoc headset is a portable

EG acquisition device providing 12 channels real-time EEG data

t a sampling rate of 128 Hz, a voltage resolution of 0.51 μV, and

bandwidth between 0.2–43 Hz, connecting to a data-acquisition

omputer via a wireless Bluetooth link ( EPOC + , 2019 ). We modi- ed the headset to allow the electrodes to be placed over frontal

nd parietal lobes of the scalp ( Fig. 2 ), whereas the original Epoc

eadset only allowed electrode coverage over frontal and occipi-

al areas within a rigid plastic spider-web cap format. Thus, the

ositions of the electrodes used in this work were F3-Fz-F4-C3-Cz-

4-T3-T4-T5-T6-Pz in the standard 10–20 system. In the EEG de-

ice, 4 leads (here identified by the locations T3, T4, T5, and T6)

ere used to supply current and establish the EEG reference and

ould not be used for data collection. The acquired data from the 7

eads were identified by F3, F4, Fz, C3, C4, Cz, and Pz. The raw EEG

ata was acquired from the Epoc device using a custom Matlab

cript developed based on the sample program eeglogger.m, which

as shipped with the data-acquisition API in Research License of

he Epoc’s software. Wet electrodes’ impedance was checked at the

tart and the end of the experiments. If the electrodes’ impedance

as not at the desired level at the end of the experiment, the ex-

eriment was repeated.

156 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

Table 1

Sample data.

Cnt. Intp. Channel F3 Channel FZ Channel T3 Channel F4 Channel C3 Channel CZ Channel C4 Channel T4 Channel PZ X Y

80 0 4020.51 4915.90 4036.92 4328.72 4285.13 4035.90 4250.77 4304.62 4102.56 1570 1719

81 0 4021.03 4913.85 4039.49 4329.23 4283.59 4035.90 4250.26 4304.62 4107.18 1569 1720

82 0 4016.41 4909.23 4038.97 4327.69 4277.95 4035.38 4250.26 4304.62 4106.67 1568 1721

83 0 4009.23 4907.69 4036.41 4325.64 4270.77 4028.21 4246.15 4304.62 4105.64 1569 1719

84 0 4003.08 4905.64 4036.41 4326.15 4264.10 4017.44 4245.13 4304.62 4107.18 1568 1722

85 0 3994.36 4902.56 4036.41 4323.59 4255.38 4010.26 4243.59 4304.62 4106.67 1569 1721

86 0 3988.21 4904.10 4034.87 4322.05 4248.72 4006.15 4236.41 4304.62 4106.15 1567 1721

87 0 3988.72 4903.08 4035.38 4324.62 4252.31 4003.08 4234.36 4304.62 4107.69 1568 1722

88 0 3991.28 4902.05 4034.36 4327.18 4258.46 4002.56 4238.46 4304.62 4106.15 1567 1721

89 0 3991.79 4903.08 4034.36 4326.67 4261.03 40 0 0.0 0 4236.92 4304.62 4101.03 1566 1721

90 0 3991.28 4898.97 4034.36 4325.64 4260.00 3994.87 4232.31 4304.62 4097.44 1566 1720

91 0 3990.77 4893.33 4034.87 4325.64 4255.90 3991.79 4234.36 4304.62 4093.85 1565 1719

92 0 3988.72 4893.85 4035.38 4324.10 4254.87 3991.79 4233.85 4304.62 4090.26 1564 1717

93 0 3988.21 4895.38 4034.36 4322.05 4257.44 3992.31 4230.77 4304.62 4090.77 1563 1717

94 0 3994.36 4894.36 4034.36 4325.64 4261.03 3994.36 4234.87 4304.62 4092.82 1562 1715

Cnt = Sample counter. Intp. = Indicate if data is interpolated. X = Gyroscope X -axis. Y = Gyroscope Y -axis.

Fig. 2. The modified headset.

Fig. 3. The block diagram of the mental attention state detector.

ω E

(

2.3. Data preprocessing and feature extraction

We developed an algorithm for detecting the subjects’ men-

tal attention states based on the EEG signals represented in the

time-frequency domain. The general block diagram of detecting

the subjects’ mental attention states based on the EEG signals is

shown in Fig. 3 . During the feature extraction stage, we calculated

the spectrograms of the EEG signals in each EEG channel, using

the short-time Fourier transform (STFT) and the Blackman win-

dow ( Almeida, 1994; Lobos & Rezmer, 1997 ). Briefly, STFT encodes

a time-dependent (temporally localized) distribution of the power

in the EEG signal frequency spectrum over a small-time interval

defined in (1) ,

X ST F T ( t, ω ) = ∞ ∑

t ′ = −∞ x ( t ′ ) w

( t ′ − t

) e − jωt

′ (1)

Here, x ( t ) is the EEG signals in a single EEG channel in the time

domain, w ( t ) is the so-called “windowing” function that differs

from zero only in a small neighborhood of t ′ = t and enforces the localization of EEG signals to a small t ′ -interval around t ′ = t . The spectrogram is defined as the square of the STFT amplitudes. S ( t ,

) = | X STFT ( t , ω)| 2 and quantifies the frequency composition of the EG signals near a given time point.

The raw EEG data was acquired from the Epoc Emotiv head-

et in 7 channels at a sampling frequency of Fs = 128 Hz. The TFT calculation was performed separately for each channel. STFT

as computed using �T = 15 second fragments of EEG signals and = 1024 fast discrete Fourier transform (DFT). The Blackman win-

owing function was used to make the EEG signal taper at both

nds of each fragment. The Blackman windowing function is de-

ned by (2) .

( ˆ t )

= {

0 . 42 − 0 . 5 cos 2 π ˆ t M−1 + 0 . 08 cos 4 π

ˆ t M−1 , 0 ≤ ˆ t < M

0 , otherwise (2)

here M is the total number of time points within the window

M = Fs · �T = 1920) and ˆ t = 0,1,…, M −1 is a discrete time-index ithin the window. STFT was then calculated at a time step of 1 s

roducing a set of time-varying DFT amplitudes X STFT ( t , ω) at 1 s ntervals within each input EEG channel.

After calculating STFT in each EEG channel, the absolute squares

f the DFT amplitudes were calculated to construct the time-

ependent power spectrum (that is, spectrogram) of the EEG

Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166 157

Fig. 4. The block diagram of the feature extraction step.

ω T

(

“

ignal S ( t , ω) in each channel as discussed above. Due to m = 1024 oints used in DFT, the obtained spectrum characterized the

ower distribution in the EEG signal over m /2 + 1 = 513 frequencies k = kFs / m = 0.125 k Hz, where k changed between 0 and m /2 = 512. hese were subsequently binned into 0.5 Hz frequency bands by

sing average, thus, evaluating an average spectral power in each

.5 Hz frequency band from 0 to 64 Hz. The frequency range was

hen restricted to 0–18 Hz so that only 36 frequencies, �k = k · 0.5 z, k = 1,…,16, were retained in the dataset. The constant compo- ent � = 0 Hz was discarded. Finally, the binned and frequency- estricted spectrograms S ( t , �) were temporally smoothed by using

15 s-running average.

The calculation above produced for each channel smoothed

ime-dependent power spectra S ( t , �) of the EEG signal at frequen-

ies �k = k · 0.5 Hz, k = 1,…,36, at 1 s time intervals. The final eature vector was then formed by converting the power values

t each time-point t into decibel form and combining the spectra

rom all 7 input EEG channels into a single, joint feature vector

3) ;

f̄ ( t ) = (10 log 10 S c ( t, �) , c = 1 , . . . , 7 , � = k · 0 . 5 Hz , k = 1 , . . . , 36) , (3)

here c enumerates the input EEG channels.

An important parameter of the feature extraction procedure is

he temporal span of the STFT’s windowing function w ( t ) and the

idth of temporal smoothing. A larger window and wider tempo-

al smoother provide a greater degree of smoothing in the result-

ng power spectra, thus offering a better noise suppression while

acrificing the temporal resolution and the responsiveness of any

ossible detector downstream. A smaller window and shorter time

moother offer a higher temporal sensitivity and responsiveness,

ut at the expense of greater amounts of noise in the STFT ampli-

udes.

We experimented with different choices of these parameters,

onsidering 1, 5, 15, 30 and 60-s intervals for both the Blackman

indow and the temporal running filter. 1- and 5-s intervals pro-

ided a rapid response to EEG changes. 30- and 60-s intervals pro-

ided a greater degree of noise suppression but also incurred 30-

o 60-s delay before any changes in the EEG signals could appear

n the spectra or affect the detector. We observed that the choice

f the 15-s interval for the window size and the running average

rovided a good compromise between the two effects above and

as therefore adopted in this study. The flowchart of the feature

xtraction step is shown in Fig. 4 .

.4. Mental state detection

We implemented the mental state detector using the machine

earning approach of the Least Squares Support Vector Machines

SVM) ( Suykens & Vandewalle, 1999 ). To develop an SVM detec-

or, firstly the spectra of the time-varying EEG signal were reorga-

ized into a feature vector. This was done by combining the power

pectra calculated for every time point from all input EEG chan-

els, therefore producing a vector with a dimensionality of 252,

haracterizing the distribution of the power of the EEG signal over

ll EEG channels and the frequencies from 0 to 18 Hz at a 0.5 Hz

tep. After that, an SVM classifier was trained to detect each men-

al state separately. The training data was composed of a specified

umber of feature vectors chosen at random time points from the

EG spectral data. As SVM is a two-class classifier, multiple SVM

lassifiers were required to be combined for the discrimination of

ore than two mental states. Specifically, the first SVM classifier

as trained to detect the occurrence of the “focused” state in the

EG data against all the rest. A second SVM classifier was trained

o detect the occurrence of the “unfocused” state against all the

est.

.5. Design of the SVM classifiers

The SVM classifiers were trained using the svmtrain function

f the Matlab Statistics Toolbox. The tuning parameters were used

s their default values. The auto-scaling parameter (reducing the

ean and the variance of the training data to zero and one, re-

pectively) was enabled, and box constraint (the cost of misclassi-

cations in SVM) was set to 1. The linear kernel function and the

least squares” method were used for solving the SVM optimiza-

ion problem caused by the large size of the training data involved

n this study.

SVM is based on the result of a linear convolution of a feature

ector characterizing the temporally-local EEG signal with a vec-

or with weights of W , y = ∑ i W i f i , where f i represents the spec- ral power features and i represents the index enumerating such

eatures as well as the corresponding weights W i . As the feature

ector of each time-point is classified as either + 1 (present) if y ≥ over a certain threshold b , or as −1 (absent) otherwise, the sign nd the magnitude of the individual weights W i provide informa-

ion about the contribution and the importance of each feature to

he decision process of the detector. By the construction of the fea-

ure vector f̄ (t ) , the weights W i indicate the frequencies and the

lectrodes contributing to the discrimination of particular mental

tates, both in a positive sense when W i > 0 and negative sense

hen W i < 0.

A plot showing a typical example of the weight vectors W for

articipant S1 over the three most distinctive EEG channels of F3,

4 and Fz is given in Fig. 5 . As can be seen there, the examina-

ion of SVM weight vectors indicates that elevated EEG power in

he frequency range of 1–5 Hz is treated as a positive evidence of

he presence of “focused” state by the detector, while the elevated

ower in the frequency range of 10–15 Hz is taken as an indication

f the “drowsing” state. These observations are in agreement with

he results in Section 3 , as well as with the findings concerning the

EG signature of drowsy and alert mental states in the literature.

he indicator of the “unfocused” or disengaged mental state is the

eduction in the EEG power at both 1–5 Hz and 10–15 Hz frequen-

ies ( Fig. 5 B). This corresponds to the learning of the EEG spectrum

n the disengaged mental state.

Consequently, 3 SVM classifiers were developed to classify the

focused”, “unfocused”, and “drowsing” states in the EEG data.

he outputs of the classifiers were combined by using an XOR-

ggregation to produce final detections. If any classifier responded

ith a “+ 1” (present) at a given time point when the other classi- ers responded with a “−1” (absent), then such a time-point was ategorized according to the label of the classifier that responded.

f two or more classifiers responded with a “+ 1” or all classifiers esponded with a “−1”, then such a time point was categorized as unclassified” and became an error.

158 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

Fig. 5. An example of SVM weight vectors for focused (A), unfocused (B) and

drowsing (C) mental states.

(

2.6. Performance evaluation

The performance of the SVM-based mental state detector is

evaluated using random hold-out cross-validation. Cross-validation

is a standard approach for evaluating the accuracy of classification

and regression models in machine learning. In cross-validation, a

subset of available data is withheld from training so that the ma-

chine learning model cannot see that data or have the knowl-

edge of the information contained in the withheld dataset. Af-

ter the training has finished, using whichever machine learning

algorithm, an unbiased estimate of the performance of the fi-

nal model was obtained by applying it to the withheld validation

dataset and evaluating the accuracy of the model there. In random

hold-out cross-validation, a certain percentage of randomly chosen

data points was chosen and withheld prior to training and, subse-

quently, used for evaluation of the accuracy of the final detector.

Here, two evaluation paradigms (i.e. subject-specific and

ommon-subject) were evaluated: In the subject-specific paradigm,

he mental state detector was trained individually for each partic-

pant based on the data collected for that participant only. 80% of

andom time points from the EEG data from all experiments of

hat individual were used for training the SVM state-classifiers. The

ccuracy of the produced individual detector was then evaluated

n the remaining 20% of the data of all trials that was not used

n the training. Here, the training data of each participant gener-

lly consisted of 60 0 0 data points randomly pulled together from

ifferent trials of the same subject. Respectively, 1500 data points

ere used for the evaluation of the performance of the final detec-

or. In this approach, a different mental state detector was trained

or each participant.

In the case of the common-subject paradigm, a single detector

as jointly trained for all participants. 80% of the data from all

oint EEG recordings of all participants was randomly selected for

raining. The performance of the “mixed” or “generic” detector was

hen evaluated against the dataset of mixed data, as well as against

he data of each participant individually, which was not seen in the

raining. The cross-validation here was performed both jointly for

ll participants and individually for each participant to obtain the

oodness of the generic mental state detector both overall, as well

s on per-participant basis.

. Results

All experiments in this study were performed with healthy vol-

nteer participants chosen among students. All participants signed

he informed consent form after receiving the instructions about

he objectives and procedures of the experiments.

Fig. 6 shows a sample EEG data before the SVM-based classifier

raining part. The differences in the EEG signals associated with

he different attention states can be seen in the spectrograms. Red

ashed lines indicate the different attention state periods corre-

ponding to the time-intervals of 0–10 min (focused), 10–20 min

unfocused), and 20–35 min (drowsy).

The preliminary examination of the collected data revealed that

he changes in the individuals’ attention states could be related to

he changes in the EEG spectral power distribution, appearing as

hanges at certain frequencies in certain EEG channels. The vary-

ng composition of EEG signals with respect to the location of the

lectrodes can be seen in Fig. 7 . In the figure, sections (A) and

B) show sample EEG spectrograms collected during the continu-

us performance task from the EEG electrodes Fz (A) and Pz (B)

or participant S1. Section (C) indicates EEG spectrogram collected

or participant S2 from the electrode Fz.

More specifically, focused and unfocused attention states ap-

eared to be associated with enhanced or suppressed EEG activity

t 1–10 Hz frequency bands in the frontal EEG channels F3, F4, and

z. The drowsing state could be observed as continuous or inter-

ittent power spouts at alpha-band frequency in the EEG signals

t 10–15 Hz and the EEG channels C3, C4, Cz, and Pz. This exami-

ation suggests that focused, unfocused, and drowsing states of the

articipants could be distinguished in the EEG data based on the

ocal spectrum of the signal in various EEG channels.

We then constructed an SVM-based algorithm for the detec-

ion of the above mental states from the EEG data, as described

n Section 2.5 . The examination of the spectral power features in

uch a detector supported the preliminary observations above. In

articular, Table 2 lists some of the features of the mental state

etector and its statistical properties, organized according to the

ntra class Correlation Coefficient (ICC) of such features with the

arget detector states. ICC is a statistical measure of relatedness of

continuous predictor variable with a discrete outcome variable,

Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166 159

Fig. 6. A sample EEG data before the SVM-based classifier training part.

Table 2

Statistical properties of the 25 most significant features based on the ICC with the target mental state variable.

Rank Channel Frequency (Hz) Mean STD Mean focused Mean unfocused Mean drowsy ICC

1 C3 12 7.3701 6.6015 3.0069 2.8568 15.258 0.77489

2 F3 12 7.3351 7.1219 2.6434 2.4826 15.817 0.76973

3 C4 12 7.0274 6.9183 2.5108 2.2899 15.251 0.76694

4 Cz 12 7.1504 6.9895 2.5024 2.4753 15.436 0.76248

5 F4 12 7.7564 6.9367 3.2259 3.0626 15.954 0.75787

6 C3 11.5 7.2411 6.4331 2.6736 3.2756 14.824 0.75544

7 C4 11.5 6.8756 6.5842 2.2998 2.7417 14.616 0.75063

8 F3 11.5 7.1857 6.7928 2.3314 3.0661 15.161 0.74986

9 Pz 12 5.6787 5.9377 2.0338 1.5049 12.627 0.74438

10 Fz 12 6.24 7.2337 1.4592 1.4907 14.709 0.7438

11 F4 11.5 7.5599 6.7017 2.8145 3.5413 15.348 0.73483

12 Cz 11.5 7.0045 6.7105 2.3403 2.8885 14.807 0.73479

13 Fz 11.5 6.1487 7.0016 1.181 2.1192 14.144 0.71065

14 Pz 11.5 5.866 6.0477 1.9401 2.0206 12.772 0.70766

15 C4 12.5 5.9152 6.233 1.9 1.9717 12.988 0.6987

16 F3 12.5 6.3208 6.3337 2.2655 2.2949 13.502 0.69763

17 C3 12.5 6.2436 5.8986 2.4349 2.5351 12.924 0.69603

…

43 F4 3 10.72 3.4544 14.252 9.0921 9.0286 0.50204

44 F4 3.5 9.5554 3.2217 12.813 8.2334 7.8354 0.49144

45 F4 2.5 11.889 3.4799 15.402 10.138 10.323 0.49129

46 F4 4 8.86 3.049 11.921 7.5095 7.3397 0.48387

47 F4 4 8.1726 2.8992 11.054 6.8883 6.7534 0.47408

48 F4 2.5 11.351 3.2627 14.548 9.691 9.9839 0.46457

…

(

elated to the simple Pearson correlation, and defined by (4)

CC = v ariance o f group means f ull v ariance

(4)

Table 2 shows two types of differences in the EEG signals as-

ociated with different mental attention states – an increase in al-

ha band power at frequencies 8–13 Hz, especially over the parietal

obe (electrodes C3, C4, Cz and Pz), and a decrease at lower delta

nd theta band frequencies 1–4 Hz, especially over the frontal lobe

electrodes F3, F4 and Fz).

The performance of the SVM-based mental state classifier is

valuated in terms of the rate of accuracy measure given by (5) .

ccurracy = Number o f corr ect pr edictions T otal number o f predictions

(5)

Tables 3 and 4 show the accuracy results of the SVM-based

ental state classifier for subject-specific and common-subject

aradigms, respectively. As can be seen from the tables, we suc-

eeded in distinguishing the mental attention states for all partic-

pants in the subject-specific paradigm with an average accuracy

anging from 88.60 to 96.70%. The identification of mental states

or the common-subject paradigm could be achieved at 71.80 to

8.80% accuracy.

To demonstrate the efficiency of the results obtained from the

VM-based classifier, the mental attention states were classified by

160 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

Table 3

The accuracy results of the attention state for the subject-specific paradigm with SVM using

7 electrodes.

Subject Fold #1 Fold #2 Fold #3 Fold #4 Fold #5 Average of five folds

S1 98% 99% 93% 95% 97% 96.70%

S2 90% 87% 91% 90% 88% 89.70%

S3 88% 88% 88% 86% 91% 88.60%

S4 91% 95% 92% 96% 92% 93.50%

S5 90% 81% 93% 91% 93% 90.10%

Table 4

The accuracy results of the attention state for the common-subject paradigm with SVM using

7 electrodes.

Subject Fold #1 Fold #2 Fold #3 Fold #4 Fold #5 Average of five folds

S1 75% 70% 71% 75% 68% 71.80%

S2 81% 72% 72% 74% 76% 75.00%

S3 82% 68% 76% 77% 72% 75.00%

S4 79% 80% 78% 79% 78% 78.80%

S5 76% 69% 74% 72% 73% 72.20%

Table 5

Comparison of average 5-fold cross validation results of Machine Learning methods for subject-

specific paradigm.

Method S1 S2 S3 S4 S5 Average of five participants

kNN 82.46% 76.15% 73.35% 79.88% 76.98% 77.76%

ANFIS 85.41% 81.51% 79.84% 82.67% 78.33% 81.55%

SVM 96.70% 89.70% 88.60% 93.50% 90.10% 91.72%

Fig. 7. Examples of EEG signal spectrograms from the EEG electrodes: Fz (A), Pz (B)

for participant S1 and Fz (C) for participant S2.

sing two other well-known classifiers in the literature and com-

ared with the best results of the SVM-based classifier. In addition

o SVM, kNN is a common method that a has been used for EEG

lassification recently ( Hu, Li, Sun, & Ratcliffe, 2018; Ibrahim, Dje-

al, Alsuwailem, & Gannouni, 2017; Shah & Ghosh, 2018 ). Accord-

ng to the results given in Table 5 , the kNN-based classifier yielded

n average accuracy ranging from 73.35 to 83.46%, which was the

orst result of all methods. Although not as widespread as kNN

nd SVM, Adaptive Neuro-Fuzzy System (ANFIS) is a method used

n EEG classification problems ( Bozkurt, Seçkin, & Co ̧s kun, 2017;

eivasigamani, Senthilpari, & Yong, 2016; Madhusmita, Mousumi,

arayan, & Kumar, 2019 ). The ANFIS-based classifier in our study

ielded a performance (78.33% to 85.41%) between kNN-based

nd SVM-based methods’. As the results indicate, the SVM-based

ethod achieved the best performance on mental state detection.

Note that the common-subject mental state detector is clearly

ore useful than a subject-specific detection approach. However,

e empirically observed that the common-subject detector could

ot classify the mental attention states in the EEG data very

ell. Accordingly, the necessity for subject-specific EEG decoders

s much more mentioned in EEG BCI literature.

Finally, the contribution of each EEG channel in discriminating

he different mental attention states is evaluated. For this task, the

ackward step-wise elimination procedure, which ranks the impor-

ance and the contribution of different EEG channels into the de-

ection of the participants’ mental states, was used. Specifically, the

ental state detector with all EEG electrodes included was trained,

nd the drop of the average accuracy of the detector for all partici-

ants was quantified if an electrode was removed from the dataset.

hen, the electrode with the smallest drop in the accuracy was

ermanently removed from the dataset. The procedure was then

epeated until only one electrode remained. The results of this pro-

edure (i.e. training/testing errors and average accuracy results vs.

he number of EEG electrodes) are shown in Fig. 8 . The order of the

lectrodes in the figure is F3-Fz-Cz-F4-Pz-C4-C3, left-to-right, thus

dentifying the ranks of the EEG electrodes in a decreasing order

f importance. In Fig. 8 , training and testing errors were calculated

sing MATLAB’s loss function.

Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166 161

Fig. 8. Training/testing errors and average accuracy results using SVM vs. the number of EEG electrodes used in the EEG dataset for the subject-specific paradigm.

. Discussion

In this study, the problem of detecting mental states in human

ubjects using EEG data was investigated. The problem of discrim-

nating drowsing from attentive states using EEG data has been

tudied in the past in the context of car driving. Different from

uch studies, the present study performed the discrimination of

he mental states engaged in passive observation or supervision

asks. The absence of active involvement in the task introduced

ubstantial differences in this setting. Furthermore, three distinct

162 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

mental states – focused, drowsing, and unfocused or detached, the

latter being the state where subjects do not explicitly doze but lose

the ability to respond to events due to the loss of focus nonethe-

less, were discriminated. The latter state has not been studied in

the literature, although it presents a significant risk for process

control and is significantly more interesting and difficult to detect,

because such a “detached” mental state may not be manifested in

any clear form via secondary, visually or otherwise observable cues

or indicators. In that sense, we call the latter distinguished state a

“pure” mental state. The methodology developed in the paper to-

wards distinguishing this “pure” mental state can be applied more

generally to discriminate subject states in different tasks and cir-

cumstances, as well as for the discrimination of different sets of

states.

Many past similar studies focused on drowsiness of car drivers

and, in another part, the assessment of stress in individuals en-

gaged in mental workload tasks. Along with EEG monitoring, other

methods such as video and movement monitoring were employed

in that context. Although having been successful in previous stud-

ies, many such methods do not allow ready translation in different

circumstances. The contexts in which individuals remain idle or

passive present a special challenge for video- and movement-based

alertness monitoring. EEG-based passive BCI offers a natural ap-

proach to address this problem by providing an easily transferrable

methodology to monitor the mental conditions of the subjects. EEG

signals connect directly with the neural activity of the brain and

offer the opportunity to directly monitor the neural signatures of

different mental states, avoiding the pitfalls of other state monitor-

ing systems, such as those based on physical, visual, or physiologi-

cal cues.

To study the problem of mental state detection and monitoring

in this study, an original EEG dataset characterizing the engage-

ments of the participants during a passive observation task was

collected. An SVM-based method was proposed for detecting the

changes in attention mental states. The SVM detector was trained

on the samples of EEG data collected for subjects while being in

target mental states. An ensemble of machine learning SVM clas-

sifiers were then used to implement the detection via an XOR-

aggregation of multiple state-specific classifier outputs. The de-

veloped detector was shown to discriminate the mental attention

states of focused, unfocused and drowsing with high accuracy us-

ing EEG signals only.

Table 6 summarizes the results of the previous studies on de-

tecting mental attention states chronologically. As seen in Table 6 ,

our study had better results than most previous studies. Analyzing

the studies that gave close results, it is seen that they either used

Table 6

Comparison of previous studies in predicting the human’s attention state.

Reference Dataset Method

Li et al. (2011) EEG KNN

Liu et al. (2013) EEG SVM

Lee et al. (2014) EEG and respiration

data

SVM

Ke et al. (2014) EEG SVM

Wang et al. (2015) EEG SVM

Djamal et al. (2016) EEG SVM

Myrden and Chau

(2017)

EEG SVM

Alirezaei and Sardouie

(2017)

EEG SVM

Nuamah and Seong

(2018)

EEG engagement index SVM

Acı et al. (2019) only EEG SVM

xtra data that support the EEG signal or simply classified the at-

ention using two states.

We found that it is important for the mental state detectors to

e trained individually for subjects. We observed that a “generic”

ental state detector could perform significantly worse than the

ubject-specific detectors, losing as much as 20–30% in detection

ccuracy over the 3 mental states mentioned above.

We examined which features in the EEG signals contributed

o the identification of the mental states above. We found that

he differences between focused and unfocused states were most

learly manifested in the low-frequency EEG activity, between 1

nd 10 Hz. Namely the focused or engaged state was associated

ith elevated EEG power in 1–10 Hz band and disengaged or un-

ocused state was associated with the emptying of the EEG spec-

rum on this frequency band, as well as on other frequency bands.

hese changes were most clearly manifested over the frontal lobe.

he relationship between delta-band EEG activity and the concen-

ration of individuals has been reported in EEG literature in the

ast. However, we found it interesting that no significant signs of

hanges were seen in higher frequency EEG activity, such as beta

r theta, which EEG literature associates with mental workload

asks. Drowsing state manifested itself via intermittent or continu-

us alpha-band power spikes at 10–15 Hz, consistent with the find-

ngs in the literature regarding the relationship between alpha EEG

aves and drowsing.

We ranked the relative importance of different EEG electrodes

n discriminating focused, unfocused, and drowsing mental states.

e found that over 90% of the performance could be recovered

ith only 4 EEG electrodes, namely F3, F4, Fz, and Cz. Three of

hese electrodes were located over the frontal lobe. Neural activ-

ty in the frontal lobe has conventionally been associated with

he individual’s concentration. Thus, the frontal electrodes effec-

ively distinguished between the subjects who were mentally en-

aged versus those not engaged. The parietal lobe is known to

e associated with the development of alpha waves. Therefore,

z electrode can be expected to facilitate the detection of the

nset of drowsiness. In addition, in this study, SVM was used

o classify the “unfocused” section and efficiency of electrodes

reas.

The parameters of EEG signals relevant to the discrimination

f different mental states are important for EEG signal acquisition

ubsystems of such mental state monitoring systems. Factors such

s signal’s bandwidth, resolution, and electrodes all affect the de-

ign and the cost of such systems. Our analysis indicates that a

uccessful determination of the three mental attention states can

e performed by using the EEG signals restricted to a frequency

Mental states predicted Accuracy (%)

3 different attention levels 57.00

2 states (i.e. attentive or inattentive ) 76.82

6 levels (i.e. awake, slightly drowsy,

moderately drowsy, extreme drowsy,

sleep, deep sleep)

98.60

3 levels (i.e. attention, no attention and

rest)

76.19 −85.24

2 states (i.e. driving or math task) 84.6 ± 5.8–86.2 ± 5.4 2 states (i.e. attentive or inattentive ) 77.0 0–83.0 0

3 states (i.e. fatigue, frustration,

attention)

71.6–84.8

2 states (i.e. attentive or inattentive ) 92.80

2 states (i.e. attentive or inattentive ) 93.33 ± 8.16

3 levels (i.e. focused, unfocused,

drowsy)

96.70 (best) 91.72

(avg.)

Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166 163

Fig. 9. Examples of the feature and their basic statistical properties for participant S1.

and below 20 Hz and using only 4 or 5 electrodes at F3, F4, Fz, Cz

nd possibly Pz locations. This indicates that the design of passive

EG BCIs for monitoring mental attention states can be done with

ardware providing signal acquisition at the relatively low frequen-

ies of 40–50 Hz and only 4 electrodes in the configuration above.

A big complication for practical applications of EEG-based tech-

ologies is the susceptibility of EEG BCI to motion artifacts and

xternal electromagnetic interferences. EEG headsets are notori-

us for being easily affected and the signal for being distorted by

ubject movements or working nearby electrics and electronic de-

164 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

vices. These factors are much less impactful in the context of this

study, where the property of the individuals to remain motion-

less and in controlled indoor environments alleviated many of such

problems.

At the same time, one of the key advantages of the method-

ology used here is its reliance on the identification of target

states directly on the neural activity. Direct use of neural signa-

tures learned while subjects implemented the desired target states

makes such approach difficult to circumvent, confuse, or mislead,

which is a significant risk when secondary signatures such as video

or movement monitoring are used for mental state estimation.

Moreover, our approach allows the detection of mental states that

are not necessarily manifested clearly via physiological or visual

signatures, or “pure” mental states as we term them. Finally, the

methodology here can easily be generalized and applied to the de-

tection of different mental states in a variety of different applica-

tions with minimal modifications.

More specifically, the method for detecting mental states in

this work was investigated in the context of a specific continuous

attention task. At the same time, the approach used is very general

and can be applied to the problem of the detection of different

mental states in different circumstances as long as individuals

remain predominantly passive. Such applications may include,

for example, the observation of in-patients and/or paralyzed and

immobile patients in hospitals, etc. The key elements of the

proposed methodology are: (i) obtaining samples of the EEG data

associated with the mental states of interest using controlled

Fig. 10. Examples of the EEG recordings associated with different detected mental stat

imulation of such states, (ii) using machine learning to construct

etectors for such states based on EEG data and in subject-specific

nd mental state-specific manners; (iii) combining the outputs of

ultiple machine learning detectors within a passive EEG BCI via

OR or similar aggregation technique. The analysis of the param-

ters of such constructed machine learning detectors can offer

nsights about the representation of different mental states in EEG

ignals.

The definition of the different mental attention states in the

ontext of this methodology is done by means of self-reporting.

he practice of self-reporting is widely accepted in psychological

ractice. In this study, the states of interest were defined as “fo-

used: remaining concentrated, focused and actively paying atten-

ion to a task”, “unfocused: remaining awake but not paying at-

ention or reacting to a task”, and “drowsing: dozing off with eyes

losed or open”. As such, the participants were instructed to im-

lement those target states while the EEG data was collected. The

ubjects implemented such states over indicated intervals of time

s they understood them.

In general, objective measures that can be used independently

o identify the target mental states should be used whenever they

xist. In our case, two factors precluded an introduction of such

bjective controls. First, given that one of the states to be iden-

ified in this study was disengagement, we found that any active

robing of the state of the subjects resulted in the destruction

f that state and the nullification of the experiment. Second, we

ould identify no clear visual metrics or signatures that could be

es for participant S1 and electrode Pz: (A) ‘focused’ (B) ‘unfocused’ (C) ‘drowsy’.

Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166 165

ssociated with participants staying passively focused or really be-

oming disengaged. Therefore, the practice of self-reporting of the

ubjects was used as the primary means of defining the mental

tates in the context of this study.

On the other hand, the inspection of the EEG signals is shown

n Figs. 9 and 10 . In Fig. 9 , left column: the mean (bar-plots) and

he standard deviations (red line) of different features shown with

espect to their corresponding EEG channel and frequency. Right

olumn: mean and standard deviations of different features within

ifferent mental state periods. Blue is the ‘focused’ state, orange is

he ‘unfocused’ state, and yellow is the ‘drowsy’ state. Error-bars

how the within-group standard deviations of each feature.

Figs. 9 and 10 reveal that there are clear, objective, and quan-

ifiable differences in the shape and characteristics of the EEG sig-

als observed over the 3 periods of time where the subjects were

nstructed to implement the 3 different attention states. The ob-

ective differences were mapped to the periods of time when the

ubjects self-identified themselves as being focused / concentrated,

isengaged / unfocused, and drowsing, respectively.

. Conclusions

In this study, we developed a passive EEG BCI for monitoring

set of mental states of human individuals. We collected a new

EG dataset related to individuals engaging in a passive supervi-

ion task. We demonstrated with high accuracy the detection of 3

ental attention states, namely passive attention, disengagement,

nd drowsiness of passive individuals in a task. The accuracy of

he discrimination of engaged or focused, disengaged or unfocused,

nd drowsing states reported in this work reached 96.70% (best)

nd 91.72% (avg.) and can be of use for driver security applica-

ions. The SVM-based EEG BCI approach used in this study allows a

eady-to-use machine learning model, generalized for other scenar-

os, including the detection of different mental states and a variety

f different circumstances. The analysis of the parameters of con-

tructed mental state detectors can provide new insights into the

epresentation of such states in the EEG signals.

One of the particularly interesting areas of the generalization of

his study is related to clinical applications that require assessment

r monitoring of the mental states of subjects. One, although not

ecessarily the only example of such an application, is the bispec-

ral index monitoring (BIS), one of the technologies for monitoring

he depth of anesthesia. BIS is a known, albeit proprietary, tech-

ology that relies on monitoring the electroencephalographic brain

ignals to produce a depth of anesthesia index that helps anes-

hesiologists reduce the incidence of intraoperative awareness of

atients. Although the algorithms behind BIS are proprietary and

ave not been disclosed, BIS is an empirically derived measure

omputed as a weighted sum of several electroencephalographic

arameters calculated in the time domain, frequency domain, and

igh order spectral representations ( Chalela et al., 2018 ). A single

alue ranging from 0 (equivalent to EEG silence) to 100 is thus de-

ived to indicate the level of general an anesthesia, based on the

lectroencephalographic activity of the brain. The construction of

ispectral index is similar to the approach used in this study and

an be extended for novel clinical applications in the future.

eclaration of Competing Interest

None.

redit authorship contribution statement

Çi ğdem İnan Acı: Supervision, Formal analysis, Conceptualiza-

ion, Writing - original draft, Writing - review & editing. Murat

aya: Data curation, Software, Funding acquisition, Writing - origi-

al draft. Yuriy Mishchenko: Supervision, Conceptualization, Writ-

ng - original draft.

cknowledgments

All experiments in this work were performed with healthy vol-

nteer participants chosen among the students of the Faculty of

ngineering at Toros University (Mersin, Turkey). All participants

igned the informed consent form after receiving the instructions

bout the experiments’ objectives and procedures, in accordance

ith the ethical guidelines of Mersin University. This study is

upported by Mersin University Department of Scientific Research

rojects (Project Code: 2018-3-TP2-3064 ). This academic work was

inguistically supported by the Mersin Technology Transfer Office

cademic Writing Centre of Mersin University.

upplementary materials

Supplementary material associated with this article can be

ound, in the online version, at doi: 10.1016/j.eswa.2019.05.057 .

eferences

cı, Ç., Kaya, M., & Mishchenko, Y. (2019). Attention state dataset Retrieved

April 10, 2019, from https://www.kaggle.com/inancigdem/eeg- data- for- mental- attention- state- detection .

lirezaei, M., & Sardouie, S. H. (2017). Detection of human attention using EEG sig- nals. In 2017 24th National and 2nd international Iranian conference on biomedical

engineering (ICBME) (pp. 1–5). IEEE. https://doi.org/10.1109/ICBME.2017.8430244 .

lmeida, L. B. (1994). The fractional Fourier transform and time-frequency rep- resentations. IEEE Transactions on Signal Processing, 42 (11), 3084–3091. https:

//doi.org/10.1109/78.330368 . ricò, P., Borghini, G., Di Flumeri, G., Colosimo, A., Pozzi, S., & Babiloni, F. (2016). A

passive brain–computer interface application for the mental workload assess- ment on professional air traffic controllers during realistic air traffic control

tasks. Progress in Brain Research, 228 , 295–328. https://doi.org/10.1016/BS.PBR.

2016.04.021 . rico, P., Borghini, G., Di Flumeri, G., Sciaraffa, N., Colosimo, A., & Babiloni, F. (2017).

Passive BCI in operational environments: Insights, recent advances, and future trends. IEEE Transactions on Biomedical Engineering, 64 (7), 1431–1436. https://

doi.org/10.1109/TBME.2017.2694856 . orghini, G., Astolfi, L., Vecchiato, G., & Mattia, D. (2014). Measuring neurophysi-

ological signals in aircraft pilots and car drivers for the assessment of mental

workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews, 44 , 58– 75. https://doi.org/10.1016/J.NEUBIOREV.2012.10.003 .

ozkurt, F., Seçkin, A. Ç., & Co ̧s kun, A. (2017). Integration of IMU sensor on low- cost EEG and design of cursor control system with ANFIS. International Jour-

nal of Engineering Trends and Technology, 54 , 162–169. https://doi.org/10.14 4 45/ 22315381/IJETT- V54P223 .

halela, R., Gallart, L., Pascual-Guardia, S., Sancho-Muñoz, A., Gea, J., & Orozco-

Levi, M. (2018). Bispectral index in hypercapnic encephalopathy associated with COPD exacerbation: A pilot study. International Journal of Chronic Obstructive

Pulmonary Disease, 13 , 2961. https://doi.org/10.2147/COPD.S167020 . eivasigamani, S., Senthilpari, C., & Yong, W. H. (2016). Classification of focal and

nonfocal EEG signals using ANFIS classifier for epilepsy detection. International Journal of Imaging Systems and Technology, 26 (4), 277–283. https://doi.org/10.

1002/ima.22199 .

jamal, E. C., Pangestu, D. P., & Dewi, D. A. (2016). EEG-based recognition of at- tention state using wavelet and support vector machine. In 2016 International

seminar on intelligent technology and its applications (ISITIA) (pp. 139–144). IEEE. https://doi.org/10.1109/ISITIA.2016.7828648 .

POC + . (2019). EMOTIV Retrieved April 15, 2019, from https://www.emotiv.com/ epoc/ .

ashemi, A., Saba, V., & Resalat, S. N. (2014). Real time driver’s drowsiness detection

by processing the EEG signals stimulated with external flickering light. Basic and Clinical Neuroscience, 5 (1), 22–27. Retrieved from http://www.ncbi.nlm.nih.gov/

pubmed/25436080 . sieh, H.-Y., Liang, S.-F., Ko, L.-W., Lin, M., & Lin, C.-T. (2006). Development of a

real-time wireless embedded brain signal acquisition/processing system and its application on driver’s drowsiness estimation. In IEEE International conference

on systems, man and cybernetics (pp. 4374–4379). IEEE. https://doi.org/10.1109/ ICSMC.2006.384822 .

u, B., Li, X., Sun, S., & Ratcliffe, M. (2018). Attention recognition in EEG-based af-

fective learning research using CFS + KNN algorithm. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 15 (1), 38–45. https://doi.org/10.1109/

TCBB.2016.2616395 . brahim, S. W., Djemal, R., Alsuwailem, A., & Gannouni, S. (2017). Electroencephalog-

raphy (EEG)-based epileptic seizure prediction using entropy and K-nearest

166 Ç. ̇I. Acı, M. Kaya and Y. Mishchenko / Expert Systems With Applications 134 (2019) 153–166

neighbor (KNN). Communications in Science and Technology, 2 (1), 6–10. https: //doi.org/10.21924/cst.2.1.2017.44 .

Ke, Y., Chen, L., Fu, L., Jia, Y., Li, P., Zhao, X., et al. (2014). Visual attention recog- nition based on nonlinear dynamical parameters of EEG. Bio-Medical Materials

and Engineering, 24 (1), 349–355. https://doi.org/10.3233/BME- 130817 . Lee, B.-G., Lee, B.-L., Chung, W.-Y., Lee, B.-G., Lee, B.-L., & Chung, W.-Y. (2014). Mobile

healthcare for automatic driving sleep-onset detection using wavelet-based EEG and respiration signals. Sensors, 14 (10), 17915–17936. https://doi.org/10.3390/

s141017915 .

Li, Y., Li, X., Ratcliffe, M., Liu, L., Qi, Y., & Liu, Q. (2011). A real-time EEG-based BCI system for attention recognition in ubiquitous environment. In Proceedings

of 2011 international workshop on ubiquitous affective awareness and intelligent interaction - UAAII ’11 (pp. 33–40). ACM Press. https://doi.org/10.1145/2030092.

2030099 . Liu, N.-H., Chiang, C.-Y., & Chu, H.-C. (2013). Recognizing the degree of human at-

tention using EEG signals from mobile sensors. Sensors, 13 (8), 10273–10286.

https://doi.org/10.3390/s130810273 . Lobos, T., & Rezmer, J. (1997). Real-time determination of power system frequency.

IEEE Transactions on Instrumentation and Measurement, 46 (4), 877–881. https:// doi.org/10.1109/19.650792 .

Madhusmita, M., Mousumi, B., Narayan, P. D., & Kumar, M. S. (2019). A novel method for epileptic EEG classification using DWT, MGA, and ANFIS: A real time application

to cardiac patients with epilepsy (pp. 525–534). Singapore: Springer. https://doi.

org/10.1007/978- 981- 13- 0617- 4 _ 51 . Mardi, Z., Ashtiani, S. N. M., & Mikaili, M. (2011). EEG-based drowsiness detection

for safe driving using chaotic features and statistical tests. Journal of Medical Signals and Sensors, 1 (2), 130–137. Retrieved from http://www.ncbi.nlm.nih.gov/

pubmed/22606668 . Myrden, A., & Chau, T. (2017). A passive EEG-BCI for single-trial detection of changes

in mental state. IEEE Transactions on Neural Systems and Rehabilitation Engineer-

ing, 25 (4), 345–356. https://doi.org/10.1109/TNSRE.2016.2641956 .

uamah, J. K., & Seong, Y. (2018). Support vector machine (SVM) classifica- tion of cognitive tasks based on electroencephalography (EEG) engagement

index. Brain-Computer Interfaces, 5 (1), 1–12. https://doi.org/10.1080/2326263X. 2017.1338012 .

esalat, S. N., & Saba, V. (2015). A practical method for driver sleepiness detec- tion by processing the EEG signals stimulated with external flickering light.

Signal, Image and Video Processing, 9 (8), 1751–1757. https://doi.org/10.1007/ s11760- 015- 0760- x .

hah, M., & Ghosh, R. (2018). Classification and prediction of human cognitive skills

using EEG signals. In 2018 Fourth international conference on biosignals, images and instrumentation (ICBSII) (pp. 206–212). IEEE. https://doi.org/10.1109/ICBSII.

2018.8524729 . hangkai, G., Yijun, W., Xiaorong, G., & Bo, H. (2014). Visual and auditory brain–

computer interfaces. IEEE Transactions on Biomedical Engineering, 61 (5), 1436– 1447. https://doi.org/10.1109/TBME.2014.2300164 .

Simon, M., Schmidt, E. A., Kincses, W. E., Fritzsche, M., Bruns, A., Aufmuth, C., et al.

(2011). EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions. Clinical Neurophysiology, 122 (6), 1168–1178. https://doi.org/10.

1016/J.CLINPH.2010.10.044 . uykens, J. A. K., & Vandewalle, J. (1999). Least squares support vector machine

classifiers. Neural Processing Letters, 9 (3), 293–300. https://doi.org/10.1023/A: 1018628609742 .

ang, Y.-K., Jung, T.-P., & Lin, C.-T. (2015). EEG-based attention tracking during dis-

tracted driving. IEEE Transactions on Neural Systems and Rehabilitation Engineer- ing, 23 (6), 1085–1094. https://doi.org/10.1109/TNSRE.2015.2415520 .

eo, M. V. M., Li, X., Shen, K., & Wilder-Smith, E. P. V. (2009). Can SVM be used for automatic EEG detection of drowsiness during car driving. Safety Science, 47 (1),

115–124. https://doi.org/10.1016/J.SSCI.2008.01.007 .