helpfn

A Seamless Virtual Emotion System with Replaceable Detection toward Proper

Emotion-based IoT Services Hyunbum Kim∗, Jalel Ben-Othman†‡, Lynda Mokdad§, Kiho Lim¶

∗Department of Computer Science, University of North Carolina at Wilmington, Wilmington, NC 28403, USA [email protected]

†L2S Lab CNRS, Centralesupélec, Université Paris Sud, France [email protected] ‡Université de Paris 13, France

§Department of Computer Science, LACL Lab, University of Paris-Est, Créteil, France [email protected]

¶Department of Computer Science, University of South Dakota, Vermillion, SD 57069, USA [email protected]

Abstract—In this paper, a seamless virtual emotion system is introduced toward appropriate emotion-based services in advanced IoT-enabled smart cities. The proposed system is supported by virtual emotion barrier with replaceable detection and hole-free property, which recognizes human emotion using wireless signal. Also, we formally define a problem with ILP formulation whose goal is to maximize the lowest detection accu- racy with weak hole in replaceable barrier in order to construct seamless replaceable hole-free virtual emotion barrier ultimately. Then, a novel scheme is proposed to solve the problem. Moreover, we perform the proposed approach through extensive simulations with various scenarios and its performance is evaluated with detailed discussion.

Index Terms—seamless, emotion, IoT, replaceable, detection.

I. INTRODUCTION

Basically, emotion should be an original property of human being and it will give a critical affection to interpersonal communication in real life of people. So, inferring human emotion recognition has earned much interests of various communities from academic researchers, industry areas such as haptic game applications, affective computing, medical activities, etc [1], [2], [3], [4], [5].

It is known that the emotion recognition research has been proceeded by three approaches largely. First, the emotion can be inferred by audio and visual data such as vocal tone, facial expression [6], [7], [8]. Second, it also can be recognized by physiological signals including blood pressure, body temperature [9], [10]. In particular, the third approach is devised recently by extracting features from information through wireless signal and its reflection so as to derive human emotion and activity consequently [11], [12].

Besides, it is highly expected that Internet of Things (IoT) [13] will serve as one of the promising in smart cities

because IoT enables people to get proper services through a large quantity of information such as emotion data, social connections, etc. Specially, inferring human emotion correctly should be indispensable factor for the realization of advanced smart cities. It follows that people are able to receive appropri- ate emotion-based services promptly if the emotion derivation with possible high detection accuracy can be performed in IoT environment and such a correct emotion information also can be utilized for various purposes including intelligent driver assistant system, medical services [14].

On the other hand, the study of barrier-coverage has gained lots of attractions since it can be used for various applications such as intrusion detection, surveillance for public safety [15], [16], [17], [18], [19]. Basically, if a barrier is formed in the region of interest (RoI), it is guaranteed that any penetration of movement into RoI is to be detected by at least one device in the barrier. Also, applying barrier-coverage to emotion detection is highly appropriate in IoT environment where each IoT device equipped with wireless signal is utilized. But, there may have a weak detection part, called as weak emotion detection hole, because wireless signal and its reflection can be weakened depending on the distance. Hence, the issue of minimizing detection hole should be considered critically to provide people with proper emotion-based services.

Based on the above motivations, we introduce a seamless virtual emotion system which is supported by virtual emotion barrier with replaceable detection property. So, the proposed system creates a seamless replaceable hole-free virtual emo- tion barrier so that the emotion of any person moving from one position to another location is to be detected by at least one IoT device such that weak emotion detection hole by the use of wireless signal is removed possibly in the system. Also,

Performance Evaluation of Communications in Distributed Systems and Web based Service Architectures, 19

978-1-7281-2999-0/19/$31.00 ©2019 IEEE 1051Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:49:09 UTC from IEEE Xplore. Restrictions apply.

with a Integer Linear Programming (ILP) formulation [20], we formally define a MaxLDA problem whose goal is to maximize the lowest detection accuracy. Also, we develop a novel Replaceable-Maximum-Lowest-Edge approach based on Initial-Rep-Bar procedure to solve the problem. Then, we analyze the performance of the proposed schemes after implementing them through extensive experiments.

The remaining of the paper is organized as follows. In Section II, we describe the proposed seamless virtual emotion system with replaceable property for IoT services as well as formally define the MaxLDA problem with ILP formulation. In Section III, we develop novel schemes to solve the prob- lem. Also, after extensive experiments with various scenarios, the performance of the proposed approach is analyzed with detailed discussion in Section IV. Then, in Section V, we conclude this paper finally.

II. A SEAMLESS VIRTUAL EMOTION SYSTEM WITHOUT WEAK DETECTION HOLE

The proposed virtual emotion system is introduced for the construction of seamless replaceable hole-free virtual emotion barrier in this section. Furthermore, we formally define a problem using ILP formulation.

A. System Model and Problem Definition

The proposed virtual emotion system considers the below assumptions. • Each IoT device equipped with wireless signal can detect

human emotion through wireless signal and its reflection. • With the reflected signal, every IoT device is able to

derive various types of human emotion through pre- determined procedures for heart segmentation and res- piration.

• IoT devices have different maximum signal ranges. • The accuracy of emotion can be estimated through wire-

less signal and its reflection strength. So, the accuracy decreases as the distance between human and IoT device increases.

Now, we define important definitions which are used in the proposed system.

Definition 2.1 (replaceable barrier): After forming a tempo- rary barrier consisting a sequence of devices in the given area, a portion of the barrier can be replaceable depending on the system’s requirement. Then, a replaceable barrier (Rep-Bar) is a barrier with a flexibility, which a part of the barrier can be replaceable with other devices in order to pursue a specific objective of system.

Definition 2.2 (seamless replaceable hole-free virtual emo- tion barrier): Given a square-shaped IoT-based space area S, there is a set of IoT devices T where those devices have differ- ent maximum signal ranges. A seamless replaceable hole-free virtual emotion barrier (SmRep-VEmoBar) is a replaceable barrier that can recognize the human emotion without weak detection hole with a very low emotion detection accuracy.

Then, we formally define the MaxLDA problem that is to be solved in the proposed model.

Definition 2.3 (MaxLDA): Suppose that a set of IoT devices is positioned in S and a detection accuracy information accord- ing to a distance between device and person is given. Then, the maximum low detection accuracy (MaxLDA) problem is to maximize the lowest detection accuracy with weak hole in replaceable barrier so that seamless replaceable hole-free virtual emotion barrier is constructed continuously.

Fig. 1(a) shows how to derive emotion using wireless signal and the possibility of weak detection hole depending on the distance between IoT device and human. As it can be seen, IoT device is able to be recognize emotion through internal deriva- tion processes. Also, when a virtual emotion barrier consisting of K1,K2,K3,K4,K5 is created in space area S, there may exist a weak detection hole with a low accuracy if human is moving between K1 and K2. Fig. 1(b) depicts the creation of seamless replaceable hole-free virtual emotion barrier. After we find three virtual emotion barriers D1,D2,D3, the lowest accuracy of each barrier can be estimated initially. (i.e, D1 with 0.3, D2 with 0.3, D3 with 0.4). Then, if we can find any replaceable edge which provide a better accuracy, the seamless replaceable hole-free virtual emotion barrier is constructed by applying the replaceable property of the barrier. For example, after initial D2 with lowest accuracy 0.3 of K7, it can be replaced with K17 with a better accuracy 0.7.

B. ILP Formulation

First, we define notations that are utilized in the proposed system. Then, the MaxLDA problem is formally represented based on Integer Linear Programming (ILP). n: the number of IoT devices which are located within S. K: a set of devices with heterogeneity, K = {K1,K2, ...,Kn}. R: a list of signal range of IoT devices, R = {r1,r2, ...,rn}. h: an index of IoT device and signal range (1 ≤ h ≤ n). i: an index of IoT device and signal range (1 ≤ i ≤ n). ω: the minimum detection accuracy value in the barrier. ψh,i: detection accuracy value between Kh and Ki.

Moreover, the integer variables in ILP are defined as fol- lows.

Ah,i =

{ 1, if Kh and Ki have detection edge by signals 0, otherwise.

Bh,i =

 

1, if Kh and Ki are chosen as a part of SmRep-VEmoBar

0, otherwise.

C =

{ 1, if SmRep-VEmoBar is formed completely 0, otherwise.

An objective function (1) is to maximize the lowest de- tection accuracy with weak hole in the seamless replaceable hole-free virtual emotion barrier. Hence, the objective function can be presented as follows.

Performance Evaluation of Communications in Distributed Systems and Web based Service Architectures, 19

1052Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:49:09 UTC from IEEE Xplore. Restrictions apply.

(a) An example of virtual emotion detection with accuracy using wireless signal and weak detection in hole.

(b) An example of creating seamless replaceable hole-free virtual emotion barrier.

Fig. 1. An example of virtual emotion detection weak hole through wireless signal and of generating seamless replaceable hole-free virtual emotion barrier.

Maximize ω (1)

Subject to: n∑

i=1

Ah,i ≤ 1,(∀h) (2)

n∑ i=1

Bh,i ≥ 1,(∀h) (3)

Bh,i ≤ Ah,i,(∀h,∀i) (4)

ψh,i ·Ah,i ·Bh,i ·C ≥ ω,(∀h,∀i) (5)

Constraint (2) forces that two devices Kh and Ki have at most one edge and constraint (3) also represents that the completed SmRep-VEmoBar covers the edge between Kh and Ki at most once. Also, by constraint (4), it is confirmed that there is an edge between Kh and Ki so as to select it as a portion of SmRep-VEmoBar. Also, it is imposed that ω value is the lowest detection accuracy value if SmRep-VEmoBar is built in the system.

III. THE PROPOSED APPROACH

In this section, we introduce two proposed schemes, Initial- Rep-Bar and Replaceable-Maximum-Lowest-Edge in order to solve the MaxLDA problem.

A. Initial-Rep-Bar

With given information including IoT space area S, a set of devices K with their maximum signal ranges, we first implement Initial-Rep-Bar for executing system initialization, which is accomplished by the below steps.

• Identify a virtual source U and a destination O. Also, verify that n number of IoT devices set K with a list of heterogeneous wireless signal ranges R are located in S.

• Generate an initial accuracy graph GK = (V (GK),E(GK)) where V (GK) where V (GK) is the set of vertices, E(GK)) is the set of edges. Then, add K, S, O to V (GK).

• Each device Kh learns a neighbor Ki by checking if euclidean distance euclidean(Kh,Ki) between two devices is at most a sum of their wireless signal ranges rh + ri where Kh,Ki ∈ K, rh,ri ∈ R, h 6= i. Then, every neighbor relationship between two devices Kh and Kj is added to E(GK) as an edge e(Kh,Ki).

• Also, for every device, estimate a device accuracy value zKh by referring pre-determined accuracy table X. More- over, if the device Kh and Ki are neighbor, not only their minimum value (i.e. min[zKh,zKi]) is considered as the edge accuracy value z(Kh,Ki) but also the edge e(Kh,Ki) with z(Kh,Ki) is added into E(GK).

• Then, generate current status graph GK′ = (V (GK′),E(GK′)) by transferring from GK = (V (GK),E(GK)). Using GK′, search for every node- disjoint path (or independent path) from U to O [21].

• If a node-disjoint path Dj is found, add it to the set of Rep-Bar candidates D. Return D ultimately if there is no more new node-disjoint path.

The pseudocode of Initial-Rep-Bar is described in Algo- rithm 1 in more detail.

Performance Evaluation of Communications in Distributed Systems and Web based Service Architectures, 19

1053Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:49:09 UTC from IEEE Xplore. Restrictions apply.

Algorithm 1 Initial-Rep-Bar Inputs: S,K,R,n, Output: D

1: set GK = (V (GK),E(GK)) = ∅; 2: set GK′ = (V (GK′),E(GK′)) = ∅; 3: set D = ∅; 4: add source U and destination O to V (GK); 5: for h = 1 to n do 6: set V (GK) ← V (GK)∪Kh; 7: end for 8: for h = 0 to n−1 do 9: for i = h + 1 to n do

10: if Euc(h,i) ≤ rh + ri and h 6= i then 11: set E(GK) ← E(GK) ∪ e(Kh,Ki) with an edge

accuracy value z(Kh,Ki) = min[zKh,zKi]; 12: end if 13: end for 14: end for 15: set GK′ ←GK; 16: while GK′ 6= ∅ do 17: find a new node-disjoint path from GK′ from U to O; 18: if a new node-disjoint path Dj is found then 19: set D ← D ∪Dj ; 20: else 21: break; 22: end if 23: end while 24: return D

B. Replaceable-Maximum-Lowest-Edge

After implementing Initial-Rep-Bar, we perform Replaceable-Maximum-Lowest-Edge which returns seamless replaceable hole-free virtual emotion barrier to the system.

• Create a set of detection accuracy value candidates Z. • From D, calculate the lowest accuracy edge value ZDj

of each node-disjoint path Dj and add it to Z. • Search for the node-disjoint path with the highest accu-

racy edge value from Z and then update the founded path as Dmax. Also, update the lowest accuracy edge value in Dmax as Zmax.

• Then, the below sub-steps are iterated.

– Within Dmax, verify if there exists an edge which is replaceable with a bigger accuracy than Zmax such that Dmax is still maintained from U to O for the replacement.

– If the replaceable edge value Zrep is found where Zrep > Zmax, then update Zrep as Zmax.

• After the above sub-steps, set Dmax as SmRep-VEmoBar and also update Zmax as ω.

• Return ω as a final result.

Also, the pseudocode of Replaceable-Maximum-Lowest- Edge approach is presented in Algorithm 2 in more detail.

Algorithm 2 Replaceable-Maximum-Lowest-Edge Inputs: D,GK′,n, Output: ω

1: set ω = 0; 2: set Z = ∅; 3: set Zmax = 0; 4: set SmRep-VEmoBar = ∅; 5: for j = 0 to |D| do 6: calculate the lowest accuracy edge value ZDj of each

node-disjoint path Dj ; 7: set Z ← Z ∪ZDj ; 8: end for 9: search for node-disjoint path with the highest accuracy

edge value from Z; 10: update the founded path as Dmax and its lowest accuracy

edge value as Zmax; 11: for j = 0 to |Dmax| do 12: check if there exists an edge that is replaceable with

bigger accuracy for every edge within Dmax; 13: if the replaceable edge value Zrep is found where

Zrep > Zmax then 14: set Zmax ← Zrep; 15: end if 16: end for 17: set SmRep-VEmoBar ← Dmax; 18: set ω ← Zmax; 19: return ω;

IV. PERFORMANCE EVALUATION

In this section, the performance of the proposed ap- proach is evaluated. For the utilized experiment settings, after Initial-Rep-Bar procedure, we have implemented Replaceable- Maximum-Lowest-Edge approach with different square-shaped areas such as 500 × 500 m2, 500 × 400 m2. And, n number of IoT devices ranging from 100 to 200 have random locations within the square-shaped region initially. Also, the maximum wireless signal range of heterogenous IoT device is between 60 and 85 so that each IoT device has different signal ranges. We note that each numerical result represents the average minimum detection accuracy value of 100 different graph sets.

For the considered group of simulations, we implemented Replaceable-Maximum-Lowest-Edge approach with n = 100 in 500 × 500 m2, 500 × 400 m2, respectively. As it can be seen in Fig. 2(a) and Fig. 2(b), it is verified that as the maximum wireless signal range increases, the minimum detection accu- racy value ω decreases because the bigger signal ranges can cause the longer path in Dmax. Furthermore, for the second group of experiments, we performed Replaceable-Maximum- Lowest-Edge scheme with n = 200 in 500 × 500 m2, 500 × 400 m2. By Fig. 2(c) and Fig. 2(d), we have checked that the minimum detection accuracy value ω decreases when the maximum wireless signal range increases.

Performance Evaluation of Communications in Distributed Systems and Web based Service Architectures, 19

1054Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:49:09 UTC from IEEE Xplore. Restrictions apply.

60 65 70 75 80 85 Maximum signal range

0.5

0.6

0.7

0.8

0.9

1

M in

im um

d et

ec tio

n ac

cu ra

cy

Replaceable-Maximum-Lowest-Edge

(a) n = 100 in 500 × 500 area

60 65 70 75 80 85 Maximum signal range

0.5

0.6

0.7

0.8

0.9

1

M in

im um

d et

ec tio

n ac

cu ra

cy

Replaceable-Maximum-Lowest-Edge

(b) n = 100 in 500 × 400 area

60 65 70 75 80 85 Maximum signal range

0.5

0.6

0.7

0.8

0.9

1

M in

im um

d et

ec tio

n ac

cu ra

cy

Replaceable-Maximum-Lowest-Edge

(c) n = 200 in 500 × 500 area

60 65 70 75 80 85 Maximum signal range

0.5

0.6

0.7

0.8

0.9

1

M in

im um

d et

ec tio

n ac

cu ra

cy

Replaceable-Maximum-Lowest-Edge

(d) n = 200 in 500 × 400 area

Fig. 2. The minimum detection accuracy value ω by Replaceable-Maximum-Lowest-Edge approach with different number of IoT devices and area sizes

V. CONCLUDING REMARKS

In this paper, we introduced a virtual emotion system which was supported by seamless virtual emotion barrier with re- placeable detection avoiding weak hole. The MaxLDA problem was formally defined using ILP formulation and we then pro- posed Replaceable-Maximum-Lowest-Edge scheme based on Initial-Rep-Bar procedure. Also, we analyzed the performance of the proposed approach through extensive experiments. As future work, we plan to extend the current approach and perform it in practical experiments with various expanded areas and scenarios.

REFERENCES

[1] J. Gratch and S. Marsella, “Evaluating a computational model of emo- tion,” Journal of Autonomous Agents and Multiagent Systems, pp. 23-43, 2005.

[2] Z. Zeng, M. Pantic, G. I. Roisman and T. S. Huang, “A survey of affect recognition methods: Audio, visual, and spntaneous expressions,” IEEE Transactions on Machine Intelligence, vol. 31, pp. 39-58, 2009.

[3] X. Sun, C. Zhang, G. Li, D. Sun, F. Ren, A. Zomaya and R. Ranjan, “Detecting users’ anomalous emotion using social me- dia for business intelligence,” Journal of Computational Science, https://doi.org/10.1016/j.jocs.2017.05.029, 2017.

[4] H. Kim, J. Ben-Othman, S. Cho and L. Mokdad, “A Framework for IoT-enabled Virtual Emotion Detection in Advanced Smart Cities,” IEEE Network, 2019.

[5] Z. Kowalczuk and M. Czubenko, “Computational approaches to modeling artificial emotion an overview of the proposed solutions,” Frontiers in Robotics and AI, April, 2016.

[6] K. R. Scherer, “Vocal communication of emotion: A review of research paradigms,” Speech Communication, vol. 40, pp. 227-256, 2003.

[7] M. Suk and B. Prabhakaran, “Real-Time Mobile Facial Expression Recognition System - A Case Study,” Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 132- 137, 2014.

[8] S. E. Kahou, X. Bouthillier, P. Lamblin, C. Gulcehre, V. Michalski, K. Konda, S. Jean, P. Froumentt, Y. Dauphin, N. Boulanger-Lewandowski, et al., “Emonets: Multimodal deep learning approaches for emotion recognition in video,” Journal on Multimodal User Interfaces, 2015.

[9] F. Agrafioti, D. Hatzinakos and A. K. Anderson, “ECG pattern analysis for emotion detection,” IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 102-115, 2012.

[10] D. S. Quintana, A. J. Guastella, T. Outhred, I. B. Hickie and A. H. Kemp, “Heart rate variability is associated with emotion recognition: direct evidence for a relationship between the autonomic nervous system and social cognition,” International Journal of Psychophysiology, vol. 86, pp. 168-172, 2012.

[11] M. Zhao, F. Adib and D. Katabi, “Emotion recognition using wireless signals,” Proc. of ACM MobiCom, pp. 95-108, 2016.

[12] W. Wang, A. X. Liu, M. Shahzad, K. Ling and S. Lu, “Device- free human activity recognition using commercial WiFi devices,” IEEE Journal on Selected Areas in Communications, vol. 35, pp. 1118-1131, 2017.

[13] B. Kang, D. Kim and H. Choo, “Internet of everything: A large-scale autonomic IoT gateway,” IEEE Transactions on Multi-Scale Computing Systems, vol.3, pp. 206-214, 2017.

[14] M. Chen, P. Zhou and G. Fortino, “Emotion communication system,” IEEE Access, vol.5, pp. 326-337, 2016.

[15] H. Kim and J. Ben-Othman, “HeteRBar: Construction of heterogeneous reinforced barrier in wireless sensor networks,” IEEE Communications Letters, vol. 21, pp. 1859-1862, 2017.

[16] S. Kumar, T.H. Lai and A. Arora, “Barrier coverage with wireless sensors,” in Proc. of the 11th Annual International Conference on Mobile Computing and Networking (MobiCom), pp. 284-298, 2005.

[17] H. Kim, H. Oh, P. Bellavista and J. Ben-Othman, “Constructing event- driven partial barriers with resilience in wireless mobile sensor networks,” Journal of Network and Computer Applications, vol. 82, pp. 77-92, 2017.

[18] H. Kim, L. Mokdad and J. Ben-Othman, “Designing UAV surveillance frameworks for smart city and extensive ocean with differential perspec- tives,” IEEE Communications Magazine, vol. 56, pp. 98-104, 2018.

[19] S. Kumar, T.H. Lai, M.E. Posner and P. Sinha, “Maximizing the lifetime of a barrier of wireless sensors,” IEEE Transactions on Mobile Computing (TMC), vol. 9, no. 8, August, 2010.

[20] M. Benichou, J. M. Gauthier, P. Girodet, G. Hentges, G. Ribiere and O. Vincent, “Experiments in mixed-integer linear programming,” Mathematical Programming, vol. 1, pp. 76-94, December, 1971.

[21] J. Edmonds and R.M. Karp, “Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems,” Journal of the ACM, vol. 19, issue 2, pp. 248-264, 1972.

Performance Evaluation of Communications in Distributed Systems and Web based Service Architectures, 19

1055Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:49:09 UTC from IEEE Xplore. Restrictions apply.