helpfn

IEEE COMMUNICATIONS LETTERS, VOL. 25, NO. 6, JUNE 2021 2073

A Virtual Emotion Detection System With Maximum Cumulative Accuracy in Two-Way Enabled Multi Domain IoT Environment

Hyunbum Kim , Member, IEEE, and Jalel Ben-Othman

Abstract— In this letter, we introduce a multiplex cumulative virtual emotion detection system for two-way enabled multi domain IoT environment, which avoids detection holes in all IoT domains. Formally, we define a problem whose objective is to maximize cumulative accuracy of detection holes on con- dition that perpendicular barriers or horizontal barriers are constructed in every domain area. After presenting the defined problem with ILP formulation, we devise a novel scheme to solve the problem. Moreover, we evaluate the obtained results through extensive experiments.

Index Terms— Emotion, detection, system, multi, the IoT.

I. INTRODUCTION

ESSENTIALLY, the emotion is an original feature ofhuman, which distinguishes from machines. With such an original property, numerous researchers focused on the study of emotion recognition with much interest. It is a fact that if we can recognize emotion such as joy, pleasure, sadness, anger with possible high accuracy, it will affect a wide range of applications and services requiring emotion-based informa- tion with high accuracy including medical services, intelli- gent vehicular services, etc. [1]. To support those necessary emotion-based services, emotion recognition can be extracted by the scheme with facial expression, motion, voice. Also, physiological signals by wearable devices allows emotion to be derived [2]. Recently, it is possible that emotion is derived by wireless signal and its reflection [3]. Because the emotion detection scheme by wireless signal has clear advantages such as expanded coverage, improved privacy when compared with other approaches, it is highly anticipated that the emotion detection method by wireless signal will be utilized widely so as to provide emotion-based services to people consequently.

On the other hand, a barrier research branch has been studied widely by many researchers. The initial concept of barrier was proposed by [4], which allows the barrier with line-based formation in the given Region of Interest (ROI) to detect any penetration into the ROI. So, if we can build multiple barriers in ROI, it is guaranteed that any device in the

Manuscript received December 2, 2020; revised January 18, 2021; accepted February 10, 2021. Date of publication February 22, 2021; date of current version June 10, 2021. This work was supported by Incheon National University (International Cooperative) Research Grant in 2020. The associate editor coordinating the review of this letter and approving it for publication was M. Erol-Kantarci. (Corresponding author: Hyunbum Kim.)

Hyunbum Kim is with the Department of Embedded Systems Engi- neering, Incheon National University, Incheon 22012, South Korea (e-mail: [email protected]).

Jalel Ben-Othman is with Laboratoire des signaux et systémes, Université Paris-Saclay, CNRS, CentraleSupélec, 91190 Gif-sur-Yvette, France, and also with the University of Sorbonne Paris North, 93430 Villetaneuse, France (e-mail: [email protected]).

Digital Object Identifier 10.1109/LCOMM.2021.3060737

barriers can sense and detect any penetration by attackers as well as any movement by mobile objects from one to another side. Thanks to such a useful function, the concept of barrier is applied to various research branches such as security, UAVs, emotion system [5]–[7]. Also, Internet of Thing (IoT) will play an important role to realize successful advanced smart cities because various services in smart cities can be supported by numerous IoT devices and applications including medical services, transportation system, etc [8].

It is anticipated that for advanced smart cities, various emotion-based services can be facilitated in public and private areas including private home, private autonomous vehicles, public transportation system, public institutions, public parks with different objectives such as security, smart life, medical services, etc. For example, when the emergent emotion type covering anger and fury is detected in public area, the patrol service can be reinforced to prevent any terror or criminal in the area. With those high potential and applicabilities, we can apply the concept of two-way enabled barriers for emotion detection [9], which guarantees the detection of perpendic- ular and horizontal movement by person. Also, we should deliberate on multi domain environment where independent or incorporated emotion barriers are built among multi domains. However, because appropriate emotion-based services depend on the detection accuracy and the barriers with wireless signal may have detection holes with low detection accuracy, the issue of maximum accuracy without those holes should be handled carefully in two-way enabled multi domain IoT.

Based on the above observations, we introduce a multiplex cumulative virtual emotion detection system for two-way enabled multi domain IoT environment. So, our proposed model builds multi domain two-way enabled virtual emotion barriers such that perpendicular barriers or horizontal barriers are constructed in every domain region so that any detec- tion holes with low detection accuracy are resolved. Also, we formally define a problem with Integer Linear Program- ming (ILP) formulation whose goal is to maximize cumulative detection accuracy of multi domain two-way enabled virtual emotion barriers. Then, we develop a novel approach to solve the problem. Furthermore, we implement the proposed scheme through extensive simulations with various scenarios and evaluate its performance with detailed discussion.

We organize the remaining of the letter as follows. Section II introduces the proposed system and then, a problem based on ILP formulation is represented formally. In Section III, a novel approach is devised to solve the defined problem. After evaluating the simulation results of the proposed scheme in Section IV, this letter is concluded in Section V finally.

1558-2558 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:48:34 UTC from IEEE Xplore. Restrictions apply.

2074 IEEE COMMUNICATIONS LETTERS, VOL. 25, NO. 6, JUNE 2021

Fig. 1. Description of two-way enabled virtual emotion with cumulative accuracy.

II. A MULTIPLEX CUMULATIVE VIRTUAL EMOTION DETECTION SYSTEM FOR TWO-WAY ENABLED

MULTI DOMAIN IOT

In this section, we introduce the proposed multiplex cumu- lative virtual emotion detection system in two-way enabled multi domain IoT environment. Then, we formally define a problem with a consideration of ILP formulation.

A. Problem Definition

We present important definitions that are used in our system. Definition 2.1: multi domain two-way enabled virtual emo-

tion barriers(MultiDo-VEmoBar): Assume that we have a set of IoT domains D within the square-shaped whole area S. A set of IoT devices T are randomly located in S, which can detect emotion using wireless signal and its reflection. Then, multi domain two-way enabled virtual emotion barri- ers (MultiDo-VEmoBar) are barriers such that h number of horizontal barriers or p number of perpendicular barriers are constructed in every domain. So, MultiDo-VEmoBar can guar- antee that each domain detects emotion of person with at least one movement type, horizontal or perpendicular movement.

Definition 2.2 (CAMax-MultiDo): Given that a set of IoT devices T with different wireless signal ranges R within S and required numbers of horizontal and perpendicular barriers for each domain, the cumulative accuracy maximization of multi domain two-way enabled virtual emotion barriers (CAMax- MultiDo) problem is to maximize cumulative accuracy of emotion detection on condition that the required MultiDo- VEmoBar without detection holes are built in every domain.

Fig. 1 describes two-way enabled virtual emotion with two horizontal barriers and two perpendicular barriers in IoT space S. The detection holes with low accuracy may affect the cumulative accuracy in both horizontal barriers and perpendicular barriers. Fig. 2 shows the example of MultiDo- VEmoBar in multi domain IoT environment. As seen in Fig. 2, there are four domains D1, D2, D3, D4 where each domain has different square-shaped sizes and preferences of horizontal or perpendicular virtual emotion barriers in S. For example, the domain D1 is a square created by four corner points D1P1, D1P2, D1P3, D1P4 and has preference of h = 3 number of horizontal virtual emotion barriers.

Fig. 2. Example of multi domain two-way enabled virtual emotion barriers.

B. ILP Formulation

Here, based on Definition 2.1 and 2.2, we represent CAMax- MultiDo problem by ILP using the below notations. m: the total number of domains within S, (m ≤ n). n: the total number of IoT devices. f : the total number of MultiDo-VEmoBar candidates. q: the total number of required MultiDo-VEmoBar. D: a set of domains D = {D1, D2, . . . , Dm}. T : a set of IoT devices T = {t1, t2, . . . , tn}. R: a set of signal range of IoT devices, R = {r1, r2, . . . , rn}. B: a set of potential MultiDo-VEmoBar, B = {b1, b2, . . . , bf}. h: an index of the domain (1 ≤ h ≤ m). i: an index of IoT device and detection range (1 ≤ i ≤ n). j: an index of IoT device and detection range (1 ≤ j ≤ n). k: an index of MultiDo-VEmoBar candidate (1 ≤ k ≤ f ). ai,j : the detection accuracy between ti and tj . λ: the cumulative accuracy of MultiDo-VEmoBar.

Then, integer variables are defined as follows.

Ui,j =

⎧⎪⎨ ⎪⎩

1, if ti and tj have detection edge using signals

0, otherwise.

Wi,j,k =

{ 1, if ti and tj are part of bk 0, otherwise.

Xk =

{ 1, if bk is selected as MultiDo-VEmoBar 0, otherwise.

Z =

⎧⎪⎨ ⎪⎩

1, if the required MultiDo-VEmoBar is generated successfully in every domain

0, otherwise.

To solve the problem, an objective function is to maximize the cumulative accuracy of emotion detection such that the required MultiDo-VEmoBar are created in every domain. Then, the objective function (2) of CAMax-MultiDo problem is to:

Maximize λ (1)

Subject to: n∑

j=1

Ui,j ≤ 1, (∀i, i �= j) (2)

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:48:34 UTC from IEEE Xplore. Restrictions apply.

KIM AND BEN-OTHMAN: VIRTUAL EMOTION DETECTION SYSTEM WITH MAXIMUM CUMULATIVE ACCURACY 2075

Algorithm 1 Multi-Domain-Initialization Input: S, D, T , Output: Gh 1: verify every domain Dh within S where 1 ≤ h ≤ m; 2: identify four corners DhP1, DhP2, DhP3, DhP4; 3: set Gh = (V (Gh), E(Gh)) = ∅; 4: for each domain Dh in D do 5: identify four corners DhP1, DhP2, DhP3, DhP4; 6: if the domain Dh requires horizontal barriers then 7: find the maximum number of independent paths Bh

between DhP1DhP3 and DhP2DhP4; 8: set Gh ← Gh ∪ Bh; 9: else

10: search for the maximum number of node-disjoint paths Bh between DhP1DhP2 and DhP3DhP4;

11: set Gh ← Gh ∪ Bh; 12: end if 13: end for 14: return Gh

f∑ k=1

Wi,j,k ≤ 1, (∀i,∀j, i �= j) (3)

f∑ k=1

Xk · Z = q, (4)

Ui,j ≤ Wi,j,k, (∀i,∀j,∀k, i �= j) (5) f∏

k=1

m∏ h=1

ai,j · Wi,j,k · Xk · Z = λ,

(i �= j, λ > 0) (6) Constraint (2) restricts that two devices between ti and

tj have at most one edge. By constraint (3), it is forced that the edge between ti and tj is included at most once in MultiDo-VEmoBar. Also, constraint (4) imposes that the required MultiDo-VEmoBar is constructed successfully in every domain. Constraint (5) represents there exists an edge between ti and tj to select it as a part of MultiDo-VEmoBar bk. Furthermore, constraint (6) verifies that λ value is cumulative accuracy of emotion detection when the required number of MultiDo-VEmoBar are built completely in multi domain.

III. THE PROPOSED APPROACH

To solve the CAMax-MultiDo problem, two algorithms are driven with correlation. We first implement Multi-Domain- Initialization algorithm which returns MultiDo-VEmoBar can- didates for every domain with initialization graph. Then, Multi-Domain-Max-Cumulative-Accuracy-Selection algorithm is performed, which also returns the maximum cumulative accuracy value λ in multi domain.

A. Multi-Domain-Initialization

We note that MultiDo-VEmoBar candidates are generated by Multi-Domain-Initialization below so that those candidates can be utilized to solve the CAMax-MultiDo problem.

• Verify every domain with within S and its barrier con- struction requirement option.

Algorithm 2 Multi-Domain-Max-Cumulative-Accuracy- Selection Input: S, D, T, Gh, Output: λ 1: set B ←∅; 2: set λ = 1; 3: identify the required number of perpendicular or horizontal

barriers for each domain; 4: while True do 5: select bk with highest cumulative accuracy from Gh; 6: set B ← B ∪ bk; 7: set Gh ← Gh − bk; 8: calculate λ = λ × ak; 9: if the required MultiDo-VEmoBar is built for every

domain then 10: break; 11: end if 12: end while 13: return λ

• Then, the below procedures are iterated for each domain Dh where 1 ≤ h ≤ m, (∀h).

– Create a multi domain initialization graph Gh = (V (Gh), E(Gh)).

– Identify four corners DhP1, DhP2, DhP3, DhP4. – For each domain Dh, the below is implemented.

* If the domain Dh requires horizontal barriers, find the maximum number of independent paths Bh between DhP1DhP3 and DhP2DhP4. Then, the found paths are added to Gh.

* If the domain requires perpendicular barriers, search for the maximum number of node-disjoint paths Bh between DhP1DhP2 and DhP3DhP4. Also, the discovered paths are added to Gh.

• Return Gh. The pseudocode of Multi-Domain-Initialization is expressed

in Algorithm 1. The total number of iterations will be (4 × m) + (m × n2). So, the complexity of Algorithm 1 is O(n2).

B. Multi-Domain-Max-Cumulative-Accuracy-Selection

After implementing Multi-Domain-Initialization with its result with B, we execute Multi-Domain-Max-Cumulative- Accuracy-Selection scheme with the below procedures.

• Create a set of MultiDo-VEmoBar B. • Verify the required number of perpendicular or horizontal

barriers for each domain. • Then, the below procedures are repeated until the required

MultiDo-VEmoBar is built for every domain. – Choose the barrier bk with highest cumulative accu-

racy from MultiDo-VEmoBar candidates set Gh. – Add bk to B and then remove bk from Gh. – Update λ by considering cumulative accuracy value

ak of bk. • Return λ. Furthermore, Algorithm 2 provides a more detailed explana-

tion with the pseudocode of Multi-Domain-Max-Cumulative- Accuracy-Selection scheme. Because the maximum size of Gh,

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:48:34 UTC from IEEE Xplore. Restrictions apply.

2076 IEEE COMMUNICATIONS LETTERS, VOL. 25, NO. 6, JUNE 2021

Fig. 3. Comparison for cumulative accuracy of emotion detection λ by different number of devices with various total number of domains m in the area 500 × 500.

|Gh|, can be n, the total number of iterations for all domains will be m×n. Hence, the complexity of Algorithm 2 is O(n).

IV. EXPERIMENTAL EVALUATION

In this section, we evaluate the performance of the proposed Multi-Domain-Max-Cumulative-Accuracy-Selection scheme. For simulation settings, all experiments were executed in square-shaped area 500×500 m2 where the area 500×500 is divided into multiple domains whose sizes are different and various. So, the size of D1 is utilized as 150 × 150 and the size of D2 is 200 × 200. Also, D3 has its size as 200 × 200 and D4 takes its size as 120×120. The number of IoT devices n is ranging from 100 to 175, which have random positions initially. Then, the maximum sensing range of devices sets a scope from 40 m to 65 m. Also, each domain has own preferred virtual emotion barriers types (i.e. horizontal virtual emotion barriers or perpendicular virtual emotion barriers). In our experiments, horizontal virtual emotion barriers are exploited for D1 and D3 and perpendicular virtual emotion barriers are preferred in D2 and D4. Note that all numerical results obtained from experiments are an average of 100 suc- cessful results where h = 2 and p = 2.

Fig. 3(a) covers n = 100 with h = 2, p = 2 and then Fig. 3(b) show the result in case of n = 125, h = 2, p = 2. Also, Fig. 3(c) represents the performance when n = 150, h = 2, p = 2 and Fig. 3(d) include the result for n = 175, h = 2, p = 2, respectively. As it can be seen in Fig. 3, we verified that as a whole, a cumulative accuracy λ increases as the maximum sensing range of devices increases. Also, when the total number of domains increase, the cumulative accuracy λ decreases. For example, the cumulative accuracy λ with m = 1 is greater than the case of m = 2, It follows that when compared with m = 1, the cumulative accuracy λ with m = 4 decreases significantly. Moreover, we have checked that the cumulative accuracy λ increases as the total number of devices increases, which were utilized in multi domain.

V. CONCLUDING REMARKS

In this letter, we introduced a multiplex cumulative virtual emotion detection model for two-way enabled multi domain

IoT environment. We formally defined CAMax-MultiDo prob- lem with ILP formulation. To solve the problem, a novel scheme was developed with a consideration of Multi- Domain-Initialization implementation. Moreover, the proposed approach was performed by extensive experiments. As future works, we will envision various target areas for virtual emotion detection including convex hulls and the size of domains can be heterogeneous with real-life condition and the simulation can be expanded with confidence interval. Also, we will deliberate on the proposed system with location services and localization. In addition, the privacy issue and security concerns for virtual emotion information should be handled.

REFERENCES

[1] J. D. Hart, T. Piumsomboon, L. Lawrence, G. A. Lee, R. T. Smith, and M. Billinghurst, “Emotion sharing and augmentation in coop- erative virtual reality games,” in Proc. Annu. Symp. Comput.- Hum. Interact. Play Companion Extended Abstr., Oct. 2018, pp. 453–460.

[2] F. Agrafioti, D. Hatzinakos, and A. K. Anderson, “ECG pattern analysis for emotion detection,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 102–115, Jan. 2012.

[3] M. Zhao, F. Adib, and D. Katabi, “Emotion recognition using wireless signals,” in Proc. ACM MobiCom, Oct. 2016, pp. 95–108.

[4] S. Kumar, T. H. Lai, and A. Arora, “Barrier coverage with wireless sen- sors,” in Proc. 11th Annu. Int. Conf. Mobile Comput. Netw. (MobiCom), 2005, pp. 284–298.

[5] H. Kim, J. Ben-Othman, L. Mokdad, and K. Lim, “CONTVERB: Continuous virtual emotion recognition using replaceable barriers for intelligent emotion-based IoT services and applications,” IEEE Netw., vol. 34, no. 5, pp. 269–275, Sep./Oct. 2020.

[6] H. Kim, J. Ben-Othman, L. Mokdad, G. Neilson, and P. Bellavista, “Delay-bounded virtual emotion recognition using IoT barriers in advanced smart environment,” in Proc. IEEE GLOBECOM, Dec. 2019, pp. 1–6.

[7] X. Fan, F. Hu, T. Liu, K. Chi, and J. Xu, “Cost effective directional barrier construction based on zooming and united probabilistic detec- tion,” IEEE Trans. Mobile Comput., vol. 19, no. 7, pp. 1555–1569, Jul. 2020.

[8] E. Luo, M. Z. A. Bhuiyan, G. Wang, M. A. Rahman, J. Wu, and M. Atiquzzaman, “PrivacyProtector: Privacy- protected patient data collection in IoT-based healthcare systems,” IEEE Commun. Mag., vol. 56, no. 2, pp. 163–168, Feb. 2018.

[9] H. Kim, J. Ben-Othman, S. Cho, and L. Mokdad, “A framework for IoT-enabled virtual emotion detection in advanced smart cities,” IEEE Netw., vol. 33, no. 5, pp. 142–148, Sep. 2019.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on July 24,2021 at 03:48:34 UTC from IEEE Xplore. Restrictions apply.