reading review

The International Journal of Advanced Manufacturing Technology (2019) 101:119–135 https://doi.org/10.1007/s00170-018-2788-x

ORIGINAL ARTICLE

Gesture-based human-robot interaction for human assistance in manufacturing

Pedro Neto1 · Miguel Simão1,2 · Nuno Mendes1 · Mohammad Safeea1,2

Received: 13 June 2018 / Accepted: 28 September 2018 / Published online: 30 October 2018 © Springer-Verlag London Ltd., part of Springer Nature 2018

Abstract The paradigm for robot usage has changed in the last few years, from a scenario in which robots work isolated to a scenario where robots collaborate with human beings, exploiting and combining the best abilities of robots and humans. The development and acceptance of collaborative robots is highly dependent on reliable and intuitive human-robot interaction (HRI) in the factory floor. This paper proposes a gesture-based HRI framework in which a robot assists a human co-worker delivering tools and parts, and holding objects to/for an assembly operation. Wearable sensors, inertial measurement units (IMUs), are used to capture the human upper body gestures. Captured data are segmented in static and dynamic blocks recurring to an unsupervised sliding window approach. Static and dynamic data blocks feed an artificial neural network (ANN) for static, dynamic, and composed gesture classification. For the HRI interface, we propose a parameterization robotic task manager (PRTM), in which according to the system speech and visual feedback, the co-worker selects/validates robot options using gestures. Experiments in an assembly operation demonstrated the efficiency of the proposed solution.

Keywords Human-robot interaction · Collaborative robotics · Gesture recognition · Intuitive interfaces

1 Introduction

Collaborative robots are increasingly present in manufactur- ing domain, sharing the same workspace and collaborating with human co-workers. This collaborative scenario allows to exploit the best abilities of robots (accuracy, repetitive work, etc.) and humans (cognition, management, etc.) [1, 2]. The development and acceptance of collaborative robots in industry is highly dependent on reliable and intuitive human-robot interaction (HRI) interfaces [3], i.e., making robots accessible to human beings without major skills in robotics. Collaborative robots and humans have to under- stand each other and interact in an intuitive way, creating a co-working partnership. This will allow a greater pres- ence of collaborative robots in industrial companies which are struggling to have ever more flexible production due to consumer demand for customized products [4]. For exam- ple, a human-robot collaborative platform for constructing

� Pedro Neto [email protected]

1 Department of Mechanical Engineering, University of Coimbra, Coimbra, Portugal

2 Arts et Métiers, Lille, France

panels from preimpregnated carbon fiber fabrics in which the human and robot share the workspace promoting sit- uation awareness, danger perception and enrichment of communication [5].

Instructing and programming an industrial robot by the traditional teaching method (text and teach pendant based methods) is a tedious and time-consuming task that requi- res technical expertise [6]. In addition, these modes of robot interfacing are hard to justify for flexible production where the need for robot re-configuration is constant. Recently, human-robot interfaces based in robot hand-guiding (kines- thetic teaching) and haptic interfaces demonstrated to be in- tuitive to use by humans without deep skills in robotics [7]. Advanced and natural HRI interfaces such as human ges- tures and speech still lack in reliability in industrial/unstruc- tured environment [8]. An interesting study reports the im- pact of human-robot interfaces to intuitively teach a robot to recognize objects [9]. The study demonstrated that the smartphone interface allows non-expert users to intuitively interact with the robot, with a good usability and user’s experience when compared to a gesture-based interface. The efficiency of a conventional keyboard and a gesture- based interface in controlling the display/camera of a robot is presented in [10]. The gesture-based interface allowed smoother and more continuous control of the platform,

120 Int J Adv Manuf Technol (2019) 101:119–135

Fig. 1 Overview of the proposed gesture-based HRI framework

while the keyboard provided superior performance in terms of task completion time, ease of use, and workload.

Making an analogy with the way humans interact and teach each other, allows us to understand the importance of gesture-based HRI. Static gestures are human postures in which the human is static (small motion like body shaking can occur) and dynamic gestures are represented by a dynamic behaviour of part of the human body (normally the arms). Gestures can be used as an interface to teleoperate a robot, allowing to setup robot configurations and combine with other interfaces such as kinesthetic interface and speech. For instance, a human co-worker can point to indicate a grasping position to the robot, use a dynamic gesture to move the robot to a given position and use a static gesture to stop the robot [11, 12]. This scenario allows the human co-worker to focus on the process task and not in the robot programming [13].

Figure 1 illustrates the proposed framework. Static and dynamic gesture data are acquired from upper body IMUs, segmented by motion, and different ANNs are employed to classify static and dynamic gestures. Recognized gesture patterns are used to teleoperate/instruct a collaborative robot in a process conducted by a parameterization robotic task manager (PRTM) algorithm. The system provides visual and speech feedback to the human co-worker, indicating to the user what gesture was recognized, or if no gesture was recognized.

Depending on the industrial domain and the company itself, the shop floor presents restrictions to the technologies used in the manufacturing processes. The implementation of human-robot collaborative manufacturing processes is today a main challenge for industry. Beyond the related human factors, the advanced human-robot interfaces (gestures, speech, hybrid, etc.) are constrained by the shop floor conditions. In noisy environments the human- human verbal communication is difficult to achieve or prohibitive in some cases, especially when the workers are

using earplugs. In this scenario speech interfaces are not efficient and gesture interfaces are a valid alternative. On the other hand, confined spaces hamper the use of arm gestures. In these conditions, the design of the collaborative robotic system has to be adapted according to the specific manufacturing conditions.

In this study, we assume that the shop floor environment is noisy and not confined in space, so that gestures are used to interface with the robot. Our proposed approach brings benefits and it is practically relevant in the context of flexible production in small lot sizes [8, 14], namely:

1. The human co-worker and robot work in parallel, while the robot is ready to assist the human when required;

2. The use of the robot reduces the exposition of the human co-worker to poor ergonomic conditions and possible injuries (through hand-guiding the robot can be adjusted online to the human body dimensions);

3. The use of the robot reduces error in production since the work plan is strictly followed and managed by the PRTM;

4. The robot assists the human in complex tasks that cannot be fully automated, reducing the cycle time;

5. The introduction of the collaborative robot improves the quality of some tasks when compared with human labor;

6. The collaborative robot allows to reduce drastically the setup time for a new product or variant of a product. This is critical in small lot production.

This work was developed according to the needs of the project ColRobot,1 which intends the development of a collaborative robot for assembly operations in automotive and spacecraft industry. The robot should be able to assist workers, acting as a third hand, by delivering parts and tools for the assembly process.

1https://www.colrobot.eu/