UTILISATION OF KINECT SENSORS FOR THE DESIGN OF A HUMAN-ROBOT COLLABORATIVE WORKCELL

The paper deals with a present topic of utilisation of nonconventional sensors for solving the tasks of a human-robot collaboration within a shared workcell. The attention is primary focused on an exploring of possibilities of utilisation of low-cost Kinect sensors. The paper presents the methodical steps of solution of a sensor subsystem proposal within conditions of the laboratory robotised workcell, methods applied for the sensors intrinsic parameters calibration and their verification.


INTRODUCTION
Future trends in the area of a robot industrial application are directed to building workplaces intended for different levels of a human-robot collaboration.This fact results from demands of an industrial practice to achieve the highest possible flexibility of a production system to accommodate to product production change quickly.Utilisation of a synergy of robot properties (precision, repeatability, force) and human´s intelligence, flexibility and skill has its justification especially in the case of a small-scale production, where a flexible re-configuration of production system is a big advantage.The main problem of human-robot collaboration (HRC) is a safety of employees within a shared workspace.Solving of this problem is a subject of nowadays research activities of the worldwide research and development workplaces and also it follows the project Industry 4.0 vision.The research activities are focused on development of multi-level sensor systems and intelligent software that allow industrial robot to be able to detect a human presence in monitored zone in a timely manner and to adapt its activity to a level of a common col-laborative space distortion by a human.Different vision-based approaches are applied for detecting a human´s entire silhouette and/or human body´s significant parts (head, limbs).Machine and computer vision systems play there an important role.The application of suitable vision technology depends on requirements of the target application.In the case of light-weight robots it is possible to use sensors, such as: cameras, lasers, Microsoft Kinect devices, and so on [27].
The means of computer aid are effective tool used within the process of robotized workcell designing.Their utilisation enables a shortening the time needed for a workcell design and its entering into a production plant.Implementation of the first collaborative robots into production plants causes an increasing of demands put on a collaborative workcell design.The main problem is that there a human and robot collaborate within a shared common workspace to perform the intended tasks [12,23,24].The presence of a human near an industrial robot is an increased safety risk.The recommendations related to reduction of this type of risks are listed in international standards and technical specifications [8,10].The result safety level of a collaborative robotized work-cell is given by used safety elements and the way of their controlling [3].Design of collaborative robotized workcell requires an utilisation of innovative technologies consisting of functions that allow a robot to sense a position of objects in its surroundings in real time [5,17,18].The different types of 3D camera systems are used for examining a position and identification of objects freely moving in 3D space in the area of robotic systems [1,6].The current market offers many types of 3D camera systems.The most affordable solutions are RGB-D sensors, as follows: Microsoft Kinect or Asus Xtion Pro Live.Kinect devices for its low start-up costs are often used in numerous research projects [7,13,16,20].Their utilisation for solving the problem of a human-worker capturing within the monitored space of the collaborative workcell are described in [2,29].
To achieve a high-quality image data obtained from RGB-D sensors for subsequent processing by the computer vision tools it is recommended to perform a calibration process [15,19].Used calibration method and conditions at which a calibration was performed have influence on a calibration process outcome.
The paper presents our approach to the design of collaborative workcell designed for exploring the possibilities of the robot Fanuc safe control with utilisation open-source platform ROS and Kinect devices within the laboratory conditions.The aim of this paper is to highlight the influence of properties of RGB-D sensors on the safety system developed proposal.

MOTIVATION
To ensure safety at the collaborative robot workcell, authors proposed the safety system whose structure is shown in Fig. 1.Safety system consists of three safety levels.Each of them uses its own rgb-d sensor.The roles of rgb-d sensors within a single safety levels are different.The rgb-d sensor intended for the first safety level detects big moving objects, i.e. human or mobile robotic arm, that are present in the monitored collaborative space.The rgb-d sensor intended for the second safety level detects static and dynamic small or medium objects, i.e. parts, with which the human manipulates.The parts can be placed on the work desk in the different positions.The sensor intended for the third safety level senses and identifies the big objects moving within the wider surroundings of the collaborative space.
The safety system proposed for solving a human-robot safe collaboration was based on utilization of the Robotic Operating System (ROS).The reason for this decision was its openness on area of development of the new robotic applications through integration of knowledge of a robot controlling, computer vision, collision states solutions, etc. Open-source platform ROS offers the database of partial packages that accelerate and simplify a design and creation of new custom robotic applications.There is also important the mention about the open-source project ROS-Industrial that covers activities related to extending advanced capabilities of ROS for industrial applications, including area of industrial robotics [4,21].Our aim is create a method that allows designing of different kinds of collaborative robotic workcell using the Kinect sensor and Robot Operating System.

ANALYSIS OF THE POSSIBILITY OF USING THE KINECT DEVICE Description of Microsoft Kinect device
The Microsoft Kinect is low-cost device belonged to RGB-D sensors primarily intended for entertainment industry.In recent years, this device found wide application in the development of new application, particularly in robotics.The device consists of infrared (IR) projector, IRcamera and an RGB camera (see Fig. 2).The main part of device is PrimeSense sensor.Applied technology "Light Coding" based on active triangulation sensing method allows perceiving depth of sensed environment.The IR-projector emits a single beam that is split by the diffraction into multiple beams.Beams create constant pattern projected onto the sensed space.The pattern is captured by infrared camera and is correlated with a reference pattern stored in memory of the sensor.When the beams are projected on an object whose distance to the sensor is smaller or larger than that of the reference plane, the position of the speckle in the infrared image will be shifted in the direction of the baseline between the laser projector and the perspective centre of the infrared camera.These shifts are measured for all speckles by a simple image correlation procedure that yields a disparity image [11].Within the framework of a basic research we considered two types of Kinect devices: Kinect Xbox 360 and Kinect One.We compared them with regard to their frame-rate, hardware demands and their compatibility within the proposed safety system functional structure.On the basis of the testing results we select Kinect Xbox 360 for further testing the safety system.Specification of this one is shown in Table .1

Determination of the device operating parameters
Operating parameters of cameras are divided into three groups: intrinsic parameters, extrinsic parameters and distortion coefficients.The intrinsic parameters include: Focal Length (fx, fy), Focal Centre (x0, y0) and Skew Coefficients (s).Matrix of intrinsic parameters is expressed by relation (1).Extrinsic parameters represent the location of a camera in the real scene.Matrix of extrinsic parameters consists of rotation matrix R and translation vector t.They are expressed by relation (2).Interdependence between intrinsic and extrinsic parameters is expressed by relation (3).This relation is based on ideal pinhole camera model which represent ideal camera without lens.This means that ideal camera has no distortion.In relation (3) w is the scale factor; x, y are coordinates of the projection point in pixels; X, Y, Z are coordinates of 3D point in the world coordinate space.[14,24,28 [|] = [ Expressions of imperfection of real cameras are given by radial and tangential distortion.For radial factor correction are applied relations: For tangential factor correction are applied relations: Above relations take into account five distortion coefficients expressed by matrix D, where k1, k2, k3 are radial distortion coefficients and p1, p2 are tangential distortion coefficients: For determination of intrinsic parameters and distortion coefficients was used package for calibration of monocular cameras available on ROS websites [22].Used calibration program is based on algorithms offered by OpenCV library.Detailed description of the algorithms is given on OpenCV websites [14].Environment of the calibration program is shown in Fig. 3a.The layout of the calibration workplace is shown in Fig. 3b.The position of the calibrated device was not change during the calibration process.Its uniqueness was guaranteed by using a tripod.The calibration pat-tern was fixed on the flat surface board that was kept by the industrial robot´s gripper.The robot´s manipulator ensures required movement of the calibration pattern in the space.Calibration measurements were carried out according to procedure prescribed by the package´s author.During the calibration process the multiple calibration patterns with checkboard patterns having square width of 20 mm, 25 mm, 30 mm and 108 mm were used.For calibration the pattern with 108 mm square size was used other calibration equipment because of its large size.The calibration pattern was mounted on the board with a smooth surface.This board was placed on a mobile carriage and its measuring position in the workspace was ensured by a human.Calibration of IR-camera required disabling of IR-projector that creates light reflection on calibration pattern and makes the calibration impossible.
On the basis of the results of calibration of different types calibration patterns performed by two calibration techniques we can summarize obtained knowledge as follows: • in generally, values determined bay calibration do not differ more significantly,  • utilisation of the robot Fanuc for ensuring a movement with the calibration patterns brought an advantage not only of reduction of number of calibration samples needed for creation the final calibration file but also a reduction of this file computing time, • use of the intrinsic calibration parameters was not significantly reflected on a sensed image quality, • with increasing a square size of the calibration pattern the distance at which the camera was able to recognize calibration pattern was enlarging.
Calibration files with values obtained from calibration processes of calibration patterns of both 20 mm and 108 mm square widths were selected for their utilisation within the real collaborative workcell design.Table 2 and Table 3 show comparison of intrinsic parameters and distortion coefficients of the cameras.There the values of the first row represent the default calibration values given by the device´s producer.The values of the second row represent the values obtained after calibration of the first Kinect device that is intended for capturing small objects placed on the table working plate.The values of the third row represent the calibration values used for the other two Kinect devices that are intended for capturing larger objects like human or mobile robotic platform within the two monitored zones.

Creation of virtual representation of collaborative robotic workcell
For determination of the device´s extrinsic parameter it was necessary to create a virtual representation of a workcell.For this purpose the program Creo Parametric 2.0 was used.In the robotic workcell design we have taken into account the following factors: • Detection range of the device (see Table 1).
• Field of view of the device in horizontal and vertical directions (see Table 1).

• Position of the industrial robot FANUC LR
Mate 200iC within the workcell and also its workspace envelope.• Human´s position intended within the collaborative workspace.
On the basis of the Kinect device technical parameters specified by manufacturer (see Tab. 1) the virtual representation of its field of view was designed.Minimum, ideal and maximum ranges of field of view of the devices were distinguished by shades of grey and green colours (see Fig. 5).The boundaries of a human-robot collaborative workspace were established on the basis of an intersection of both human and robot workspace envelope surfaces and the upper plane of the table working plate.Workspace envelope for the industrial robot was created in Creo Parametric 2.0 with using the simulation tools of its CAE module Mechanism.There was considered the maximum possible reach of the robot arm equipped with gripping effector oriented to grip the object in the direction perpendicular to the table working plate (see Fig. 4a).The robot kinematic model was created with using a limited range of motions in each of its joints, defined according to the robot technical specification.On the basis of the special kinematic link defined between the gripper´s TCP and the plane parallel to the table working plate and also specifically defined movement for the robot´s J1 and J3 axes it was possible to simulate movement of TCP point.The TCP point described the widening circles of graduated layers.Totally 12 layers in 50 mm distances were used.By folding the end points and using contour surface, we get the final shape of the robot workspace envelope.Human body was replaced by a mannequin in the simulation environment.[26]Working envelope of the human was created on the basis of the data contained in the standards of workplace ergonomics [9].A discrete area representing the established collaborative workspace is shown in Fig. 4b.Virtual models of the working table and the facilities placed within The Laboratory Workplace of Automated Assembly have been created based on dimensions of the real objects.A com-puter-aided design of the collaborative robotized workcell including a preliminary draft of the sensors arrangement is shown in Fig. 5.

Placement of sensors at the real collaborative workcell
On the basis of both camera´s internal parameters determined after the calibration processes and the created 3D model of collaborative workcell it was possible to adjust the Kinect device´s extrinsic parameters.For this purpose it has been suggested some methodic steps, described below.
A process of the extrinsic parameters calibration was carried out with using the ROS´s visualizer tool Rviz.For the proper process of calibration was necessary to load all relevant nodes and properly configure RViz.This process was performed in four main steps.The first one represented a creation of files that contain an initial approximate position of Kinect devices in the global coordinate system.The point located in the intersection of the robot´s joint axis J1 and the table working plate was defined as an imaginary centre of the virtual environment.This point represents the centre of so-called base coordinate system of a robot.The second step included an adding the physical models of the robot Fanuc and other facilities located at the workplace of automated assembly.In the third step, we launched communication with Kinect with necessary configuration parameters of OpenNI driver.Thus it was possible visually compare the model representation of the table working plate with the real one that is sensed within the real workcell.The final step included a measurement of displacement between points of the model representation and their associated points in cloud.The differences between the points can be determined by using RViz measure tool and adjusted by varying the value in the original file.Environment of Rviz visualisation tool is shown in Fig. 6.
One of key parameters of the Kinect devices proper placement within the laboratory workspace is their recognition ability.On the basis of finding the dependence between the size of the calibration pattern and the recognition distance of Kinect device it was possible to identify the maximum distance for recognizing an object of required dimensions.Human body or a mobile robotic device belongs to group of objects with larger dimensions.For purpose of their recognition in monitored zones the first and the third safety levels were defined.The distance between Kinect device and the plane on which lies the furthest recognizable point was defined by calibration pattern with 108 mm square width.For the second safety level where will be performed manipulation with the smaller parts, was used the same procedure with 20 mm square calibration pattern.Recognition of 20 mm square calibration pattern has not been possible in more distant image areas.Along with decreasing mounting height of device was reduced size of the monitored zone.In order to capture the whole area of collaborative workspace and recognize calibration pattern it was necessary to use 30 mm square calibration pattern.Observed maximum distances were taken into account in determining the best suitable position of each Kinect device.Proposed location of device for Safety Level 2 was not confirmed.To

CONCLUSION
Created design of collaborative robotic workcell is contemplated using of innovative ways of ensuring cooperating entities.To compensate imperfections of the devices used to monitor subject behaviour was performed a set of calibration processes.On the basis of the results of performed measurements it was found that the use of different sized calibration patterns affects the value of maximum distance in which the Kinect is still able to recognize the object.Insufficient resolution capability of the used device significantly affects its usability within the intended application.Creating of the model representation of the collaborative robotic workcell in the program Creo Parametric 2.0 and its transfer into the program Rviz allowed a verification of all settings of the Kinect devices needed for the future testing of the proposed safety system.In further work we want to continue with creation of an automated system for extrinsic calibration process.

Fig. 3 .
Fig. 3. Calibration of RGB and IR cameras intrinsic parameters; a) the calibration pattern with a checkboard pattern; b) the calibration workplace with robot Fanuc

Fig. 4 .
Fig. 4. Visualisation of both human and industrial robot working envelopes (a) and the discrete area of the collaborative space (b); Creo Parametric 2.0

Table 2 .
Comparison of calibration values of RGB-camera

Table 3 .
Comparison of calibration values of IR-camera