Intelligent Visual Quality Control System Based on Convolutional Neural Networks for Holonic Shop Floor Control of Industry 4.0 Manufacturing Systems

The article presents research on industrial quality control system based on AI deep learning method. They are a part of larger project focusing on development of Holonic Shop Floor Control System for integration of machines, machine operators and manufacturing process monitoring with information fl ow in whole production process according to Industry 4.0 requirements. A system connecting together machine operators, machine control, process and machine monitoring with companywide IT systems is developed. It is an answer on manufacture of airplane industry requirements. The main aim of the system is full automation of information fl ow between a management level and manufacturing process level. Intelligent, fl exible quality control system allowing for active manufacturing optimization on the base of achieved results as well as a historical data collection for further Big Data analysis is the main aim of the current research. During research number of selected AI algorithms were tested for assessing their suitability for performing tasks identifi ed in real manufacturing environment. As a result of the conducted analyzes, Convolutional Neural Networks were selected for further study. Number of built Convolutional Neural Networks algorithms were tested using sets of data and photos from the production line. A further step of research will be focused on testing a system in real manufacturing process for able possible construct a fully functional quality control system based on the use of Convolutional Neural Networks.


INTRODUCTION
Manufacturing companies that have to compete on the open global market are more and more focused on continues quality and production systems performance improvement. Reduction of cost and optimisation of manufacturing processes are more important than in the past. Quick development and implementation of Industry 4.0 solutions are shortest the way to make possible a radical improvement of productivity, quality and resources utilisation [1].
Radical change in production systems organisation paradigm is forced also by rapid increasing of products complexity and new hard to manufacture materials development [2]. Growing sophistication of production process and connected with it technical risk is necessary [3]. Also business risk is growing rapidly because of law and fi nancial responsibility for wrong products or pour quality never was so high [4].
Such factors force manufacturing companies into rapid development of production systems, also by implementation solutions comply with Industry 4.0 idea [5]. One of the most important areas are integration of information fl ow by sophisticated IT solutions based on Artifi cial Intelligence (AI) [6]. It became especially important in time of rapid development of Internet of Things (IoT) [7], Big Data [8], cloud computing [9], Artificial Intelligence (AI), Machine Learning (ML) [10], etc. Integration of various company systems in the frame of production process and wider whole supply chain into one system is necessary [11]. Looking in the past number of ideas of manufacturing systems organisation, like agile [12], holonic [13], fractal [14], etc. were developed. At that time it was difficult to implement them because of lack of proper, easy to use sophisticated software, control and communication technology. All of pointed manufacturing systems organisation ideas are based on the common concept of a distributed and dynamic shop floor, where autonomous assets interact to overcome disturbances and problems during orders performance. One of the fundamental problems is assurance of the highest product quality and application of data and analyses results to on-line control and optimisation of production process and machines operation.
The aim of the paper is to present research carried out on development of Intelligent Visual Quality Control system based on Convolutional Neural Networks. In the future it will be a part of for Holonic Shop Floor Control of Industry 4.0 Manufacturing Systems that is during research.

INDUSTRY 4.0
The need to radical increase manufacturing systems performance and rapid development of data processing technologies, Artificial Intelligence (AI) and Machine Learning (ML) algorithms, as well as miniaturization of electronic systems force the emergence of a new paradigm -the idea of Industry 4.0. [15] This idea identifies the areas and technologies through which a radical revolution will take place in the coming years. Wide automation, data digitization, development of intelligent decision support systems will be main domains of it [16].
Full implementation of Industry 4.0 demand radical development in various areas, like control automation, drastic reduction of human workload, automatic process control, automatic monitoring of the entire life cycle of the product [17]. It will require construction of new control, supervision and monitoring systems and the related new approach to integration and intelligent data management.
Efficient use of technologies developed within Industry 4.0 systems, such as multi-criteria Big Data analytics and advanced AI and ML algorithms requires real-time acquisition of large amounts of data [18]. It have to be obtain directly from machines and their components, process, tools and manufactured parts. It is necessary to develop new monitoring and also quality control systems. They should be based on AI algorithms for data acquisition and processing.

HOLONIC SHOP FLOOR CONTROL SYSTEMS
Strong competition, changes in customer requirements and growing complexity of manufacturing systems makes flexibility of Shop Floor Control Systems more and more important. Currently IT systems became the most stiff and difficult for reconfiguration changes part of the production systems. It seems that the best idea for management of complex manufacturing and market imposed production changes, is to use small, intelligent, pattern on human behavior, goal orientated semi-autonomous units. Such units at the basic level of manufacturing should be supported by suitable AI and ML based software agents.
In the past years several new manufacturing paradigms were proposed. The most important of them are: Computer Integrated Manufacturing (CIM) [19], Intelligent Manufacturing Systems (IMS), [20] Fractal Manufacturing [14], Bionic Manufacturing [21], Holonic Manufacturing (HMS) [22]. Most of proposed paradigms were focused on manufacturing systems' improvement by development of two factors: information management and flexibility. At that time it was very difficult to implement proposed new paradigms because of lack of proper IT, computer, communication and AI technology. Built IT systems were rather demonstrators of technology, then properly operating ready to use systems. Because of rapid development of IT technology, AI/ ML algorithms, IoT and Cyber-Physical Systems (CPS) it is possible to come back to those paradigms. The holonic idea base on small autonomous, intelligent and cooperating units is very suitable for solving most of problems addressed by modern manufacturing systems requirements and Industry 4.0 postulations.
Presented in research are focused on development technology of visual intelligent control system that will be suitable for new holonic manufacturing system fulfi lling requirements of Industry 4.0 philosophy, that is currently built in the frame of larger project.

INTEGRATION OF INFORMATION FLOW IN THE PRODUCTION SYSTEMS
The integration of information and data fl ows into production control systems is crucial for future manufacturing systems. Industry 4.0 systems will generally require a large amount of diverse data. Various types of data, analyzed by artifi cial intelligence and machine learning algorithms will be necessary to optimize the control of machines, material fl ow and production processes in the entire supply chain. Such a system is developed at WUT by paper author [23]. The concept of it is show on the below fi gure.
One of the key areas of data acquisition is assurance and monitoring of products quality. Automation of the quality supervision and control process should be based on the use of artifi cial intelligence methods for data analysis and inference to make it more effi cient or even possible to do. The data obtained and the conclusions made should be compared with information from monitoring systems of machines and manufacturing processes, as well as data and conclusions from machine operators. The analysis of such a wide spectrum of data should be used to optimize on--line control of machines and manufacturing processes. These data should also be the basis for process optimizing and design of products. In the long term, the data and analysis results should allow for the reconfi guration of production systems in order to optimally adapt them to current and planned production tasks.

INTELLIGENT VISUAL QUALITY CONTROL
Quality control systems are based on various types of measurements of the degree of compliance with the parameters specifi ed by the designer of a given product. Geometric dimensions, material parameters and visual parameters are the most often measured features. The measurement of visual parameters is particularly diffi cult to automate. In this way, it is possible to verify the condition of the surface of a given product, as well as the correctness of processing or the completeness of a given product. In addition, the test can most often be carried out without the need to stop the transported objects and their manipulation. However the problem is the analysis of the acquired data and presume its correctness. Comparing with a pattern is ineffi cient. The most promising are artifi cial intelligence algorithms, including in particular deep machine learning algorithms based on neural networks. Interesting research in this area was carried out by Żabiński et al. [24]. Convolutional neural networks were used for analysis of measurement results in an intelligent condition monitoring system. The platform for rapid prototyping of intelligent condition monitoring systems and advanced computational intelligence methods was developed. Tools can be used for novelty detection as well as multiclass classification. As an example, CNC milling tool head mechanical imbalance prediction system developed with the use of deep learning is described.
Visual quality control systems can potentially be used to assess the quality of fi nished products, the quality of manufactured components as well as to assess the correct functioning of workstations and production lines. They can also be used as sources of information to control automated production stations.
Training and using machine learning models is still not eff ective in a large areas of applications [25]. It requires big data sets to teach, which is a signifi cant limitation in unit or small-lot production. These algorithms also require expensive tors integration in manufacturing system [23] computers with high computing power. However, the technology of image analysis methods is gaining a number of new tools. Since the fifties of the last century, artifi cial intelligence has remained almost exclusively a purely academic issue due to the problem with its implementation on the computers of that time [26]. Today, however, many algorithms created by previous years are available. Research presented in the paper were focused on selecting AI algorithms and tools for intelligent visual control system. Various kind of AI models were tested. Finally a convolutional neural networks were chosen. Convolutional neural networks are one of the subtypes of algorithms included in the fi eld of deep learning. They are a useful tool in tasks related to image analysis [27].

CONVOLUTIONAL NEURAL NETWORKS -AI BASED ALGORITHM CHOSEN FOR VISUAL DATA ANALYSIS
Deep learning owes its eff ectiveness in image analysis to the convolutional fi lter. The result is a new image, showing edges of any orientation or rounding. This is achieved by analyzing the input data using a convolution matrix, the values of which determine the image features detected by the fi lter. This section discusses convolutions on the example of a transformation that detects the vertical edges of an image (Vertical Sobel kernel with a size of 3×3).
Assuming the input data in the form of a 5×5 array: We apply a convolution fi lter, multiplying each of the input values by the corresponding elements from the convolution matrix. The methodology for applying the fi lter is shown on the fi gure below.
It is worth noting that using this operation reduces the size of the data to 3×3. It is possible to apply a fi lter that does not change the size of the data array by adding an artifi cial "frame" of values around it (so-called stride). For the example discussed above: The fi lter has specifi ed areas of the input image where there are vertical edges. Using other convolution matrices, it is possible to detect straights of any orientation, as well as rounding. These operations occurring individually do not allow to recognize Fig. 2. Convolution fi lter -the idea of work [28] complex features, but the convolution layers created from them are already capable of this. The result of transforming the sample image using the Vertical Sobel kernel is shown in the fi gure below.
Convolutional layers consist of many convolutional fi lters, the values of which are selected when teaching the model in the regression process. This is a signifi cant advantage compared to the fi lters predetermined by the programmer in classical image analysis, because the algorithm has full freedom in choosing features that are helpful for classifi cation -the program is able to notice more dependencies occurring between the input data than a human is able to do.
There can be distinguished low, mid and high level features. For example 32 fi lters returns lowlevel features, i.e. lines at any angle or rounding. This is not enough to obtain a reliable image classifi cation result on their basis. However, there have been found that overlapping multiple convolutional layers allows the algorithm to "see" high-level features. Following fi gure shows a visualization of the features detailed by the subsequent hidden layers.
As it can be seen, the features of the medium level are characterized by much more complex shapes. The features of the high level are able to detect even such elements of the image as the wheels of the car. They depend on the learning data and what the algorithm will pay the most attention to during the learning process -for example, a model that recognizes faces in its last layers will reveal human eyes, ears or mouth.

DATASET AND PREPROCESSING OF CASTING PRODUCT IMAGES
The problem being solved consists in the quality control of cast products -impellers of submersible pumps. The issue consists in the classifi cation of elements that do not meet the  [30] quality requirements, i.e. have defects typical of the foundry process. The data set containing images of these parts was made available by Pilot Technocast and made available by the user Ravirajsinh Dabhi on the www.kaggle.com portal. [31] A dataset consists of two classes: • Products without defects Collection size: 3137 images • Products with defects Collection size: 4211 images In order to develop an algorithm for a convolutional neural network, it was necessary to separate the data into two sets: teaching and testing. After this process, the distribution of the data is as follows ( Table 1). The next step in preparing a data set is preprocessing. One of the biggest challenges when creating a neural network algorithm is to obtain a suffi ciently large set of data of satisfactory quality, which is often too time-consuming or expensive. To deal with this issue, a technique called data augmentation is used.
TensorFlow Keras provides tools for this task that automatically perform operations to transform images. When choosing algorithms that will be used in the learning process, you should pay attention to whether they are suitable for the problem being solved. For example, for the data set under consideration, copying and rotating images with a random angle in the range from 0 to 360° is very benefi cial. The fronts of the impellers of submersible pumps have an axis of symmetry passing perpendicular to them, and defects can be on their surface in any position and orientation. Using this technique, it is possible to increase the number of valuable examples in the learning set using a single line of code, thus increasing the chances of creating an algorithm of satisfactory quality.
In the process of training the neural network, the following treatments were used: • Rotation by random angle from 0 to 360°; • Rotation in the horizontal axis; • Vertical axis rotation; • 10% zoom in/out; • Shear mapping by 20%.

DEVELOPMENT OF ALGORITHM BASED ON CONVOLUTIONAL NEURAL NETWORKS
In the process of solving the problem of classifi cation of rotors, 6 models were created. Depending on their hyperparameters (network architecture, optimizer, input size), the following results were obtained: Model number six was chosen as the fi nal algorithm. Only the highest precision model on the test set is presented. Model No. 6 is summarized using the TensorFlow library as follows: There are 10 920 769 of all parameters in six model. The size of the input image is 300×300 pixels. The model architecture consists of the following layers: • input layer in the form of an image with dimensions (300, 300), not included by the Ten-sorFlow library in the above summary. • conv2d -the fi rst convolution layer creating 16 fi lters, the data is in the form of a three-dimensional array (298, 298, 16) with 160 trained parameters. • max_pooling2d -a grouping layer aimed at reducing the size of the data by half, i.e. also reducing the number of calculations performed.   The architecture of the created convolutional neural network can also be presented graphically (Fig. 8).

TESTS OF DEVELOPED DEEP LEARNING ALGORITHM
Tests of developed model shows that it achieves a precision on the test set 99.820%. With the help of a simple equation, it is possible to determine how many images from the test set were classifi ed incorrectly: 2) The defect is invisible in the photo.
3) The camera used to take the photo does not offer a suffi ciently high quality image.
Taking into account how high the precision of the model is and analyzing the only mistake made by it, it can be safely said that the algorithm is ready for use. Despite the satisfactory performance, however a tests on data from real production line have to be done.

FURTHER RESEARCH DIRECTION
The created algorithm is implemented in the form of a computer program. With its use, it is necessary to expand and equip a system using physical devices built in an industry standard. It is necessary to transfer it to the edge device and integrate it with actuators. In the case of complex installations, it is necessary to create Fig. 8. Visualization of the created model a network of edge devices operating at separate production stations along with the centralization of the acquisition of statistics or the calculations themselves. Convolutional neural networks are characterized by high parallelism of calculations, which means that GPUs cope with them much better than CPUs [32]. Such processors must be used in the finished system. Further research will consist in checking the limit numbers of the size of learning sets, at which the algorithm maintains satisfactory accuracy of prediction. This is necessary to be able to apply this type of solutions for small-series production.

CONCLUSIONS
The aim of the research was to develop intuition about the operation of convolutional neural networks at a detailed level and to solve the problem of their application to the visual quality control system, which in the future may be an element of the Holonic Control System being built. As a part of the system test, the quality of the impeller castings of submersible pumps moving on the belt feeder was evaluated.
Using the TensorFlow Keras library in the Python environment, 6 neural networks with different models were trained and their precision was compared. For the best rated, its architecture was presented, analyzed and presented.
The solution has been critically evaluated for putting it into service. Further directions of project development were also proposed, consisting in the implementation of a neural network model in the form of a complete automation system and the creation of a network of devices for product quality control located in many places in the factory.
The neural network created for visual quality control of impellers of submersible pumps is characterized by a precision of 99.820% (returns erroneous prediction of product quality 18 times out of 10,000 examples). The only example from the data set for which an error appears was considered to be an incident not resulting from the imperfection of the created model.
The results of the research presented in the article are part of a research project aimed at developing a system of data integration in manufacturing systems [23,33,34].