An automated optical shelf welding pre-inspection system

Since Germany proposed Industry 4.0 in 2013, it has driven the industry toward the goal of automation and smart manufacturing. The COVID-19 epidemic in the past two years has accelerated this development trend. Manufacturing production lines will inevitably develop toward automation and intelligence in the future. In the development of the manufacturing industry, welding technology has always been the most indispensable part of the overall manufacturing process. In the general traditional manual welding operation, the operator can weld and inspect at the same time to ensure the quality of the welding. However, our implemented system applied an automated optical inspection technique to pre-inspect the metal wire of the shelf before welding with the parts. The operators use this system for collaborative inspection, which reduces the workload of manual inspection and improves the work efficiency and quality of shelf welding. After the on-site experiment, the system's functions are also expanded according to the production line requirements, especially the automatic parameter function provided to reduce the complexity and time of on-site parameter adjustment.


INTRODUCTION
According to the industrial production statistics of the Department of Statistics of the Ministry of Economic Affairs (R.O.C.) in January 2022, the production index of Taiwan's manufacturing industry increased by 10.69% compared with the same period last year (MOEA, 2022).Under the population decline and the COVID-19 epidemic, manufacturing production lines are bound to develop towards automation and intelligence in the future.Welding technology has always been the most important component of the manufacturing process as it has evolved.The technology can be widely used in various large and small manufacturing fields, such as aerospace, machinery, bicycles, daily necessities, and other fields.
This research cooperation with a storage equipment manufacturing company.The company produces a variety of storage equipment, which can be applied to restaurants, warehousing, medical care, stores, refrigeration supply chains, and households.One of the company's production processes for a storage shelf is shown in Fig. 1.There are two processes to form a finished product, which are metal mesh processing and metal tube processing.In the metal mesh processing, the metal wire rod is cut and formed into a mesh shape and waveshape.Next, the operator uses welding to fix the shape and combine the support parts.Then, a single-layer metal mesh of the shelf is done.In another process, the metal tube cuts the straight tube into a particular length, then rolls around, deburr, plugs, and taps the tube.Finally, a finished product will be finished after combining the metal mesh and tube.All the inspection and welding processes are done manually in the company's production line.
A manual manufacturing line will always have some precision issues, producing several low-quality products.If there are some precision issues when cutting or forming the shape before welding, the welding process will become bad.The Not Good (NG) welding result provided by the company is shown in Fig. 2.There are various reasons for the NG welding issues.For example, the wave line on the short side is uneven or deviates outward, the wave line is offset, the length of the welded mesh is insufficient, and there are more forming precision issues before welding.Therefore, we developed an automated optical preinspection system to measure and inspect the shelf's quality before welding to prevent NG welding.The system needs to alarm the on-site operator to stop continuing welding to prevent the NG welding result.In simple terms, the motivation of this research is to develop a system that can automatically inspect if a mesh is suitable for the next welding operation.The proposed system helps prevent the waste of reproduction time and money caused by dealing with too many defective products during the welding process.
The proposed system uses a mono-color industrial camera to take two diagonal corners of the four corners of the single-layer mesh shelf before welding the support parts.The capture method will be from bottom to top, and then external I/O to use 3.3-12 V triggers the camera to capture an image.The length difference between the end of the upper metal wire and the lower metal wire at the corner of the shelf mesh can be captured in the image, as shown in Fig. 3.Then, the system measures the captured metal wire images and calculates the value of the length difference between the upper and lower metal wire.The measured result can be used to make decisions by the on-site operator.The system reduces the time of manual measurement and improves the production efficiency and quality of the shelf.
Previously, in the growth of the manufacturing industry, the traditional manual technique was used to inspect and validate manufactured products to assure production quality.However, automated optical inspection technology can replace humans to perform inspections and measurements on the industrial manufacturing site, identification, and judgment tasks.It can reduce workers' mistakes caused by work fatigue and also help to improve accuracy and production efficiency.
In the weaving machine industry, defects are a considerable problem in controlling the quality of fabrics, and defects may appear in the production stage.Vladimir et al. (2019) proposed a fabric defect inspection method based on image processing.On the other way, the process will cause some defects on the surface of the wire due to steel production.Zhang et al. (2010) proposed a wire surface detection method based on image processing to detect the surface defects of the wire.
Industrially produced electrification grids are bulky and have many connection points.Aiming at traditional manual inspection's poor reliability and low efficiency, He et al. (2020) proposed a machine vision-based defect detection method for energized power grids.The experimental results show that the proposed scheme can detect energized power grid defects with an accuracy rate of about 97.5% to meet industrial production needs.
In the ceramic tile industry, it is challenging to manually monitor the quality of each tile in the tile industry.This requires a lot of human resources and takes much time to implement.Shire et al. (2011) proposed a method based on image processing.Automated defect detection and classification technology improved tile quality and productivity during manufacturing.Wu et al. (2015) proposed an AOI system to inspect golf clubs.The inspection rate can reach 88%, which is close to human visual inspection.Due to the solid reflective properties of steel, subtle defects, and slender and tapered body, it is challenging to take high-quality images for automatic inspection.
In addition to the applications highlighted previously, the following paragraphs delve into other literature discussing the utilization of AOI systems in industrial manufacturing applications.Sidla et al. (2001) proposed a machine vision system for automatically straightening steel bars up to 6000 mm long using two cameras as the 3D camera to measure the bar's shape.In addition, Zhang et al. (2018) developed a high-precision quality inspection system for steel bars by utilizing the sub-pixel boundary location method (SPBLM) and fast stitch method (FSM) on the construction site.This system demonstrates exceptional accuracy in calculating steel bar diameter, spacing, and quantity, with an absolute error as low as 0.04 mm, a relative error of 0.002%, and zero counting error.Those two system setup is complex, and it will be hard to deploy into multiple production lines.Tan et al. (2021) proposed a method to improve machining accuracy by measuring shaft diameter in realtime during the finishing process.They implemented a structured light system with a laser linear light source and a camera to create elliptical light stripes on the shaft's surface.The accuracy is validated through testing in a maximum average measurement error of 0.019mm at 36.162mm shaft diameter and 1250r/min speed.However, the measurement accuracy of the model is affected by the precision of detected points on the ellipse, which can be influenced by surface roughness and shaft rotation speed.Frustaci et al. (2020) proposed a machine vision system designed for precise and cost-effective in-line geometric inspection of assembly processes, focusing on the catalytic converter assembly process.The system utilized image segmentation to identify the region of interest (ROI) and a geometrical model to detect possible defects caused by interface shifts.Their prototype on a Raspberry Pi 3 demonstrates a cost-effective, real-time quality check method.Martinez et al. (2019) introduced a vision based on the manufacturing of steel frame assembly pre-inspection system.The proposed method successfully identifies, validates, and rectifies frame assembly issues by comparing industrial camera data with building information model (BIM) information.The proposed system offers potential improvements and efficiencies in the construction industry.The computation time of pre-inspection in the manufacturing process is long but not essential; as the authors stated, the material waste and time are much more important to them.Thakrea et al. (2019) presented the development of a machine vision system for directly measuring the flank wear of carbide cutting tool inserts in the machining industry.The proposed system utilizes a camera, a light source, and image processing to capture and analyze images of carbide inserts, extracting tool wear parameters.Experimental validation showed a 3% average error in measuring tool wear width compared to digital microscope measurements.Yu et al. (2021) also proposed a machine vision-based method for measuring chisel edge wear of drills, utilizing image enhancement and contour extraction techniques to calculate the wear area.The effectiveness of the proposed methods in tool wear measurement surpasses other traditional methods.
Moreover, various methods and hardware architectures, including cameras, light sources, sensors, controllers, and communication devices, have been proposed for enhancing the measurement and inspection systems.Xiao et al. (2019) developed a system to calculate the straightness and roundness of the spun workpieces.In the image processing method design, the authors design with regions of interest, image deblurring with Wiener filter, image de-noising with Gauss filter, and edge detection by Canny operator.Chen Ting (2021) also developed a system for measuring workpieces.The author introduces the industrial camera, lens, and library to develop the system.The library that the author utilizes is the HALCON Vision library, which is a fully functional library for machine vision system development.Using the industrial machine vision library can improve system stability and speed while also expediting development and deployment for practical use.Besides the industrial library, more researchers in difference fields (Yan, 2018;Du, 2021;Fang, 2021;Saif, 2021;Yang, 2021;Balasubramanian, 2022;Hadiyoso, 2022;Xiong, 2022) also implemented the system with the OpenCV library, which is an open-source library.The utilization of the open source has the flexibility to modify the base function.It is still crucial to create a system that can accurately measure various types of workpieces or objects.Various types of cases can be found in different works., for example, hole saw caps (Lu, 2021), small machined parts (Zhang, 2021), conical spun workpiece (Zhang, 2022), milled workpiece roundness holes (Saif, 2022), outer dimensions of the workpiece (Liu, 2023), etc.
The previous research found that most of the study is based on general AOI systems with multiple machine vision methods in many cases.Although the machine vision techniques are mature, there is still a need to design an optimized method to fit the application scenario.However, no research is the same as our case study in the inspection of shelf mesh's metal wire, and we are trying to take advantage of the previous research as a reference to design our system in this study.Different products to be inspected need to design a set of different inspection processes to fully comply with the characteristics or production process of the inspected product.However, some studies are done on highreflection material detection, which we can reference to avoid misjudgment problems caused by high reflection.This research focuses on the inspection before welding to ensure that the product will not need to be remade due to direct welding without inspection, resulting in wasting a lot of time.
The following sections are organized as follows: Section 2 introduces and explains our proposed system architecture.In Section 3, we conducted various on-site experiments and presented data to demonstrate the system's viability.Section 4 shows and introduces the system interface we implemented.Section 5 discusses the proposed system.Finally, a summary of the research is given.

PROPOSED METHOD
In this section, we introduce the method of the proposed system, which includes the important hardware we selected and the algorithm we implemented.

System Architecture
The hardware architecture of the system is shown in Fig. 4. Firstly, we connect the 5V voltage from the power supply to the trigger signal.Then, the camera is controlled to capture images through the 5V trigger signal of the foot switch.Finally, the camera sends the captured images of the corners of the shelf mesh to the personal computer (PC) for inspection.The camera applied in this system is a DMK 33GX264e GigE black-and-white industrial camera (The Imaging Source, 2023) and installed with an MI-1620-10M megapixel lens (Sure Technology Corporation, 2023).Our development language is C#, and the integrated development environment uses Microsoft Visual Studio 2019 through the black-and-white industrial camera software development kit (The Imaging Source 2, 2023).Then, we apply the industrial image processing library Open eVision (Euresys, 2023) to develop a measurement and inspection method.
Fig. 5 shows a flowchart of the proposed system.The operator steps on a foot switch to trigger the external I/O and control the camera to acquire the pre-inspection image.The camera sends the image to the inspection system, and then starts the image processing and inspection functions.
The detailed steps description of the flowchart are as follows: 1. Image calibration: In the part of the measurement, it is necessary to place a dot calibration chart with an interval of 2mm on the area to be inspected because the length of the object in reality and the image is different.The operator uses the calibration function of Easy Gauge in the Open eVision library to calibrate the coordinates

EXPERIMENTAL RESULT
In the experiment, we place the shelf mesh on the table and attach it to the corner where we will measure.Next, set up a light-absorbing, non-reflective black cloth on the bottom of the mesh corner, then change the light source's illumination location and capture the pre-inspection image from top to bottom with the camera, as shown in Fig. 9.

Fig. 9. Metal mesh corner inspection setting with lightabsorbing black cloth
Initially, we defined that the length from the reference point to the lower metal wire is called the A value; the length from the reference point to the upper metal wire is called the B value; the length between the upper and lower metal wire is called the C value.The reference point is the point defined by the user.We used a vernier caliper to measure the length difference between the experiment's upper and lower metal wire.We set the compensation value of the system C value based on the actual length to reduce the error between the system measurement values and the actual value.
In the experiment, we completed an actual measurement on the company's manufacturing line.Firstly, we captured 50 images for the test and made statistics on the measurement results.Figs. 10 (a) and (b) are bar charts of the distribution of C values of all vertical and horizontal metal wires, respectively.The vertical axis is the number of images; the horizontal axis is the range of C values, which can be seen in these two charts in the part of the vertical side, the number of C values of 2.5-3 mm is the largest, followed by 3-3.5 mm.In the part of horizontal metal wire, the number of C values of 2-2.5 mm is the largest, and the second is 2.5-3 mm.From these two distributions, we can realize the general distribution of the vertical and horizontal side's C values in the production line.In the next experiment, we measured 10 metal meshes.Each mesh measured 4 C values at one corner and its opposite corner.After the measurement is completed, the 10 metal mesh are sent to the robotic arm for welding, and the welding quality of the finished product after welding is judged manually.The judgment result is compared with the corresponding C value to analyze the appropriate value for the threshold.
Currently, only 10 pieces of mesh are used for testing and analysis, and the analysis results have not been able to see the correlation between the size of the C value and welding quality, as shown in Table 1.More test data may be required to continue analyzing in the future.

IMPLEMENTATION
In this section, we introduce the functions of some essential user interfaces of our implemented system.Fig. 12 is the main interface of the system.Each of the functional areas on this interface is divided into A-D areas.The details are as follows.1.The area is where to set the length threshold of every inspection item.The threshold can be prevented from accidental change through the [Lock] button at the bottom right side.The error from the actual size can be corrected by setting the compensation value.Finally, press the [Set to save] button to save the settings.2. This area shows the number of images inspected by the system, and it can be reset to zero by pressing the [Counter Reset] button.If the user finds a false inspection in the current inspection result, they can press the [False Inspection] button to save the false inspection image to the false inspection folder and record it in the CSV file.
A false inspection function can let the engineer know there are some problems with the inspection result.The engineer can fine-tune the parameters later.Lastly, the user can press the [Start Inspection] button to start or stop the inspection function.3. The status column shows the two cameras' status.The green color shows the camera is normal, while the red color means the camera is abnormal.4. The inspection results of Camera 1 and Camera 2 are divided into blue and purple areas.If a C value exceeding the threshold is detected vertically or horizontally, an "NG" result will be displayed in the upper-left or lowerleft corner of the image.Otherwise, an "OK" result will be displayed if the C value does not exceed the threshold.Step 1, the user presses the [Set Target Point] button.Then, the user needs to set the position of the target point on the image.Step 2, the user needs to press the [Parameter Auto Adjustment] button to adjust the parameter automatically.Finally, the threshold and the minimum amplitude value will adjust automatically to fit the measurement condition of the target point.The function of automatic parameter setting is only available in Point Gauge because the correct black-and-white turning point position is successfully marked.To implement the function of automatic parameter setting, 100 images of vertical metal wire were used for testing.Firstly, try to adjust the parameters of the Point Gauge to determine the primary influence on finding the turning point of black and white.The critical parameters found after trying are the threshold value and the minimum amplitude, and then the ideal value of the automatic parameter setting is set to the intermediate value between the threshold value and the minimum amplitude for the next test.After the user sets all the parameters of vertical or horizontal side measurement, the inspection system can be started by pressing the "Start Inspection" button in Fig. 12.If the result is a misjudgment, the user can fine-tune the parameters again by using the parameters setting interface.

DISCUSSION
In Section 4, we did some experiments to verify the feasibility of this detection system.The system is available online and is a reference for whether the subsequent robotic arm can continue welding.Due to the production line's limitations and the object's size in the experimental environment, the results of our experiments could not be obtained immediately with more accurate standard optical measuring instruments for the actual measurement.However, the standards and conditions provided by the manufacturer have been met, and the current system has no issues in use.
Moreover, to reduce the trouble of on-site operators adjusting parameters, we have implemented automatic adjustment parameters on the system, such as threshold value and minimum length value, which significantly reduces the problem that it takes much time to adjust every product specification.Our automatic parameter adjustment's convenience is better than the previous studies we studied before.

CONCLUSION
In this research, we designed a foot switch I/O to control the mono-color industrial camera to capture the image of the metal wire at the corner of the shelf and then used the image processing technique to measure the length difference between the upper and lower metal wire.The measurement result will be judged according to the threshold set by the user to check whether it meets the manufacturing requirements.
Overall, we have sorted out the significant contributions of this research as follows: 1. Development of AOI system: The research aimed to create a foot-switch-controlled system using an industrial camera to capture images of metal wires on shelves.These images were then analyzed to measure the length difference between upper and lower wires, with userdefined thresholds for quality assessment.2. Improved inspection accuracy and efficiency: The system outperforms traditional manual inspections by providing more precise pre-inspection of welding conditions.It reduces manual workload and inspection time, enhancing manufacturing quality and efficiency.3. Data collection and analysis: The system collects images and data for subsequent analysis.Statistical analysis focused on the C value distribution for vertical and horizontal wires and the distribution of qualified and unqualified parts with a threshold range of 1.75-2.75mm.

Enhanced functionality with automatic parameter setting:
The system became more user-friendly through optimizations and adding new features, particularly with automatic parameter setting.Operators at manufacturing sites experience simplified and quicker on-site parameter adjustments.5. Practical application on-site: The system has been successfully tested and proven to be practical in real manufacturing environments, offering a valuable tool for improving product quality and production efficiency.The system has been experimentally proven to be usable in the manufacturing field.However, this system only partially solves all the production automation of the manufacturer, which is a preinspection before the welding process.We have planned and conducted experiments to achieve robotic arm-assisted welding in future work.The results measured in this study will be used as parameters of the robotic arm to automatically perform welding based on the distance parameters provided by the system.Therefore, the workload and training of professional welding workers can be reduced, and the factory automation capabilities can be improved in the future.

Fig. 4 .Fig. 5 .Fig. 6 .Fig. 7 .
Fig. 4. Proposed system architecture Fig.11shows the distribution of all vertical and horizontal metal wire inspection results.The vertical axis is the number of images; the horizontal axis is the result of the metal wire.The inspection results are divided into two types: OK and NG.The OK result determines whether the length meets the threshold value of 1.75-2.75mm; otherwise, it is NG.The figure is divided into three types of results: vertical side OK/NG, horizontal side OK/NG, and both sides OK/NG.It can be seen that half of the results of the vertical metal wire are OK, while there are only 5 OK results for the horizontal metal wire part.The proportion of horizontal metal wire NG is too high; the OK results are reduced to only 4 in the final result.In the next experiment, we measured 10 metal meshes.Each mesh measured 4 C values at one corner and its opposite corner.After the measurement is completed, the 10 metal mesh are sent to the robotic arm for welding, and the welding quality of the finished product after welding is judged manually.The judgment result is compared with the corresponding C value to analyze the appropriate value for the threshold.Currently, only 10 pieces of mesh are used for testing and analysis, and the analysis results have not been able to see the correlation between the size of the C value and welding quality, as shown in Table1.More test data may be required to continue analyzing in the future.

Fig. 11 .
Fig. 11.Distribution of vertical, horizontal, and both sides inspection result

Fig. 13
Fig.13shows the parameter setting interface for vertical/horizontal side metal wire measurement.Since the parameter setting interface for vertical and horizontal metal wire measurement is the same, only the vertical metal wire measurement is demonstrated as an example.The interface included a calibration map setting, ROI of the vertical/horizontal side, EasyMatch model file importation, PointGauge parameters, and LineGauge parameters.Each functional area on this interface is divided and explained in the A-E area as follows.1.The user can set parameters or calibration images in this area.In the [Set Parameter Setting], press the [Read Measurement Image File] button to read the image from the camera.Then, the image will be shown on the righthand side.Press the [Set Reference Point] button to click on the image to set the reference point.2. The user presses the [Read Parameter File] button to read the existing parameter file.Then, the setting of the ROI range can be directly dragged on the image with the mouse or manually entered in the X, Y, width, and height fields.The user presses the […] button to load the model file.The model file included the contour feature points of the metal wire, which is pre-learned in the development phase.The model file is used to find the position of the vertical or horizontal side metal wire in the inspection phase.3. The user sets the Point Gauge and Line Gauge parameter values in this area.The point and line measurement tools represent Point Gauge and Line Gauge, respectively.The user can also set the measurement position by pressing the [Set Target Point] button.Press the [Automatic Parameter Adjustment] to automatically adjust the threshold and the minimum amplitude of the Point Gauge to a suitable value.4.You can choose whether to display Point Gauge, Line Gauge, and measurement results on the image, respectively.Press the [Measure] button to measure the part of the vertical or horizontal metal wire.Press the [Parameter Save] button to save the parameter settings in the front.After pressing the […] button at the bottom, the user can set the storage location of the parameter file. 5.The image coordinates (unit: pixel) and world coordinates (unit: mm) of the mouse on the image will be displayed on the status bar.The coordinates are updated in real-time if the user moves in the image area.Fig. 14 shows the steps of setting the parameter automatically.Three steps of description correspond to the figure as follows.Step 1, the user presses the [Set Target Point] button.Then, the user needs to set the position of the target point on the image.Step 2, the user needs to press the [Parameter Auto Adjustment] button to adjust the parameter automatically.Finally, the threshold and the minimum amplitude value will adjust automatically to fit the measurement condition of the target point.The function of automatic parameter setting is only available in Point Gauge because the correct black-and-white turning point position is successfully marked.To implement the function of

Table 1 .
The ratio of OK/NG in every range