Automated Visual Inspection System for Factory Products
Our group’s project proposal is for an automated visual inspection system utilizing a conveyor belt with items to be inspected and a camera to accept or reject items as seen in a factory setting.
In industrial applications, defects on the manufacturing line is a common issue, where if not spotted quickly, can increase downtime and result in bad products. Our system uses an Intel RealSense Camera powered by an 8GB Raspberry Pi Board and Machine Learning (ML) algorithms to spot defective products that are running on a designed mechanical conveyor belt. There is a mechanical actuation system present to push out a product if it is defective. This actuator system and the motor for the conveyor belt is controlled by a Programmable Logic Controller (PLC) which is typical in factory settings. We believe this system could be beneficial to industry replication, increase efficiency for reject/accept systems, and educate others on PLC, mechanical design and ML basics.
Finished Work Overview
System Working Video
Introduction:
There are numerous examples on manufacturing lines where defects have to be taken care of manually. Such examples include sheet metal stamping procedures where quality control needs to be taken very seriously, given the metal is used in engines, fuel injectors, bayonet assemblies, military devices etc. [2] Defects are taken care of manually for these materials where if automated, could increase accuracy and consistency while also being able to work continuously in hazardous environments. [2]
Another example includes bottle inspection, where inspection speed is expected to be 400-600 bottles per minute while the bottles move at the speed of 1 meter per second. [10] In this case, it is much more efficient and accurate to use an automated visual inspection system rather than manual inspection.
In these cases, if manual inspection fails or is too slow, downtime is increased which can significantly affect costs and profits for a manufacturing company that needs to run as consistently and often as possible. If defects are not spotted, this can result in bad products to the customers which will decrease reliability in the brand or even result in safety/quality issues and risk of injury. Furthermore, buying visual inspection systems off the market can prove to be very expensive.
Our research is to show with the advances in computer vision and machine learning technology, automated visual inspection systems can be produced with low cost and effectively detect bad products for a wide range of manufacturing industries. Our visual inspection system will also allow interfacing with programmable logic controllers (PLC) which is standard for many manufacturing industries that use logic to drive other machine components.
System Overview:
Our subsystems are connected via the diagram shown in Figure 1 above. The Intel RealSense Camera is controlled by the Raspberry Pi running an ML algorithm to visually accept or reject a part based on certain criteria. The PLC will then take this output result of accepting or rejecting and activate the linear actuator (in this case a solenoid) via a conditional relay. The actuator will then push out the part while the DC motor is continuously running the conveyor belt (also controlled by the PLC).
Design Exploration Sketch
In the early phase of the project, determining the functionality and the design of the conveyor belt system is crucial. As a team, we have several expectations for our system, including:
1. Vision camera is mounted on top with adjustable height that could achieve clear vision and focual distance to different products.
2. Solenoid should be mounted on the side and near the conveyor belt for push-out action.
3. Belt structure should be based on 80-20 aluminum bar for the best stability and manufacturability.
4. driving motor should be mounted on the system with proper connection to the belt shaft to transmit power without any risk of falling or breaking.
With all the thoughts, we have created the design exploration sketch listed below:
Design & System Troubleshooting Footprint
CAD Mechanical Design
To bein with, our group would like to make the vision system as replicable and as easy-to-access as possible. The reason of this aim is to help other group of student that have more complex research goals to easily replicate our design and build upon it.
There are several essential component in the CAD design, which includes the belt driving system, the belt tensioning mechanism, the camera placement as well as the solenoid mount. Which will all be covered in this section.
The following picture shows the whole system assemly of our vision inspection system:
To get into details, the first highlighted section is the belt driving system. This part is done by a servo motor mounted on the lower part of the 80-20 bar. The motor is carried by a FDM 3D printed sliding housing, it connects to the main conveyor belt by two keyed pullys and a timed belt.
One of the technical challenge that we face is that since the motor shaft diameter is different from the ID of keyed pully, we have to print a shaft adapter in the very middle in order to transmit the power, as shown below:
In order to make the design easy to replicate and in order to stay within budget, the belt tensioning mechanism is aimed to be simple, effective, and purly mechanical. Therefore, the ball bearing mounting gear is designed in a elongated way with a M5 screw clearence hole oponed on its very top. This allows the mount to freely slide for a distance of 50mm on each end of 80-20 bar, while can be tightened by screw and metal insert, as shown below:
The camera placement is designed to connect on a single aluminum bar at the very top of the conveyor. The custom made 3D print camera mount is steady to be slided onto the 80-20 bar, while having a screw-insert component below it for adjustable height. The camera itself is connected to the mounting part by a single screw on its back. An agvantage of utilizing 8020 bar for camera placement is that a light shell can also be applied by sliding to 8020, if needed.
Computer Vision & Mechatronics
1. Raspberry Pi
In the center of our system, we are using a Raspberry pi as control which communicates with the camera
1.1 Intel RealSense 435i Camera
At the heart of the system is the Intel RealSense 435i camera, which integrates RGB, depth sensing, and laser-based imaging. This ensures:
•High detection accuracy by capturing critical object features.
•Reliable imaging for real-time inspection.
Intel Realsense 435i
1.2 OpenCV Integration
The system employs OpenCV for image processing.
•Lightweight algorithms tailored for single-board computers.
•Optimization prioritizes simple shape and color detection to balance speed and efficiency.
This design enhances performance on resource-constrained devices while maintaining precision.
2. Circuit
2.1 Hardware Integration with Arduino
The Arduino acts as a central controller for actuation, interfacing with the Raspberry Pi and other components. Key features include:
•Voltage Management: A 3V/5V output adapted to a 24V solenoid using an amplifier circuit.
•Safety Mechanisms: Protective circuit design prevents damage to hardware components.
•Precision Actuation: Delay mechanisms for precise timing.
System Functionality
3.1 Object Detection
The detection system distinguishes between “good” and “NG” (not good) parts based on pre-defined visual attributes:
•Efficient processing enables fast classification of objects.
•Focused design on essential features avoids overloading the processing unit.
Good Part Simulated
Good Part in Testing
Not Good Part Simulated
Not Good Part in Testing
3.2 Actuation and Control
After identifying an object’s classification, the Arduino triggers the corresponding actuation mechanism to sort or handle the object appropriately.
4. Implementation and Results
•Integration Efficiency: Combining Raspberry Pi and Arduino ensures seamless communication and control.
•Detection Accuracy: The depth camera captured essential object details, supporting reliable detection even in variable lighting conditions.
•Durability and Safety: Prototyping and robust circuit design prevented component failures during extended operation.
Programmable Logic Controller (PLC), Motor and Solenoid
Arduino provides a fundemental educational and engineering real-world troubleshooting component to electrical and software design projects [3]. Due to this educational learning and ease of access, the Arduino Opta PLC kit was used in order to supply our industrial PLC.
The corresponding version Arduino Opta PLC IDE software was downloaded from: https://www.arduino.cc/pro/software-plc-ide.
The PLC was connected to the correct port for communication and the bootloader was downloaded to the board. This allows programming to be accomplished using the IDE software. The Arduino Opta PLC software allows scripting logic that can be used to control connected inputs and outputs in multiple languages, including ladder logic, structure text and C++. Ladder logic was chosen in order for the educational element of the project to become apparent. Inputs would connect to the Raspberry Pi vision system in order to read the input when it is supplied with voltage and the corresponding output was the solenoid actuator connected to the first output relay. Both these inputs and outputs were defined in the first routine program as variables local to the program and the input was used as a bit to turn on a positive coil which would control the solenoid conditionally. Bits were defined based on these inputs, any instruction blocks were placed for specific logic and the output coils were programmed accordingly.
The first test was to toggle the solenoid on or off based on a switch input. This connection was successful as can be shown below in Figure X while the motor was continuously running with a complete circuit. The output solenoid coil was defined as a positive sensing coil, allowing a pulse output. This solenoid was also tested and successfully proved capable to push out the parts we had printed for visual inspection.
Figure X: First PLC tested Connections
The next step was to communicate successfully with the camera result of rejecting or accepting the part. The connection was tested with the Raspberry Pi but the solenoid would not actuate. Through troubleshooting it was found that the relay needed a higher input voltage than the Raspberry Pi could provide in order to actuate. This was addressed by using a transistor circuit. The Arduino was then used to take the input from the Raspberry Pi when the Raspberry Pi saw a reject and activate the solenoid valve.
It was then used to control the DC motor. This was originally connected to the positive and negative terminal ends directly but was found to have issues driving the motor. Due to this, it was connected to a motor driver where the speed could be controlled by turning a knob. The system electrical components were then integrated together successfully as can be seen in the video.
Education Levels
- Set up PLC using step by step guide
- Understand basics of how PLCs are used in industry
- Specify which types of motors, valves etc. to use
- Design mechanical conveyor system components in SolidWorks
- Write ladder logic to control components with the PLC
- Assemble physical conveyor together and refine design based on mechanical observations
- Apply Machine Learning and Computer Vision techniques to identify a part to reject
- Integrate complex electrical circuits together with mechanical components
- Troubleshoot Motor, PLC, Solenoid, Raspberry Pi and Camera communication
Conclusions
Overall, we have achieved our primary goal of designing and buiding a graudate-level automated vision inspection system for industrial usage. We have successfully designed, sourced, and construct the physical assembly of the system. We have managed to integrate PLC controller to activate, control, and making decisions to Good and NG parts. We have trained our own AI-based vision inspection algorithm to monitor the colors of the part and integrated a solenoid push-off unit to exit the non-ideal color of the inspected object.
The system that we built and the documentations that we provide are aimed for sharing this easy-to-replicate design process for other groups of students to build and improve based on our achievement. Any component of our assemly, including conveyor belt system, vision camera, or PLC are fundmental but potentially essential for a system with higher complexity and purpose.
Looking back to our design process and finished work, we do have some lessions learnt, and some future improvement wishes. One of which is the belt. A enclosed belt would benefit this project even more since tapped belt would affect required torque and the operational speed of the conveyor when the rough section hits the roller. The second future improvement would be the inspection capacity of our AI model. With limited time and resources, we managed to have the system accurately differentiate colors. Our goal goes beyond this. We would like the system to differentiate shape, surface texture, and eventually, small surface defects on the level of milimeters. This would be a practical requirement if the system were put in a manufacturing production floor and create profits.
Thank you for watching our design portfolio of the automated vision inspection system.
Reference
- [1] Automated Vision Inspection with Conveyor Belt and Rejecting Arm Szepietowski, Charles. Automated Vision Inspection with Conveyor Belt and Rejecting Arm. GitHub, 2021, https://github.com/CharlesSzepietowski/Automated-Vision-Inspection-with-Conveyor-Belt-and-Rejecting-Arm.
- [2] Automatic visual inspection system for stamped sheet metals (AVIS3M) A. R. Rababaah and Y. Demi-Ejegi, 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE), Zhangjiajie, China, 2012, pp. 661-665, doi: 10.1109/CSAE.2012.6272855. keywords: {Inspection;Visualization;Metals;Surface treatment;Humans;Image segmentation;Classification algorithms;automatio;machine vision;stamped sheet metal;visual inspection}
- [3] Collaborative robotic educational tool based on programmable logic and Arduino P. Plaza, E. Sancristobal, G. Fernandez, M. Castro and C. Pérez 2016 Technologies Applied to Electronics Teaching (TAEE), Seville, Spain, 2016, pp. 1-8, doi: 10.1109/TAEE.2016.7528380. keywords: {Robots;Hardware;Field programmable gate arrays;Internet;Microcontrollers;Random access memory;Remote laboratories;Arduino;Raspberry Pi;FPGA;Robotics;Education;STEM},
- [4] Deep Learning for Automated Visual Inspection in Manufacturing and Production Smith, John, et al. “Deep Learning for Automated Visual Inspection in Manufacturing and Production.” MDPI, 2023, https://www.mdpi.com/2571-5577/7/1/11.
- [5] Deep Transfer Learning for Industrial Automation: A Review and Discussion of New Techniques for Data-Driven Machine Learning Models: Luo, Wei, et al. “Deep Transfer Learning for Industrial Automation: A Review and Discussion of New Techniques for Data-Driven Machine Learning Models.” IEEE Xplore, 2021, https://ieeexplore.ieee.org/document/9328227.
- [6] Leveraging Robust CNN Architectures for Real-Time Object Recognition on Conveyor Belts Patel, Raj, et al. “Leveraging Robust CNN Architectures for Real-Time Object Recognition on Conveyor Belts.” IEEE Xplore, 2024, https://ieeexplore.ieee.org/document/10212380.
- [7] Machine Vision Systems for Industrial Quality Control Inspections “Machine Vision Systems for Industrial Quality Control Inspections.” Springer, 2022, https://link.springer.com/content/pdf/10.1007/978-3-030-01614-2_58.pdf.
- Object Detection Using Deep Learning in a Manufacturing Plant to Support Quality Control Rivera, Luis, et al. “Object Detection Using Deep Learning in a Manufacturing Plant to Support Quality Control.” IEEE Xplore, 2021, https://ieeexplore.ieee.org/document/9486529.
- [9] Toward Industrial Densely Packed Object Detection: A Federated Semi-Supervised Learning Approach Zhang, Jun, et al. “Toward Industrial Densely Packed Object Detection: A Federated Semi-Supervised Learning Approach.” IEEE Xplore, 2023, https://ieeexplore.ieee.org/document/10634893.
- [10] The development of prototype of automated bottle caps visual inspection system lU. Kaewmorakot, L. Poolperm, P. Prongphimai and S. Tansriwong, 2022 International Electrical Engineering Congress (iEECON), Khon Kaen, Thailand, 2022, pp. 1-4, doi: 10.1109/iEECON53204.2022.9741567. keywords: {Meters;Visualization;Software design;Image processing;Prototypes;Interference;Inspection;Automated Visual Inspection System;Machine Vision;Image Processing;LabVIEW;Bottle Caps}
