Automation of Industrial Pick and Place Pipelines Using Computer Vision 3-Dimensional Robotic Bin Picking
By Navya Vepakomma
Overview
The focus of today’s industries is lean management where production times and response times are being reduced, and the aim is to increase overall efficiency. Bin picking is a target solution for automating manual, labor intensive, and repetitive tasks where parts need to be picked up from a filled bin and placed at a target destination. Highly repetitive tasks being performed by humans are more prone to high error rates, thus decreasing productivity.
Bin picking technology can optimize production in several ways. It can be used in environments which are hazardous to a human workforce. It also replaces redundant and labor-intensive work, reducing error and increasing productivity and efficiency. Typically, bin picking can be used in applications where it is necessary to:
a. Automate processes where it is not possible to receive objects from a previous process or sub-supplier (handle un-ordered items),
b. Automate processes which require low cycle times and precise placement of picked objects into a further process,
c. Pick up randomly placed objects from a bin or container,
d. Pick up a high number of small parts quickly and robustly (screws, washers, etc.),
e. Process or handle parts of varied sizes, shapes, and weights.
1. What is Robotic Bin Picking?
Robotic bin picking is a niche complex process in which a robotic manipulator is guided by a 3D vision system and empties objects from a bin. The aim of bin picking is to pick up objects which are randomly placed and randomly oriented within a bin, using a vision system, a robot, and a gripper. Breaking down the bin picking process, a robot must locate each part within the bin, compute its position and orientation, plan a complete path from picking the object to placing it at the target position without any collisions, and execute the pick and place operation.
The vision system is developed to recognize and locate each individual object inside the bin and provide the robot with the perfect position and orientation in which the object is to be picked. The robotic system must plan often complicated paths and maneuver into the bin in unique orientations to avoid collision with the bin picking setup: the bin itself and other objects within the bin. The robotic arm needs to replan the object’s path towards the designated target position and place the object precisely in the required orientation. A crucial element of bin picking is that other parts within the bin or any other items in the surroundings are not damaged by the robotic arm that is in motion.
2. Challenges of Bin Picking
a. Precise determination of objects’ poses: The pose of an object is the object’s location in three-dimensional space with respect to a coordinate system. The 3D location (pose) is defined by a vector of six values, Tx, Ty, Tz, Rx, Ry, Rz, where Tx, Ty, and Tz values describe the translation parameters and Rx, Ry, and Rz values define the rotation or orientation from the reference axis. Determining the pose of the objects within the bin is challenging because the objects may significantly overlap and lay in inconsistent orientations. Moreover, once an object is successfully picked from the bin, the other objects may shift and create a new topography. The vision system will need to re-compute bin objects’ location for every iteration. The system must also decide which part is most suitable for picking so that the object is moved without colliding with the bin and damaging the object or the surroundings.
b. Availability of precise depth data: Precise depth data is necessary to enable accurate pose computation of the objects in various conditions. In order to compute objects’ orientation and position with high precision, the vision system needs to perform independent of lighting conditions. Small objects can cause shadows over other objects and will also have high overlap. Moreover, industrial objects are often quite shiny and highly reflective, which distorts the depth data from most vision systems. Vision systems that can obtain precise depth data for reflective surfaces and in different lighting conditions are generally based on laser scanning and are quite expensive. Performing bin picking with lower cost vision systems is still a major challenge.
c. Accurate robotic path planning and collision avoidance: It is extremely important that the robot, with a gripper, can pick up each object within the bin without hitting any other items in its surroundings. It is also crucial that the remaining objects within the bin are not damaged when the gripper enters the bin and picks up an object. Damage is more likely if the objects are fragile. It is therefore important to have a precise picking strategy and accurate path planning with collision avoidance. It is necessary for robots to adapt to different types of grippers based on the application and type of object within the bin. If the objects within the bin are of totally different shapes and natures, it can also be necessary to automatically change the gripper for each type of object within the bin.
d. Emptying the bin: Another major challenge with bin picking is that objects may be difficult to reach within a bin. It is usually manageable to pick up about 85 % of the objects within the bin, whereas emptying the complete bin can be difficult if the robot is not able to reach the objects towards the walls and corners of the bin. In such cases, it is sometimes necessary to automatically shake the bin (using an external motor) in order to rearrange the objects within the bin, enabling pickup of the remaining parts. Some objects can have complicated shapes and can entangle with each other. In such cases, it is more complicated to pick up individual items, and again, the system needs to be able to avoid picking up entangled parts or automatically re-arrange the bin to detangle the objects before picking up.
e. Requirement of low cycle time: Typical bin picking applications have a requirement of low cycle times (less than 10 seconds). Most bin picking systems are present within a complete automation pipeline, and the pick and place operation must be completed before the next sequence of operation is ready. The complete sequence of obtaining precise point clouds from the vision system, computing poses of multiple objects within a bin, planning the paths to different objects, deciding the best one to be picked, and executing the pick and place operation can be computationally intensive. These processes need to be executed on high power computing systems and extensively optimized to complete each sequence within the low cycle time requirement.
The task of picking and placing objects might seem simple for the human brain because it can intuitively and instantaneously judge an unorganized environment and make quick decisions. However, providing the same decision-making power and intelligence to a robotic-vision machine or system has taken years of technological innovation.
3. Our Solution
At Wipro we have developed a complete bin picking solution which can address the pick and place operations in industry. As a part of the solution, the end user can configure all the details of the bin picking system. Our solution has high-quality object localization with an efficient and well tested path planning module. Moreover, the solution does not require any previous training time for a new object that needs to be handled. The complete pick and place cycle time is relatively low, and the system can be set up with less effort.
System Architecture
a. User Interface:
The end user, typically a floor operator, can configure the complete bin picking system using a web interface. The user needs to upload the CAD model (3-dimensional mesh that defines the object shape) of the object that will lie inside the bin. The user also has the option to fine tune the Gripper TCP (Tool Central Point). The TCP is the location on the gripper to which the object will be grasped. The user can identify a few specific ways in which it is ideal to grip the object so that the final placement pose is feasible.
The picked object often needs to be placed into another machine for further processing. The user can specify the exact position and orientation in which the object should be placed within the environment. The user further sets up different items that form the environment of the bin. This enables the robot to be aware of the obstacles in its surroundings and avoid hitting them during path planning.
b. Algorithm Modules:
Point cloud capture
Point cloud is a set of data points in space, which represents a 3D shape or object. A highly accurate and precise 3D scanner is used to capture a static point cloud of the bin with the objects. The high-resolution point cloud is further processed to estimate the objects’ position within the bin.
Point cloud cropping and noise removal
To optimize the solution and reduce processing time, the point cloud is clipped to the area of the bin. Small areas of noise are also filtered to avoid incorrect detections.
Cluster generation and ranking
The data points in the point cloud are grouped together, based on certain conditions, to create clusters of points that might define objects lying within the bin. The created groups of points are assumed to be the individual objects within the bin, and the pose of these objects is estimated. Further, a ranking process is established to discard clusters that may seem too big or too small based on the CAD model of the object. Clusters that are about the same size as the CAD model are ranked higher and will be evaluated first.
6D Pose detection
The most important part of bin picking is the 6D pose estimation of the objects in the bin. The pose of the object is defined by its translational position and rotational orientation of the way it is lying within the bin. Pose computation is achieved by first extracting certain features from the object’s CAD model. Similar features are then extracted from the individual clusters that have been generated from the point cloud of the bin. A detailed feature matching takes place, at the end of which the CAD model of the object is coarsely aligned with the object within the bin.
This coarse alignment is fine-tuned by computing the distance between points in the model and the target object. Iteratively, the distance between points is reduced by moving the model closer to the object. The lowest distance provides the final pose of the object within the bin.
Final picks pose computation
The object pose computed in the previous step is in reference to the camera coordinate system. For the robotic arm to be able to reach and pick this object, the pose needs to be generated in the robotic coordinate system. The pose of the object in the robotic system is computed by applying the hand eye calibration matrix to the object pose. This is the pick pose of the object.
Octomap generation
The environment in which bin picking is being executed, along with the latest bin scan, forms the collision surfaces for the robot. These surfaces are generated as an octomap, which is a 3D occupancy grid mapping approach.
Path planning
To plan a path into the bin and check the possibility of object pick up, the path planning module takes into consideration the octomap and the robotic arm’s structure.
c. Interface
Interfaces are used for connectivity between the software and hardware components such as the laser scanner and the robotic arm. Robotic Operating System (ROS) has been used by robot operators for synchronization and connectivity between the components.
d. Hardware
Hardware for a system like this consists of a laser scanner, a robotic arm with gripper, and the computing system. The rest of the environment, including the bin and camera fixture also comprise the hardware.
5. Future scope
Bin picking is an ideal way to increase efficiency in industrial pipelines where pick and place operations are involved. With Wipro’s bin picking solution, we aim to replace manually operated cycles to decrease production time.
Our current solution for bin picking is performed with high-end laser scanners and is compatible with the Universal Robot series. We aim to extend our support to various other robotic arms and co-bots to better suit a wider variety of industrial applications. Extensive research and development are being carried out to implement precise object pose detection (localization) with lower cost depth sensors and to optimize further the current cycle time.
Pick and place operations have been an integral part of industries for many decades. However, labor shortage has become a major current issue. Moreover, finding employees to cover multiple shifts to keep production running is an even bigger challenge. Automating these operations through robotic bin picking will enable production to run all round the clock with minimal errors.
