Computer Vision for Pick-and-Place: Detecting Objects in Chaos

Pick-and-place robots sound simple until you encounter a bin of jumbled parts, inconsistent lighting, and occlusion. Your carefully trained model performs beautifully on clean datasets, then fails spectacularly when objects overlap or sit at odd angles. This is where theory meets reality—and where most computer vision implementations fall short.

Unstructured environments present a fundamentally different challenge than controlled lab settings. Parts aren't neatly arranged. Lighting changes minute to minute. Objects occlude each other. Your detection system must handle these variables or your robot spends more time failing than picking.

Why Standard Object Detection Isn't Enough

Off-the-shelf models like YOLOv8 or Faster R-CNN train beautifully on ImageNet and COCO datasets, but factory environments are different animals. You're not detecting "person" or "car"—you're detecting specific widget variants, often with minimal visual distinction, partially hidden beneath other parts.

The problem compounds when you consider:

Occlusion and Overlap

When parts stack or partially hide each other, bounding boxes become unreliable. A model confident in its detection might be looking at 60% of an object. Your gripper reaches for what it thinks is a complete part and grabs air or the wrong item.

Lighting and Reflectivity

Factory lighting rarely stays constant. Metallic or shiny parts create unpredictable reflections. The neural network trained under fluorescent overhead lights behaves differently under sunlight or when a part's surface catches a glare.

Domain Gap

Your training data probably came from a limited set of camera angles, distances, and lighting conditions. Deploy that model to a different line or facility, and accuracy drops 15-20%. This isn't a bug—it's the fundamental gap between training and deployment.

A Practical Approach

Effective pick-and-place systems combine multiple techniques. Here's a workflow that works:

1. Start with Instance Segmentation

Forget bounding boxes. Use instance segmentation (Mask R-CNN, YOLACT) to get exact pixel boundaries. This matters when parts touch—you need to know where one object ends and another begins.

pythonCopy
import cv2
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg

cfg = get_cfg()
cfg.merge_from_file(
    "COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"
)
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 5  # Your parts
cfg.MODEL.WEIGHTS = "model_final.pth"

predictor = DefaultPredictor(cfg)
outputs = predictor(image)
masks = outputs["instances"].pred_masks

2. Add 3D Information

2D detection is insufficient for grasping. Integrate depth data—either from stereo, structured light, or time-of-flight sensors. This tells your robot the actual 3D position and orientation of detected parts.

pythonCopy
# Combine RGB detection with depth map
depth_map = capture_depth_frame()

for mask in masks:
    # Get centroid in 2D
    y_indices, x_indices = np.where(mask.cpu().numpy())
    centroid_x = np.mean(x_indices)
    centroid_y = np.mean(y_indices)
    
    # Lookup depth at centroid
    z = depth_map[int(centroid_y), int(centroid_x)]
    
    # Convert to world coordinates
    world_pos = camera_to_world(centroid_x, centroid_y, z)

3. Handle Domain Shift with Active Learning

Your initial model will be wrong sometimes. Capture these failures, label them, retrain. This iterative process is where real accuracy gains happen. At LavaPi, we've found that even 50-100 carefully selected hard examples can improve performance by 8-12% in production.

4. Validate Grasping Feasibility

Don't trust detection alone. Add a secondary check: given the detected object's position and orientation, can your gripper actually reach and grasp it? Filter out predictions that are unreachable or unstable.

typescriptCopy
function isGraspable(
  position: Vector3,
  orientation: Quaternion,
  gripper: GripperSpec
): boolean {
  const reachable = checkIKSolution(position, orientation);
  const stable = checkGraspStability(position, orientation, gripper);
  return reachable && stable;
}

The Real Takeaway

Pick-and-place in unstructured environments requires more than a good detection model. Combine instance segmentation, depth sensing, continuous improvement through active learning, and gripper-aware validation. This multi-layered approach handles the real world better than any single algorithm. Start simple, measure what actually breaks in production, and iterate from there.