Vision-Guided Motion Planning and Control for Robotic Manipulators
Abstract
Low-cost robotic arms are usually driven by scripts that command each joint directly, which is adequate for a demonstration but rarely transfers to the next project and offers no protection against the arm moving through an obstacle or into itself. Robotics addresses this with two standard pieces of software: the first gives programs a common way to communicate with real or simulated motors and is called ROS 2 Control, while the second computes the joint angles needed to reach a position, checks the path for collisions, and plans a safe movement, and is called MoveIt 2. Taken together, they turn a one-off script into a reusable arm setup that transfers between simulation and hardware.
Supervision
- Prakash Aryan
- Sebastiano Panichella
Motivation
The arm used here is a low-cost open-source model called the SO-101, widely used for teaching robots by demonstration but not yet paired with a conventional setup built on the two software packages above, which limits the range of projects it can support. Adding two cameras gives the arm a clear physical goal to aim at, one mounted on the gripper itself and one looking down at the table from above, both watching for a printed marker attached to a cube. The arm must then determine where that cube lies in its own coordinate frame and move the gripper to a position above it. This is a self-contained integration problem that leaves behind a foundation other projects can build on, and it requires neither machine learning nor a second arm to copy movements from.
Goal
The student will connect a small tabletop arm to the two software packages described above, then demonstrate reaching towards a target observed by two cameras. One depth camera is mounted on the gripper while a second is fixed directly above the workspace, and both look for a printed black-and-white pattern similar to a QR code, known as an AprilTag, which tells the robot where the cube sits and which way it is turned. The overhead camera provides a stable view of the whole table while the gripper camera provides a closer view during the approach. The detected position is converted into the coordinate frame of the arm, and the planner produces a collision-free path to a point just above the cube. Reaching is the required outcome, while picking the cube up remains an optional extension.
Project Scope
- Prepare or validate the file describing the arm’s parts, joint limits, and shape for collision checking, a format known as a URDF.
- Configure the motor interface and trajectory controller so that the same planning commands work against a simulated arm and the physical one.
- Calibrate both cameras against the robot base, so that a position seen by a camera can be converted into arm coordinates, and publish the detected target.
- Plan and execute one safe reaching motion to a pre-grasp position above the cube, providing launch files, configuration, and concise notes so the result can be reproduced.
Evaluation
- Accuracy: the distance between the position the gripper was asked to reach and the position it reached, measured using each camera alone and both together across six to ten target locations.
- Reliability: how often planning succeeds, how often the motion executes, and what typically causes failures such as unreachable poses or joint limits.
- Speed: the time from observing the marker to obtaining a position, the time to plan, and the time to execute the movement.
- Simulation against reality: how far the physical arm differs from the simulated one for equivalent targets.
Requirements
The student should be comfortable with Python or C++, know the basics of ROS 2, and be undeterred by coordinate frames and transformations. Robot description files, kinematics, and camera calibration can all be learned during the project. No model training, second arm, or virtual-reality equipment is needed.
Pointers
- E. Olson, “AprilTag: A Robust and Flexible Visual Fiducial System,” in Proc. IEEE ICRA, 2011. The printed marker system used to locate the cube.
- PickNik Robotics, “MoveIt 2 documentation.” The “Getting Started” tutorial is the appropriate entry point. [Online]. Available:
- ROS 2 Control maintainers, “ros2_control and joint trajectory controller documentation.” [Online]. Available:
- MoveIt, “Hand-eye calibration tutorial,” MoveIt 2 documentation. Covers aligning the cameras with the robot.
- Intel RealSense, “RealSense ROS 2 wrapper.” [Online]. Available: