MIT engineers built a new excavator training interface that replaces joysticks with a mini robotic arm. In tests, novice operators performed as well as experts from their very first session.
Learning to operate an excavator can take years — operators have to build a mental map connecting joystick movements to the digger’s boom, arm, bucket, and cab. MIT engineers say they’ve found a way to skip that mental translation almost entirely, with a new training interface that novices can use as effectively as seasoned operators, right from their first session.
Why Excavators Are Basically a High-Stakes Claw Machine
The MIT team draws a direct comparison to an arcade claw machine: coordinating joysticks to connect their movement to a mechanical claw’s motion is a type of “mental mapping” that isn’t immediately intuitive — and it’s exactly what makes claw machines hard to win. An excavator works on the same underlying principle, just with far more complexity, since operators must direct the machine to move rocks, grade soil, clear debris, and dig foundations.
Replacing Joysticks With a Miniature Robotic Arm
Instead of joysticks, the new controller — developed by Hermano Krebs, a principal research scientist in MIT’s Department of Mechanical Engineering, and collaborators — physically resembles a miniature excavator arm and bucket. Trainees grip the device and move their own arm and hand the way an excavator would move, while a digital excavator projected on an immersive six-screen display mirrors those movements in a virtual environment in real time. The team calls the combined system the “World-Space Interface,” or WSI. “This is a more intuitive way to command the machine,” Krebs said. “With this new interface, we can eliminate a lot of the mental maps that an operator would need to build in order to operate an excavator.”
The Training Test: 15 Virtual Environments, 7 Days, 1 Hour a Day
To test the system, the researchers ran volunteers — both experts and novices — through training on both the new World-Space Interface and a traditional joystick-based simulator. The team built 15 realistic virtual excavation environments, including construction sites, highways, forest roads, riverbanks, mining areas, and urban and rural settings, each paired with tasks like scooping and dumping gravel, digging and grading trenches, clearing debris from roads, and breaking up rocks. The experiment mirrored a real-world weeklong excavator driving course: one hour a day for seven days, with task difficulty increasing over the week.
Novices Matched Experts From Day One With the New Controller
The results were stark. With the traditional joystick simulator, novices were consistently worse than experts, though they improved over the training period, as expected. With the new World-Space Interface, novices performed as well as experts right from the start. “In this case, joysticks are a non-intuitive way to control and coordinate the machine,” said study co-author and MIT postdoc Joao Buzzatto. “This is the first interface that does not require me to command the excavator with joysticks.”
On-Site Today, Remote Tomorrow
Krebs sees two practical applications for the interface: speeding up how quickly new operators are trained, and offering an entirely new way to physically control excavators, whether in the cab or from a distance. “Instead of having joysticks, you might have this miniature arm on the side, where the operator would place their own arm, kind of like an exoskeleton, which would allow them to operate the excavator in the cab,” Krebs said. “If work has to be done in a difficult or unsafe environment, you could have an operator sitting off-site in a trailer and using this arm to remotely tele-operate the excavator.”
What’s Next: Adding a Sense of Touch to the Robotic Arm
The team is now working to add haptics — a sense of touch — to the WSI’s physical arm, so that when an operator mimes picking up a pile of rocks, the arm generates resistance that mimics the actual weight, confirming the excavator is really picking material up. “Haptics would make this an even more intuitive system,” said co-author and visiting engineer Solmon Jeong.
Big Machinery Makers Are Still Building on Joysticks — This Could Change That
The research team notes that major construction equipment manufacturers, including Caterpillar, Hyundai, and Komatsu, are already developing virtual simulators to train operators and eventually enable remote operation — but these systems remain largely built around traditional joystick controls that still take time to learn. If a World-Space-style arm-and-bucket controller were built into an excavator cab or a teleoperation simulator, the researchers say even first-time operators could potentially get to work from day one.
The Research Behind It
The project began in 2018, when Krebs’ group — which typically focuses on human-robot interaction and physical rehabilitation — began collaborating with researchers at Sumitomo Heavy Industries, a Japan-based industrial machinery manufacturer that was looking for a faster way to train excavator operators amid a rapidly aging workforce of heavy machinery operators in Japan. MIT co-authors on the study include Moises Alencastre-Miranda, Joao Buzzatto, and Eran Beeri Bamani. The findings were published this week, open-access, in the Journal of Computing and Civil Engineering, and the research was supported in part by Sumitomo Heavy Industries.
Image credits: MIT News