You Can Now Tell a Robot to Move — Powered by AgenticROS, NemoClaw, and the AVerMedia SenseEdge Development Kit

Say "move forward" to most AI-powered robots today, and nothing happens.

Not because the AI didn't understand you. It did. The model can parse the sentence, recognize the intent, even hold a conversation about what it plans to do next.

The robot just doesn't move — because understanding a command and acting on it are two completely different problems.

The AI Already Understands You. It's the Robot That Doesn't Listen.

Large language models are good at this now. Ask one "what do you see?" or "move forward," and it knows exactly what you mean.

But an LLM has no hands. It can't reach into ROS2, spin a motor, or read a depth sensor on its own. Every one of those actions requires an integration layer that most teams end up building — and rebuilding — for every robot, every sensor, every new capability.

The bottleneck in robotics AI isn't language understanding anymore. It's the wiring between "I understood you" and "I did it."

An AI Agent With Unchecked Access to a Robot Is a Liability, Not a Feature

Here's the harder problem: even if you wire the AI up to the robot, should it be allowed to do whatever it decides?

An AI agent can generate tool calls you didn't expect. On a chatbot, that's an annoying bug. On a robot with wheels and a motor, that's a robot doing something nobody approved.

So the real challenge isn't just "can the AI control the robot" — it's "can the AI control the robot only in the ways we've decided are safe."

Two Pieces Close the Gap: One Gives the Robot Capabilities, the Other Decides What's Allowed

This proof of concept, running on the AVerMedia SenseEdge Development Kit, pairs two existing technologies to solve both halves of the problem:

• AgenticROS exposes ROS2 capabilities directly as tools an AI agent can call by name — move, take a snapshot, check a sensor — and lets teams package more complex, multi-step behaviors as reusable Skills, without needing to understand ROS2 internals.
• NVIDIA NemoClaw hosts the LLM that interprets the command, and governs what happens next: which Skills are available, which are allowed to run, and a record of what was executed.

Put together, a spoken command travels through a single governed path — interpreted, checked, then acted on:

Caption: one command, one governed path — from voice input through NemoClaw and AgenticROS to a physical action on the robot.

Developers don't build this pipeline from scratch. They write a Skill for the capability they want, and this path already exists for it.

We Ran It on Real Hardware. Here's What the Robot Actually Did.

"Tell me what you can see in front of you right now." This is a built-in AgenticROS tool — it wraps the camera topic directly. The agent captured a frame from the onboard camera, passed it to an on-device vision model, and described what was in front of it — a desk, a monitor, a person working — in plain language, seconds after being asked. The description held up against the actual scene.

Caption: the AI's description (left) alongside the real desk it was looking at (right) — asked what it sees, the robot answers in plain language, seconds after the request.

"How far away is whatever is directly in front of you?" Another built-in AgenticROS tool — a direct wrapper around the depth topic. It samples the RealSense camera's depth data and answers with a precise reading: 0.647 meters.

"Move forward, and stop when something is too close." This one needed more than a direct wrapper. Following the Skill examples AgenticROS provides as a template, the team built a simple custom Skill for this: proximity_guard — logic that checks the distance reading and decides when to stop, on top of the basic tools above. Told in one sentence to "turn on the distance-based auto stop, then start moving forward," the agent enabled proximity_guard first, confirmed a 0.5-meter safety threshold was active, and only then issued the move command. One request, two capabilities, checked and sequenced correctly.

Caption: one sentence, two Skills in sequence — the guard is enabled and confirmed before the robot is allowed to move.

The SenseEdge Development Kit Is Where This Stops Being a Demo

None of this works without hardware that can run AI inference, robotics middleware, and sensor processing together, in real time, at the edge.

Getting there usually means solving a hardware problem before writing a line of AI code. GMSL is the interface robotics teams reach for because it carries high-bandwidth video over long cable runs with low latency and holds up in the vibration and electrical noise of a real robot — conditions a standard USB camera connection struggles with. But wiring GMSL cameras into a Jetson-based system from scratch means adapter boards, camera synchronization, and driver integration, all before the robot does anything.

The AVerMedia SenseEdge Development Kit does that integration work up front. It combines the D317 carrier board — built for NVIDIA® Jetson™ AGX Orin — with a GMSL camera adapter board and the RealSense™ Depth Camera D457, pre-tested and validated as one system. That's what let this proof of concept move from a whiteboard architecture to a robot that actually stopped before hitting something — without the team first having to solve camera integration on their own.

For a closer look at how the carrier board, camera adapter, and RealSense camera fit together, see the SenseEdge Development Kit product page.

For developers building autonomous mobile robots, service robots, or other edge AI robotics applications, this is the same hardware platform they'd prototype and deploy on — not a lab-only setup.

The Takeaway: Natural-Language Robot Control Is Ready to Build On, Not Just Demo

AgenticROS gives a robot reusable Skills. NemoClaw governs which of those Skills an AI agent is allowed to use, and when. Together, on the AVerMedia SenseEdge Development Kit, they turn "the AI understood me" into "the robot did the right thing, safely" — validated on real hardware, not just in theory.

If you're building AI-powered robotics and want to see this running, or want to explore the SenseEdge Development Kit for your own project: contact AVerMedia to request a demonstration or discuss your next AI robotics deployment.