Pondering an Idea of LLM-Controlled Robot

/ Naixian Zhang

Studying Unitree’s Go2 EDU robot through its website and support materials, the interesting part for me is the possibility of controlling the robot with an LLM. But before getting there, I need to understand the layers underneath in some detail.

If I want to tell the robot something like, “Walk forward, look around, find a chair, and sit down,” I am pretty sure the Go2 has not yet reached the level where I can simply give it that instruction directly. Instead, I need an LLM to assist me in translating the instruction into calls to an SDK-based wrapper, written in C++ or Python, and then the robot can operate more or less autonomously.

The SDK is the source of truth. It is available in the repository, and Unitree provides both C++ and Python SDKs. The C++ SDK seems to be the more fundamental one.

For example, there is a SportClient class, which inherits from Client and exposes higher-level robot operations such as Move. Client provides the generic RPC machinery: connecting to the robot, making calls, handling timeouts, and so on. SportClient specializes that machinery for the robot’s sport-related functions.

It also provides data bundles using C++ structs. For example:

struct PathPoint
{
    float x;
    float y;
    float yaw;
    float vx;
    float vy;
    float vyaw;
    float timeFromStart;
};

You can see that this is simply a bundle of data describing a point on a planned trajectory: where the robot should be (x, y), which direction it should face (yaw), how fast it should move (vx, vy, vyaw), and when it should reach that point (timeFromStart). A planned trajectory is therefore a sequence of these points.

It is also worth noting that the Python interface mechanically mirrors the underlying C++ interface. So the C++ SDK gives us a good way to understand what the Python layer is really exposing.

Let’s look at an actual application rather than just the SDK definition.

A typical C++ application first includes the necessary header files, initializes the communication interface, creates a SportClient, and then calls functions such as Move(). The application is compiled through CMakeLists.txt, and after building, the executable can be run from the command line.

One line that caught my attention is:

ChannelFactoryInitialize(0, "eth0");

Here, eth0 is the network interface connected through Ethernet. In other words, the application needs to tell the communication layer which network adapter it should use to communicate with the robot.

Underneath this is DDS, or Data Distribution Service, which is the communication middleware. It provides the plumbing for exchanging messages between the application and the robot, ultimately over the network.

Once everything is initialized, the actual application logic can be surprisingly simple:

while (true)
{
    int32_t ret = sc.Move(0.5, 0, 0);
    std::cout << ret << std::endl;
    usleep(...);
}

This program simply keeps sending the movement command continuously.

The Go2, of course, is not simply a collection of motors waiting for commands. There are multiple layers of services. At the lower level, there is the DDS communication layer and basic motion control. Above that are higher-level services for things such as robot state, obstacle avoidance, navigation, LiDAR, SLAM, motion switching, voice and user interfaces, and AI motion services.

For example:

LiDAR
  ↓
point cloud / range data
  ↓
SLAM / navigation
  ↓
where am I?
what are the obstacles?
where should I go?

This is where the ROS framework comes into the picture. SLAM helps the robot build a map and understand its location, while the motion API tells the robot how to actually move.

It is also interesting that this version of the robot comes with an NVIDIA Jetson Orin NX onboard, with up to around 100 TOPS of AI compute. So the robot is not just a remote-controlled physical machine; it is also an onboard computing platform where our own Python code and AI services can run.

This brings me back to the LLM.

The LLM can access many capabilities through wrappers exposed as tools. In that architecture, the LLM becomes the planner, not the motor controller.

For example, instead of giving the LLM direct access to every low-level SDK function, I could expose simple tools such as:

walk()
turn()
look()
go_to()
sit()
stop()

The LLM can then decide which tool to call and in what sequence.

This is fundamentally different from asking the LLM to directly control individual motors. The LLM operates at the level of intentions and actions, while the underlying SDK and robot services handle the actual physical execution.

So it seems to me that LLM-based robot control is fundamentally a tool/API problem, very similar to building an MCP server.

The hard part is not necessarily making the LLM understand the robot’s entire SDK. The interesting engineering problem is deciding what capabilities to expose as tools, wrapping those capabilities properly, and creating a clean interface between the LLM and the robot.

The architecture starts to look something like:

Human instruction
       ↓
      LLM
       ↓
   Tool / MCP
       ↓
Python / C++ wrapper
       ↓
   Unitree SDK
       ↓
   SportClient
       ↓
      DDS
       ↓
     Go2

The LLM is the high-level brain. The SDK and robot services are the execution layer. The wrapper is the interface between the two.

That, I think, is the interesting part of studying a robot like the Go2: not simply learning how to make it move, but understanding where an LLM should sit in the stack.

Leave a Reply