Hexagon Robotics’ AEON humanoid relies on advanced spatial intelligence to understand and interpret its environment and optimise the task at hand accordingly. Spatial intelligence is achieved by integrating data from many sensors that are different in nature; what we call sensor fusion.
As AEON performs more complex tasks, the sensor fusion architecture gets more complex, and the underlying data gets richer. Hexagon Robotics is solving this complexity by using NVIDIA Holoscan: we can now deliver synchronised and pre-processed sensor data efficiently and precisely through a single connector to our main compute module. This represents a major leap in improving our perception pipeline, our physical robot design, and our compute resources allocation.
Spatial intelligence for a multipurpose humanoid
AEON relies on sensor fusion and AI for perception. This is critical given AEON’s multipurpose nature: it needs to understand the environment, detect objects, manipulate and/or inspect them, and capture the 3D environment to create a digital twin.
Our architecture required specific algorithms and computing resources to ensure synchronicity and consistency across the varied sensors data sets. We used GStreamer with NVIDIA DeepStream plugins for image transformations, AI models and classical computer-vision algorithms, including feature detectors used in simultaneous localisation and mapping (SLAM). Depth images travelled separately between processing nodes through ROS 2.
Our sensors connected directly over MIPI to fixed ports across two compute units. All 12 ports were occupied, and sensor data had to be transferred between the units. The wiring also created mechanical constraints when routing connections through the robot’s rotating neck.
Building a high-performance sensor bridge
As AEON performs increasingly complex tasks in varied environments, we needed a solution to automate and optimise our sensor data pipeline and processing, reduce processing delays, support additional sensor data types, minimise our computing needs, and ensure we could expand our sensor family seamlessly.
Achieving these objectives required coordinated changes across computing modules, hardware connectivity and software processing. We therefore built an end-to-end solution composed of NVIDIA Holoscan, our own data-processing board, receivers on the compute side, and a high-throughput connector [See Figure 1].
Figure 1. Overview of Hexagon Robotics’ sensor-processing architecture, integrating NVIDIA Holoscan Sensor Bridge.
NVIDIA Holoscan serves as one unified runtime, with Holoscan Sensor Bridge components supporting connections to inertial measurement units (IMUs) and sensor-data transport to central computing. Holoscan provides a common, consistent sensor-pipeline framework, reducing framework-specific integration as new sensors are added.
We then developed a specialised circuit board and FPGA software to control when cameras and depth sensors capture each frame, aligning their observations for further processing. Figure 2 shows how the different hardware components were integrated into AEON’s head.
Figure 2. AEON’s head, housing the sensors and hardware used to capture and synchronise data for perception.
To send the data to the compute module, we use an NVIDIA-provided IP core coupled with a fiber optic cable [See Figure 3]. We built a custom receiver to connect to NVIDIA Jetson Thor, where the hardware-accelerated MGBE interface receives the data.
Figure 3. Sensor data travels from AEON’s head to the NVIDIA Jetson Thor compute module through a fibre-optic connection, simplifying cabling through the robot’s rotating neck.
Key benefits of AEON’s new sensor-processing architecture
There are three advantages to this simplified architecture:
- It scales as additional sensors can connect through Thor’s available network interfaces, directly or through a switch, without adding another central computing unit,
- Each operation in the pipeline is done by fit-for-purpose hardware that can be optimised,
- We preserve CPU and GPU capacity for other workloads, in our case, the AI algorithms we implement to transform sensor data into spatial intelligence.
What's next?
Initial performance measurements of the new sensor pipeline show less than 1% CPU usage per sensor stream and no GPU usage for camera processing, with only a small amount of GPU computation required for time-of-flight sensors. The architecture can also support peak network bandwidth of 100 Gbit/s and eliminates buffer copies in the measured data path.
These results demonstrate how the architecture uses dedicated hardware for sensor processing, preserving CPU and GPU resources for other workloads.
In addition to enabling the integration of different sensor types to enrich our perception capabilities, we are adding more AI-based pre-processing of raw sensor data at each sensor level and at the sensor-fusion level. This enables better object detection and categorisation and provides more actionable data to our motion planning and motion control modules.
Spatial intelligence meets task intelligence.