Technology Stack

Key Features

• C++ bridge classes with Blueprint-overridable hooks for observations, rewards, resets, and actions
• TCP protocol that synchronizes Unreal simulations with Gymnasium-compatible Python environments
• Single-environment and parallel vectorized training through Stable-Baselines3
• YAML-configured PPO, A2C, SAC, TD3, and DDPG training with checkpointing and resume support
• ONNX Runtime integration for real-time model inference inside Unreal Engine

About UE Reinforcement Learning

UE Reinforcement Learning

Why

I wanted a streamlined way to train and run reinforcement learning agents inside Unreal Engine without forcing every project to rebuild the connection between a real-time simulation and the Python machine-learning ecosystem. The framework keeps the environment API accessible to both C++ developers and Blueprint users while allowing training code to use familiar Gymnasium and Stable-Baselines3 interfaces.

Interesting Technical Points

Blueprint-Friendly Environment API

The source-code layer provides reusable C++ bridge classes that define the reinforcement learning lifecycle while leaving project-specific behavior to overridable callbacks. Developers provide observations, calculate rewards, reset episodes, apply model actions, and report whether an action is still running. These hooks are exposed to Blueprint, so an environment can be implemented in C++ or assembled visually without changing the networking and training infrastructure.

The repository also includes example Blueprint .uasset assets for single- and multi-environment agents. These are configured in the Unreal Editor and demonstrate how the C++ bridge classes connect to a playable simulation.

Unreal–Python Training Bridge

Unreal runs the simulation and exposes it to Python through a TCP protocol. An initial handshake communicates the environment type, observation size, action size, and number of parallel environments. Each subsequent step exchanges observations, rewards, and episode state for the next action vector, keeping the engine simulation synchronized with the Python training loop.

On the Python side, Gymnasium wrappers translate that protocol into standard reset and step calls. This separation lets reinforcement learning algorithms treat an Unreal project like a conventional Gymnasium environment while the plugin handles engine-specific communication.

Parallel Environment Execution

The framework supports both a single Unreal environment and multiple environments collecting experience in parallel. Multi-environment training assigns each simulation its own socket, tracks it by environment ID, and connects the group to Stable-Baselines3 through SubprocVecEnv. The same training entry point selects the correct topology from handshake metadata, so experiments can scale without changing their environment logic.

Configurable Training Pipeline

Training is driven by YAML configuration rather than hard-coded experiment settings. Users can select PPO, A2C, SAC, TD3, or DDPG; configure shared and algorithm-specific hyperparameters; choose the compute device; save periodic checkpoints; and resume an interrupted run. This keeps experiment configuration separate from both the Unreal environment and the reusable training code.

In-Engine ONNX Inference

Once a trained policy is available as an ONNX model, the plugin's C++ inference layer loads it through ONNX Runtime, converts Unreal observations into input tensors, and returns action vectors directly inside the engine. A reusable C++ actor component manages the inference loop and exposes its state and action hooks to Blueprint, allowing trained agents to run without the external Python process used during training.

What I Learned

Product Design

I learned how to turn technically complex reinforcement learning infrastructure into a product that non-programmers can use. Instead of exposing networking and training internals, we designed Blueprint-facing interfaces around the concepts users actually need—observations, actions, rewards, episode resets, and inference—and provided example Blueprint .uasset assets that can be configured directly in the Unreal Editor. This required thinking carefully about API boundaries, sensible workflows, and how C++ abstractions, visual scripting, examples, and documentation work together to reduce setup friction.

Engineering

I learned how to design a cross-language systems boundary between a real-time engine and a Python machine-learning runtime. The TCP bridge begins with a compact handshake describing the environment topology and observation and action dimensions, then uses a newline-delimited key-value protocol to exchange actions, observations, rewards, resets, and episode state. Supporting parallel environments required coordinating dedicated sockets, buffering partial messages, routing data by environment ID, and keeping Python Gymnasium calls synchronized with Unreal's tick and action lifecycle. I also learned how to preserve the same modular environment interface when moving from external Python training to real-time ONNX inference inside the engine.

Next Goals

Evaluation Framework

Build a reproducible evaluation layer that runs trained policies across fixed scenarios and random seeds, compares algorithms and checkpoints, and tracks reward, stability, and generalization alongside user-defined success metrics tailored to each environment. This would make performance claims measurable and help catch regressions as the framework evolves.

Multi-Agent Training

Extend the current multi-environment architecture to support multiple agents learning and acting within the same Unreal simulation. This would provide the foundation for more complex scenarios involving cooperative and competitive behavior, configurable shared or individual rewards, coordination, and emergent behavior.