Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

Summary

Realtime-Bench evaluates real-time streaming capabilities in environments that continuously evolve. GPT-6 Astra (high) achieves the highest Pass^3 under Wall Clock at only 36%, indicating that even the strongest evaluated model struggles to complete tasks consistently.

1Introducing Realtime-Bench

Figure 1. Task examples. Top: game tasks (Gomoku, Pong, Highway 404). Bottom: web tasks (Return Receipt Exchange, Cruise Batch Closeout, Pickup Driver Handoff).

Many real-world interactive scenarios require decisions to be made under strict temporal constraints. For example, when purchasing a highly demanded ticket, detecting availability is not enough: an agent must observe the changing interface and act before the opportunity disappears.

Unlike previous benchmarks that evaluate models in static environments, Realtime-Bench evaluates real-time streaming capabilities in environments that continuously evolve. It includes 50 tasks across web and game domains. All of the tasks are designed around three key characteristics.

Dynamically evolving environment. Environment states should change over time, independently of the model's actions.
Temporally constrained interaction. Success should depend on both what actions the model takes and when it takes them.
Real-world fidelity. To capture practical real-time interaction demands, task interfaces and interaction mechanics should reflect realistic applications and games.

2Results

2.1Real-time interactions remain challenging for frontier models.

To evaluate model performance in real-world deployment settings, we evaluate all models under Wall Clock, which advances the environment according to the actual time spent on inference.

GPT-6 Astra (high) achieves the highest Pass^3 at only 36%, indicating that even the strongest evaluated model struggles to complete tasks consistently. While Jev-1.13.0 achieves the lowest latency, its Pass^3 remains low at 8%. These results suggest that low latency alone is insufficient for reliable real-time performance: models must combine timely responses with strong task-solving capabilities. Improving real-time abilities therefore requires advances in both response speed and model real-time streaming capability.

Latency vs. Pass^3 under Wall Clock — **Figure 2.** Latency and Pass^3 under Wall Clock.

2.2Reasoning effort is a tradeoff under time pressure.

Higher reasoning effort increases latency for every model in Figure 2, but its effect on success is mixed. GPT-6 Astra improves from 30% at low effort to 36% at high effort. Most other models instead drop: Gemini 3.8 Flash falls from 22% to 14%, and Claude Opus 5 from 10% to 4%. Claude Fable 5.1 and GPT Realtime 2 stay unchanged.

These contrasting outcomes illustrate the trade-off that adaptive deliberation must manage: further reasoning may improve action selection, but its temporal cost can reduce the opportunity to execute that action successfully. Ideal real-time models should decide whether to think, and adapt their reasoning to the time available to act, alongside task difficulty.

2.3Models fail to adapt reasoning length to temporal urgency.

Models show little awareness of how much time their own generation consumes. In Gomoku, each move has a 10-second countdown. Yet Gemini 3.8 Flash (high) spends 12.5 s reasoning about where to place its first stone, and loses on time before making a move. GPT-6 Astra (high) shows the same pattern in Pong: it states "I need to react quickly!" but continues analyzing earlier serve trajectories. Its response takes 5.8 s, and the game is already over when the action arrives.

Figure 3. Gomoku under Wall Clock. Gemini 3.8 Flash (high) reasons for 12.5 s against a 10-second move clock and loses on time.

2.4Turn-based models miss changes during inference.

Current models interact in a turn-based manner: they receive an observation, reason, act, and only then receive the next observation. While a model is reasoning, it cannot observe how the environment is changing, so it misses intermediate observations and acts on a stale state. In Pong, GPT-6 Astra (high) receives a new frame only every 4–7 seconds, at 7 s, 12 s, 19 s, and 25 s. Between two observations the ball crosses the court and rebounds. The model plans for the ball to keep moving up and right, although it is already returning left and downward, leaving the paddle poorly positioned. Continuous perception is therefore essential for real-time models: they must process incoming information and revise actions with sufficiently low latency.

3What's Next

Our experiments showed that real-time interaction remains challenging for frontier models. Future models should adapt their reasoning to the time available, balancing the benefits of further deliberation against the risk of acting too late. They should also support continuous perception and action, using timely observations to revise decisions as the environment evolves.

We invite the community to develop more capable and efficient real-time models, build harnesses that support timely interaction in evolving environments, and expand our benchmark with richer tasks and real-world scenarios.

View the Leaderboard · Read the Paper · View on GitHub