LfH uses a VLM to relabel failed rollouts with the tasks they actually achieved, turning failures into training signal for VLA fine-tuning.
Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves 5× improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.
Under sparse rewards, a weak policy fails nearly every rollout, so most GRPO groups have no reward variance and are discarded. LfH recovers this signal: a VLM reads an anchor failure and proposes a hindsight instruction \(g'\) describing what the robot actually did, then scores every trajectory in the group against it, and the policy trains on the relabeled and original groups jointly.
The walkthrough advances automatically; click a step to pin it.
On out-of-distribution LIBERO-PRO tasks, LfH matches GRPO's final performance in roughly a fifth of the training steps — a 5× gain in sample efficiency — and outperforms a dense progress-reward baseline. The benefit transfers to GR00T and OpenVLA-OFT backbones.
Sample efficiency. Normalized gain over the initial \(\pi_{0.5}\) policy, mean over 4 seeds and the same rollout budget for all methods.
Groups kept for training. GRPO discards groups with no reward variance; LfH recovers most of them through relabeling.
Main comparison. (a) LfH reaches higher gain with fewer training steps than GRPO and GRPO+RoboMETER. (b) LfH keeps a larger fraction of trajectory groups usable for training. (c) The benefit transfers beyond \(\pi_{0.5}\) to GR00T and OpenVLA-OFT.
We fine-tune a \(\pi_{0.5}\) policy on a Franka FR3 on the held-out task “put the green container into the bowl.” Both methods start from zero success; at 160 rollouts LfH reaches 56% success while GRPO plateaus at 22%. The overlays report the training iteration and measured success rate.
GRPO. Failed attempts count only as failures of the commanded task; success plateaus at 22%.
GRPO + LfH. Relabeled failures become supervision; success reaches 56%.
Success rate vs. rollouts collected. LfH improves real-world sample efficiency across rollout budgets.
@misc{xu2026learning,
title = {Learning More from Less: Reinforcement Learning from Hindsight},
author = {Iris Xu and Sunshine Jiang and John Marangola and Nitish Dashora and Richard Li
and Thomas Liu and Zexue He and Yuheng Zhi and Alex Pentland and Pulkit Agrawal
and Zhang-Wei Hong},
year = {2026},
eprint = {XXXX.XXXXX},
archivePrefix = {arXiv},
primaryClass = {cs.RO}
}