onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
onPanda turns the first bad token into an annotation boundary, regenerating from each correction to create on-policy supervision with less manual rewriting.
onPanda asks an annotator to find the **first inappropriate token**, replace it from candidates or free-form text, discard the remaining output and regenerate from the corrected prefix. A small controlled study reports **52% lower median annotation time** than manual post-editing.
For agent tuning, capture corrections at the moment a trajectory first diverges instead of rewriting the final answer wholesale. The resulting positions and paired alternatives can support on-policy SFT and preference data, including tool-connected trajectories.
onPanda asks an annotator to find the **first inappropriate token**, replace it from candidates or free-form text, discard the remaining output and regenerate from the corrected prefix. A small controlled study reports **52% lower median annotation time** than manual post-editing. For agent tuning, capture corrections at the moment a trajectory first diverges instead of rewriting the final answer wholesale. The resulting positions and paired alternatives can support on-policy SFT and preference data, including tool-connected trajectories. The time result comes from a small study, and the supplied material gives no breakdown by task or annotator. Whether the workflow scales to long coding trajectories without adding review overhead remains open.
This turns the prior call to mine real agent trajectories into a specific annotation primitive: intervene at the first bad token and preserve the corrected prefix as on-policy training evidence. It may make expert review cheaper than full rewrites, but the small timing study does not establish that first-token correction remains efficient or sufficient for long, branching tool-use failures.