Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
A benchmark study finds that relevance-based and rarest-first memory selection both fail in multi-agent decisions, while attribution tags can amplify social-proof effects. Memory quality depends on the evidence retained, not just the selection heuristic.
CrossProfile tests memory selection on **480 tasks** across 60 templates. Relevance-based selection can keep on-topic decoys, while rarest-first performs no better than random on average; removing or randomizing agent attribution tags raised accuracy by **13–24 points** in the classic hidden-profile setup.
Builders compressing multi-agent discussions should test whether summaries preserve the balance of supporting and opposing evidence, and whether attribution labels sway the final decision. A relevance filter alone may not protect against decoys.
CrossProfile tests memory selection on **480 tasks** across 60 templates. Relevance-based selection can keep on-topic decoys, while rarest-first performs no better than random on average; removing or randomizing agent attribution tags raised accuracy by **13–24 points** in the classic hidden-profile setup. Builders compressing multi-agent discussions should test whether summaries preserve the balance of supporting and opposing evidence, and whether attribution labels sway the final decision. A relevance filter alone may not protect against decoys. The authors also report that full transcripts in natural discussions performed worse than no notes. These findings come from specific models and diagnostic tasks, so they do not establish how every memory system will behave.
This narrows confidence in memory-selection heuristics for group decisions: topical relevance can retain decoys, rarity performs no better than random on average, and attribution tags can shift accuracy substantially in a diagnostic setup. It also reports that full transcripts can hurt in natural discussions, so preserving more discussion is not by itself a reliable memory strategy.