arXivPaperNeeds Review

EquivSVA: A Formally Verified Dataset of Behavioral Assertions Across Equivalent RTL Implementations

EquivSVA tests whether generated hardware assertions describe interface behavior rather than quirks of one RTL implementation, exposing robustness gaps hidden by single-implementation evals.

arXiv · Sep 22, 2026

Open Source Open Markdown Open JSON

Source Summary

EquivSVA organizes **120 behavior families** into four equivalent RTL implementations each, totaling **480 implementations**, 914 gold properties, and 360 mutants. Every family passes a 17-job formal-validation suite.

Practical Implication

Builders evaluating assertion agents should split by behavior family and test outputs across structurally different implementations. In the Qwen2.5-Coder-7B-Instruct case study, only **93 of 293** interface-only properties were formally sound.

Agent-Ready Context

EquivSVA organizes **120 behavior families** into four equivalent RTL implementations each, totaling **480 implementations**, 914 gold properties, and 360 mutants. Every family passes a 17-job formal-validation suite.

Builders evaluating assertion agents should split by behavior family and test outputs across structurally different implementations. In the Qwen2.5-Coder-7B-Instruct case study, only **93 of 293** interface-only properties were formally sound.

This is a specialized hardware-verification dataset, not a broad coding-agent benchmark. Sound-property counts varied across equivalent implementations in **14 of 24** test families, but the paper reports only one model demonstration.

Connected Context · Feed7 Judgment

This adds formally verified, representation-varied evaluation to the benchmark-integrity toolkit. Family-level splits reduce leakage between equivalent implementations, while cross-implementation testing reveals whether assertions capture behavior rather than syntax. The single-model result exposes substantial soundness and robustness gaps but does not establish how other assertion agents perform.

When LLM Decompilers Recompile More and Preserve LessBoth require semantic equivalence checks beyond surface success: EquivSVA formally validates assertions across equivalent RTL, while the decompiler study differentially executes reconstructed code against the original.Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma LabsEquivalent RTL implementations provide controlled environment variation, implementing the broader recommendation to vary benchmark state so fixed structural patterns cannot stand in for adaptation.When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AIThe family-level split and formal validation directly address two benchmark risks highlighted there: leakage between related cases and weak verifiers that reward shortcuts.Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware ScreenBoth show that convenient proxy metrics can misstate correctness and replace them with domain-grounded validation, formal soundness here and execution-grounded security checks for repair.

Context Map

benchmarkcoding#agent-evals#benchmark-integrity#agent-reliability

Uncertainty

This is a specialized hardware-verification dataset, not a broad coding-agent benchmark. Sound-property counts varied across equivalent implementations in **14 of 24** test families, but the paper reports only one model demonstration.