MakazhanAlpamys/Soup
Soup v0.73.2 repairs misleading release-gate scores, adds over-refusal checks and measures GPU noise floors; its layer streaming also puts 8B fine-tuning within 4 GB VRAM.
Soup trains from YAML and can stream a frozen base one layer at a time. On an RTX 3050 Laptop, Llama 3.1 8B with NF4 used **3.32 GB peak VRAM**; the reported 119.6 tok/s predates a later correctness repair and has not been rerun there.
For release decisions, **v0.73.2** fixes two misranked suites, adds mini_over_refusal to catch models that reject benign requests, and introduces **--noise-floor N** to rerun the base and discount deltas within observed GPU variation. Builders should treat these checks as part of the training recipe, not rely on a single aggregate score.
Soup trains from YAML and can stream a frozen base one layer at a time. On an RTX 3050 Laptop, Llama 3.1 8B with NF4 used **3.32 GB peak VRAM**; the reported 119.6 tok/s predates a later correctness repair and has not been rerun there. For release decisions, **v0.73.2** fixes two misranked suites, adds mini_over_refusal to catch models that reject benign requests, and introduces **--noise-floor N** to rerun the base and discount deltas within observed GPU variation. Builders should treat these checks as part of the training recipe, not rely on a single aggregate score. The project says the noise floor sizes observed variation but does not choose a significance threshold. Layer streaming also trades memory for repeated weight reads, and Python support is limited to **3.10–3.12**.
This makes evaluation integrity part of the training loop: corrected suite rankings, a benign-request refusal check, and repeated-base noise measurement all weaken reliance on one aggregate delta. It confirms that small gains need scrutiny, but does not define statistical significance; the headline throughput is also no longer decision-grade because it predates a correctness repair and was not rerun.