# MakazhanAlpamys/Soup

Source: [GitHub](https://github.com/MakazhanAlpamys/Soup)  
Feed7 permalink: https://feed7.dev/p/soup-1rqg9hp  
Published: Unknown  
Trust: Needs Review (needs_review)

## Why Included

Soup v0.73.2 repairs misleading release-gate scores, adds over-refusal checks and measures GPU noise floors; its layer streaming also puts 8B fine-tuning within 4 GB VRAM.

## Source Summary

Soup trains from YAML and can stream a frozen base one layer at a time. On an RTX 3050 Laptop, Llama 3.1 8B with NF4 used **3.32 GB peak VRAM**; the reported 119.6 tok/s predates a later correctness repair and has not been rerun there.

## Practical Implication

For release decisions, **v0.73.2** fixes two misranked suites, adds mini_over_refusal to catch models that reject benign requests, and introduces **--noise-floor N** to rerun the base and discount deltas within observed GPU variation. Builders should treat these checks as part of the training recipe, not rely on a single aggregate score.

## Agent-Ready Context

Soup trains from YAML and can stream a frozen base one layer at a time. On an RTX 3050 Laptop, Llama 3.1 8B with NF4 used **3.32 GB peak VRAM**; the reported 119.6 tok/s predates a later correctness repair and has not been rerun there.

For release decisions, **v0.73.2** fixes two misranked suites, adds mini_over_refusal to catch models that reject benign requests, and introduces **--noise-floor N** to rerun the base and discount deltas within observed GPU variation. Builders should treat these checks as part of the training recipe, not rely on a single aggregate score.

The project says the noise floor sizes observed variation but does not choose a significance threshold. Layer streaming also trades memory for repeated weight reads, and Python support is limited to **3.10–3.12**.

## Connected Context

Feed7 judgment across 462 accumulated Signals:

This makes evaluation integrity part of the training loop: corrected suite rankings, a benign-request refusal check, and repeated-base noise measurement all weaken reliance on one aggregate delta. It confirms that small gains need scrutiny, but does not define statistical significance; the headline throughput is also no longer decision-grade because it predates a correctness repair and was not rerun.

- [When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI](https://feed7.dev/p/when-will-the-benchmaxxing-plague-end-nick-heiner-surge-ai-178gqcg) — Soup operationalizes the warning that leaderboard gains can mislead by correcting misranked suites, separating refusal behavior, and discounting changes within observed run variation.
- [onepot-Bench 0: towards lab-aware in silico chemistry benchmarks](https://feed7.dev/p/2608-02595v1-0l7cc8k) — Both separate capability from safety-related refusal behavior; Soup’s benign-request suite specifically catches over-refusal, complementing onepot-Bench’s distinct safety-refusal evaluation.
- [The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping](https://feed7.dev/p/2608-06361v1-1n3dr85) — The video study shows aggregate improvements can hide failed intermediate behavior; Soup reaches a parallel implementation conclusion by requiring suite-level checks rather than trusting one combined score.
- [Verifiable Environments for AI in Biology — Kenny Workman, LatchBio](https://feed7.dev/p/verifiable-environments-for-ai-in-biology-kenny-workman-latchbio-1vs6y66) — Both show that evaluator correctness must itself be validated: Soup repaired suite ranking errors, while the biology work found brittle deterministic graders can reject valid paths.

## Context Map

- Layer: benchmark
- Domains: data
- Topics: benchmark-integrity, agent-evals, agent-reliability

## Uncertainty

- The project says the noise floor sizes observed variation but does not choose a significance threshold. Layer streaming also trades memory for repeated weight reads, and Python support is limited to **3.10–3.12**.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.