# Core dump epidemiology: fixing an 18-year-old bug

Source: [OpenAI](https://openai.com/index/core-dump-epidemiology-data-infrastructure-bug)  
Feed7 permalink: https://feed7.dev/p/core-dump-epidemiology-data-infrastructure-bug-1vwhgy3  
Published: Unknown  
Trust: Official Source (official_source)

## Why Included

OpenAI debugged rare infrastructure crashes by analyzing core dumps at fleet scale, tracing them to a hardware fault plus an 18-year-old software bug. A useful pattern for hunting non-reproducible failures.

## Source Summary

**The gist** OpenAI engineers ran **large-scale core dump analysis** — treating crashes as an epidemiology problem — to chase rare failures in their data infrastructure. The hunt surfaced two root causes: a **hardware fault** and a software bug that had survived for **18 years**.

## Practical Implication

**Why it matters** The method is the takeaway: when a crash is too rare to reproduce, aggregating **core dumps across a fleet** turns anecdotes into statistics you can correlate against hardware, kernel, and workload. The pattern scales down to any team running enough machines for **rare, non-reproducible failures** to show up.

## Agent-Ready Context

**The gist** OpenAI engineers ran **large-scale core dump analysis** — treating crashes as an epidemiology problem — to chase rare failures in their data infrastructure. The hunt surfaced two root causes: a **hardware fault** and a software bug that had survived for **18 years**.

**Why it matters** The method is the takeaway: when a crash is too rare to reproduce, aggregating **core dumps across a fleet** turns anecdotes into statistics you can correlate against hardware, kernel, and workload. The pattern scales down to any team running enough machines for **rare, non-reproducible failures** to show up.

**Watch out** The article couldn't be fetched, so the specifics — **which component** hosted the 18-year-old bug, the tooling used, and whether fixes were upstreamed — aren't covered here; read the original for the actual diagnosis.

## Context Map

- Layer: infra
- Domains: data
- Topics: observability

## Uncertainty

- The article couldn't be fetched, so the specifics — **which component** hosted the 18-year-old bug, the tooling used, and whether fixes were upstreamed — aren't covered here; read the original for the actual diagnosis.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.