GitHubGitHub RepoNeeds Review

jundot/omlx

oMLX serves local models to coding agents on Apple Silicon, combining continuous batching, persistent SSD-backed KV caching, and menu-bar control behind compatible APIs.

GitHub

Open Source Open Markdown Open JSON

Source Summary

oMLX is an Apple Silicon inference server with **continuous batching** and a two-tier KV cache spanning RAM and SSD. It exposes OpenAI- and Anthropic-compatible APIs and can configure Codex, OpenCode, Copilot, and other clients.

Practical Implication

For local agent work, persistent prefix caching can avoid recomputing prior context after eviction or restart. Pin everyday models, set TTLs for heavier ones, and verify native kernels when serving GLM-5.2, MiniMax M3, or Qwen3.5 families.

Agent-Ready Context

oMLX is an Apple Silicon inference server with **continuous batching** and a two-tier KV cache spanning RAM and SSD. It exposes OpenAI- and Anthropic-compatible APIs and can configure Codex, OpenCode, Copilot, and other clients.

For local agent work, persistent prefix caching can avoid recomputing prior context after eviction or restart. Pin everyday models, set TTLs for heavier ones, and verify native kernels when serving GLM-5.2, MiniMax M3, or Qwen3.5 families.

It requires **macOS 15+** and Apple Silicon. The repository reports GLM-5.2 prefill at **845 versus about 29 tokens/s** with native kernels versus fallback on an M3 Ultra, but this is one stated setup rather than a broad benchmark.

Connected Context · Feed7 Judgment

This moves local open-model use from generic compatible serving to Apple-specific inference infrastructure optimized for concurrent agent workloads and durable prefix reuse. It strengthens the case for keeping compatible clients while owning the serving layer, but narrows deployment to recent Apple Silicon systems. The reported native-kernel gap makes backend verification consequential, while one M3 Ultra result remains insufficient for broad model or hardware comparisons.

unslothai/unslothBoth let existing coding agents call local models through compatible APIs; oMLX narrows that pattern to Apple Silicon serving with continuous batching and RAM/SSD KV caching, while Unsloth also covers training and export.esengine/DeepSeek-ReasonixReasonix structures prompts to preserve prefix-cache reuse, while oMLX provides a server-side persistent cache that can retain prior prefixes after eviction or restart.CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon AgentsCompactionRL reduces the context agents carry, whereas oMLX reduces the computation needed to reuse retained context; the mechanisms address different costs in long-running coding sessions.Set up coding agents in one command with AI GatewayBoth offer one compatible endpoint for heterogeneous coding clients, but oMLX emphasizes local ownership and hardware-specific performance while AI Gateway emphasizes centralized routing, budgets, policy, and traces.

Context Map

infracoding#open-models#context-caching#coding-agents

Uncertainty

It requires **macOS 15+** and Apple Silicon. The repository reports GLM-5.2 prefill at **845 versus about 29 tokens/s** with native kernels versus fallback on an M3 Ultra, but this is one stated setup rather than a broad benchmark.