AI EngineerVideoSource Linked

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Edge-agent design is often memory-bound: use small prompted models when devices can spare gigabytes, or fine-tune tiny models for narrow functions when reach and latency matter more.

AI Engineer · Jul 25, 2026

Open Source Open Markdown Open JSON

Source Summary

Google’s optimized **2B model uses 841 MB for weights**, but runtime and KV cache can push it toward **2 GB of active RAM**. A fine-tuned **270M-parameter** model reached **45 tokens/sec and over 86% function-call reliability** across ten mobile actions.

Practical Implication

Start with the device’s memory and latency budget. If a small prompted model fits, use it; otherwise choose a task-specific tiny model, generate representative synthetic training data, and fine-tune it for a narrow function such as voice-to-action.

Agent-Ready Context

Google’s optimized **2B model uses 841 MB for weights**, but runtime and KV cache can push it toward **2 GB of active RAM**. A fine-tuned **270M-parameter** model reached **45 tokens/sec and over 86% function-call reliability** across ten mobile actions.

Start with the device’s memory and latency budget. If a small prompted model fits, use it; otherwise choose a task-specific tiny model, generate representative synthetic training data, and fine-tune it for a narrow function such as voice-to-action.

Tiny models are not drop-in general agents. Their reliability depends on task-specific data and tuning, while visual workloads and Raspberry Pi interactions can still miss real-time expectations. The reported figures apply to the demonstrated configurations.

Context Map

model#model-selection#tool-use#open-models

Uncertainty

Tiny models are not drop-in general agents. Their reliability depends on task-specific data and tuning, while visual workloads and Raspberry Pi interactions can still miss real-time expectations. The reported figures apply to the demonstrated configurations.