Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi
Shared agents need a different harness: action-boundary security, selective group memory, per-user privacy, and routing that decides both who receives information and when the agent should speak.
An agent deployed among friends and family for **eight months** exposed three group-specific concerns: security at the action boundary, curated shared memory, and recipient-aware routing. The proposed memory stores atomic facts, retrieves selectively, and accounts for time and hierarchy.
Design permissions around both the person and the room. Separate data from instructions, inspect tool actions deterministically, minimize retained facts, and add explicit policies for who may receive an answer and whether the agent should speak at all.
An agent deployed among friends and family for **eight months** exposed three group-specific concerns: security at the action boundary, curated shared memory, and recipient-aware routing. The proposed memory stores atomic facts, retrieves selectively, and accounts for time and hierarchy. Design permissions around both the person and the room. Separate data from instructions, inspect tool actions deterministically, minimize retained facts, and add explicit policies for who may receive an answer and whether the agent should speak at all. Several mechanisms remain proposals, including per-user LoRA adapters for privacy and a classifier for speaking turns. Model-baked permissions are not equivalent to enforceable access control, and the talk gives no measured privacy, security, or retrieval results.
This moves shared-agent context from a retrieval problem to a room-aware governance problem: useful memory must be minimal, selectively retrieved, and constrained by both speaker and audience. It reinforces bounded context and explicit human authority while narrowing shared-context proposals with recipient-level disclosure and silence policies. The eight-month deployment supplies practical concerns, but not measured assurance that the proposed controls work.