Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind
For fuzzy requests, an agent should identify the missing constraint with the most decision value, elicit it in the right modality, then choose a response format suited to the task.
The proposed loop builds a working state from conversation history, personal context, references, hard constraints, soft preferences, confidence, and live variables such as inventory. It then asks the **single highest-utility question**, such as room width before suggesting furniture.
Treat elicitation as an agent policy: identify blockers, avoid repeated questioning, and select **text or visual preference boards** according to whether the user can articulate the constraint. Map each answer into the catalog ontology before retrieval, then choose a response format such as a comparison table or inspiration images.
The proposed loop builds a working state from conversation history, personal context, references, hard constraints, soft preferences, confidence, and live variables such as inventory. It then asks the **single highest-utility question**, such as room width before suggesting furniture. Treat elicitation as an agent policy: identify blockers, avoid repeated questioning, and select **text or visual preference boards** according to whether the user can articulate the constraint. Map each answer into the catalog ontology before retrieval, then choose a response format such as a comparison table or inspiration images. This is a commerce-grounded framework, not reported performance evidence. Merchant ontologies remain important, and agent-to-agent shopping is described as **an early-stage possibility**, with direct user involvement still preferred during discovery.
This specializes context engineering into an explicit commerce elicitation policy: maintain structured state, ask the one question that removes the most uncertainty, translate answers into merchant ontology, and adapt the response medium. It reinforces process-first scoping while narrowing autonomous shopping claims, since discovery still favors direct user participation and no performance evidence is reported.