AI EngineerVideoSource Linked

"My name is... my name is...": A Linguistic Map for Voice Agents — Midam Kim, ServiceNow

Voice-agent failures span recognition, wording, turn-taking, and shared context. Debugging only ASR misses the interaction failures that make users repeat themselves or escalate to a person.

AI Engineer · Sep 15, 2026

Open Source Open Markdown Open JSON

Source Summary

The proposed framework maps voice interaction across listening and speaking channels, each covering sounds, words, interaction timing, and the evolving mental model. These layers are **interdependent** and unfold over a timeline whose spoken evidence immediately disappears.

Practical Implication

Instrument failures by layer: recognition and pronunciation, understood and chosen vocabulary, turn detection and latency, then intent and context retention. Treat corrections as updates to shared state rather than asking the user to repeat the same input.

Agent-Ready Context

The proposed framework maps voice interaction across listening and speaking channels, each covering sounds, words, interaction timing, and the evolving mental model. These layers are **interdependent** and unfold over a timeline whose spoken evidence immediately disappears.

Instrument failures by layer: recognition and pronunciation, understood and chosen vocabulary, turn detection and latency, then intent and context retention. Treat corrections as updates to shared state rather than asking the user to repeat the same input.

This is a diagnostic map, not a prescribed implementation. Dynamic handling must adapt to different speakers, emotions, and language change; the talk names **Eva benchmark** as an end-to-end check but supplies no results.

Connected Context · Feed7 Judgment

This supplies a shared diagnostic map for voice-agent failures that were previously treated as separate transcription, latency, pronunciation, or context problems. It emphasizes their interaction over an ephemeral spoken timeline and reframes correction as state repair, while remaining neutral about the architecture needed to implement or evaluate that behavior.

5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, PlivoIts transcription, data-capture, latency, and pronunciation failures become concrete instances of the map’s word, sound, and interaction-timing layers.While my guitar gently speaks — Todd Fisher, Philo VenturesThe talking-guitar system supplies an embodied example of latency and segmentation failures that the linguistic map would instrument across listening and speaking.200 Million Patient Interactions Later — Vivek Muppalla, Hippocratic AIThe clinical stack’s specialist checks and contextual recognition illustrate one possible implementation of layer-specific detection, although the map itself prescribes no architecture.

Context Map

craftaudio#interface-quality#sound-design#agent-reliability

Uncertainty

This is a diagnostic map, not a prescribed implementation. Dynamic handling must adapt to different speakers, emotions, and language change; the talk names **Eva benchmark** as an end-to-end check but supplies no results.