Real-Time, Emotion-Aware Speech-to-Speech, On-Premises

Abstract. P99 Labs is a research lab building real-time speech-to-speech models that understand and convey emotion, and that run entirely on your premises. Speech carries far more than words: timing, prosody, hesitation, feeling. Most systems flatten it into text and back, losing the conversation somewhere in the round-trip to a cloud. We model speech directly, end to end, so emotion survives; we deploy on-premises, so your data never leaves your walls; and we hold every system to a real-time latency bound, because a conversation that lags is a conversation that has already broken. This note states the principles behind the systems we build.

Emotion is the signal

Words are a lossy projection of speech. The same sentence can be a question, an apology, or a threat depending on how it is said, and a transcript discards exactly that. We treat prosody, timing, and affect as first-class signal rather than noise to be normalized away. Our models are trained to perceive emotion in the input and to render it faithfully in the output, so that what is felt in a conversation is preserved across it, not stripped at the first encoder.

Speech to speech, end to end

The conventional pipeline is a relay race: it recognizes audio to text, reasons over text, then synthesizes back to audio, and each handoff drops latency and meaning on the floor. We build speech-to-speech models that map audio to audio directly, keeping the acoustic and emotional context intact through a single system. Fewer stages means fewer places to lose the feeling of the original voice, and fewer milliseconds spent converting between representations that need never have been separated.

Keep your data on premises

Voice is among the most personal data a person produces, and most platforms stream it to servers where it can be logged, retained, trained on, and subpoenaed, often without anyone noticing. We deploy on-premises: the model runs inside your walls, on your hardware, and your audio never leaves them. Privacy becomes a property of the architecture rather than a clause in a policy. This is a hard constraint on every system we ship, not a premium tier or an optional mode.

Real-time latency

Conversation is a real-time medium; it lives or dies at the tail. A system that is fast on average but stalls at the ninety-ninth percentile still feels broken, because people remember the pause, not the mean. We engineer to a bounded latency target end to end and hold the line at the tail, the p99 we are named for, so the system stays responsive under the conditions that usually expose it.

Agents that act, not just talk

A voice that only converses is half a system. We pair our speech models with a reasoning layer that can act: confirm a caller's identity, look up an account, send a secure link, record a commitment, all by calling real tools and carrying context across the conversation. The talking stays emotive and real-time; the acting stays exact, with the decisions that touch money or compliance handled by deterministic logic rather than left to a model's guess. And it all runs on-premises, which is precisely what lets regulated enterprises, where audio and account data may not leave the building, automate voice conversations at all.

How we work

We are interested in the unfashionable version of this problem: making careful systems engineering, compact models, and exact behavior do the work the industry tends to throw scale at. We hold our systems to engineering standards rather than demo standards: bounded latency, predictable output, no silent failure modes, and no data leaving the machine. We would rather decline a feature than ship one we cannot make correct.