Show HN: Voice bots with 500ms response times Last year when GPT-4 was released I started making lots of little voice + LLM experiments. Voice interfaces are fun; there are several interesting new problem spaces to explore. I'm convinced that voice is going to be a bigger and bigger part of how we all interact with generative AI. But one thing that's hard, today, is building voice bots that respond as quickly as humans do in conversation. A 500ms voice-to-voice response time is just barely possible with today's AI models. You can get down to 500ms if you: host transcription, LLM inference, and voice generation all together in one place; are careful about how you route and pipeline all the data; and the gods of both wifi and vram caching smile on you. Here's a demo of a 500ms-capable voice bot, plus a container you can deploy to run it yourself on an A10/A100/H100 if you want to: https://ift.tt/2nZARW8 We've been collecting lots of metrics. Here are typical numbers (in milliseconds) for all the easily measurable parts of the voice-to-voice response cycle. macOS mic input 40 opus encoding 30 network stack and transit 10 packet handling 2 jitter buffer 40 opus decoding 30 transcription and endpointing 200 llm ttfb 100 sentence aggregation 100 tts ttfb 80 opus encoding 30 packet handling 2 network stack and transit 10 jitter buffer 40 opus decoding 30 macOS speaker output 15 ---------------------------------- total ms 759 Everything in AI is changing all the time. LLMs with native audio input and output capabilities will likely make it easier to build fast-responding voice bots soon. But for the moment, I think this is the fastest possible approach/tech stack. https://ift.tt/2nZARW8 June 27, 2024 at 03:21AM
Show HN: Voice bots with 500ms response times https://ift.tt/IJwQmkL
Related Articles
Show HN:Free Online Tool to Experience Microsoft's MarkItdown https://ift.tt/CPv6zO4Show HN:Free Online Tool to Experience Microsoft's MarkItdown https://… Read More
Show HN: Web worker and polling exp / HN client app https://ift.tt/wCxpByeShow HN: Web worker and polling exp / HN client app demo and src insid… Read More
Show HN: Skybear.net – A managed platform automating Synthetic HTTP API testing https://ift.tt/DwcflMHShow HN: Skybear.net – A managed platform automating Synthetic HTTP AP… Read More
Show HN: Rivet Actors – Durable Objects build with Rust, FoundationDB, Isolates https://ift.tt/fHQEMF0Show HN: Rivet Actors – Durable Objects build with Rust, FoundationDB,… Read More
Show HN: City Summit – buildings data visualization project https://ift.tt/sBFGSp9Show HN: City Summit – buildings data visualization project https://if… Read More
Show HN: Model Validation Using LLMs https://ift.tt/VGSW1tNShow HN: Model Validation Using LLMs https://ift.tt/fwCEQ30 December 2… Read More
Show HN: Get e-signatures & pay per signed doc https://ift.tt/SpKLDqfShow HN: Get e-signatures & pay per signed doc Woke up today with … Read More
Show HN: openai-realtime-embedded-SDK Build AI assistants on microcontrollers https://ift.tt/tPmwEUMShow HN: openai-realtime-embedded-SDK Build AI assistants on microcont… Read More
0 Comments: