Voice AI Stack Evaluation: A Comprehensive Benchmarking Framework for Voice Assistant Pipelines
Published in GitHub, 2025
Recommended citation: Paknezhad, M. "Voice AI Stack Evaluation: A Comprehensive Benchmarking Framework for Voice Assistant Pipelines." GitHub. (2025). https://github.com/MahsaPaknezhad/VoiceAIStackEvaluation
A comprehensive evaluation framework for voice assistants that measures speech-to-text accuracy, response quality, and latency metrics across different AI service combinations. The framework enables systematic comparison of STT/TTS/LLM combinations to optimize for accuracy, latency, and voice quality.
Key Features:
- Evaluates 40 STT×TTS service combinations across multiple dimensions
- Uses Pipecat for voice agent orchestration and Strands Agents for multi-agent conversation management
- Measures Word Error Rate (WER), LLM response quality (correctness, relevance, completeness, clarity), end-to-end latency, and voice quality (NISQA, audio LLM judges)
- Benchmarks services including AWS Transcribe, Deepgram Nova-3, NVIDIA Parakeet, Whisper (STT); AWS Polly, Cartesia, Deepgram Aura, ElevenLabs (TTS); and Amazon Bedrock Claude 3.5 Haiku (LLM)
- Top-performing combination: NVIDIA Parakeet + Cartesia (composite score 0.802)
Pre-computed results and TTS audio samples are available on Hugging Face.
Recommended citation: Paknezhad, M. “Voice AI Stack Evaluation: A Comprehensive Benchmarking Framework for Voice Assistant Pipelines.” GitHub. (2025).