Deepgram voice AI optimization with AWS Trainium2
Deepgram uses AWS Trainium2 to process up to 70% more audio per dollar for batch speech-to-text workloads.
Deepgram has implemented a Deepgram voice AI optimization by porting its Nova-3 speech-to-text model to AWS Trainium2 hardware. This move allows the company to process massive amounts of audio data more efficiently, significantly reducing the cost of high-volume batch workloads.
What's new with Deepgram and AWS Trainium2?
- Deepgram worked with the AWS Forward Deployed Engineering (FDE) team to port its Nova-3 model from standard PyTorch code to AWS Trainium2.
- The optimized configuration runs eight model replicas on a single Amazon EC2 trn2.3xlarge instance.
- The optimization specifically targets offline batch workloads, such as medical transcription and contact-center analytics.
- Evaluations showed that the new setup can process between 11% and 70% more audio per dollar than previous compute options.
Why it matters for voice AI users?
For organizations and developers processing quadrillions of audio tokens, inference efficiency is critical to managing infrastructure budgets. These gains allow Deepgram to scale its services more effectively and potentially offer lower prices to its customers. These benefits also extend to users who self-host Deepgram models within their own virtual private cloud (VPC) on Amazon SageMaker AI.
How does the technical optimization work?
Speech-to-text AI is the process of converting spoken audio into written text. Because this requires intense mathematical calculations, developers must optimize how the AI model interacts with specialized hardware chips to ensure speed and cost-effectiveness.
Deepgram utilized TorchNeuron, which is the native PyTorch integration of the Neuron Compiler. This allowed the team to use torch.compile to ensure that every model operation ran directly on the Trainium2 hardware without falling back to the CPU. The encoder and decoder were compiled as separate static-shape graphs to maximize performance.
To further increase throughput, Deepgram replaced several layers with fused kernels from the Neuron Kernel Interface (NKI) Library. These kernels combine multiple operations—such as fused QKV projections, fused feed-forward network (FFN) layers, and depthwise 1D convolution—into single, more efficient steps. The optimization followed a continuous cycle of enabling NKI kernels, profiling the workload, and then tuning or replacing kernels based on the profiling data.
What hardware is used for this setup?
The optimized Nova-3 model runs on Amazon EC2 trn2.3xlarge instances.
Source: aws.amazon.com