Note:If you encounter any issues playing the video on this page, refresh your browser and select “Watch on YouTube” in the lower-right corner of the video player.
Healthcare organizations face rising call volumes, limited staff availability, and growing patient expectations for immediate support. Patients need quick answers about appointments, office hours, insurance, prescription refills, and basic intake questions, but many clinics still struggle with missed calls, long wait times, and after-hours gaps.
To address this, we built a prototype of a real-time Healthcare Call Agent using NVIDIA PersonaPlex (a research model) using LiveKit for orchestration. The solution combines full-duplex speech-to-speech AI, NVIDIA GPU-powered model deployment, LiveKit’s IVR capabilities, and persona-driven prompt customization to create a virtual medical assistant that is available 24/7.
Important: The NVIDIA PersonaPlex real-time model integration described in this article is experimental. PersonaPlex is an experimental speech-to-speech model and an experimental feature. Its APIs, capabilities, and integration patterns may change in future releases. The implementation presented here demonstrates the current capabilities of the technology and is intended for evaluation and research purposes. It is not recommended for production deployments at this time.
Why Traditional Voice AI Falls Short
Most voice assistants rely on a cascaded pipeline: speech recognition, a language model, and text-to-speech. While modern implementations can achieve sub-second latency, each stage still operates as a discrete step which can make it harder to handle natural conversational behaviors like interruptions, mid-sentence corrections, or overlapping speech.
In healthcare, these edge cases matter. Patients may pause, clarify, or change details mid-conversation. When each stage processes sequentially, the system has added latency and less ability to respond to these moments in real time.
How PersonaPlex Improves the Experience
NVIDIA PersonaPlex is a full-duplex speech-to-speech research model designed for real-time conversation. It can listen and speak simultaneously, enabling smoother turn-taking, faster responses, interruptions, and natural conversational backchannels.
For this healthcare use case, PersonaPlex is customized with a medical assistant prompt that defines the agent’s role, tone, responsibilities, and instructions for capturing patient information. This helps the agent behave consistently as a professional virtual receptionist while supporting common patient requests.
Built on the Moshi Architecture
PersonaPlex is built on the Moshi architecture, designed for real-time, full-duplex speech interaction. Unlike traditional systems that separate speech recognition, reasoning, and text-to-speech, Moshi-style architectures support continuous speech-to-speech conversation.
PersonaPlex uses two inputs to define conversational behavior:
- Voice prompt: An audio embedding that captures vocal characteristics, speaking style, and prosody.
- Text prompt: A natural language instruction that defines the agent’s role, background information, responsibilities, and conversation context.
These inputs are processed jointly to create a coherent persona expressed through both speech and behavior. For the healthcare call agent, this means the model can sound consistent, respond naturally, and follow the medical assistant instructions throughout the conversation.
Its dual-stream design allows the agent to listen and speak at the same time, enabling faster responses, smoother turn-taking, interruptions, and more human-like patient interactions.
Fig 1: Hybrid Prompting Architecture.
How LiveKit Enables 24/7 Availability
LiveKit provides the real-time communication and IVR layer that makes the agent accessible around the clock. Its IVR capabilities offer a simple, straightforward way to connect callers to the AI agent and manage live voice interactions.
In the video, we demonstrate how the PersonaPlex model is deployed using NVIDIA GPU infrastructure, hosted, and then integrated with LiveKit. Once connected, LiveKit handles the voice interaction flow while PersonaPlex powers the real-time conversational experience.
Architecture Overview
The solution combines LiveKit, NVIDIA PersonaPlex, and Oracle Cloud Infrastructure (OCI) GPU instances to support a real-time healthcare voice assistant. LiveKit provides the telephony and real-time media layer, including support for inbound and outbound AI-powered voice calls, SIP trunks, phone numbers, and integrations with providers such as Twilio and WhatsApp. PersonaPlex serves as the speech-to-speech conversational model, enabling low-latency, full-duplex interaction with role and voice conditioning through hybrid text and audio prompts.
In this architecture, PersonaPlex is deployed on OCI GPU instances powered by NVIDIA A10 Tensor Core GPUs. The GPU infrastructure provides the acceleration needed to process speech, run the model, and generate spoken responses with lower latency during live calls. OCI also provides a scalable cloud foundation, allowing the deployment to expand capacity as call volume grows and support multiple patient conversations in parallel, subject to model serving configuration, GPU sizing, and application-level orchestration.
Though this solution was built with a research model, NVIDIA has created Nemotron 3 VoiceChat (currently in Early Access) for production deployments, with upcoming tool-calling functionality in its next iteration, as it moves towards general availability
Fig 2: OCI Architecture: 24/7 Healthcare Call Agent.
Key Advantages of Leveraging a Healthcare Agent
When built with a model that supports tool-calling, a Healthcare Call Agent can support routine patient interactions such as appointment scheduling and rescheduling, office hours and location inquiries, insurance and service questions, prescription refill requests, basic patient intake, after-hours support, and front-line handoff to human staff when needed.
By handling these common requests around the clock, the agent helps patients get timely support while reducing administrative workload for healthcare teams.
Current Limitations
PersonaPlex is powerful, but built for research and experimentation. Production deployments require supporting systems for deeper workflow integration. Current limitations of the workflow shown in Figure-2 include limited native tool-calling support in the LiveKit integration, no built-in long-term memory, no native transcript output, and the need to manage hosted infrastructure.
Organizations that require scheduling-system integration, electronic health record (EHR) access, compliance logging, or advanced automation may need to add orchestration, transcription, and backend connectors around the model.
For teams looking for production-ready voice agents, a cascaded workflow may still be the best path forward today. The Nemotron Voice Agent example from NVIDIA provides an end-to-end ASR + LLM + TTS reference implementation with sub-second latency, barge-in support, tool calling, and it includes a healthcare scenario example.
Conclusion
NVIDIA PersonaPlex, LiveKit, and Oracle Cloud Infrastructure provide a practical foundation for building always-available healthcare call agents. PersonaPlex delivers natural, full-duplex conversation, while LiveKit enables patients to reach the agent through real-time voice and telephony workflows. Deploying PersonaPlex on OCI GPU instances powered by NVIDIA A10 Tensor Core GPUs provides the acceleration needed to process speech, generate responses quickly, and support low-latency conversations during live calls.
With the model running on OCI, teams can integrate it with LiveKit or another compatible real-time voice platform, customize the medical assistant prompt, and configure the agent for healthcare workflows. The result is a real-time, human-like voice assistant that can help patients receive support when they need it, without waiting for office hours or staff availability.