Clients
The Pipecat service
pipecat-vaakini: install it, construct VaakiniTTSService, change settings mid-session, and what it reports.
For Pipecat 1.12.0 or later, on Python 3.11 or later. Tested with Pipecat 1.12.0 and 1.16.0.
Install#
pip install pipecat-vaakiniUsage#
Construct the service and put it where a TTS service goes, after the LLM.
import osfrom pipecat_vaakini import VaakiniTTSServicetts = VaakiniTTSService( api_key=os.environ["VAAKINI_API_KEY"], settings=VaakiniTTSService.Settings( voice="Monika", language="hi", ),)# In your pipeline, where a TTS service goes, after the LLM:# Pipeline([transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator])VaakiniTTSService(...)#
api_keystringRequired- Your Vaakini API key.
settingsVaakiniTTSService.SettingsRequired- The voice and how it speaks, below.
base_urlstringDefault wss://api.vaakini.com- Vaakini's URL.
Other keyword arguments go to Pipecat’s WebsocketTTSService, except two. text_aggregation_mode is refused: Vaakini splits the text into sentences itself, so the service streams the LLM’s text to it as it arrives. sample_rate is refused too: the audio is 24000 Hz, and the pipeline’s output transport resamples it to its own rate.
Settings#
VaakiniTTSService.Settings(...), which is VaakiniTTSSettings.
voicestringRequired- One of Vaakini’s voices by name, in any case (
Monika), or the id of one of your org’s custom voices (cv_…). Voices and languages lists them. languagestringOptional- The language the text is in:
as,bn,en,gu,hi,kn,ml,mr,or,pa,taorte. Left out, the model picks it from the script of each sentence. A PipecatLanguageworks too (Language.HI; a region, as inLanguage.HI_IN, is dropped). modelstringDefault "vaakini-tts-1"- The model. This is the only one.
normalizebooleanOptional; None: on- Speak numbers, currency, dates, times, OTPs, phone numbers, URLs and emails as words in the request’s language.
falseturns it off: the text is spoken as written. temperaturenumberOptional; None: the model's default- Sampling temperature, 0.1 to 1.5: lower is steadier, higher is more varied.
top_kintegerOptional; None: the model's default- Sample from the k likeliest codes, 1 to 2048.
top_pnumberOptional; None: the model's default- Nucleus sampling, above 0 and at most 1.
repetition_penaltynumberOptional; None: the model's default- Penalty on recently repeated codes, 0 to 5.
Vaakini checks the settings: a wrong one is reported as an ErrorFrame. Options says what each one does.
A whole bot#
Speech to text, an LLM and Vaakini, on Pipecat's runner.
import osfrom pipecat.audio.vad.silero import SileroVADAnalyzerfrom pipecat.frames.frames import TTSSpeakFramefrom pipecat.pipeline.pipeline import Pipelinefrom pipecat.pipeline.worker import PipelineParams, PipelineWorkerfrom pipecat.processors.aggregators.llm_context import LLMContextfrom pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams,)from pipecat.runner.types import RunnerArgumentsfrom pipecat.runner.utils import create_transportfrom pipecat.services.deepgram.stt import DeepgramSTTServicefrom pipecat.services.openai.llm import OpenAILLMServicefrom pipecat.transports.base_transport import BaseTransport, TransportParamsfrom pipecat.workers.runner import WorkerRunnerfrom pipecat_vaakini import VaakiniTTSServicetransport_params = { "webrtc": lambda: TransportParams(audio_in_enabled=True, audio_out_enabled=True),}async def run_bot(transport: BaseTransport, runner_args: RunnerArguments): stt = DeepgramSTTService( api_key=os.environ["DEEPGRAM_API_KEY"], settings=DeepgramSTTService.Settings(model="nova-3", language="multi"), ) llm = OpenAILLMService( api_key=os.environ["OPENAI_API_KEY"], settings=OpenAILLMService.Settings( model="gpt-4.1-mini", system_instruction=( "You are a receptionist at City Hospital. Speak Hindi in Devanagari script, and " "keep replies to two or three short sentences. No markdown, emojis or lists." ), ), ) tts = VaakiniTTSService( api_key=os.environ["VAAKINI_API_KEY"], settings=VaakiniTTSService.Settings(voice="Monika", language="hi"), ) context = LLMContext() user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()), ) pipeline = Pipeline( [transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator] ) worker = PipelineWorker( pipeline, params=PipelineParams(enable_metrics=True, enable_usage_metrics=True), idle_timeout_secs=runner_args.pipeline_idle_timeout_secs, ) runner = WorkerRunner(handle_sigint=runner_args.handle_sigint) await runner.add_workers(worker) @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): await worker.queue_frames([TTSSpeakFrame("नमस्ते, मैं सिटी हॉस्पिटल से बोल रही हूँ। मैं आपकी क्या मदद कर सकती हूँ?")]) @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): await runner.cancel() await runner.run()async def bot(runner_args: RunnerArguments): transport = await create_transport(runner_args, transport_params) await run_bot(transport, runner_args)if __name__ == "__main__": from pipecat.runner.run import main main()It also needs pipecat-ai[deepgram,openai,runner,silero,webrtc], with DEEPGRAM_API_KEY and OPENAI_API_KEY set. Run it, then open http://localhost:7860/client and talk to it:
python bot.pyChanging settings#
A TTSUpdateSettingsFrame changes the voice, the language or an option mid-session.
from pipecat.frames.frames import TTSUpdateSettingsFrameawait worker.queue_frame( TTSUpdateSettingsFrame(delta=VaakiniTTSService.Settings(voice="Bunty", temperature=0.6)))Replies that start afterwards use the new settings. A setting left out of the update keeps its value; None puts an option back to the model’s default (Settings(temperature=None)).
How a reply is spoken#
The service streams the LLM’s text to Vaakini as it arrives. Vaakini splits it into sentences and speaks each as soon as it is complete, so the first audio follows the reply’s first sentence.
Interruptions#
When the user interrupts, the service tells Vaakini to stop the reply. The next reply starts on the same connection.
Transcripts#
The service’s TTSTextFrames follow the audio: each sentence’s words as its audio starts.
Metrics#
With usage metrics on (enable_usage_metrics=True), each reply’s TTSUsageMetricsData is the characters Vaakini spoke, which are the ones it charges for: the reply’s sentences, without the spaces between them. An interrupted reply counts the sentences sent in full before the interruption.
Errors#
Vaakini's errors become Pipecat ErrorFrames, each with the ErrorCategory that matches its code.
| code | ErrorCategory |
|---|---|
| bad_request | INVALID_REQUEST |
| unauthorized | AUTHENTICATION |
| insufficient_credits | QUOTA |
| rate_limited | RATE_LIMIT |
| unavailable, timeout, internal | SERVER |
A connection that fails or drops is an ErrorFrame with CONNECTIVITY. Errors says what each code means.