Vaakini docs

API

The WebSocket API

/v1/stream: send a reply's text as it is written, and get each sentence's audio back as soon as the sentence is complete.

View as Markdown

This is the protocol the LiveKit plugin, the Pipecat service and the Python SDK speak. Use it directly from any other language.

Connect#

Open a WebSocket to wss://api.vaakini.com/v1/stream with your API key in the handshake.

Authorization: Bearer vk_…

A missing or wrong key is refused at the handshake with HTTP 401, and an org out of credit with HTTP 402, each with the error body every route answers.

A segment#

A connection carries segments one after another. A segment is one reply.

  1. You send start, then text any number of times as the reply is written, then flush.
  2. Vaakini splits the text into sentences. For each, as soon as it is complete, it sends a sentence message and then the sentence’s audio in binary frames.
  3. The segment ends with done, or with cancelled if you sent cancel, or with error.

Your messages are JSON text frames. One segment is open at a time; after done or cancelled, the next start may follow on the same connection.

A whole client#

In Python, with the websockets package. It speaks one reply and writes its audio to a file.

pip install websockets
reply.py
import asyncioimport jsonimport osfrom websockets.asyncio.client import connectURL = "wss://api.vaakini.com/v1/stream"# Your LLM's reply, as it writes it.PIECES = ["नमस्ते, आपका ", "ऑर्डर कल पहुँच ", "जाएगा। डिलीवरी से ", "पहले आपको कॉल आएगा।"]async def main() -> None:    headers = {"Authorization": f"Bearer {os.environ['VAAKINI_API_KEY']}"}    async with connect(URL, additional_headers=headers) as ws:        start = {            "type": "start",            "segment_id": "reply-1",            "model": "vaakini-tts-1",            "voice": "Monika",            "language": "hi",            "sample_rate": 24000,            "format": "pcm",        }        await ws.send(json.dumps(start))        for piece in PIECES:            await ws.send(json.dumps({"type": "text", "segment_id": "reply-1", "text": piece}))        await ws.send(json.dumps({"type": "flush", "segment_id": "reply-1"}))        with open("reply.pcm", "wb") as f:            async for message in ws:                if isinstance(message, bytes):                    f.write(message)  # the audio of the sentence announced last                    continue                event = json.loads(message)                print(event)                if event["type"] in ("done", "cancelled", "error"):                    breakasyncio.run(main())
{'type': 'sentence', 'index': 0, 'text': 'नमस्ते, आपका ऑर्डर कल पहुँच जाएगा।'}{'type': 'sentence', 'index': 1, 'text': 'डिलीवरी से पहले आपको कॉल आएगा।'}{'type': 'done', 'segment_id': 'reply-1', 'sentences': 2, 'characters': 64}

What you send#

start#

Opens a segment and says how it is spoken. Any field not listed here is refused.

{  "type": "start",  "segment_id": "reply-1",  "model": "vaakini-tts-1",  "voice": "Monika",  "language": "hi",  "sample_rate": 24000,  "format": "pcm",  "options": {"temperature": 0.6}}
segment_idstringRequired
Yours, 1 to 128 characters. It is echoed back in the messages that end the segment.
modelstringRequired
vaakini-tts-1.
voicestringRequired
One of Vaakini’s voices by name, in any case (Monika), or the id of one of your org’s custom voices (cv_…). Voices and languages lists them.
sample_rateintegerDefault 24000
24000 is the only rate.
languagestringOptional
The language the text is in: as, bn, en, gu, hi, kn, ml, mr, or, pa, ta or te. Left out, the model picks it from the script of each sentence.
formatstringDefault "pcm"
pcm (16-bit little-endian mono, no header) or mp3.
optionsobjectOptional
normalize, temperature, top_k, top_p and repetition_penalty, each optional: see Options.

text#

A piece of the reply, in any chunking: an LLM’s tokens as they arrive, or the whole reply at once. The text can carry emotion tags.

{"type": "text", "segment_id": "reply-1", "text": "नमस्ते, आपका "}

flush#

Ends the segment’s text, exactly once. What is left after the last end mark is spoken as the last sentence.

{"type": "flush", "segment_id": "reply-1"}

cancel#

Ends the open segment now, for a barge-in. It is answered with cancelled. Binary frames carry no segment id, so discard the frames you receive until cancelled arrives.

{"type": "cancel", "segment_id": "reply-1"}

warm#

start’s fields without segment_id, sent between segments: typically once, right after connecting. It readies the model for those settings, so the next segment’s first audio doesn’t wait for that. A valid warm is never answered. One the start would be refused for (a voice that is none of yours, a field it can’t take) is answered with that error, with a null segment_id, and the connection is closed.

{"type": "warm", "model": "vaakini-tts-1", "voice": "Monika", "language": "hi", "sample_rate": 24000}

What you receive#

sentence and its audio#

A sentence message precedes its audio: the binary frames that follow, up to the next text frame, are that sentence’s, in the segment’s format. index counts from 0 within the segment.

{"type": "sentence", "index": 0, "text": "नमस्ते, आपका ऑर्डर कल पहुँच जाएगा।"}

done#

The segment was spoken in full: sentences spoken and their characters, which is what the segment cost. The space between two sentences isn’t counted.

{"type": "done", "segment_id": "reply-1", "sentences": 2, "characters": 64}

cancelled#

Follows the last frame of a cancelled segment, and counts the sentences sent in full before the cancel.

{"type": "cancelled", "segment_id": "reply-1", "sentences": 1, "characters": 34}

error#

Ends the segment, and the connection: Vaakini closes it after an error. If retryable is true, send the reply again on a new connection. The codes are on the Errors page.

{"type": "error", "segment_id": "reply-1", "code": "rate_limited", "message": "vaakini-tts-1 is at capacity, try again", "retryable": true}

segment_id is null when no segment was open: a message that isn’t valid, or one out of order.

{"type": "error", "segment_id": null, "code": "bad_request", "message": "`text` before `start`", "retryable": false}

Limits and closes#

  • A message you send may be at most 64 KB, and a segment’s text 20,000 characters.
  • A connection that sends nothing for 30 minutes while Vaakini waits for input is closed.
  • A close with code 1012 means the server is restarting: connect again. A reply that was being spoken has to be sent again.