# The WebSocket API

> /v1/stream: send a reply's text as it is written, and get each sentence's audio back as soon as the sentence is complete.

This is the protocol the [LiveKit plugin](https://docs.vaakini.com/livekit.md), the [Pipecat service](https://docs.vaakini.com/pipecat.md) and the [Python SDK](https://docs.vaakini.com/python.md#stream) speak. Use it directly from any other language.

## Connect

Open a WebSocket to `wss://api.vaakini.com/v1/stream` with your API key in the handshake.

```http
Authorization: Bearer vk_…
```

A missing or wrong key is refused at the handshake with HTTP 401, and an org out of credit with HTTP 402, each with the [error body](https://docs.vaakini.com/errors.md#shape) every route answers.

## A segment

A connection carries segments one after another. A segment is one reply.

1. You send `start`, then `text` any number of times as the reply is written, then `flush`.
2. Vaakini splits the text into sentences. For each, as soon as it is complete, it sends a `sentence` message and then the sentence’s audio in binary frames.
3. The segment ends with `done`, or with `cancelled` if you sent `cancel`, or with `error`.

Your messages are JSON text frames. One segment is open at a time; after `done` or `cancelled`, the next `start` may follow on the same connection.

## A whole client

In Python, with the websockets package. It speaks one reply and writes its audio to a file.

```shell
pip install websockets
```

**reply.py**

```python
import asyncio
import json
import os

from websockets.asyncio.client import connect

URL = "wss://api.vaakini.com/v1/stream"
# Your LLM's reply, as it writes it.
PIECES = ["नमस्ते, आपका ", "ऑर्डर कल पहुँच ", "जाएगा। डिलीवरी से ", "पहले आपको कॉल आएगा।"]


async def main() -> None:
    headers = {"Authorization": f"Bearer {os.environ['VAAKINI_API_KEY']}"}
    async with connect(URL, additional_headers=headers) as ws:
        start = {
            "type": "start",
            "segment_id": "reply-1",
            "model": "vaakini-tts-1",
            "voice": "Monika",
            "language": "hi",
            "sample_rate": 24000,
            "format": "pcm",
        }
        await ws.send(json.dumps(start))
        for piece in PIECES:
            await ws.send(json.dumps({"type": "text", "segment_id": "reply-1", "text": piece}))
        await ws.send(json.dumps({"type": "flush", "segment_id": "reply-1"}))

        with open("reply.pcm", "wb") as f:
            async for message in ws:
                if isinstance(message, bytes):
                    f.write(message)  # the audio of the sentence announced last
                    continue
                event = json.loads(message)
                print(event)
                if event["type"] in ("done", "cancelled", "error"):
                    break


asyncio.run(main())
```

```text
{'type': 'sentence', 'index': 0, 'text': 'नमस्ते, आपका ऑर्डर कल पहुँच जाएगा।'}
{'type': 'sentence', 'index': 1, 'text': 'डिलीवरी से पहले आपको कॉल आएगा।'}
{'type': 'done', 'segment_id': 'reply-1', 'sentences': 2, 'characters': 64}
```

## What you send

### `start`

Opens a segment and says how it is spoken. Any field not listed here is refused.

```json
{
  "type": "start",
  "segment_id": "reply-1",
  "model": "vaakini-tts-1",
  "voice": "Monika",
  "language": "hi",
  "sample_rate": 24000,
  "format": "pcm",
  "options": {"temperature": 0.6}
}
```

- `segment_id` (`string`, Required): Yours, 1 to 128 characters. It is echoed back in the messages that end the segment.
- `model` (`string`, Required): `vaakini-tts-1`.
- `voice` (`string`, Required): One of Vaakini’s voices by name, in any case (`Monika`), or the id of one of your org’s [custom voices](https://docs.vaakini.com/custom-voices.md) (`cv_…`). [Voices and languages](https://docs.vaakini.com/voices.md) lists them.
- `sample_rate` (`integer`, Default 24000): 24000 is the only rate.
- `language` (`string`, Optional): The language the text is in: `as` , `bn` , `en` , `gu` , `hi` , `kn` , `ml` , `mr` , `or` , `pa` , `ta` or `te`. Left out, the model picks it from the script of each sentence.
- `format` (`string`, Default "pcm"): `pcm` (16-bit little-endian mono, no header) or `mp3`.
- `options` (`object`, Optional): `normalize`, `temperature`, `top_k`, `top_p` and `repetition_penalty`, each optional: see [Options](https://docs.vaakini.com/options.md).

### `text`

A piece of the reply, in any chunking: an LLM’s tokens as they arrive, or the whole reply at once. The text can carry [emotion tags](https://docs.vaakini.com/options.md#emotion-tags).

```json
{"type": "text", "segment_id": "reply-1", "text": "नमस्ते, आपका "}
```

### `flush`

Ends the segment’s text, exactly once. What is left after the last end mark is spoken as the last sentence.

```json
{"type": "flush", "segment_id": "reply-1"}
```

### `cancel`

Ends the open segment now, for a barge-in. It is answered with `cancelled`. Binary frames carry no segment id, so discard the frames you receive until `cancelled` arrives.

```json
{"type": "cancel", "segment_id": "reply-1"}
```

### `warm`

`start`’s fields without `segment_id`, sent between segments: typically once, right after connecting. It readies the model for those settings, so the next segment’s first audio doesn’t wait for that. A valid `warm` is never answered. One the `start` would be refused for (a voice that is none of yours, a field it can’t take) is answered with that `error`, with a null `segment_id`, and the connection is closed.

```json
{"type": "warm", "model": "vaakini-tts-1", "voice": "Monika", "language": "hi", "sample_rate": 24000}
```

## What you receive

### `sentence` and its audio

A `sentence` message precedes its audio: the binary frames that follow, up to the next text frame, are that sentence’s, in the segment’s `format`. `index` counts from 0 within the segment.

```json
{"type": "sentence", "index": 0, "text": "नमस्ते, आपका ऑर्डर कल पहुँच जाएगा।"}
```

### `done`

The segment was spoken in full: `sentences` spoken and their `characters`, which is what the segment cost. The space between two sentences isn’t counted.

```json
{"type": "done", "segment_id": "reply-1", "sentences": 2, "characters": 64}
```

### `cancelled`

Follows the last frame of a cancelled segment, and counts the sentences sent in full before the cancel.

```json
{"type": "cancelled", "segment_id": "reply-1", "sentences": 1, "characters": 34}
```

### `error`

Ends the segment, and the connection: Vaakini closes it after an `error`. If `retryable` is true, send the reply again on a new connection. The codes are on the [Errors](https://docs.vaakini.com/errors.md) page.

```json
{"type": "error", "segment_id": "reply-1", "code": "rate_limited", "message": "vaakini-tts-1 is at capacity, try again", "retryable": true}
```

`segment_id` is null when no segment was open: a message that isn’t valid, or one out of order.

```json
{"type": "error", "segment_id": null, "code": "bad_request", "message": "`text` before `start`", "retryable": false}
```

## Limits and closes

- A message you send may be at most 64 KB, and a segment’s text 20,000 characters.
- A connection that sends nothing for 30 minutes while Vaakini waits for input is closed.
- A close with code `1012` means the server is restarting: connect again. A reply that was being spoken has to be sent again.
