> ## Documentation Index
> Fetch the complete documentation index at: https://api-docs.wisprflow.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocket API

Access the Flow WebSocket API

## How it works

### Connect:

Open a WebSocket connection to the server

**API-key endpoint:** `wss://platform-api.wisprflow.ai/api/v1/dash/ws?api_key=Bearer%20<YOUR_API_KEY>`

**Client auth endpoint (Recommended):** `wss://platform-api.wisprflow.ai/api/v1/dash/client_ws?client_key=Bearer%20<CLIENT_KEY>`

We recommend using the client endpoint with client-side authentication to connect directly to our servers to lower latency.

### Send Messages:

* **Start**: Begin a new transcription session
* **Append**: Send audio chunks sequentially, ideally 1-second chunks

<Note>
  Make sure the chunks you send are of the same duration. Varying durations can cause unexpected failures.
</Note>

### Receive Responses:

Partial and final results are returned during the session.

| TYPE   | DESCRIPTION                                                         | Example                                                             |
| ------ | ------------------------------------------------------------------- | ------------------------------------------------------------------- |
| auth   | Authenticates the client with the server                            | `{ "type": "auth", "access_token": "..." }`                         |
| append | Sends a chunk of audio data and (optionally) metadata to the server | `{ "type": "append", "audio": "...chunk_audio...", "position": 0 }` |
| commit | Mark the end of audio                                               | `{ "type": "commit", "total_packets": 284 }`                        |

Once connected, the first message you send must be a start message, with any API properties you'd like to pass in. After receiving confirmation that the session started, you can send multiple chunks of audio data. You must stream base64-encoded single-channel 16-bit (int16) PCM wav data sampled at 16 KHz. When you finish sending all audio chunks, you MUST send a commit message that includes the total number of audio chunks sent.

## Step-by-Step Guide

### Establish WebSocket Connection

Connect to the WebSocket server using your API key:

```javascript theme={null}
const ws = new WebSocket("wss://platform-api.wisprflow.ai/api/v1/dash/ws?api_key=Bearer%20<YOUR_API_KEY>");

ws.onopen = () => {
  console.log("WebSocket connected");
};

ws.onerror = (error) => {
  console.error("WebSocket error:", error);
};

ws.onclose = () => {
  console.log("WebSocket closed");
};
```

### Start a Session

Send a start message to initialize the transcription session:

```javascript theme={null}
ws.send(JSON.stringify({
  type: "auth",
  access_token: "...",
  language: ["en"], // If known beforehand, set to list of languages the user speaks
  context: { // Optional contextual information used to improve dictation quality when applicable
    app: {
      name: "Weather Forecast Chatbot",
      type: "ai" // one of {email, ai, other}
    },
    dictionary_context: [], // list of uncommon names or words relevant to the context
    user_identifier: "john_doe_1",
    user_first_name: "John",
    user_last_name: "Doe",
    textbox_contents: {
      before_text: "", // text immediately before cursor position
      selected_text: "", // text highlighted by cursor for replacement
      after_text: "" // text immediately after cursor position
    },
    screenshot: null, // screenshor for when speaking about something on the screen
    content_text: null, // plaintext content of the app user is dictating in
    content_html: null, // HTML content of the app user is dictating in
    conversation: { // chatbot style history of conversation (messaging or AI app)
      id: "ai_chat_126", // conversation identifier
      participants: ["John Doe", "AI Assistant"], // list of people the user might be addresing
      messages: [ // list of messages in conversation, in chronological order
        {
          role: "user" // One of {user, human, assistant}
          content: "How is the weather in SF today?"
        },
        {
          role: "assistant"
          content: "It's partly cloudy. Do you want to know the temperature?"
        }
      ]
    }
  }
}));
```

Upon successful authentication, the server will respond with an `"auth"` message.

### Stream Audio

Send audio chunks using append messages. Ensure each chunk is Base64-encoded and includes a sequential chunk\_number:

```javascript theme={null}
ws.send(JSON.stringify({
  type: "append",
  position: 0,
  audio_packets: {
    packets: [base64Audio1, base64Audio2, base64Audio3, ...], // you can send any number of packets
    volumes: [volume1, volume2, volume3, ...], // The number of volumes must match the number of packets
    packet_duration: BUFFER_SIZE / audioContext.sampleRate, // in seconds
    audio_encoding: "wav", // pcm wav data 
    byte_encoding: "base64" // set to "binary" for reduced message size (see "Network Optimization" below)
  }
}));
```

<Note>
  If you send 4 packets in the first and second `append` messages, the third append message must have `position: 8`
</Note>

### Receive Final Transcription

When all audio chunks have been sent, send a final append message with `final: true`:

```javascript theme={null}
ws.send(JSON.stringify({
  type: "commit",
  total_packets: n
}));

ws.onmessage = (event) => {
  const response = JSON.parse(event.data);
  if (response.status == "text") {
    if (response.final) { // final transcription result (websocket will close)
      console.log("Final server response");
    } else { // intermediate transcription result (roughly every 30 seconds)
      console.log("Interim server response");
    }
  } else if (response.status == "error") { // unrecoverable (websocket will close)
    console.log("Server returned error response");
  } else { // other information
    console.log("Server returned other response");
  }
};
```

## Responses

### Connection Authenticated

```json theme={null}
{
  "status": "auth"
}
```

### Commit Acknowledgement

```json theme={null}
{
  "status": "info",
  "message": {
    "event": "commit_received"
  }
}
```

### Partial Transcriptions

```json theme={null}
{
  "status": "text",
  "position": 180,
  "final": false,
  "body": {
    "text": "This is a partial transcription...",
    "detected_language": "en"
  }
}
```

### Final Transcription

```json theme={null}
{
  "status": "text",
  "position": 205,
  "final": false,
  "body": {
    "text": "This is a partial transcription... and this is the full text.",
    "detected_language": "en"
  }
}
```

## Network Optimization

To reduce the message size being sent over the network, instead of encoding binary data (audio and screenshot) using base64, you can serialize the message into the MessagePack format and send as binary data.

To enable binary transmission mode, add the following field to the websocket header:

```typescript theme={null}
Encoding: 'msgpack'
```

Also, set `byte_encoding: "binary"` in the append messages.

<Note>
  When binary transmission mode is activated, the responses received over the websocket will also be serialized using MessagePack and will need deserialization by the client.
</Note>