Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ aai.settings.api_key = "your-key"
- `aai.TTSConfig` — Text-to-Speech options: `voice_id` (a library voice such as `aai-jane`, or a cloned voice by UUID or slug), `model_id` (`assemblyai-tts-1`), `language`, `sample_rate`, `encoding` (`aai.TTSEncoding`: `pcm_s16le` default | `pcm_mulaw` | `pcm_alaw` | `mp3`), `container` (`aai.TTSContainer`: `raw` default | `wav`), `bit_rate`, `extend_biometric_consent` (owner-only consent renewal of a cloned voice). Rejects unknown fields
- `aai.TTSResponse` — Text-to-Speech result: `.audio` (bytes) plus the response headers: `.request_id`, `.region`, `.model`, `.voice_id`, `.language`, `.encoding`, `.sample_rate`, `.container`, `.bit_rate`, `.content_type`, `.voice_expires_at`, `.billed_characters`, `.audio_duration_ms`. `TTSStream.info` carries the same fields without `audio`; `audio_duration_ms` is `None` on a stream with audio (`0` when the text had nothing to speak)
- `aai.TTSError` — Raised on any Text-to-Speech refusal: `.status_code`, `.error_code` (stable API code; `biometric_terms_not_accepted` 403 until the account accepts the biometric data terms in the dashboard, `auth_unavailable` 503 retryable; on a session these arrive as close 3006 and 1011, with `status_code` None, the close code in `.body["close_code"]` and `.retryable` from the API's Error frame), `.param`, `.request_id`, `.retryable`, `.retry_after`, `.errors`, `.body` (the whole envelope)
- `aai.Diarizer` — Diarization API: speaker labels for one audio file per request. `diarize(audio, transcript=None, config=None, *, content_type=None)`: `audio` is a path, bytes or binary file object; `transcript` is a `SyncTranscriptResponse` (request it with `timestamps=True`), a dict with `words`, or a list of words (`text`, `start`/`end` ms, optional `confidence`). Omit `transcript` for segments-only mode (`words`/`utterances` empty). Content-Type from the extension (`audio/wav` default, `audio/pcm` when `sample_rate`/`channels` set, `audio/mpeg`, `audio/flac`, `audio/ogg`, `audio/mp4`, `audio/webm`) unless `content_type=` is given. Host from `aai.settings.diarization_base_url` (`https://diarization.assemblyai.com`; `https://diarization.us.assemblyai.com` / `https://diarization.eu.assemblyai.com` for data residency)
- `aai.AsyncDiarizer` — Asyncio counterpart of `Diarizer`; `diarize()` is a coroutine. Owns an HTTP pool: use `async with` or `await aclose()`
- `aai.DiarizationConfig` — Diarization options: `sample_rate`, `channels` (raw PCM only), `language_code` (default `en`). Rejects unknown fields
- `aai.DiarizationResponse` — Diarization result: `.words` (`DiarizedWord`: `text`, `start`, `end`, `confidence`, `speaker`, `speaker_confidence`), `.utterances` (`DiarizationUtterance`: `speaker`, `text`, `start`, `end`, `confidence`, `speaker_confidence`, `words`), `.segments` (`DiarizationSegment`: `start`, `end`, `speaker`, `speaker_confidence`), `.speakers`, `.audio_duration_ms`, `.session_id`, `.request_time_ms`
- `aai.DiarizationError` — Raised on any Diarization API refusal: `.status_code` (401 bad key, 400 validation, 411/413/415, 5xx), `.error_code` (snake_cased problem title, e.g. `unauthorized`, `bad_request`), `.retry_after`
- `assemblyai.streaming.v3.RealTimeTranscriber` — Real-time streaming with event-based API (threaded)
- `assemblyai.streaming.v3.AsyncRealTimeTranscriber` — Asyncio-native counterpart; same options/events
- `aai.LLMGateway` — LLM Gateway client. Resources: `models.list()`, `chat.completions.create()` (incl. `stream=True`), `understanding.create()`/`.validate()`, `decisions.create()`
Expand Down
76 changes: 76 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1098,6 +1098,81 @@ print(response.voice_expires_at) # Unix seconds

</details>

### **Diarization Examples**

`aai.Diarizer` labels the speakers in an audio file with one request to the Diarization API (`POST /v1/diarize`). Pass the audio with its transcript to get each word's speaker and the utterances; leave the transcript out to get only the speaker turns, so diarization can run alongside transcription instead of after it. Speakers are labelled `A`, `B`, and so on, up to 8.

The audio is a local path, bytes or a binary file object (not a URL). Its Content-Type, which picks the decoder, comes from the file extension: `audio/wav`, `audio/mpeg`, `audio/flac`, `audio/ogg`, `audio/mp4`, `audio/webm`, or `audio/pcm` for `.pcm`/`.raw` and whenever `sample_rate`/`channels` are set on the `DiarizationConfig`. Audio with no known extension is sent as WAV; pass `content_type=` to set it yourself. `DiarizationConfig` takes `sample_rate` and `channels` (raw PCM only) and `language_code` (default `en`), and rejects unknown fields.

Requests go to `aai.settings.diarization_base_url`, `https://diarization.assemblyai.com` by default; set it to `https://diarization.us.assemblyai.com` or `https://diarization.eu.assemblyai.com` to keep them in the US or EU data zone.

<details>
<summary>Label the words of a sync transcript</summary>

```python
import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

# Word timestamps are needed to place each word: request them.
transcript = aai.SyncTranscriber().transcribe(
"./call.wav", config=aai.SyncTranscriptionConfig(timestamps=True)
)

result = aai.Diarizer().diarize("./call.wav", transcript=transcript)

for utterance in result.utterances:
print(f"Speaker {utterance.speaker}: {utterance.text}")

for word in result.words:
print(word.text, word.start, word.end, word.speaker, word.speaker_confidence)
```

`transcript` also takes a dict with a `words` list, or the list of words alone, each with `text`, `start` and `end` in milliseconds and an optional `confidence`; other fields are ignored. A word without timings gets them from the words next to it. An empty list returns empty results.

</details>

<details>
<summary>Speaker turns only (no transcript)</summary>

```python
result = aai.Diarizer().diarize(
pcm_bytes,
config=aai.DiarizationConfig(sample_rate=16000, channels=1),
)

print(result.speakers) # ["A", "B"]
for segment in result.segments:
print(segment.speaker, segment.start, segment.end, segment.speaker_confidence)
```

Without a transcript, `words` and `utterances` are empty and `segments` and `speakers` are filled.

</details>

<details>
<summary>Asyncio and errors</summary>

```python
import asyncio
import assemblyai as aai

async def main():
async with aai.AsyncDiarizer() as diarizer:
try:
result = await diarizer.diarize("./call.mp3")
except aai.DiarizationError as error:
print(error.status_code, error.error_code, error)
return
print(result.speakers, result.audio_duration_ms)

asyncio.run(main())
```

`aai.AsyncDiarizer` owns an HTTP pool: use `async with` or `await aclose()`, or pass an `aai.AsyncClient` to share one. Refusals raise `aai.DiarizationError` with the HTTP `status_code`, the snake_cased problem `title` as `error_code` (`unauthorized` for a missing or invalid key, `bad_request` for an unknown config field or a transcript without `words`, `audio_too_large`, `unsupported_media_type`, `capacity_exceeded`, `inference_timeout`), and `retry_after` when the API sends `Retry-After`. Requests wait up to `aai.settings.diarization_http_timeout` (330 s by default).

</details>

### **Asyncio Examples**

`aai.AsyncTranscriber` is the asyncio counterpart of `aai.Transcriber`. Every method
Expand Down Expand Up @@ -2224,6 +2299,7 @@ All custom SDK exceptions inherit from `aai.AssemblyAIError`, except the streami
| `aai.TranscriptError` | Raised when a prerecorded transcript API call fails at the HTTP level or polling times out. | `status_code: Optional[int]` |
| `aai.SyncTranscriptError` | Raised when synchronous transcription fails (`aai.SyncTranscriber`). | `status_code: Optional[int]`, `error_code: Optional[str]`, `retry_after: Optional[int]` |
| `aai.TTSError` | Raised when a Text-to-Speech request fails (`aai.TextToSpeech`) or a session is refused or fails (`aai.TTSSession`, where `status_code` is `None`, `request_id` is the session id and `body["close_code"]` is the WebSocket close code). | `status_code: Optional[int]`, `error_code: Optional[str]`, `param: Optional[str]`, `request_id: Optional[str]`, `retryable: Optional[bool]`, `retry_after: Optional[int]`, `errors: Optional[List[Dict[str, Optional[str]]]]`, `body: Optional[dict]` |
| `aai.DiarizationError` | Raised when a Diarization API request fails (`aai.Diarizer`, `aai.AsyncDiarizer`). | `status_code: Optional[int]`, `error_code: Optional[str]`, `retry_after: Optional[int]` |
| `assemblyai.types.RedactedAudioIncompleteError` | Raised when requesting redacted audio before processing completes. | `status_code: Optional[int]` |
| `assemblyai.types.RedactedAudioExpiredError` | Raised when requesting redacted audio that has expired and is no longer available. | `status_code: Optional[int]` |
| `assemblyai.types.RedactedAudioUnavailableError` | Raised when requesting redacted audio that is unavailable at the given URL. | `status_code: Optional[int]` |
Expand Down
18 changes: 18 additions & 0 deletions assemblyai/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
from .__version__ import __version__
from .async_client import AsyncClient
from .client import Client
from .diarization.v1 import (
AsyncDiarizer,
DiarizationConfig,
DiarizationError,
DiarizationResponse,
DiarizationSegment,
DiarizationUtterance,
DiarizedWord,
Diarizer,
)
from .dictation.v1 import (
AsyncDictationTranscriber,
DictationConfig,
Expand Down Expand Up @@ -156,6 +166,7 @@ def __getattr__(name: str):
"ActionItemsResponse",
"AssemblyAIError",
"AsyncClient",
"AsyncDiarizer",
"AsyncDictationTranscriber",
"AsyncLLMGateway",
"AsyncSyncTranscriber",
Expand All @@ -174,6 +185,13 @@ def __getattr__(name: str):
"ContentSafetySeverityScore",
"CustomFormattingRequest",
"CustomFormattingResponse",
"DiarizationConfig",
"DiarizationError",
"DiarizationResponse",
"DiarizationSegment",
"DiarizationUtterance",
"DiarizedWord",
"Diarizer",
"DictationConfig",
"DictationError",
"DictationResponse",
Expand Down
2 changes: 1 addition & 1 deletion assemblyai/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "1.10.12"
__version__ = "1.11.0"
31 changes: 31 additions & 0 deletions assemblyai/diarization/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
"""The Diarization API: audio (and optionally its transcript) in, speaker labels out.

The implementation lives in ``diarization/v1/``, mirroring the ``tts/v1/`` and
``dictation/v1/`` layout. This ``__init__`` re-exports the public surface so
``from assemblyai.diarization import Diarizer`` works alongside the versioned
path.
"""

from .v1 import (
AsyncDiarizer,
DiarizationConfig,
DiarizationError,
DiarizationResponse,
DiarizationSegment,
DiarizationUtterance,
DiarizedWord,
Diarizer,
TranscriptInput,
)

__all__ = [
"AsyncDiarizer",
"DiarizationConfig",
"DiarizationError",
"DiarizationResponse",
"DiarizationSegment",
"DiarizationUtterance",
"DiarizedWord",
"Diarizer",
"TranscriptInput",
]
23 changes: 23 additions & 0 deletions assemblyai/diarization/v1/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
from ._base import TranscriptInput
from .async_client import AsyncDiarizer
from .client import Diarizer
from .models import (
DiarizationConfig,
DiarizationError,
DiarizationResponse,
DiarizationSegment,
DiarizationUtterance,
DiarizedWord,
)

__all__ = [
"AsyncDiarizer",
"DiarizationConfig",
"DiarizationError",
"DiarizationResponse",
"DiarizationSegment",
"DiarizationUtterance",
"DiarizedWord",
"Diarizer",
"TranscriptInput",
]
91 changes: 91 additions & 0 deletions assemblyai/diarization/v1/_base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
from __future__ import annotations

from typing import Any, Dict, Mapping, Optional, Sequence, Tuple, Union

from ... import _audio, types
from ..._audio import AudioInput
from ...dictation.v1._base import _CONTENT_TYPES
from .models import DiarizationConfig

__all__ = [
"AudioInput",
"TranscriptInput",
"check_config",
"resolve_request",
]

# A transcript to label: the sync API's response (request it with
# `timestamps=True`), a dict with a `words` list, or the list of words alone.
# Each word is a `SyncWord` or a dict with `text`, `start` and `end` in
# milliseconds and an optional `confidence`; other fields are ignored.
TranscriptInput = Union[
types.SyncTranscriptResponse,
Mapping[str, Any],
Sequence[Union[types.SyncWord, Mapping[str, Any]]],
]


def check_config(owner: str, config: Optional[DiarizationConfig]) -> None:
"""Raises unless `config` is a `DiarizationConfig` or `None`."""

_audio.check_config(owner, config, DiarizationConfig)


def _dump(model: Any) -> Any:
if hasattr(model, "model_dump"):
return model.model_dump(exclude_none=True)
if hasattr(model, "dict"):
return model.dict(exclude_none=True)
return model


def _transcript_json(transcript: Optional[TranscriptInput]) -> Optional[Dict[str, Any]]:
"""The JSON `transcript` part, or `None` to leave the part out."""

if transcript is None:
return None
if isinstance(transcript, Mapping):
return dict(transcript)
words: Any = getattr(transcript, "words", transcript)
if isinstance(words, (str, bytes, bytearray)) or not isinstance(words, Sequence):
raise TypeError(
"transcript must be a SyncTranscriptResponse, a dict with `words`, "
f"or a list of words, not {type(transcript).__name__}"
)
return {"words": [_dump(w) for w in words]}


def resolve_request(
owner: str,
audio: AudioInput,
transcript: Optional[TranscriptInput],
config: Optional[DiarizationConfig],
content_type: Optional[str],
) -> Tuple[bytes, str, str, Optional[dict], Optional[dict]]:
"""
Reads the audio and builds the parts of a diarize request.

The audio's Content-Type comes from `content_type` when given, else from
the file extension (`.mp3` -> `audio/mpeg`, ...), else `audio/pcm` when
`sample_rate`/`channels` are set, else `audio/wav`.

Returns: `(audio_bytes, filename, content_type, transcript_json, config_json)`.
"""

check_config(owner, config)
config = config or DiarizationConfig()
data, filename, inferred = _audio._resolve_audio(
audio,
sample_rate=config.sample_rate,
channels=config.channels,
owner=owner,
config_name="DiarizationConfig",
content_types=_CONTENT_TYPES,
)
return (
data,
filename,
content_type or inferred,
_transcript_json(transcript),
_audio._config_to_json(config),
)
98 changes: 98 additions & 0 deletions assemblyai/diarization/v1/api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
import json
from typing import Any, List, Optional, Tuple

import httpx

from ... import types
from ...sync.v1.api import _error_from_response as _shared_error_from_response
from .models import DiarizationError, DiarizationResponse

ENDPOINT_DIARIZE = "/v1/diarize"


def _error_from_response(response: httpx.Response) -> types.AssemblyAIError:
"""
Builds a `DiarizationError` from a non-200 response.

The Diarization API answers with the same RFC 9457 problem-details body
(`{"status", "title", "detail"}`) as the sync API, so the parsing is
shared and only the exception class differs.
"""

return _shared_error_from_response(response, DiarizationError, "diarization")


def _files(
*,
audio: bytes,
filename: str,
audio_content_type: str,
transcript: Optional[dict],
config: Optional[dict],
) -> List[Tuple[str, Tuple[Optional[str], Any, str]]]:
"""
The multipart parts of a diarize request.

Buffered, so httpx sends `Content-Length`, which the API requires (411).
An absent `transcript` leaves its part out: segments-only mode.
"""

files: List[Tuple[str, Tuple[Optional[str], Any, str]]] = []
if config is not None:
files.append(
("config", (None, json.dumps(config).encode(), "application/json"))
)
if transcript is not None:
files.append(
("transcript", (None, json.dumps(transcript).encode(), "application/json"))
)
files.append(("audio", (filename, audio, audio_content_type)))
return files


def _parse(response: httpx.Response) -> DiarizationResponse:
if response.status_code != httpx.codes.OK:
raise _error_from_response(response)

return DiarizationResponse.parse_obj(response.json())


def diarize(
client: httpx.Client,
*,
base_url: str,
audio: bytes,
filename: str,
audio_content_type: str,
transcript: Optional[dict],
config: Optional[dict],
timeout: float,
) -> DiarizationResponse:
"""
Posts a diarize request.

Args:
client: the HTTP client (carries the `Authorization` header).
base_url: the Diarization API base URL, e.g. `https://diarization.assemblyai.com`.
audio: the audio bytes.
filename: name for the audio multipart part.
audio_content_type: the audio part's Content-Type; selects the decoder.
transcript: the JSON `transcript` part, or None to leave it out.
config: the JSON `config` part, or None to leave it out.
timeout: the httpx timeout in seconds.

Raises: `DiarizationError` on any non-200 response.
"""

response = client.post(
base_url.rstrip("/") + ENDPOINT_DIARIZE,
files=_files(
audio=audio,
filename=filename,
audio_content_type=audio_content_type,
transcript=transcript,
config=config,
),
timeout=timeout,
)
return _parse(response)
Loading
Loading