Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Changelog

## [4.44.0]

- Add the Diarization API: `client.diarization.diarize({ audio, transcript?, config?, contentType? }, { timeout?, signal? })` posts audio, and optionally a transcript with word timings in ms, to `/v1/diarize` and returns `DiarizationResponse` (`words` and `utterances` with speaker labels, `segments`, `speakers`, `audio_duration_ms`, `session_id`, `request_time_ms`). A `SyncTranscriptResponse` can be passed as the transcript; without one only `segments` and `speakers` are filled. The audio content type is inferred from the file extension (`audio/wav`, `audio/mpeg`, `audio/flac`, `audio/ogg`, `audio/mp4`, `audio/webm`, raw `audio/pcm` with `config.sample_rate` and `channels`). Failures throw `DiarizationError` (`status`, `errorCode`, `retryAfter`). New client option `diarizationBaseUrl` (default `https://diarization.assemblyai.com`; `https://diarization.us.assemblyai.com` and `https://diarization.eu.assemblyai.com` for the data zones)

## [4.43.4]

- Add `assemblyai-stt-3-7-realtime` (note taking) and `assemblyai-stt-3-7-realtime-agent` (voice agents) to the streaming `speechModel` options (`StreamingSpeechModel`)
Expand Down
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ const client = new AssemblyAI({
- `client.dictation.transcribeLive(audio, config?, options?)` — Dictation: upload a short spoken note as it is spoken and get one finished transcript back, optionally rewritten by an LLM
- `client.dictation.openLive(config?, options?)` — Open a dictation upload that audio is pushed into (`write()` / `close()` / `result()`), for callback-driven sources
- `client.dictation.warm()` — Pre-open the connection to the dictation API so the next request skips the handshake
- `client.diarization.diarize({ audio, transcript?, config?, contentType? }, { timeout?, signal? })` — Speaker labels for audio: with a transcript (e.g. a sync response requested with `timestamps: true`) every word gets a speaker and `utterances` are filled; without one only `segments` and `speakers` come back. Throws `DiarizationError`. Host: `diarizationBaseUrl` (`https://diarization.assemblyai.com`; data zones `https://diarization.us.assemblyai.com`, `https://diarization.eu.assemblyai.com`)
- `client.streaming.transcriber(params)` — Create a real-time streaming session
- `client.llmGateway.chatCompletions(request)` — OpenAI-compatible chat completions
- `client.llmGateway.listModels()` — List models available on the LLM Gateway
Expand Down
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -757,6 +757,57 @@ can be correlated with the request.

</details>

### Diarize speakers

`client.diarization` labels who spoke when. Post the audio together with a
transcript that has word timings, such as a sync transcript requested with
`timestamps: true`, and every word comes back with a speaker (`A`, `B`, ...),
grouped into utterances. Requests go to `diarizationBaseUrl`,
`https://diarization.assemblyai.com` by default; pass
`https://diarization.us.assemblyai.com` or
`https://diarization.eu.assemblyai.com` to keep them in the US or EU data zone.

```typescript
const transcript = await client.sync.transcribe("./call.wav", {
timestamps: true,
});
const { utterances } = await client.diarization.diarize({
audio: "./call.wav",
transcript, // only `words` is sent
});
for (const u of utterances) console.log(`${u.speaker}: ${u.text}`);
```

Leave out `transcript` to diarize the audio alone, for example in parallel with
transcription: `words` and `utterances` come back empty, and `segments` (speaker
turns with `start`/`end` in ms) and `speakers` are filled.

<details>
<summary>Audio formats, config and errors</summary>

`audio` takes a local file path, a data URL, raw bytes, a Blob/File or a
readable stream (read whole before sending), but not a URL. Its content type
picks the decoder and is inferred from the file extension: `audio/wav` (also
the default), `audio/mpeg`, `audio/flac`, `audio/ogg`, `audio/mp4`,
`audio/webm`, or raw `audio/pcm`, which needs `sample_rate` and `channels` in
`config`. Pass `contentType` to override it.

```typescript
const result = await client.diarization.diarize(
{
audio: pcmBytes,
config: { sample_rate: 16_000, channels: 1, language_code: "en" },
},
{ timeout: 120_000 },
);
```

Failures throw `DiarizationError` with `.status` (401 for a missing or invalid
key, 400 for a validation failure), `.errorCode` (the snake_cased
problem-details title, e.g. `unauthorized`) and `.retryAfter`.

</details>

### Transcribe streaming audio

Refer to [AssemblyAI's streaming documentation](https://www.assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio) for full code examples.
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "assemblyai",
"version": "4.43.4",
"version": "4.44.0",
"description": "The AssemblyAI JavaScript SDK provides an easy-to-use interface for interacting with the AssemblyAI API, which supports async and real-time transcription and text-to-speech.",
"engines": {
"node": ">=18"
Expand Down
1 change: 1 addition & 0 deletions src/services/diarization/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
export * from "./service";
180 changes: 180 additions & 0 deletions src/services/diarization/service.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
import { readFile } from "#fs";
import { BaseService } from "../base";
import {
BaseServiceParams,
DiarizationAudioInput,
DiarizationRequest,
DiarizationRequestOptions,
DiarizationResponse,
} from "../..";
import { DiarizationError } from "../../utils/errors/diarization";
import { getPath } from "../../utils/path";
import {
basename,
concatChunks,
dataUrlToBytes,
deadlineSignal,
describe,
errorFromResponse,
extname,
isIterable,
isWebReadableStream,
liveChunks,
multipartBoundary,
multipartClosing,
multipartHead,
readAsyncIterable,
resolveFormat,
} from "../../utils/live";

const diarizeEndpoint = "/v1/diarize";
// The extension picks the content type, which picks the server's decoder.
// `.pcm`/`.raw` map to `audio/pcm` inside resolveFormat; unknown extensions
// fall back to `audio/wav`.
const contentTypes: Record<string, string> = {
".wav": "audio/wav",
".mp3": "audio/mpeg",
".flac": "audio/flac",
".ogg": "audio/ogg",
".opus": "audio/ogg",
".m4a": "audio/mp4",
".mp4": "audio/mp4",
".webm": "audio/webm",
};
const encoder = new TextEncoder();

/**
* The Diarization API: audio in, speaker labels out.
*
* Posts the audio, and optionally a transcript with word timings, to the
* client's `diarizationBaseUrl` (`https://diarization.assemblyai.com` by
* default; `https://diarization.us.assemblyai.com` and
* `https://diarization.eu.assemblyai.com` keep requests in the US or EU).
*
* @example
* ```ts
* const transcript = await client.sync.transcribe("./call.wav", { timestamps: true });
* const { utterances } = await client.diarization.diarize({ audio: "./call.wav", transcript });
* ```
*/
export class DiarizationService extends BaseService {
constructor(params: BaseServiceParams) {
super(params);
}

/**
* Label the speakers in `request.audio`.
*
* With `request.transcript`, every word gets a speaker and the response
* groups them into utterances. Without it, only the model runs:
* `segments` and `speakers` are filled and `words` and `utterances` are
* empty.
* @param request - The audio, and optionally the transcript, config and
* audio content type.
* @param options - Client-side options: a deadline and an abort signal.
* @returns A promise that resolves to the speaker labels.
* @throws Error when `audio` is a URL, or when raw PCM is missing
* `sample_rate` or `channels`.
* @throws DiarizationError when the API refuses the request.
*/
async diarize(
request: DiarizationRequest,
options: DiarizationRequestOptions = {},
): Promise<DiarizationResponse> {
const config = request.config ?? {};
const { bytes, filename, suffix } = await readAudio(request.audio);
const format = request.contentType
? { filename: filename ?? "audio", contentType: request.contentType }
: resolveFormat(config, suffix, filename, contentTypes, "config");
if (
format.contentType === "audio/pcm" &&
(config.sample_rate === undefined || config.channels === undefined)
) {
throw new Error(
"raw PCM audio requires both sample_rate and channels in config",
);
}

const boundary = multipartBoundary();
const transcriptPart = request.transcript
? `--${boundary}\r\n` +
'Content-Disposition: form-data; name="transcript"\r\n' +
"Content-Type: application/json\r\n\r\n" +
// Only `words` is sent: a sync response carries much else.
JSON.stringify({ words: request.transcript.words }) +
"\r\n"
: "";
// One buffered body rather than a stream: the API requires
// Content-Length (411 otherwise).
const body = concatChunks([
encoder.encode(transcriptPart),
multipartHead({ boundary, config, ...format }),
bytes,
multipartClosing(boundary),
]);

const response = await this.fetchResponse(diarizeEndpoint, {
method: "POST",
body,
headers: { "Content-Type": `multipart/form-data; boundary=${boundary}` },
signal:
options.timeout !== undefined
? deadlineSignal(options.timeout, options.signal)
: options.signal,
});
if (response.status !== 200) {
throw await errorFromResponse(response, DiarizationError, "diarization");
}
return (await response.json()) as DiarizationResponse;
}
}

/**
* Read any accepted audio input into bytes, with the file name and extension
* it carries, if any. URLs are rejected: the API has no URL ingestion.
*/
async function readAudio(
input: DiarizationAudioInput,
): Promise<{ bytes: Uint8Array; filename?: string; suffix: string }> {
if (typeof input === "string") {
if (/^https?:\/\//i.test(input)) {
throw new Error(
"DiarizationService does not accept URLs. Pass a local file path, " +
"audio bytes, a Blob or a stream.",
);
}
if (input.startsWith("data:")) {
return { bytes: dataUrlToBytes(input), suffix: "" };
}
const path = getPath(input) ?? input;
return named(
await readAsyncIterable(liveChunks(await readFile(path))),
path,
);
}
if (input instanceof Uint8Array) return { bytes: input, suffix: "" };
if (input instanceof ArrayBuffer) {
return { bytes: new Uint8Array(input), suffix: "" };
}
if (input instanceof Blob) {
// File instances carry a name; the File global itself needs Node >= 20.
return named(
new Uint8Array(await input.arrayBuffer()),
(input as { name?: string }).name,
);
}
if (isWebReadableStream(input) || isIterable(input)) {
// fs.ReadStream carries the path it was opened from.
const path = (input as { path?: unknown }).path;
return named(
await readAsyncIterable(liveChunks(input)),
typeof path === "string" ? path : undefined,
);
}
throw new TypeError(`unsupported audio input type: ${describe(input)}`);
}

function named(bytes: Uint8Array, path: string | undefined) {
const filename = path ? basename(path) : undefined;
return { bytes, filename, suffix: filename ? extname(filename) : "" };
}
19 changes: 19 additions & 0 deletions src/services/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { BaseServiceParams } from "..";
import { SyncTranscriber, SyncLiveSession } from "./sync";
import { DictationTranscriber, DictationLiveSession } from "./dictation";
import { DiarizationService } from "./diarization";
import { LlmGatewayService } from "./llm-gateway";
import {
TextToSpeechService,
Expand All @@ -11,6 +12,7 @@ import { TextToSpeechError } from "../utils/errors/tts";
import {
SyncTranscriptError,
DictationError,
DiarizationError,
LlmGatewayError,
} from "../utils/errors";
import {
Expand Down Expand Up @@ -38,6 +40,7 @@ const defaultBaseUrl = "https://api.assemblyai.com";
const defaultStreamingUrl = "https://streaming.assemblyai.com";
const defaultSyncUrl = "https://sync.assemblyai.com";
const defaultDictationUrl = "https://dictation.assemblyai.com";
const defaultDiarizationUrl = "https://diarization.assemblyai.com";
const defaultLlmGatewayUrl = "https://llm-gateway.assemblyai.com";
const defaultTtsUrl = "https://tts.us.assemblyai.com";

Expand Down Expand Up @@ -72,6 +75,11 @@ class AssemblyAI {
*/
public dictation: DictationTranscriber;

/**
* The Diarization service: speaker labels for audio, with or without a transcript.
*/
public diarization: DiarizationService;

/**
* The LLM Gateway service.
*/
Expand Down Expand Up @@ -119,6 +127,15 @@ class AssemblyAI {
baseUrl: dictationBaseUrl,
});

let diarizationBaseUrl = params.diarizationBaseUrl || defaultDiarizationUrl;
if (diarizationBaseUrl.endsWith("/")) {
diarizationBaseUrl = diarizationBaseUrl.slice(0, -1);
}
this.diarization = new DiarizationService({
...params,
baseUrl: diarizationBaseUrl,
});

let llmGatewayBaseUrl = params.llmGatewayBaseUrl || defaultLlmGatewayUrl;
if (llmGatewayBaseUrl.endsWith("/")) {
llmGatewayBaseUrl = llmGatewayBaseUrl.slice(0, -1);
Expand Down Expand Up @@ -147,6 +164,8 @@ export {
DictationTranscriber,
DictationLiveSession,
DictationError,
DiarizationService,
DiarizationError,
LlmGatewayService,
LlmGatewayError,
TextToSpeechService,
Expand Down
Loading
Loading