Primate IntelligencePrimateIntelligence

Streaming

Streams analyze live video in real time: point a camera (or any MediaStream) at the API and get per-frame detection results back over a WebRTC data channel, with the same prompt semantics as file analyses.

Lifecycle

POST /v1/streams → queued|ready → (WS join → offer/answer/ICE) → live → ended
  1. Create the stream server-side with your secret key:
curl -s -X POST https://api.primateintelligence.ai/v1/streams \
  -H "Authorization: Bearer $PRIMATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Is there a person in the frame?"}'

The response carries everything the client needs: signaling.url (a WebSocket), ice_servers (STUN/TURN with credentials), limits (max_session_s, warn_at_remaining_s), and queue_position/estimated_start_s when capacity is busy.

  1. Mint a client token for the browser/device (the signaling WS never accepts secret keys):
curl -s -X POST https://api.primateintelligence.ai/v1/client_tokens \
  -H "Authorization: Bearer $PRIMATE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"scopes": ["streams:signal"], "stream_id": "'"$STREAM_ID"'", "ttl_s": 300}'
  1. Connect from the client. With the TS SDK this is one call:
import { connectStream } from '@primate-intelligence/sdk/browser';

const session = await connectStream({
  stream,                    // the Stream resource (relay it from your server)
  clientToken,               // pvct_… minted in step 2
  mediaStream: await navigator.mediaDevices.getUserMedia({ video: true }),
  onResult: (r) => console.log(r.frame_num, r.detections),
  onWarning: (remainingS) => showCountdown(remainingS),
  onEnd: (reason) => cleanup(reason),
});
// change the question mid-stream:
session.updatePrompt('Is there a forklift moving?');
// done:
session.end();

Under the hood: WS join → ready (or queued {position}) → WebRTC offer/answer + ICE trickle → media flows → result messages per analyzed frame.

Signaling protocol (build your own client)

Messages you send: join, offer {sdp}, ice {candidate}, update_prompt {prompt}, end, ping. Messages you receive: queued {position, estimated_start_s}, ready, answer {sdp}, ice {candidate}, live, result {frame_num, detections}, metering {…}, warning {…}, end {reason}, error {code, message}, pong.

There are two kinds of warning: the credit-exhaustion countdown (warning {remaining_s}, no code field) and diagnostic warnings that carry a code field — see No-media warning below. Distinguish them by the presence of code.

A Python/aiortc backend example (robotics-style, no browser — including a "stream a file" recipe and a relay-only datacenter/CI profile) ships in the public examples repo, alongside a self-contained browser webcam example.

Trickle ICE is bidirectional: after the answer, the server may keep sending ice {candidate} messages as late srflx/relay candidates finish gathering — keep consuming and adding them. Clients on UDP-blocked networks (datacenters, CI, strict NATs) complete via the TURN servers in ice_servers; TURN-over-TCP:443 is supported.

Result contract (identical to file analyses)

Each result.detections[] row uses the same enums and casing as the file API: answer is lowercase yes|no|indeterminate, confidence is 0..1, and prompt is echoed exactly as you submitted it. Latency telemetry (per-stage breakdowns) lives under a timing object.

Result sampling — results are sampled, not per-frame

Results are sampled, not emitted for every source frame — inference cadence is adaptive (roughly every 8th source frame under load). Two different axes to keep straight:

  • frame_num on each live result is the source-frame index the result was computed on. Gaps between consecutive frame_num values are normal and expected — don't treat them as dropped results.
  • results_summary.result_frames on the ended stream resource counts the number of result events emitted over the session — NOT a source-frame count.

Expect roughly 8–15 results per second depending on load; never assume a fixed cadence.

Alias note: results_summary.frames is a supported alias of results_summary.result_frames (identical value), superseded by result_frames. Prefer result_frames in new code; existing reads of frames keep working.

Metering ticks

While live, a metering message arrives every 5 seconds:

{"type": "metering", "elapsed_s": 45, "billed_s": 45, "session_remaining_s": 555, "balance_s": 555}
  • session_remaining_s — seconds remaining in this session's credit reservation (the session cap). This is NOT your account credit balance — that lives at GET /v1/billing/credits (balance_seconds).
  • balance_s — supported alias of session_remaining_s (identical value), superseded by session_remaining_s. Prefer session_remaining_s in new code; existing reads of balance_s keep working.
  • elapsed_s / billed_s — live-clock seconds elapsed = billed (identical by construction; join, queueing, and negotiation are free).

No-media warning (the server tells you when your media is the problem)

If the WebRTC transport connects but no decodable video frame arrives within 5 seconds, the server pushes a diagnostic warning on the signaling WS (re-warned once at 15s, then quiet; canceled by the first decoded frame):

{"type": "warning", "code": "no_media_frames", "transport_connected": true,
 "elapsed_s": 5, "packets_received": 312, "frames_decoded": 0, "hint": "…"}

Read packets_received to self-diagnose:

  • packets_received: 0 — your client is not sending media (track not attached, muted, or the capture source is dead).
  • packets_received > 0 with frames_decoded: 0 — media is arriving but is not decodable (wrong codec — must be VP8 or H.264 — or the source produces no real frames, e.g. a canvas capture without an active draw loop).

If the session then ends without ever going live, the terminal resource records end_reason: "media_timeout" with the same media counters in failure_diagnostic, and bills 0.

Honest end reasons

end {reason} and the terminal resource's end_reason distinguish real failures — a session that never went live is never completed:

Reason Meaning Billed
completed Normal end after live live seconds
canceled Ended before going live (client action) 0
ice_failed ICE never connected — resource carries failure_diagnostic (server candidate summary) 0
media_timeout Transport connected but no media flowed — failure_diagnostic carries the media counters 0
insufficient_credits Balance exhausted mid-stream live seconds
timeout limits.max_session_s reached live seconds
error Server-side failure live seconds (0 if never live)

failure_diagnostic — debugging a failed session from the API alone

When a session ends without ever going live on a transport failure, the terminal Stream resource carries a nullable failure_diagnostic object (and the session bills 0):

  • On ice_failed it holds the server-side ICE candidate summary:
{"end_reason": "ice_failed",
 "failure_diagnostic": {
   "local_candidates": ["host 203.0.113.7", "srflx 203.0.113.7", "relay 198.51.100.20"],
   "hint": "Server advertised public host + srflx/relay candidates; check the client's path to the TURN servers in ice_servers."
 }}

If local_candidates shows only private addresses (172.x, 10.x), the fault is on our side — contact support. If it shows srflx/relay candidates, check your client's network path to the TURN servers in ice_servers (TURN-over-TCP:443 and turns: TLS relays are available for locked-down egress networks).

  • On media_timeout it holds the media counters from the no-media warning (packets_received, frames_decoded) so you can tell "never sent media" apart from "sent undecodable media" after the fact.

failure_diagnostic is null on successful sessions and on ends unrelated to transport.

Credits & limits

  • One active (queued/ready/live) stream per account by default — a second create returns 409 stream_already_active
  • Sessions reserve credits up front; metering ticks every 5s (see Metering ticks for the field contract); you get warning {remaining_s} before funds run out, then end {reason: "insufficient_credits"}
  • limits.max_session_s is the hard per-session cap from your plan
  • Your account-wide balance is at GET /v1/billing/credits — metering ticks only report the session reservation
  • After ended, GET /v1/streams/{id} shows duration_s, usage (reconciled to the second), and results_summary (result_frames = count of result events — see Result sampling)

End a stream

From the client: send end (or just session.end()). Server-side / cleanup:

curl -s -X POST https://api.primateintelligence.ai/v1/streams/$STREAM_ID/end \
  -H "Authorization: Bearer $PRIMATE_API_KEY"

Idempotent — safe to call on an already-ended stream. stream.ended fires as a webhook if subscribed.

Prefer raw text? This page is available as markdown.