Media layer: LOC packaging and MSF catalogs

This document started on 2026-09-18 as a proposal for the media helpers both SDKs would take over from the Video.js example. Most of it is now implemented:

• Stage 1, the format helpers, is implemented: the format packages (@caprid/moq-media, moq-media), the shared vectors, the protocol pins, and the Python example’s migration. behavior.md is their contract.
• Stage 2, media publishing, is partly implemented. The catalog service, a document service for replaceable documents, and the frame writer exist in both product packages: caprid_relay.media, and @caprid/relay/media as createCatalogService, createDocumentService and createFrameWriter. behavior.md is their contract, and the Video.js and WebCodecs examples are their first callers. The register_media handle that would compose them with the catalog builder is still a proposal.
• Stage 3, the readers, is implemented. The catalog reader ships in both format packages as parseMsfCatalog and parse_msf_catalog (issue #164), and the Python example’s --verify logs its track list. The object-to-frame reader ships as the frame sequencer in both format packages and as the frame reader in both product packages (issue #303, Frame reader).

Both format packages also ship the shared timeline and playout clock (issue #285), the interrupt track’s format helpers (issue #320), and the Media Timeline track’s format helpers and timeline accumulator (issues #291 and #292), which both examples publish. The WebCodecs example publishes audio on the shared timeline, measured live in browsers (issue #288). vocabulary.md records the public names of the implemented stages.

The before and after sketches below describe the eventual API, register_media included. They are not runnable. See the format packages’ READMEs for current usage. Stage 1 keeps VideoTrackInfo and AudioTrackInfo, adds DataTrackInfo, and uses build_msf_catalog and buildMsfCatalog for UTF-8 bytes. The TypeScript catalog reader is the Video.js parser’s independent-catalog path, promoted to parseMsfCatalog. Player presentation projection and delta updates are not part of this extraction.

TypeScript input integration

Stage 1 accepts native WebCodecs chunks and decoder configs through @caprid/moq-media: chunkToLocFrame, videoTrackFromDecoderConfig, and audioTrackFromDecoderConfig. The direct browser example connects VideoEncoder output to LOC/MSF and Relay without a media engine. The pure helpers copy chunk bytes without converting the bitstream. Encoding, callback buffering, and capture remain application concerns.

The optional @caprid/moq-mediabunny adapter handles file-input packets, demonstrated by the MP4 example. Mediabunny owns demuxing. Standard decoder config mapping, LOC properties, MSF catalogs, and source URLs belong to @caprid/moq-media, with no Mediabunny dependency. The adapter retains compatibility re-exports of the config mappers.

TypeScript consumes supplied AVC configuration and four-byte-length packets. Annex B splitting, SPS/PPS reconstruction, and conversion are Python-only, for the PyAV example. The config mappers support AVC and Opus. Grouping, demand, backlog bounding and catalog serving are SDK code in @caprid/relay/media (createFrameWriter, createCatalogService, createDocumentService), which both TypeScript examples call. Pacing remains example code: the WebCodecs example skips capture while the encoder is busy, the frame writer has a backlog, or video has no demand, and the Mediabunny example paces file packets in real time. The stage 2 register_media handle that would compose these is not implemented.

What a media publisher has to know today

This section records the Python Video.js example as it stood on 2026-09-18, before stage 1 moved its helpers into the format packages. The product SDKs stop at bytes, groups, and demand. behavior.md says so on purpose: “The SDK adds no default timescale, timestamp, or LOC media packaging.” Everything a Video.js MoQ player needs above that line is rebuilt inside the example:

Step	What the example does	Where the knowledge comes from
Timing	Declares Timescale 1,000,000 as a track property on every track and puts Timestamp and Timescale on every object, with Video Config on keyframes	draft-ietf-moq-loc-04 registry; issue #23 for why the track property is mandatory against the relay
Decoder configuration	Opens the encoder with global headers, splits the Annex B extradata into NAL units, keeps SPS and PPS, builds an avcC, and reads avc1.PPCCLL out of its bytes 1 to 3	ISO/IEC 14496-15, RFC 6381 §3.3
Bitstream contract	Re-frames every access unit from Annex B start codes to 4-byte length prefixes and drops SPS, PPS, and AUD, because avc1 promises parameter sets out of band	ISO/IEC 14496-15
Catalog document	Assembles version, a slash-joined namespace per track, packaging: "loc", isLive, role, codec and dimensions, and initDataList with base64 inline entries and per-track initRef; leaves out isComplete, which would declare the live broadcast over (#284)	draft-ietf-moq-msf-01 §5, read against the Video.js parser for the exact dialect
Catalog track	Registers a track named catalog and re-sends the document on every relay subscription, retrying transient failures while demand remains	Video.js parse-source.ts; the demand semantics in behavior.md
Group mapping	Opens a video group on each keyframe and appends deltas; sends audio as one group per frame; after any failure or loss of demand, discards deltas until the next keyframe	MSF group mapping; Video.js track-publisher.ts
Source URL	Composes moqt://host/path#msf:<ns>--catalog&c4m=<token>, escaping every byte outside [A-Za-z0-9_] in each namespace field as .xx	msf-01 §11.1; Video.js parse-source.ts
Relay reality	Carries the avcC in the catalog even though LOC puts it on keyframes, because the relay drops odd-typed object properties	Issue #23, observed against relay 0.14.17

That is 347 lines in moq_media.py, none of it encoder or network code, plus about 250 lines of catalog serving and group mapping in stream.py, before a single frame is encoded. None of it is specific to this example. A GStreamer, ffmpeg, or WebCodecs publisher writes the same code, and the Video.js publisher branch has in fact written it again in TypeScript (build-catalog.ts, loc-packaging.ts, avc-codec.ts, composeMoqSource in parse-source.ts). Issue #29 is the symptom: the Python README’s own media snippet calls loc_properties(), a function that exists only in the example.

Where the line goes

Three tiers. The SDK owns the bottom two.

Tier	Owner	Contents
Encoding	Application	Capture, encoder configuration, pacing, time-based buffering and drop policy, reconnection. PyAV, GStreamer, ffmpeg, WebCodecs.
Media packaging	SDK, exists except the register_media handle	LOC object properties, the MSF catalog and its serving loop, the source URL, decoder configuration records, keyframe-to-group mapping
Transport	SDK, exists	Connections, tracks, groups, demand, deadlines, cancellation

The boundary is the encoded frame: bytes, a timestamp, and a keyframe flag. That is what a WebCodecs EncodedVideoChunk carries and what Video.js’s loc-packaging.ts consumes. The SDK never configures an encoder and never decides that a frame is too old to send. The example keeps its LiveQueue. The SDK does turn an encoder’s output into something a LOC/MSF player can play, and the one place the tiers touch, decoder configuration, gets helpers in the packaging tier because an avc1 catalog entry is meaningless without an avcC and PyAV hands back Annex B.

What the example would look like

The Python example, with the encoder, test pattern, pacing, and LiveQueue unchanged:

from caprid_relay import connect_relay_publisher
from caprid_relay.media import AudioTrack, VideoTrack, register_media  # provisional
from moq_media.h264 import H264Config

publisher = await connect_relay_publisher(url, namespace=namespace, get_token=lambda: token)

# libx264's extradata is Annex B SPS/PPS. This yields the avcC and the codec string it declares.
h264 = H264Config.from_annexb(encoder.extradata)

media = register_media(
    publisher,
    video=VideoTrack(codec=h264.codec, width=640, height=360, framerate=30, bitrate=1_500_000, init_data=h264.avcc),
    audio=AudioTrack(codec="opus", bitrate=96_000),  # 48 kHz and channel_config "2" are the Opus defaults
)
print(media.source_url(url, token=token))

for sample in encoded_samples():  # the application's encoder loop, however it is built
    if sample.kind == "video":
        await media.video.write_frame(h264.to_avcc(sample.payload), timestamp_us=sample.pts, key=sample.key)
    else:
        await media.audio.write_frame(sample.payload, timestamp_us=sample.pts)

media.close()
await publisher.close()

Behind those calls:

• three tracks registered with a microsecond timescale
• a catalog built with packaging: "loc", the initDataList entry and initRef, and the avc1 entry withheld until an avcC exists
• a catalog service that re-sends the document on every relay subscription
• frame writers that open a group on each keyframe, append deltas, drop frames while the relay has no demand, and resume at the next keyframe after any failure
• a source URL with the namespace escaped the way the player decodes it

The same in a browser, where WebCodecs reports the avcC with the first encoded chunk rather than up front, which is why the catalog has to be updatable:

import { connectRelayPublisher } from '@caprid/relay';
import { registerMedia } from '@caprid/relay/media'; // provisional

const publisher = await connectRelayPublisher({ url, namespace, getToken });
const video = { codec: 'avc1.42E01F', width: 640, height: 360, framerate: 30, bitrate: 1_500_000 };
// The video entry is advertised once initData is known; the audio entry is advertised now.
const media = registerMedia(publisher, { video, audio: { codec: 'opus', bitrate: 96_000 } });
console.log(media.sourceUrl(url, { token }));

const encoder = new VideoEncoder({
  output: (chunk, metadata) => {
    const description = metadata?.decoderConfig?.description;
    if (description) media.update({ video: { ...video, initData: new Uint8Array(description as ArrayBuffer) } });
    void media.video.writeChunk(chunk); // key: new group; delta: append; LOC properties attached
  },
  error: console.error,
});
encoder.configure({ ...video, avc: { format: 'avc' } });

The pieces

Format helpers

Pure functions and small records. No I/O, no writer types, usable from a test or a script that never opens a connection. Each row names what it replaces in the example and the Video.js module it mirrors, so the TypeScript side could start as an attributed snapshot, as @caprid/moq-core did.

Helper	Replaces in the example	Video.js module
LOC property registry (0x10 Timestamp, 0x08 Timescale, 0x0D Video Config, 0x0F Audio Config) and the microsecond default	The constants block in moq_media.py	loc.ts
Track properties for a LOC track (declared timescale) and object properties for one frame (timestamp, timescale, config on keyframes)	TRACK_TIMESCALE_PROPERTIES, loc_properties()	loc-packaging.ts
Catalog builder taking video, audio, and data track descriptions and a namespace, returning the MSF draft-01 JSON bytes; any number of video and audio tracks, in render and alternate groups (several media tracks)	VideoTrackInfo, AudioTrackInfo, build_msf_catalog()	build-catalog.ts
Source URL composer, and the namespace and track-name escaping it depends on	compose_player_url(), encode_namespace_name()	composeMoqSource, encodeNamespaceName in parse-source.ts
H.264 configuration: SPS/PPS from Annex B, avcC from SPS/PPS, avc1.PPCCLL from an avcC, and Annex B to AVCC re-framing for each access unit	extract_parameter_sets(), build_avcc(), avc_codec_string(), annexb_to_avcc()	avc-codec.ts (the codec string; the browser does the re-framing)

The catalog builder is where most of the value is, because it can refuse to build what a player cannot play. The rules come straight from what the example and the Video.js publisher learned:

• An avc1 or avc3 entry with no init data is not emitted. WebCodecs never puts parameter sets in band for avc format, and the relay drops the per-keyframe Video Config property, so a track advertised without initRef is one a player configures from once and never recovers from.
• The H.264 codec string is derived from the avcC when one is supplied, not copied from the requested profile: an encoder may honor the profile and still choose its own constraint flags and level, and the string is the only place the player learns the bitstream contract.
• Opus declares samplerate: 48000 whatever the encoder was fed, because that is the rate an Opus stream decodes at (RFC 7845 §5.1), and channelConfig is a string.
• Members of one alternate group share a role and a render group (see several media tracks).
• Data tracks are emitted with name and role only. A media role on an entry with no codec would round-trip into an undecodable renderable track.
• Fields the caller left out are omitted, never nulled, and the document is compact JSON, so the bytes are stable enough to compare and to put in spec/vectors/.

Media publishing

The stateful part, which wraps RelayTrackWriter and therefore lives beside the product API. One call registers the catalog track and every media track with the LOC timescale declared, builds the first catalog, and starts serving it. It returns a handle with one frame writer per media track, the raw catalog writer for anyone who wants it, and a close() that ends the catalog service and the tracks. A deliberate close() follows msf-01 §11.3: it ends the media tracks, then sends the end-of-broadcast catalog (isComplete: true, no tracks) before closing the catalog track. A failure does not send it.

Each media track is registered with its default delivery timeout and publisher priority from mediaTrackProperties / media_track_properties (issue #302, issue #316), which the handle would compute from the track’s kind and keyframe interval. A viewer subscribes to each media track with mediaSubscribeParameters / media_subscribe_parameters for the matching subscriber priority (issue #337), the only one moq-relay 0.17 orders delivery by.

The catalog service is the loop the example’s serve_catalog() ran on 2026-09-18, generalized:

• On every True from the catalog track’s demand iterator, write the current document as one independent group. The repeated True for a replacement relay subscription matters most, because a viewer joining after a relay reconnect needs it.
• update() rebuilds the document from new track descriptions. Identical bytes send nothing. Changed bytes go out as a new group at once if demand is present, and on the next subscription otherwise. Every catalog object is a random-access point, which is what the MSF independent catalog form requires.
• A transient write failure is retried after a pause while demand remains, keeping only the latest document. A closed track or connection ends the service and is reported once.
• The catalog advertises only tracks the handle has registered. Adding a track through update() registers it first. Removing one from the catalog does not free its name on this connection, which behavior.md already forbids, so the handle says so rather than letting a later update() fail on the relay.

The frame writer is the mechanism the example’s publish_media() had on 2026-09-18, without the example’s time-based staleness policy:

• Video: a keyframe closes the open group and opens a new one. A delta appends to the open group. While the relay has no demand, frames are counted and not sent, without waiting on the write deadline, because an encoder loop cannot block for ten seconds. When demand returns, and after any group failure, deltas are discarded until the next keyframe.
• Audio, and any track marked as one group per frame: every frame is its own single-object group, the LOC mapping for independently decodable frames.
• LOC object properties are attached from the frame’s timestamp and the track’s declared timescale. The track’s current init data rides keyframes as Video Config or Audio Config, for consumers that receive odd-typed properties, and the catalog stays the copy players rely on.
• Calls on one writer are serialized in arrival order, so a callback-driven encoder can call it without tracking a pending write. Backpressure is bounded in frames, not groups: Video.js’s three queued groups are seconds of video, too much for live. When the backlog is full, keyframe grouping drops it with the open group and resumes at the next keyframe, and frame grouping drops the oldest frame. The writer says when it needs a keyframe so the encoder can force one (issue #263). Whether to discard a frame because it is late against a clock stays with the application.
• Each write reports what became of its frame: sent, skipped, stale, dropped, interrupted, or ended (FrameOutcome.kind in both SDKs). The writer keeps only groups, objects, backlog, and keyframeNeeded / keyframe_needed. Video.js’s track publisher keeps a fixed set of counters, but the writer does not. An application counts the outcomes it cares about, grouped the way it wants, and adds frame.payload’s length if it wants bytes. The Python example’s finished line does this (issue #263).
• An interruption resets the group and leaves the track open. A track that ends on its own is an ended outcome, for that frame and every later one, and the writer’s ended holds the error. It is not an exception, because the encoder keeps producing frames. Only a write after the application’s own close() raises CLOSED. An application that wants the example’s stop-the-run policy stops on the first ended outcome.

Reading side

Before stage 3, the example’s --verify mode and any non-browser consumer did the inverse by hand. Three readers, symmetric with read_text and read_json:

• Parse a catalog object into track descriptions, in the independent catalog form. Delta updates can wait until a publisher in this repository emits them. Shipped as parseMsfCatalog / parse_msf_catalog (see behavior.md).
• Turn a track’s objects into frames in decode order within a latency budget: the frame reader below.
• Turn an object plus the track’s received timescale into a frame: payload, timestamp in microseconds, keyframe flag from object ID 0, and any config property that survived the relay. Video.js’s toLocFrame is the reference, including its acceptance of the draft-03 timestamp ID. Its timestamp part ships as locTimestampUs / loc_timestamp_us (issue #290): an object’s Timestamp in microseconds, scaled by the first Timescale it finds on the object, then in the track properties (SUBSCRIBE_OK’s, as the relay sends them). It does not read the catalog’s timescale: msf-01 §5.2.21 never ties that field to the LOC Timestamp, and a catalog that set it for another purpose would silently rescale a Timestamp LOC defines as microseconds. With no Timescale, the Timestamp already is microseconds since the epoch (LOC §2.3.1.1). Reading 0x10 as raw microseconds works against moq-relay only because it rescales every timestamp to microseconds and declares Timescale 1,000,000. A relay that passes the publisher’s Timescale through needs the scaling.

Several media tracks

Issue #351. A broadcast can carry simulcast layers and more than one source, such as a camera and a screen share. msf-01 describes both with two integers on a track entry:

• renderGroup (§5.2.11): tracks “designed to be rendered together”, such as a camera and its microphone.
• altGroup (§5.2.12): “alternate versions of one-another”, such as the same camera at three bitrates. “A subscriber typically subscribes to one track” of the group, and alternates “MUST have matching media time sequences”.

Building

buildMsfCatalog / build_msf_catalog takes a media list of video and audio track descriptions, each with an optional renderGroup / render_group (default 1) and altGroup / alt_group. In TypeScript each entry names its role ({ role: 'video', ... }). In Python it is a VideoTrackInfo or an AudioTrackInfo. The single video and audio inputs remain, as shorthand for one-entry lists listed before media, so existing callers build the same bytes. A group participant publishing a camera in three layers, a microphone, and a screen share with its audio lists:

Track	Role	renderGroup	altGroup
video-hi, video-mid, video-lo	video	1	1
audio	audio	1	—
screen	video	2	—
screen-audio	audio	2	—

The builder refuses:

• a duplicate track name, across media, data and the media timeline, as before
• a render or alternate group that is not a non-negative safe integer
• an alternate group whose members differ in role or render group. Alternates are one content, so they play with the same companions. A viewer picks one track per alternate group within a render group, which is only well defined when the alternate group sits inside one render group.

Each H.264 entry is still withheld until it has init data, so an alternate group can list fewer members than the caller passed. The media timeline’s default depends is the first video track then the first audio track. Name the tracks to index others. Alternates are time-aligned but their group IDs are not (see switching tracks), so a timeline’s locations point into its first dependency only.

Reading and choosing

The catalog reader returns altGroup / alt_group beside renderGroup / render_group, as declared. groupMsfTracks / group_msf_tracks turns a parsed track list into the sets a viewer chooses from, in order of each set’s first track: tracks sharing a render group and an alternate group form one set, and every other track is a set of its own, alternate group or not. A viewer plays one track from each set of the render group it wants, and moves between alternates with want(..., { until }) (want(..., until=...)), which already holds the old track until the new one has started.

The helper picks nothing: bandwidth estimation, layout and which render group is on screen belong to the player. It skips no track either: a track with no alternate group, a data track among them, comes back as a set of one. Tracks sharing an alternate group but declaring no render group form one set.

Keeping the catalog current

A publisher’s encoder configuration can change mid-broadcast: a camera switch, a resized shared window, a codec fallback from H.264 to VP8. Each change is a new catalog. Serve the catalog with createDocumentService / DocumentService instead of the fixed catalog service, and update() it with the rebuilt bytes when the configuration changes. The new document goes out as a new group at once while the relay is subscribed, and identical bytes send nothing. Adding a track this way registers it first. A track removed from the catalog keeps its name on the connection.

msf-01 §5.3 fixes a track’s attributes to its namespace and name: “To modify any attribute, a new track with a different Namespace|Name tuple is created”. The rule sits among the delta-update rules, but its wording is general, so the conforming way to change a codec, its init data or its dimensions is a new track: register video-2, list it in place of video in the next catalog, and let viewers move to it with want(..., { until }). Rewriting a listed track’s entry in place works only with viewers that reconfigure their decoder on every catalog, and is outside msf-01.

Either way, a viewer can receive a key frame before the catalog that describes it, because the catalog and the media are separate tracks. Until relays forward object properties (issue #262), the catalog is the only copy of the decoder configuration a viewer can rely on, so a viewer that sees a key frame it cannot decode with its current configuration waits for the next catalog and drops frames until then. Once relays forward them, the LOC Video Config property on each key frame (0x0D) is the earlier copy, and a viewer can prefer it. To shorten the wait, a publisher updates the catalog before it sends the first key frame of the new configuration.

Not covered: temporalId / spatialId (§5.2.16, §5.2.17, the temporal and spatial layer a scalable-coded track carries) and delta catalog updates.

Shared timeline and playout clock

Issue #285. Implemented in both format packages. behavior.md holds the testable rules.

One timeline, anchored at the Unix epoch

A broadcast is the set of media tracks one MSF catalog lists. Its publisher stamps every one of them from one broadcast clock:

• The LOC Timestamp of each object is the presentation time of its first sample: its capture time, in microseconds since the Unix epoch on the publisher’s system clock. Each track declares Timescale 1,000,000 (locTrackProperties / loc_track_properties). That is LOC’s own default unit and zero (draft-04 §2.3.1.1: without a Timescale, the Timestamp is wall-clock microseconds since the epoch). The Timescale is declared anyway because moq-relay replaces the Timestamp of a track without one with its own clock (see below). With a Timescale, LOC §2.3.1.2 leaves the anchor to the application, and this convention fixes it at the epoch.
• Audio and video share that anchor. A track never starts its own count at zero: audio stamped from a sample counter is anchored at the broadcast time of its first sample, and video stamped from capture timestamps is mapped onto the same epoch as everything else.
• Every broadcast on this convention shares one unit and one zero, so timestamps line up across catalogs as well as within one: huddle participants, a third-party data track, several viewers playing in lockstep. That is the only binding between catalogs that the drafts define (msf-01 §5.2.8 targetLatency, §7 Media Timeline, §8 Event Timeline, all in wall-clock time). How well two broadcasts line up is bounded by how far apart their publishers’ system clocks are, and by each publisher’s capture-latency error (see Capture latency).
• Issue #290 chose this over #285’s original convention, which zeroed each broadcast at its own monotonic origin. Under that convention a player could reach wall-clock time only through the millisecond-resolution Media Timeline, which rules out honouring targetLatency per frame and leaves syncing between broadcasts undefined. The cost is that a publisher with a wrong system clock now produces wrong wall-clock times, and that nothing on the wire signals which anchor a broadcast uses. The epoch anchor is a convention between the SDKs and players that follow this document. A receiver that does not know it sees only an application-defined anchor (LOC §2.3.1.2).
• The catalog puts the video and audio entries in one render group unless the caller names another: renderGroup: 1 (msf-01 §5.2.11, “tracks with the same group number SHOULD be rendered simultaneously”). That is msf-01’s way to say tracks play together, and moq-rs catalogs already send it. A second source, such as a screen share, takes its own render group (see several media tracks). There is no timeline field, because msf-01 defines none and a player ignores fields it does not know. The builder sets no target latency or buffer, which §5.2.8 and §5.2.9 require to be identical within a render group. The catalog reader returns renderGroup as declared. The Video.js MoQ player parses it but does not act on it yet.
• A supervised publisher’s next generation is a new broadcast with a new broadcast clock. Its timestamps continue on the same epoch, less any step in the system clock between the two clocks’ creation. A player still resets its playout clock wherever it resets its decoders: at a Restart, at a Move, and on a new catalog.
• The playout clock below syncs the tracks of one broadcast. Playing several broadcasts in lockstep (msf-01 §5.2.9: “Players with identical buffer lengths are likely to be synchronized”) needs no more than a shared target latency on the viewer’s own wall clock, and is left to the player.

What the relay does to timestamps is in Relay property handling. A track with a declared timescale keeps the publisher’s timestamps, rescaled to microseconds. A track without one, or an object without a Timestamp, gets the relay’s own clock instead. A received Timestamp cannot tell you which of these happened, so a player must not use relay-clock stamps to sync, and a publisher must declare the timescale on every media track. The format helpers do this by default. None of these rewrites is permitted by MOQT-22 §8.4, which says a relay must forward properties unmodified (see Spec gaps).

Broadcast clock (publisher)

createBroadcastClock() / BroadcastClock() pins one moment on a monotonic source to the system clock, and maps the common timestamp sources onto the epoch timeline from there. It reads an injected monotonic microsecond source. The default source is performance.now() in TypeScript and time.monotonic_ns() in Python. The pinned moment is the origin: originUs on the source and originEpochUs on the epoch (see The origin’s wall-clock time for how the system clock is read). A time on the source maps to originEpochUs + (t − originUs), so the timeline advances at the monotonic source’s rate and an NTP step during the broadcast does not move it.

Every other clock a publisher reads has its own origin, and often its own rate. A source timeline maps one of them onto the broadcast timeline. The application calibrates it with measured pairs: “source time s happened at at on the broadcast clock’s source”. A pair replaces the current anchor when there is none, or when the anchor predicts it more than toleranceUs off (default 5 ms): the source drifted, or it jumped. Between anchors, timestamps keep the source’s own spacing. A timeline never calibrated anchors its first timestamp at the moment it arrives.

Operation	TypeScript	Python	Result
Origin, on the source clock	originUs	origin_us	Integer µs; defaults to the source’s time at creation, rounded
Origin, as wall-clock time	originEpochUs	origin_epoch_us	Integer µs since the Unix epoch; defaults to the system clock at creation less the time since the origin
Time on the timeline now	now()	now_us()	originEpochUs + source − originUs, integer µs
Map a time on the clock’s own source	fromSourceUs(t)	from_source_us(t)	originEpochUs + t − originUs, rounded to integer µs, ties upward
Map a clock with another origin	sourceTimeline({ toleranceUs, captureLatencyUs })	source_timeline(tolerance_us=, capture_latency_us=)	A mapper; tolerance defaults to 5,000, capture latency to 0
Measured pair	timeline.calibrate(s, at?)	timeline.calibrate(s, at_us=)	Re-anchors (returns true) when unanchored or off by more than the tolerance
Source time to timestamp	timeline.timestampUs(s)	timeline.timestamp_us(s)	anchor + round(s − anchor s) − captureLatencyUs; unanchored, the first call anchors at now()
Change the capture latency	timeline.captureLatencyUs = n	timeline.capture_latency_us = n	Integer µs, at least 0; applies to later timestamps, not to calibration
Anchor a sample counter	sampleTimeline(rate, { firstSampleUs })	sample_timeline(rate, first_sample_us=)	A mapper; the anchor defaults to now()
Sample index to timestamp	timeline.timestampUs(n)	timeline.timestamp_us(n)	anchor + floor(n × 1,000,000 / rate)

Browser capture

The recipe has to work in Chromium, Firefox and Safari. MediaStreamTrackProcessor cannot be the default, because Firefox has none and Safari’s handles video only, in workers (MDN compat data, 2026-10). The portable path:

• Video. Draw the camera into a canvas, or wrap a <video> element in a VideoFrame, and stamp each frame with clock.now() when it is captured. That clock is performance.now(), which every browser has.
• Audio. Route the microphone through an AudioContext (MediaStreamAudioSourceNode into an AudioWorkletNode), and stamp each chunk with the context time of its first frame, in microseconds, through a source timeline calibrated from the context’s render clock.
• Calibrating for capture. The worklet sees fresh microphone input when the context renders. So the calibration pair is a context time and the performance.now() time at which it was rendered. The worklet posts each block with the context frame rendered so far (currentFrame plus the quantum). The main thread times each block’s arrival, and over a sliding one-second window the block that arrived soonest after its render gives the pair: calibrate(renderedUs, arrivalUs). Messages only ever arrive late, so the earliest is the best estimate. Calibrating on every block costs nothing, because a pair re-anchors only when it disagrees by more than the tolerance. The example’s render-clock.ts and capture-microphone.ts implement this.
• Not getOutputTimestamp(). The engines disagree on what it returns. Chromium (147) follows the Web Audio spec: contextTime is reaching the speaker at performanceTime, so render time is contextTime + baseLatency + outputLatency. Firefox (148) returns currentTime − outputLatency and performance.now() − outputLatency, which is the render pair shifted back by the output delay, and it steps with each audio callback. The formula calibrate((contextTime + baseLatency + outputLatency) × 1e6, performanceTime × 1000) agrees with the render clock to within about 15 ms in Chromium (median 5 ms). In Firefox it stamps audio early by the output delay: 40–50 ms, measured on a PulseAudio null sink.
• Contiguous blocks. Start a new block wherever currentFrame is not the frame expected next. On a new node’s first process() call, Chromium (147) reports currentFrame 0 and only later the real frame. A block stamped from that 0 starts tens of milliseconds before its samples.
• Why the AudioContext. The context runs at one fixed sample rate whatever the microphone is, because the browser resamples. When the user switches devices, say from a laptop microphone at 48 kHz to Bluetooth earbuds at 16 kHz, the count does not change rate. The context also keeps running through the gap while the device switches, emitting silence, so the count never stalls.
• Drift. The context clock follows the output device, so it drifts against performance.now(), and it changes when the output device does. Regular calibration absorbs both. A new AudioContext restarts its time at zero: give it its own render clock and source timeline.
• Encoders. Do not trust an audio encoder to carry input timestamps through. Chromium’s Opus encoder counts samples from its first input and ignores later gaps. Firefox’s stamps each chunk 6.5 ms (its lookahead) before the input it came from. Feed the encoder one contiguous sample count, and map each chunk back to context time through the block its timestamp falls in. Encode every block, with or without demand, so the count never skips. A new context needs a new encoder.
• The Chromium alternative. Where MediaStreamTrackProcessor exists, its VideoFrame and AudioData timestamps count from the system’s monotonic zero, not from performance.now()’s (Chromium). Map them through an uncalibrated source timeline. The first frame’s capture-to-callback delay becomes a constant offset of a few milliseconds.

Capture latency

No browser API reports a microphone’s own input latency. A timeline built from when samples reach the page cannot see the time before that, so audio is stamped late by it, a player plays it late against video, and on the shared epoch it is late against every other participant too. Chromium’s MediaStreamTrack.getSettings().latency does not help: it read 2.7 ms for the built-in microphone and 5.3 ms for AirPods below, against real delays of about 30 ms and 190 ms.

A source timeline subtracts a capture latency the application supplies, captureLatencyUs / capture_latency_us, from every timestamp it returns, so the Timestamp keeps LOC’s meaning: the presentation time of the first sample. Calibration pairs are unaffected. Set it when the source timeline is created and again whenever the input device changes. A change applies from the next timestamp. The helpers ship no device table. An application that knows its devices sets the latency per device. The WebCodecs example estimates from the microphone’s label: 190 ms for a Bluetooth headset (a label containing “AirPods” or “Bluetooth”), 30 ms otherwise.

Folding the latency into the Timestamp is allowed (LOC draft-04 defines it only as the “Timestamp of the encoded media frame”) and no draft has a separate field for it, but it has costs:

• Audio is stamped at its corrected capture time, video at the time the frame reached the page, unless the application also subtracts a camera latency. The two are equally “capture time” only as far as both corrections are right.
• A relay or player that judges an object’s age against its own wall clock sees corrected audio as that much older: 30 ms for a laptop microphone, 190 ms for AirPods. draft-lcurley-moq-timestamp-01 §4.1 measures age against the newest object on the same track instead, where a constant correction cancels out. Only a change in the correction, at a device switch, moves it.
• The label heuristic under-corrects AirPods in call mode (470–600 ms measured, below) by about 280–410 ms. Only an application that can tell call mode apart can do better.

Measured with the device latency lab on macOS 26.6 (MacBook Air, Chrome and Firefox, 2026-10-13; raw results on issue #290). The figure is how late the capture recipe stamps a click, net of what the browser reports for the output path:

Input	Output	Chrome	Firefox
Built-in microphone	Built-in speakers	30 ms	35–38 ms
AirPods microphone	Built-in speakers	162–192 ms	— (no clean run)
Built-in microphone, AirPods mic open (call mode)	AirPods	470 ms	117 ms, with the output misreported
AirPods microphone, call mode	AirPods	600 ms	511 ms

The AirPods microphone delivers about 160 ms later than the built-in one. A reference-microphone run (161.8 ms) and a clap run (157.8 ms) agree. Call mode, which macOS enters whenever an application opens the AirPods microphone while playing to them, is 470–600 ms off in both browsers, and its output delay moves too, so no constant corrects it well.

The same runs show the output side, which the playout clock reads from outputLatency. On AirPods in music mode, Chrome reports 171 ms and misses about 30 ms more. Firefox keeps reporting 34 ms and misses about 320 ms. After a switch back to the speakers, Firefox’s outputLatency stays at the AirPods value (165 ms), and Chrome keeps about 44 ms of extra error. Both are player-side errors: the playout clock plays audio that much late, or early, and no publisher correction reaches them.

Stamping

Stamp the encoder’s input (new VideoFrame(frame, { timestamp }), the AudioData timestamp) and the encoded chunks carry the timeline, so chunkToLocFrame passes it through unchanged. WebCodecs timestamps are 64-bit microseconds, so epoch values fit. If a chunk still carries the clock’s own source time, timestampOffsetUs: clock.originEpochUs - clock.originUs maps it. Both are whole microseconds, so that offset is valid. A camera has a capture latency too, typically a frame or two. Subtract it from the video timestamps if you know it. A sample timeline suits sources with no clock of their own to calibrate against, such as a file or a synthetic tone. A live device’s counter drifts against the monotonic source by up to about 100 ppm, and it changes rate and stalls when the device changes. Use a calibrated source timeline for live capture instead.

Playout clock (player)

createPlayoutClock() / PlayoutClock() answers one question for the video renderer: when should the frame with this timestamp be shown? It is DOM-free and transport-agnostic, with the same injected time source as the broadcast clock, so a Python player or a test drives it with numbers.

Audio is the master. The audio path calls reportAudio(timestampUs, atUs?) / report_audio(timestamp_us, at_us=None) with the timeline timestamp that is audible at local time at (default: now). That is the timestamp leaving the speaker. In a browser, take the context time audible now from the render clock: currentTime sampled against performance.now(), min-filtered as for capture, minus baseLatency + outputLatency. The player maps that context time to the media timestamp it scheduled there, then reports reportAudio(mediaUs, nowUs). getOutputTimestamp() gives the same answer in Chromium, but in Firefox (148) its pair is the render pair, and reporting from it plays audio late by the output delay (33–36 ms measured). Read the latencies fresh for every report: the output delay changes with the device. Switching to Bluetooth earbuds can add 150 ms or more, and a value cached at startup leaves video that far ahead. Where an application reports at handover time instead, it subtracts the jitter-buffer depth, the current outputLatency, and any extra path, such as a WebRTC loopback for echo cancellation. A step in the output delay moves the position once. Video then holds frames, or drops them, by that step and stays in sync after it. Between reports the clock extrapolates at real-time rate: position = reported + (now − at). Reports re-anchor it, so the audio device’s drift never builds up. Audio silence is measured from when the latest report arrived, not from its at, so a report that describes a moment slightly in the past still counts as fresh.

The clock has three modes:

Mode	When	Position
idle	Created, or after reset(), with nothing reported or asked yet	None; the next due() anchors wall pacing
audio	A report arrived within audioTimeoutUs	Extrapolated from the latest report
wall	No audio has ever been reported, after stopAudio() / stop_audio(), or once reports have been silent for audioTimeoutUs	Wall clock from an anchor: the first frame asked about from idle, or the audio position when audio went quiet, so nothing jumps

The next audio report returns the clock to audio.

due(timestampUs) compares the frame with the position: offset = timestamp − position, where positive means the frame is early. The decision is one of three:

Decision	When	The renderer
wait, with waitUs = ceil(offset)	0 < offset ≤ maxVideoDelayUs	Holds the frame and asks again after the wait
show	−lateDropUs ≤ offset ≤ 0, or a re-sync (below)	Shows it now
drop	offset < −lateDropUs	Discards it; audio has already moved past it

Every decision carries offsetUs, rounded to integer µs, so the application can log its sync error. From idle, the first due() anchors wall pacing on that frame and returns show with offset 0.

Re-sync with maxVideoDelayUs. In audio mode, a frame more than maxVideoDelayUs early is shown now and not held: video that far ahead of audio means the two tracks are not on one timeline, and freezing the picture would not fix that. The offset reports the gap. In wall mode, a frame more than maxVideoDelayUs early, or more than the target latency late, re-anchors the clock on itself and is shown. This covers a timeline jump and recovery after a network stall, which would otherwise make every later frame late. A frame reader never passes a frame that late, so behind one the late re-anchor happens only at its resync. A late frame in audio mode is always judged against audio and dropped. If video is late consistently, the audio path needs a deeper buffer, and the clock cannot add one.

Option	TypeScript	Python	Default	Why
Late-drop threshold	lateDropUs	late_drop_us	40,000	A late frame makes audio effectively early; EBU R37 puts the audio-early limit at 40 ms
Lead before re-sync	maxVideoDelayUs	max_video_delay_us	1,000,000	Above a deep audio jitter buffer plus output latency
Audio silence before wall pacing	audioTimeoutUs	audio_timeout_us	500,000	Reports come every 10–20 ms; half a second without one is a stall or a mute
Target latency	targetLatencyUs	target_latency_us	2 × the longest of the last four reported keyframe intervals; 2,000,000 before any	Two GOPs: room to wait for a late group and still decode up to the next keyframe; see Frame reader. For a broadcast with no video, pass AUDIO_ONLY_TARGET_LATENCY_US (500,000); see Audio player
Time source	now	now_us	performance.now() µs / time.monotonic_ns() // 1000	Injected so tests are exact

The clock does not buffer, decode, or render frames, and it never changes how much audio is buffered. The target it is built for is the broadcast guidance of ITU-R BT.1359 and EBU R37: audio within about ±45 ms of video, and not audibly early.

A player’s decoders do not carry timestamps through either. After a gap in its input, Chromium’s Opus decoder (147) keeps counting from before the gap, so reset the audio decoder wherever the next timestamp is not the one expected. Chromium does not trim Opus’s 6.5 ms pre-skip, so its decoded audio plays that much late.

Measured in browsers

Issue #288. mise run ts:test:av-sync publishes the WebCodecs example’s canvas and microphone and plays them back in the same page, through the production Relay in Chromium and through the in-memory relay hub in Firefox (and, since Firefox 155, through the Relay too, as described below). Every 600 ms the canvas flashes white and a synthetic microphone beeps. The A/V offset is how much later the beep reached the player’s output after it was emitted than the white frame was shown after it was captured. Frame quantization does not count, because the white frame’s own capture time is the video reference. The figures are net of the beep’s path to the capture worklet. That path stands in for a real microphone’s input latency, which no browser reports (#290), and it was 0–55 ms here. “Settled” leaves out events emitted or played within 2 s of any switch. Results from 2026-10-07, in ms, positive when audio is late:

Phase	Chromium 147, Relay: settled mean [min, max]	Firefox 148, hub: settled mean [min, max]
Baseline	3.9 [−6.1, 22.3]	−2.4 [−13.6, 0.4]
Second microphone (16 kHz)	2.0 [−13.9, 10.0]	−0.4 [−1.6, 0.8]
Replaced capture context	2.5 [−6.7, 10.3]	−0.5 [−2.2, 2.4]
Output delay +200 ms	8.4 [−4.1, 28.9]	−0.2 [−6.9, 4.1]
Audio resumed after a 3 s gap	8.3 [−6.0, 27.1]	−4.3 [−6.2, −0.9]

Across all phases, Chromium’s mean was 4.7 ms (p95 17.5, worst 28.9) and Firefox’s was −1.5 ms (p95 5.6, worst −13.6). Transients stayed within the same bounds: the worst event in the 2 s after any switch was −14.1 ms in Chromium and −8.2 ms in Firefox. The output-delay step held video once for about 230 ms, as designed. During the audio gap the clock paced video by the wall clock, with no dropped frame and no pause between frames longer than 44 ms. With the defaults (lateDropUs 40 ms, maxVideoDelayUs 1 s, audioTimeoutUs 500 ms), no frame was dropped in any phase. The one outlier came at startup: the first flash, emitted before any audio had reached the player, was off by −246 ms while the clock was still pacing by the wall clock.

A re-run on 2026-10-14, after #290 moved every timestamp onto the Unix epoch, gave the same picture with 8 s phases: settled means between 2.2 and 5.3 ms in Chromium through the Relay and between −3.6 and 0.8 ms in Firefox through the hub, with no frame dropped in either.

The same runs show how far getOutputTimestamp()’s capture formula lands from the render clock. In Chromium it was +5 ms (median; p5 −1, p95 16). In Firefox it was −48 ms (p5 −50, p95 −42), which would stamp audio that much early. With players reporting from getOutputTimestamp(), Firefox played audio 33–36 ms late, and Chromium’s settled offsets stayed within ±17 ms.

Firefox reaches the Relay over WebTransport from 155 (stable 2026-09-01), which added the protocols option and the protocol attribute. Firefox 148 has neither, and the SDK refuses a session that negotiates no protocol (transport-adapter.md, TA-7). A run on 2026-10-19 (issue #322), with Playwright’s Chromium 153 and Firefox 155 both through the production Relay (sjc), passed every phase, in ms:

Phase	Chromium 153, Relay: settled mean [min, max]	Firefox 155, Relay: settled mean [min, max]
Baseline	6.1 [1.1, 15.6]	−1.8 [−3.1, 0.5]
Second microphone (16 kHz)	4.2 [−2.3, 11.2]	−2.5 [−4.8, 0.5]
Replaced capture context	3.2 [−4.9, 13.3]	−4.3 [−6.1, 1.2]
Output delay +200 ms	5.0 [−12.3, 20.8]	−3.5 [−5.5, −0.5]
Audio resumed after a 3 s gap	4.8 [−19.3, 29.4]	−2.0 [−5.1, 1.2]

Each dropped at most one frame in the whole run. The same day’s Firefox run through the hub failed, in 148 and in 155 alike: its player has judged nearly every video frame late since the frame reader landed (issue #339).

Over QMux over WebSocket (@caprid/relay-websocket), Firefox 148 and Playwright’s WebKit both passed the live data scenario against a local moq-relay 0.17.5 on 2026-10-17 (QMux over WebSocket). The A/V sync run has not been repeated over WebSocket. Firefox also needs an audio server: without one, its AudioContext stays suspended. Its H.264 VideoEncoder (148, Linux) holds about 41 frames, 1.4 s, even with latencyMode: 'realtime', while VP8, VP9 and AV1 return each frame within milliseconds. So its video arrives that much after its audio. The playout clock keeps audio the master and would drop video that late, so the Firefox run gives the player a 1.6 s audio jitter buffer. Firefox 155 still needs it: with a 150 ms buffer it showed no video through the Relay. Playwright’s WebKit (Safari 26.4’s engine, on Linux) runs the hub scenario with short phases (settled offsets 4–7 ms), but at full length it stops responding partway through. It cannot open a WebTransport session to the Relay (it can reach it over WebSocket, above), and its getOutputTimestamp() is seconds off. It is opt-in (AV_SYNC_RUNS=webkit:hub) and says nothing about Safari on macOS.

Frame reader

Issue #303. Implemented in both SDKs. behavior.md holds the testable rules.

Each group travels on its own stream, and streams are not ordered relative to each other. After a 2 s stall through the impairment proxy, the live A/V test’s player, which fed every video object to one decoder in arrival order, crashed its decoder on … g10/o0, g6/o1 … g6/o4, g9/o0, g9/o1, g10/o1: the tail of group 6’s stalled stream arrived after group 10’s keyframe. Nothing was lost. The same stall left about 2.3 s of audio scheduled ahead, against about 0.14 s normally, because the player re-anchored its jitter buffer on the first late frame.

The frame sequencer is the decision engine, a pure state machine in the format package like the playout clock: events in, frames out, and how long until a wait can end. The frame reader in the product package is the glue. It is a subscription’s or a want’s handlers, feeds every event to the sequencer, and polls again when the wait ends. The product package cannot import the format package, so the reader takes the sequencer as an argument.

• One budget. Lateness is measured against the playout clock’s position, the media time that should be playing now. It advances in real time through a stall, so a burst that arrives after one is already behind it. Nothing depends on the publisher’s and viewer’s system clocks agreeing. The budget is the clock’s targetLatencyUs, so all tracks of a broadcast and the clock’s own late re-anchor share one number.
• The default follows the stream. The video sequencer reports the interval between the keyframes of consecutive groups. The clock keeps the last four and takes twice the longest, so a forced keyframe (onKeyframeNeeded) does not shrink the budget. Before two consecutive keyframes have arrived it is 2 s, twice the examples’ one-second keyframe interval. msf-01 has no catalog field for keyframe spacing, and its targetLatency (§5.2.8) means end-to-end wall-clock latency, so neither is read.
• Video waits only while it is worth it. In-order frames go out at once. A gap, or a still-open older group, holds newer frames until the last delivered frame’s timestamp plus the target latency, because the missing frame is later than that one. A group that has not been seen may never come, since a failed group’s ID is not reused, so the next keyframe waits for it only while it is not yet due. Past the deadline the sequencer skips to the next keyframe within the budget and marks a gap, and the player resets its decoder there.
• Audio is not held. No audio frame depends on another, so a frame more than the clock’s lateDropUs behind the position is dropped at once, and nothing waits for a missing one. A budget of two GOPs for audio would keep almost all of a 2 s blackout’s burst, and the player would re-anchor on it and stay that far behind.
• Resync. If frames keep arriving but none within the budget for one whole target latency, and their lateness holds steady, the path delay has grown for good. Lateness that keeps growing is a stall, and lateness that shrinks is its backlog draining. Neither resyncs, because the frames the clock would anchor on are about to be caught up with. The sequencer delivers the newest late keyframe group (or audio frame), marks it resync, and resets the clock so it re-anchors there, rather than leave the player black or silent.
• Resync forward. A clock anchored on old frames, by a resync onto a delay that later recovered, sees fresher frames arrive more than the target latency ahead of its position. That is a resync too, forward, so the old delay does not become lasting latency. The player drops the audio and video it had scheduled on the old anchor.
• Restart and Move. A want’s Restart or Move restarts the sequencer, since group IDs start over, and marks the next frame restart.

Measured live on 2026-10-15: the A/V test (mise run ts:test:av-sync) in Chromium 147 against sjc.relay.mux.global, through a UDP impairment proxy that, 4 s into playback, either blacked out both directions for 2 s or capped the uplink at 80 kbit/s (1 s queue) for 4 s. The publisher uses #302’s default delivery timeouts. The player reads both tracks through frame readers, keeps a 150 ms audio jitter buffer, drops audio whose slot has passed instead of re-anchoring on it, and re-anchors at a resync, a restart, or a schedule more than 100 ms beyond its depth. “Ahead” is the audio scheduled ahead of the context’s time at each later phase boundary, the first about 6 s after the impairment:

Impairment	Before (the player decoding in arrival order)	With the frame reader, final round
None	Passes; 130–185 ms ahead	3 of 3 passed; 130–158 ms ahead
2 s blackout	0 of 2: EncodingError from an old group’s tail after a newer keyframe, or about 2.6 s ahead for the rest of the run	3 of 3 passed; 120–170 ms ahead
Uplink capped for 4 s	0 of 2: EncodingError, or about 3.5 s ahead for the rest of the run	3 of 3 passed; 126–197 ms ahead

Over the rounds before the final one, after the rebase onto #302: 14 of 15 unimpaired runs, 11 of 12 blackouts and 11 of 11 capped runs passed. Unimpaired, the readers’ counters stayed at zero skips and resyncs, with at most two audio frames dropped as late. The two failures were an unimpaired video stall of 3.2 s, before the player’s last fixes and not seen again, and the WebCodecs example’s own encoder-queue guard (“The audio encoder cannot keep up”) tripping after one blackout.

Getting here took three rules found in these runs: a resync only on a steady delay, a resync forward when the clock is behind what arrives, and the player’s bounded jitter buffer.

The publisher side of the same stall is issue #302’s delivery timeouts. They derive from the same keyframe interval, about one interval for video, under the reader’s two, so a publisher stops spending bandwidth on frames the reader would drop. Nothing links the two on the wire.

Timestamp playout

TypeScript only (the frame sequencer’s timestampPlayout), off by default. A text-to-speech agent generates its reply faster than real time, and MoQ lets it send each frame as it is generated, stamped with its slot on the timeline, rather than pace it as WebRTC must. A player then has to hold what arrives early and play each frame at its timestamp. The sequencer without it does not: a frame more than the clock’s target latency ahead of the position (AUDIO_ONLY_TARGET_LATENCY_US, 500 ms, for audio) resyncs it forward, as after a stall, which drops what the player holds and re-anchors on the newest frame, and the jitter buffer trims what it holds beyond its target. relay-dev’s offline replay of bursty TTS traces through it discarded about 90% of the clip (moq-netbench plans/README.md, “Stage 4 wave 2’s agent cells”).

const input = player.addInput({ clock, decoderConfig });
const sequencer = createFrameSequencer({
  clock,
  grouping: 'frame',
  timestampPlayout: { leadUs: () => input.playoutLeadUs },
});

• A frame, and every frame after it, waits until its timestamp is within leadUs of the clock’s position. The poll’s waitUs says when, so a frame reader releases each frame on time.
• The lead is what the player holds beyond what is audible: input.playoutLeadUs, the output delay (baseLatency + outputLatency + player.outputDelayUs) plus the jitter target now, keeps the jitter buffer at its target. A longer constant lead leaves more held than the target, and the buffer’s trim drops the excess every second; a shorter one leaves the buffer short of its target. Default 100 ms, the player’s default target and a 40 ms output delay.
• While the clock has no position (nothing has played yet, or after a reset), the first frame anchors a position of the sequencer’s own on the clock’s local time. The clock’s takes over as soon as the player reports.
• Only a frame more than horizonUs (default 30 s) ahead of the position, the clock’s or the sequencer’s own, resyncs forward, as a timeline jump. The sequencer holds up to a horizon of frames (at 2.5 ms each), beyond the 1,024 it holds otherwise.
• A live talker’s frames are never early, so they play as without it: late drop, resync and catch-up are unchanged. stats.early counts the frames held. A restart (a want’s Restart or Move) drops every frame held.
• Delay added by underruns is given back. Each underrun leaves the clock further behind local time, by the silence it played. Without timestamp playout, frames that then arrive early fill the jitter buffer past its target and its trim gives that delay back. With it, frames reach the buffer only a lead ahead, so the buffer never holds more than its target and never trims: a queue that built and drained would leave its delay for good. So the sequencer keeps the least delay seen from the clock’s position to its local time since the clock was last reset, with the lead in force then, and releases held frames early by however much the delay now exceeds it, less any growth of the lead since (a jitter target that grew holds more on purpose), beyond 40 ms (a frame or two of report noise). The buffer then holds that excess and trims it, as it does without timestamp playout. A TTS talker whose frames came early all along has added nothing, and plays at its timestamps as before.

Measured on relay-dev’s voice benchmark. Stage 4ai-tts (relay-sdks build tag netbench/stage4-ai, 5a4bd33, the same code without the give-back above; moq-relay 0.17.9; runs run-20261029-054413-95db41, run-20261029-102627-5276c7, run-20261029-105904-7de8ee and run-20261029-112630-e14a13; finding C-3 in relay-dev moq-netbench/results/voice/FINDINGS.md): the agent’s TTS fed at four times real time, each frame stamped with its slot. With timestamp playout the listener played 100% of the audio on a clean path, 99.7% with 60 ms loss bursts and 98.1% on LTE with a download (R 89.2, 87.9 and 71.6). Without it, the same player played 5% (R 11.2 clean): it resyncs on early frames. WebRTC fed the same bursts played 39.2% clean (38.3% on LTE) and dropped the rest, whatever its jitter buffer target; paced, it played 100%. Wave 2’s agent cells (finding R-AG-01) had shown the player without it failing the same feed live: every clean round suspect, and on LTE 2,497 and 2,159 ms mouth-to-ear with 41.4 and 24.1 drops a minute, R 0.

The same runs found the give-back’s reason (finding A-16): on LTE with a download, a paced talker through timestamp playout sat at 960 ms mouth-to-ear (R 43.5), against 378 ms for the bursty talker and 540 ms for paced WebRTC; with 60 ms loss bursts, 252 against 146 ms. A simulation of the sequencer and the playout clock with a model of the jitter buffer (its grow, shrink, trim and hard bound; a paced 20 ms talker; a queue growing to 700 ms over 30 s, then draining) reproduces it: the median mouth-to-ear 10 to 30 s after the drain was 120 ms without timestamp playout and 840 ms with it, as the clock never caught up. With the give-back it is 140 ms. The bursty talker through the same queue played 98.1% at 300 ms after the drain without the give-back, 97.4% at 120 ms with it; on a path without a queue neither talker changed (bursty 100% played, paced 100 ms). The bench has not run the give-back yet.

Measured locally on 2026-10-29 (relay-3cnp.399, before the give-back): moq-relay 0.17.9 (30fd9d450) built from source and the talker in one network namespace of relay-dev’s moq-netbench rig (rig local), the listener in the other behind the rig’s profile (rig apply, the downlink shaped on the peer), both on this SDK over native QUIC. The listener is relay-sdks’ own receive chain composed as the example listener (frame reader, frame sequencer, audio input, its worklet and the WASM Opus decoder), with Web Audio stood in for in real time (558-frame render callbacks, a 20 ms output delay); a frame counts as played when its decoded audio reaches the output. A TTS agent sends six 8 s turns, 2 s apart, each handed over at four times real time from its first frame’s slot (2,400 frames of 20 ms), with and without timestampPlayout (leadUs: () => input.playoutLeadUs). Without it the player discarded about 91% of the clip on every link: the sequencer dropped most frames as late and the jitter buffer trimmed about 6 s, and what it did play came up to 5.9 s before its slot. With it, every frame that arrived played, at a constant delay from its slot within each turn (the listener’s own delay plus the lead), with no trim and no resync; QUIC’s resends on the lossy link arrived while their frames were still held. The one frame lost in a second lossy run was 100 ms into the first turn, sent 75 ms ahead of its slot: too little lead for a resend over a 50 ms round trip, as for any frame a live talker sends.

Link (rig profile)	Played without it	Its losses	Played with it	Mouth-to-ear from the slot, with it
Clean	203 of 2,400 (91.5% dropped)	1,938 late, 6.18 s trimmed	2,400 of 2,400	40 ms (53 ms max)
loss-random-2 (50 ms RTT, 2% loss)	206 (91.4%)	1,942 late, 6.20 s trimmed	2,400; 2,399 in a second run	64 ms (77 ms)
narrow (30 ms RTT, 2/1 Mbit/s, 500 ms FIFO)	210 (91.2%)	1,935 late, 6.16 s trimmed	2,400	47 ms (60 ms)
narrow, a bulk CUBIC download filling the FIFO	215 (91.0%)	1,921 late, 6.61 s trimmed, 12 resyncs	2,398 (2 late, 1 underrun)	473 ms (513 ms)

Audio player

Issue #314. TypeScript only, in @caprid/moq-media: createAudioPlayer({ context }) plays any number of inputs, usually one per speaker, into one AudioContext. Each player.addInput({ clock, decoderConfig }) takes frames in decode order from a frame reader (onFrame: input.push), decodes them with WebCodecs, holds them in a jitter buffer in an AudioWorklet, and reports what is audible to its broadcast’s playout clock. Every input goes through its own gain into player.output, which the application connects. The player has no UI and no transport, so the Video.js renderer can use the same core behind its own frames-in, clock-out seam.

Decoding. The decoder is built from the catalog entry (audioDecoderConfigFromTrack), or defaults to Opus at 48 kHz. Decoders count output timestamps on from their first input (see Playout clock), so the input resets its decoder wherever a frame does not start where the previous one ended: at a gap, a resync or a restart, and at any jump of more than 1 ms. For Opus the previous frame’s length is read from its TOC byte (RFC 6716 §3.1), so the check does not wait for decoder output. At a gap or a jump the old decoder is flushed, not closed: a frame reader can deliver frames and the gap after them in one poll, before any of them is decoded, so the old decoder first outputs what it was given, and the frames after the gap wait for the new decoder until it has. A frame that starts before the previous one ended is behind it and is dropped as late. A resync or restart closes the decoder, drops its pending output, and flushes what the buffer holds, all of which belongs to the replaced timeline. A restart also forgets earlier cuts. A resync keeps them, because audio a talker interrupted can still arrive behind one. A context at another rate than the decoder’s gets linearly resampled audio.

Loss concealment. WebCodecs’ AudioDecoder has no packet-loss concealment and cannot decode the in-band FEC (LBRR) an Opus encoder puts in the next packet, so a frame that never came would play as silence. With lossConcealment on (the default), an Opus input decodes instead with libopus 1.5.2 compiled to WebAssembly (decoder only, 116 KB, imports nothing; built by ts/packages/moq-media/scripts/build-opus-wasm.sh from pinned sources and committed as base64, with its SHA-256 checked by a test). It decodes synchronously on the main thread with one decoder for the input’s life. At a gap (frames skipped: lost, or too late for the sequencer) the next frame’s FEC rebuilds the frame just before it when that packet carries LBRR, and libopus’s concealment synthesizes the rest of the gap, up to 120 ms, from the decoder’s own state. Past that, the worklet plays silence, as for any gap. The filled audio carries the missing timestamps, so the buffer plays on without a hole. The main thread fills every gap up to 1 s that no cut spans, and the decoder follows on from the concealment. The worklet holds filled audio only where it continues held audio (playing, or filling to its target) or playback (still playing, a quantum having just taken the last sample), as its own silence fill works inside held audio; once the buffer has run dry, been flushed or never filled, it drops the fill, which would play late and add its length to the delay. The worklet decides, not the main thread, whose reports of the buffer’s state are up to one report (about 20 ms) old: deciding there filled gaps that came after the buffer ran dry but before its report, and left a loss while the buffer was filling to its target as silence. A gap that spans a cut or exceeds 1 s is not filled, and a resync or a restart starts a new timeline: each resets the decoder, so the frame after it is not decoded from state that belongs before the gap. synthesizedUs and recoveredUs count what was filled, less, from the next report, what the worklet dropped. lossConcealment: false, a codec other than Opus, or a realm without WebAssembly decodes with WebCodecs as described above. Measured offline on stage-2 netbench arrivals at 2% random loss (relay-dev moq-netbench/docs/moq-optimisation.md): drops from about 56 a minute to 16 with concealment alone and 3 with the publisher’s in-band FEC, at no added delay.

Jitter buffer. A ring in the worklet, in the context’s frames, starting from the numbers Huddle’s player worklet runs with (demos/huddle):

Rule	Default	Why
Hold the target before playing, at the start and after running dry	60 ms	Three 20 ms Opus frames: room for one late frame and a burst of two. Huddle runs on the Relay with this start. The A/V test’s fixed 150 ms was sized for video and Firefox’s encoder, not for voice
Grow the target after an underrun	+40 ms	Two Opus frames: react hard, since the shrink below gives it back within seconds when the spike was a one-off (relay-dev relay-3cnp.170; Huddle grew 20 ms and gave it back in 16 s)
Count an underrun only when audio comes back within the ceiling, and not after a flush		A longer dry spell is the talker going quiet, which no buffer would cover; audio after a flush starts a new timeline
Shrink the target after 2 s with no underrun	−10 ms, to 20 ms	The jitter is measured where it shows, in what the buffer never used (the trim below), so the floor only has to cover a render callback and timing noise. Measured on network arrivals replayed offline, not on a busy page: a main thread that stalls delays decoded audio on its way to the worklet, and a lower floor leaves less room for that before an underrun (which then grows the target). Huddle’s 40 ms floor and 8 s cadence held audio 20–60 ms longer than WebRTC’s NetEQ on every stage-2 netbench path, from the same arrivals (relay-dev moq-netbench/docs/moq-optimisation.md)
Ceiling	400 ms, and never more than the clock’s target latency less the output delay and 40 ms	Huddle’s ceiling; a frame further ahead of the clock than its target latency makes the frame sequencer resync forward, which would flush the buffer
Trim, each second, held audio that never dipped to the target	when ≥ 5 ms, and not while the playback rate is below 1	A burst after a stall would otherwise stay as lasting latency, since audio arrives as fast as it plays; a rate below 1 gains that delay on purpose
Hard bound	2 × target + 60 ms, under the ceiling	Huddle’s backlog bound
A frame that never came, inside held audio	silence for its length, up to 1 s	Keeps later audio on time. With loss concealment the main thread fills an Opus gap before it reaches the worklet (above), so this covers what concealment leaves and the WebCodecs path

Joining sooner. The recommended join for a listener whose time to first audio matters, such as a voice conversation; the WebCodecs example’s listenAudio joins this way. The parts are optional API, and a player that uses none of them joins as the sections above describe. They exist for relay-dev’s voice benchmark (relay-3cnp.351, stage 4.3), which found MoQ’s first audio about 200 ms behind WebRTC’s on a clean path, all of it between the catalog and the audio SUBSCRIBE. Profiled in Chromium, that time is the listener’s own: the benchmark page waited for its diagnostic tap’s first block plus 200 ms (about 285 ms) before subscribing, and the page built the AudioContext (about 30 ms, synchronous), loaded the tap’s and the player’s worklet modules and libopus one after another after the catalog (about 25 ms more). The relay’s part, SUBSCRIBE to SUBSCRIBE_OK, was two round trips, as for the catalog. A join that is quick:

1. Starts the connect, then builds the audio side while the handshake and the catalog are on their way: the AudioContext, context.resume() and warmAudioPlayer(context), which loads the jitter buffer’s worklet module and compiles libopus at once. A player created on that context later loads neither again. Its libopus load decodes the embedded module with the engine’s Uint8Array.fromBase64 where there is one (loadWasmOpus({ nativeBase64: true })), instead of a JavaScript loop that holds the main thread for several milliseconds, tens on a busy page. Order matters: the context’s constructor blocks the main thread, so built after the catalog it also delays the SUBSCRIBE_OK and the first objects behind it (measured: 55 ms later). The warm belongs inside the user’s gesture at the connect: an AudioContext created outside a gesture stays suspended, so the page creates the context in the click and warms it there. Warming at page load measured no faster (289 ms audible against 284 for warming at the connect).
2. Subscribes to the audio track before the player exists, with the frame reader delivering into createFrameHold(). When the input exists, hold.release(input.push) passes every frame kept, in order, and every later one straight on. Nothing is dropped or judged by the hold: the sequencer has already delivered those frames. A listener that knows the track’s name by convention (audio) subscribes at once with the catalog’s subscription, saving the catalog’s round trips (the fastest row below); the catalog still decides the decoder, so the input is added when it arrives. One that waits for the catalog to name the track subscribes after it. A talker’s interrupt track is subscribed as soon as the catalog lists it, not after the player exists: audio already waits in the hold, and an interrupt sent meanwhile must cut it. Interrupts that come before the input exists are applied (input.cut) before the hold is released, so audio the talker had cut never plays.

A shorter hold before the first audio (joinJitterTargetUs, 40 ms against the 60 ms target) was measured in relay-dev’s benchmark (stage 4.3b, finding R-43B-01) and left out: first audio at 242 ms against 249 clean, but 2 drops in the first 10 s on clean, and 8 against 0 at 2% loss. The play start stays at the jitter target.

The input records its start in input.joinPhases (AudioInputJoinPhases), on its clock’s time source: addedUs, firstPushUs, firstDecodedUs (the first decoded audio handed to the worklet), firstPlayUs (when the worklet’s first play start renders: each report carries the context frame where it happened) and firstAudibleUs (that plus the output delay clock reports add). player.joinPhases (AudioPlayerJoinPhases) has when the worklet module and libopus were ready, on performance.now(). The rest of a join, the transport’s ready, the catalog, the SUBSCRIBE sent, its SUBSCRIBE_OK and the first object, the application sees in its own callbacks.

Measured on a Sprite (relay-3cnp.351): headless Chromium 153 joining a moq-relay at moq-dev main 6298cbcdf 20 ms round trip away (a network namespace with 10 ms of netem each way; the talker, the bench’s own, reaches the relay over the same path, as a near cell’s does), the benchmark’s shipped player, the benchmark page’s order and the joins above, 8 cold joins each (16 for the first row), a fresh browser each. Milliseconds from the connect’s start, medians; “audible” is firstAudibleUs or, for the page’s own order, the first clock report’s audible time:

Join	SUBSCRIBE sent	SUBSCRIBE_OK	First object	Audible	Range
The page’s order (73de7ab)	481	525	536	624	599–650
The page’s order without the tap’s wait	206	249	258	349	327–360
Built after the catalog, subscribed at it, held	140	239	256	354	311–406
Warmed at the connect, subscribed at the catalog, held	140	184	193	284	272–290
Warmed when the page loaded, before the join (24 joins)	139	184	193	289	270–291
Warmed, subscribed at once with the catalog’s subscription	94	140	144	234	216–254

The transport was ready at 67 ms and the catalog parsed at 139–140 in every row. The first row matches the benchmark’s stage 4.3 cells on a clean near path (connect to audible 634, catalog 161, subscribe 492, first object 554), where WebRTC was audible at 422. The hold held nothing in these runs: the warmed player was ready before the first object came. Three of about 155 joins that built the audio side at the connect had an AudioContext constructor that took 1.2–6.8 s; none of about 100 that built it after the catalog, nor 24 that built it when the page loaded, nor 24 more at the connect in a repeat. With no added delay (the relay on loopback) the page’s order was audible at 470 ms and the warmed join at 164; there the audio side cannot hide under the network: built at the connect it delayed the transport’s ready from 6 to about 55 ms, and built after the catalog the join was audible at 175.

On relay-dev’s voice benchmark, stage 4.3b (run run-20261028-194400-949931, relay-sdks build tag netbench/stage4-ttfa = 8eb9246, moq-relay 0.17.9, 4 rounds per cell), the median time from the connect’s start to the first audible audio, shipped player against quick join: clean 627 ms against 249 (WebRTC 419; MoQ over WebSocket with the quick join 399); at 2% random loss 957 against 503 (WebRTC 861); at a handover 880 against 456. The quick join had 0 drops in the first 10 s on clean (finding R-43B-01, relay-dev moq-netbench/results/voice/FINDINGS.md).

Clock. About every 20 ms the worklet posts the context frame it is rendering and the timeline timestamp of the sample there. The main thread times those posts with a render clock, on the playout clock’s own time source, and reports reportAudio(timestamp, renderedAt + baseLatency + outputLatency + player.outputDelayUs, input.stats), reading the latencies fresh each time. While the buffer is dry it reports nothing, so the clock runs on by itself and frames that missed their slot are dropped as late instead of played late.

The input calls four members of its clock, targetLatencyUs, localNow, reportAudio and stopAudio (AudioInputClock), so a player can pass its own clock instead of a PlayoutClock (issue #329). The Video.js renderer does: its clock holds where the buffer ran dry rather than handing over to wall pacing, which would step the position back when audio resumes. Such a clock needs to know what the worklet has played, which it cannot rebuild from the frames pushed and the timing of reports, so each report carries the input’s stats, refreshed with it:

Stat	Meaning
heldUntilUs	Timeline timestamp just past the newest held sample: the end of decoded audio, or null when nothing is held
dryAtUs	Timeline timestamp of the read position where the buffer last ran dry while playing, until it plays again; null otherwise, and after a flush or a cut that empties it
playStarts	Bumped each time playback starts: at the start, after running dry, after a flush or a cut that emptied it

A report whose playStarts is newer than the one before is of audio played after the buffer last ran dry. A report with the same count, still in flight when it ran dry, is the tail before it. A clock can hold at dryAtUs while it is set and release on the first report of a newer playStarts. PlayoutClock ignores the third argument: its behavior is pinned by the shared vectors, and it paces by wall clock while the buffer is dry.

Playback rate. input.playbackRate (and the playbackRate option), from 0.5 to 2, default 1, is how much held audio each output frame plays, from the next render quantum. A player that owns its latency, as the Video.js renderer’s controller does with ±5 % for seconds at a time, nudges it: above 1 sheds latency, below 1 gains it, where trimming would cut audibly and running dry would play silence. The worklet keeps a fractional read position and reads between held samples by linear interpolation, as the resampler does, carrying the fraction across quanta and rate changes, so a change does not click. Pitch shifts with the rate (about 0.85 semitone at 5 %), and there is no time-stretching. Held audio, the jitter target, trims, the hard bound and underruns stay in held audio, and bufferedUs stays timeline time. Below rate 1 the input does not trim: held audio that never dips to the target is the latency being gained, and trimming it would cut out what the slower rate added. Only the hard bound limits it then, and back at rate 1 the usual trim applies. A player that keeps more latency than that holds it upstream and meters frames in, as the Video.js renderer does with its subscriber buffer. Reports carry the timestamp at the fractional read position of the reporting quantum’s first sample, so a PlayoutClock extrapolating at rate 1 between reports is off by at most |rate − 1| × 20 ms, 1 ms at 5 %. Rate 0 is not a rate: pausing is context.suspend().

Audio-only target latency. The clock’s 2 s default waits for keyframe intervals that an audio-only broadcast never reports. AUDIO_ONLY_TARGET_LATENCY_US, 500 ms, is the smallest budget that still holds the 400 ms jitter-buffer ceiling plus a wired output delay (about 50–60 ms in Chromium’s headless runs below), and it cuts the silence before a resync from 2 s to half a second. A Bluetooth output delay lowers the ceiling instead: to 260 ms at a 200 ms delay. ITU-T G.114 puts the limit for most conversation at 400 ms one way, so a jitter buffer near the ceiling is already the network’s failure, not a setting to plan for. The default stays a constant the application passes, rather than a clock preset or an automatic default, because the clock’s behavior is pinned by shared vectors in both SDKs.

Cut. input.cut({ fromUs, untilUs }) drops received audio in a timeline range: what the buffer holds, output still in the decoder, and, when the range has an untilUs, frames that arrive later inside it, until a restart. The first audio from the cut’s start on is joined on without silence for the range cut out. Audio from before the cut’s start that was still decoding can land first and plays as usual, and a frame lost outside the range is still filled with silence. With no range it drops everything held. A talker’s interrupt, read from the interrupt track, is a cut range as it is. The worklet reports at once on the quantum after it applies a cut, and input.stats.cutHeardAtUs then says when that quantum reaches the speaker (its render time plus the output delay, on the clock’s time source), so the cut audio has stopped there; onCutHeard(atUs) says so as it happens. It is null from each cut() until the buffer has applied it. Track replacement times a barge-in with it.

Echo cancellation. Chromium’s echo canceller takes only audio that WebRTC plays as its reference. createEchoCancellationLoopback(context, player.output) sends the output through two local RTCPeerConnections into a media element. Firefox and Safari cancel echo from all page output and need none. The loopback adds its receiver’s buffering, which no API reports: set player.outputDelayUs if video must stay in sync with it.

Measured live on 2026-10-16 with mise run ts:test:audio-player: headless Chromium plays the WebCodecs example’s synthetic microphone (a steady tone) through the example’s audio-only listener, 8 s steady, then a cut of everything held and the next 300 ms of the timeline. “Delay” is the broadcast clock’s now less the playout clock’s position: capture to audible, with no microphone latency. Medians [p95]:

Path	Frames	Underruns	Buffer, ms	Delay, ms	Output latency, ms
In-memory hub	399	0	55 [70]	124 [132]	59.6
Relay (sjc)	401	0	64 [84]	183 [191]	51.6

In both runs the target shrank from 60 to 50 ms during the steady phase, playback resumed after the cut with no underrun, and the frame sequencer resynced nothing.

Rerun on 2026-10-19 with two phases between the steady phase and the cut: playback rate 0.95 for a second, then 1.05 for a second (issue #329). “Delay moved” is the playout delay at the end of the phase less at its start. A second at ±5 % should move it 50 ms. “Largest step” is the largest difference between two consecutive output samples, read through an AnalyserNode every 20 ms (a rerun after the fix for running dry at a fractional rate, which none of these phases did, moved the delay by +49.2 / −49.1 ms on the hub and +49.0 / −49.2 ms on the Relay): the 0.3 tone’s own steepest is 0.0174 at rate 1 and 0.0181 at 1.05, and a click would be far larger. Neither phase trimmed or concealed audio, and the clock’s position never stepped back between 20 ms ticks:

Path	Phase	Underruns	Delay moved, ms	Buffer p50 [p95], ms	Largest step
In-memory hub	0.95	0	+49.1	76 [111]	0.0174
In-memory hub	1.05	0	−49.0	80 [105]	0.0183
Relay (sjc)	0.95	0	+49.1	74 [92]	0.0174
Relay (sjc)	1.05	0	−49.4	68 [96]	0.0183

The steady phases in the same runs: hub 399 frames, buffer 46 [56] ms, delay 101 ms, output latency 42 ms; Relay 400 frames, buffer 43 [53] ms, delay 142 ms. The hub’s steady phase concealed 25 ms of frames that came late, which no rate phase did.

Interrupt track

Issue #320. A talker that stops mid-utterance, a voice bot the user talks over, has already sent audio its listeners hold in their jitter buffers and have in flight. Stopping its writes does not stop that audio. The interrupt track tells listeners to drop it.

The track. A data track in the broadcast’s namespace, named interrupt by default (INTERRUPT_TRACK), listed in the catalog by name with no role, so a listener subscribes only to a talker that can interrupt. Each interrupt is one object in a group of its own, its payload compact UTF-8 JSON (buildInterrupt / build_interrupt):

{"track":"audio","fromUs":1760000000000000,"untilUs":1760000000300000}

Field	Meaning
track	The audio track in the same namespace whose audio to drop. Required, a non-empty string.
fromUs	Optional. Drop from this timestamp on, on the broadcast timeline. Absent: from the oldest audio the listener holds.
untilUs	Required. Drop before this timestamp: the end of the last audio the talker sent.

Timestamps are non-negative safe integers in microseconds on the broadcast timeline the audio is stamped on, and fromUs, when present, is before untilUs. A reader (parseInterrupt / parse_interrupt) refuses a payload that breaks those rules and ignores fields it does not know, so later versions can add some.

The talker sends an interrupt when it stops before the end of what it has sent, with untilUs the end of the last audio frame it wrote: for a Python talker on OpusEncoder, the last packet’s timestamp_us plus its duration_us. It drops any audio it has encoded and not sent, and starts its next audio at or after untilUs. On a live clock that comes on its own. It leaves fromUs out to drop everything the listener has not played yet, which is what barge-in wants: the talker cannot know how far behind each listener plays. With fromUs set, a talker can let a listener finish audio up to a point, such as the end of a word.

The listener applies an interrupt for an audio track it plays as a cut, input.cut(interrupt): it drops what the buffer holds and what is still decoding in the range, and frames that arrive later inside the range, which covers audio still in flight on the audio track when the interrupt overtakes it. Audio from before a fromUs still plays, wherever it is. The listener ignores an interrupt for a track it does not play and reports one that does not parse.

No sequence number. A cut can only drop audio before its untilUs, and the talker’s next audio starts at or after it, so an interrupt that arrives late, twice, or out of order with a newer one drops nothing it should not. Group IDs give the order anyway. For the same reason a listener that joins and is handed the track’s latest object (a relay cache) drops only audio older than what it will play. Nothing about an interrupt needs the relay to deliver it before the audio it cuts.

What it does not do. Detecting that the user is talking (VAD, turn detection) is the talker’s business, as is cancelling whatever produced the audio. The convention cuts audio only. A listener applies the same interrupt to the transcript of the audio it cuts.

Measured live on 2026-10-19 with mise run ts:test:audio-player, which now sends its cut as an interrupt (everything held and the next 300 ms of the timeline) from the publisher page to the listener: the listener applied it 0.4 ms after the write through the in-memory hub, and 50.7 ms after it through the Relay (sjc). It cut 60 ms and 39 ms of held audio, and playback resumed with no underrun.

The voice bot example is a talker built this way: a Python bot answers a browser page and interrupts itself when talked over, leaving fromUs out. Its test runs it over the in-memory hub. Against sjc on 2026-10-19, with the page in headless Chromium on a scripted microphone, over four runs of three turns each, every interrupt reached the page’s player 32 to 50 ms after the bot’s last audio, and each run cut 117 to 140 ms of held reply audio, with no underrun.

Track replacement

The moq.dev post ‘OpenAI still has a WebRTC Problem’ (2026-10-27) proposes interruptions by replacing the audio track: the agent starts a new track for its new reply and drops the old one, and the player discards the old track’s buffer and switches to the new one by timestamp. A text-to-speech reply is generated faster than real time, so a player can hold seconds of it, more than an interrupt has to cut. TypeScript and Python publish it; the TypeScript example listener follows it.

The publisher. createReplaceableTrack(publisher, { track, catalog, ... }) in @caprid/relay/media, ReplaceableTrack(publisher, track=..., catalog_service=..., build_catalog=...) in caprid_relay.media, wraps a frame writer (frame grouping by default) over the current track. replace() registers the next track, audio-2, audio-3 and so on (or the application’s name(n)), lists it in the catalog in the old one’s place, drops what the old track has not sent and closes it. The drop happens synchronously in the call, before anything else runs, so no frame of the old track goes out after the switch. msf-01 §5.3 makes a changed track a new name, and a name is not reused on a connection, so a new track is the conforming way. A supervised publisher’s new generation would replace the whole connection, so the track name it is. The listener subscribes to the new track once the new catalog reaches it, about a round trip, and a frame writer skips frames the relay has no subscription for, so frames written to a replacement wait, in order, for its first relay subscription (demandWaitMs / demand_wait, default 2 s), then go out.

The listener keeps the catalog subscription open. When a catalog no longer lists the track it plays and lists another of the same role, render group and namespace (findReplacementTrack / find_replacement_track), it closes the old track’s frame reader and subscription, restarts the frame sequencer (dropping every frame it holds; the clock resets), cuts everything the player holds (input.cut()), starts a new transcript (sequence numbers belong to one track), and subscribes to the new track, whose first frame then anchors the clock: the switch is by timestamp. A replacement that comes while the listener is still setting up is followed once it has started. The listener depends on the catalog subscription for this, so an error of it after the first catalog is reported (onError). A replacement whose subscription the relay rejects, which arrives asynchronously after the old track was closed, leaves the new track current and silent: the next catalog that still lists it subscribes to it again, the same steps without a replacement event. A listener that joins later takes the catalog’s audio track whatever its name: it drops the subscription the quick join made by the conventional name audio, whose failure (a retired track) then fails nothing. The new track must have the old one’s codec, since the input keeps its decoder. The example’s onTrackReplaced reports three steps on the clock’s time source: seen (the catalog arrived and the old audio was cut), stopped (onCutHeard: the old audio has stopped at the speaker) and started (the first clock report of the new track’s audio, after the cut is heard). Until the cut is heard, after a replacement or a retried subscription alike, the listener passes no report to the clock: one the worklet posted before it applied the cut describes the old track’s audio and would re-anchor the clock the restart reset.

Against the interrupt track. An interrupt cuts a range of one track and leaves it open, so audio still in flight inside the range is dropped on arrival; a replacement stops reading the old track altogether, so nothing of it can arrive late, and its new reply needs no timestamp past the old one’s end. Either stops the old audio once its message, the interrupt object or the catalog, has reached the listener.

Measured locally on 2026-10-29 (relay-3cnp.399): moq-relay 0.17.9 (30fd9d450) built from source and the talker in one network namespace of relay-dev’s moq-netbench rig (rig local), the listener in the other behind the rig’s profile (rig apply, the downlink shaped on the peer), both on this branch’s SDK over native QUIC. The talker is a TTS agent: 20 ms Opus frames handed over at four times real time, each stamped with its slot. The listener is relay-sdks’ own receive chain composed as the example listener (frame reader, frame sequencer, audio input, its worklet and the WASM Opus decoder), plus timestamp playout in the sequencer (the frame sequencer’s timestampPlayout, a separate opt-in change; it holds a reply that arrives faster than real time), with Web Audio stood in for in real time (558-frame render callbacks, a 20 ms output delay); a frame counts as played when its decoded audio reaches the output, read from markers the harness puts in place of the PCM. The harness is not in this repository (relay-3cnp.399’s task directory).

Each run replaces the reply eight times, each 1.7 s into a 20 s turn, when the listener holds about 5.2 s of the old reply ahead of what it plays (6 s behind the full FIFO), nearly all of it in the sequencer (the jitter buffer held 23 to 80 ms). The new reply starts at the replace’s own time, on audio-2, audio-3 and so on, at the same speed. Times are from the talker’s replace() call, median and greatest of the eight, on one host’s clock; “stopped” and “first heard” are read from the output, the signals beside them from onTrackReplaced:

Link (rig profile)	Catalog seen	Old audio stopped	New audio first heard	stopped / started signals
Clean	2 ms (4 ms)	31 ms (37 ms)	43 ms (51 ms)	26 / 30 ms
loss-random-2 (50 ms RTT, 2% loss)	27 ms (28 ms)	48 ms (56 ms)	118 ms (124 ms)	50 / 116 ms
narrow (30 ms RTT, 2/1 Mbit/s, 500 ms FIFO)	17 ms (19 ms)	41 ms (47 ms)	87 ms (92 ms)	41 / 84 ms
narrow, a bulk CUBIC download filling the FIFO	447 ms (587 ms)	478 ms (617 ms)	919 ms (1,091 ms)	474 / 911 ms

The old audio stops one way after the replace, the catalog’s trip to the listener, plus at most a render callback and the output delay (31.6 ms here). The new audio starts about a round trip later: the subscription to the new track reaches the relay and the talker, whose waiting frames then travel the same way, and the jitter buffer fills to its target. In all 32 replacements the old track played only until the catalog arrived (1 to 3 frames after the call, 17 to 31 behind the full FIFO) and none of its remaining 5 s of held audio played later. A queue in front of the listener delays both by its depth, as it delays everything on that link. After the listener stopped passing reports from before the cut to the clock, a rerun gave the same numbers within a few milliseconds (clean: stopped 27 ms, first heard 43 ms; loss-random-2: 51 and 119 ms).

Measured on the bench: relay-dev’s voice benchmark, scenario 4ai-tts, on the build tagged netbench/stage4-ai (relay-sdks 5a4bd33; runs run-20261029-054413-95db41, run-20261029-102627-5276c7, run-20261029-105904-7de8ee and run-20261029-112630-e14a13), finding C-4. Its interrupt comes when the listener already has 5 s of the old reply. The listener following the replacement stopped the old audio a median 107 ms after the interrupt on the clean link (380 ms on LTE with a download), played at most 60 ms of stale audio, and started the new reply at 107 ms. A paced WebRTC agent did the same (96 ms, no stale audio; 560 ms on LTE against MoQ’s 329), while WebRTC fed bursts through a 2 s jitter buffer played up to 540 ms of stale audio. The benchmark’s listener also ran timestamp playout (a separate opt-in change), which track replacement does not need.

Transcript track

Issue #320. The text of what a talker says, such as a voice bot’s reply from its TTS, for a listener to show as the audio plays. A data track written one group per write has no order across groups and no tie to the audio, and a transcript of audio an interrupt cut would still claim the cut words were said. The transcript track places each piece of text on the audio’s timeline and lets an interrupt retract it.

The track. A data track in the broadcast’s namespace, named transcript by default (TRANSCRIPT_TRACK), listed in the catalog by name with no role. Each object is one version of one segment, in a group of its own, its payload compact UTF-8 JSON (buildTranscriptSegment / build_transcript_segment):

{"seq":12,"track":"audio","fromUs":1760000000000000,"untilUs":1760000000480000,"text":"Hello there.","final":true}

Field	Meaning
seq	Required. The segment’s number. A later version of the same segment repeats it; a talker numbers its segments in timeline order.
track	Required. The audio track in the same namespace whose audio this is the text of.
fromUs	Required. Where the segment’s audio starts, on the broadcast timeline.
untilUs	Optional. Where it ends, after fromUs. Absent while the talker does not know yet.
text	Required, possibly empty. A later version replaces it whole.
final	Required. true: no later version follows. A final segment has untilUs.

Numbers are non-negative safe integers, timestamps in microseconds on the timeline the audio is stamped on. A reader (parseTranscriptSegment / parse_transcript_segment) refuses a payload that breaks those rules and ignores fields it does not know.

Versions. A segment can be sent more than once: interim versions while its text or its end is not known, then a final one. An STT talker sends its partial hypotheses this way. A TTS talker sends an interim version just before the segment’s first audio, so that a listener has it when the audio plays, and the final one when its last audio is sent. A listener keeps a segment’s final version against any later one, and otherwise the last version it read. Groups can arrive out of order, so an interim version read after a newer interim one shows until the final one replaces it.

Interrupts retract. A talker that interrupts its audio sends no final version for the segment it cut. The interrupt ends it. A listener applies the interrupt to its transcript at its playout position: audio before the position was heard, so the cut starts at the later of the interrupt’s fromUs and the position. It drops the segments that start inside the cut, and ends the one playing at the cut’s start there and marks it cut. It remembers the cut, so a version that arrives later is treated as if it had arrived first, as the audio player treats audio that arrives late: a segment that starts inside the cut is dropped, one that runs into it is ended at its start, and an interim version of a cut segment changes nothing. The talker’s final version of a cut segment, delayed past the interrupt, is clipped the same way, so a segment whose audio ended before the cut is no longer marked cut.

Showing it. A listener shows a segment once its playout clock reaches fromUs: the text appears as the audio it belongs to plays, not when it arrives, and a segment the interrupt dropped is never shown.

Transcript (createTranscript() / Transcript()) is that listener’s state for one audio track: apply(segment) keeps a version and says whether it changed anything, cut(range, positionUs) applies an interrupt (a parsed Interrupt is the range), and linesAt(positionUs) / lines_at(position_us) gives the segments whose audio has started, in timeline order, each a TranscriptLine with a cut flag. It is pure, so both SDKs hold it to the same vectors. A segment is known by its seq within one run of the talker’s track, so a listener uses a new Transcript when the talker starts over, and a talker that starts over should not reuse its last run’s numbers (the example bot starts at the clock’s millisecond). The transcript keeps every segment it is given. A long-lived listener that wants a bound starts a new one.

No ordering track or sequence across segments. The timeline gives the order (fromUs), and seq only names a segment so that its versions find each other. A relay that hands a late joiner the track’s latest object hands it the latest version of one segment, which is all the transcript it can have: earlier segments are history the talker would publish another way.

The voice bot example publishes the text of each part of its replies, and its page shows each line as the audio reaches it, marked where it was talked over. Measured against sjc on 2026-10-19, with the page in headless Chromium on a scripted microphone, over four runs of three turns each: each interim segment reached the page 65 to 72 ms before its audio played (14 segments; the first of a run is not measured, while the clock is idle), every line appeared with its audio, and every reply the user talked over was shown marked cut, with no underrun. The margin is about the jitter buffer’s depth, since the bot sends the text just before the audio: a talker that knows its text earlier, as a TTS does, can send it sooner.

Media Timeline track

Issue #291. Format helpers implemented in both format packages. behavior.md holds the testable rules. Both product packages publish the track with their document service, and both examples do (issue #292).

msf-01’s Media Timeline track (§7) is an optional track, announced in the catalog, whose records map media time and an MOQT location to wall-clock time. It is MSF’s counterpart of HLS PROGRAM-DATE-TIME. Since #290 the broadcast timeline is itself wall-clock time, so a player that knows this convention does not need the track to learn when a frame was captured. One that does not can still read it here. What the track also gives is the map from time to location: seeking behind the live edge, and generating HLS or DASH playlists, need to know which group to fetch for a given time, and a LOC timestamp is only visible once the object has arrived.

Entries

A timeline object’s payload is a JSON array of records, each [mediaTimeMs, [group, object], wallclockMs] (msf-01 §7.1.1):

• mediaTimeMs is the media time of the first sample in the referenced object, floored to whole milliseconds: floor(timestampUs / 1000) of its LOC timestamp on the broadcast timeline.
• [group, object] is that object’s location in the track the timeline indexes.
• wallclockMs is milliseconds since the Unix epoch when that media was captured, floored. On the epoch-anchored broadcast timeline that is the timestamp itself, so the two numbers are equal. An entry helper caller with another anchor passes the wall-clock time explicitly, or 0 for unknown, as §7.1.1 allows.

The wall-clock time is the capture time of the indexed frame on the publisher’s system clock. It is not when the encoder produced the frame, which §7.1.1 names, and the difference is the encoder’s latency. Capture time is what lines broadcasts up with each other and what a packager writes as PROGRAM-DATE-TIME. This departs from §7.1.1’s wording (see Spec gaps).

Cadence

One entry per group of the indexed track, for the group’s first object. Under keyframe grouping that is one entry per video keyframe group, so there is an entry at every random-access point and nowhere else. That is what a seek needs, at one record per GOP, typically every one or two seconds.

The entries’ locations point into video, the first track in depends. Audio is one group per frame, 50 or more a second, and an entry for each would multiply the document for no gain: every track in the render group is stamped on the same broadcast timeline, so a player that has found the video location for a media time finds the audio for it by timestamp. A location cannot address audio anyway: it is a bare [group, object], and audio’s group numbers (one per frame) do not line up with video’s (one per GOP). msf-01’s example 5.6.10 gives each track its own timeline for that. A player here seeks audio by timestamp instead. msf-01 §7.2 still requires depends to list every track the timeline applies to, so audio is listed after video. An audio-only broadcast indexes its audio track at most once a second.

The §7.4 template, which states the mapping as a start and a constant stride, is not used. A live encoder’s GOPs vary in length (scene cuts, keyframe requests), and §7.4 requires the explicit track whenever durations vary.

Catalog entry

buildMsfCatalog / build_msf_catalog takes an optional mediaTimeline / media_timeline description and lists the track after video and audio, before data tracks:

{"namespace":"room/live","packaging":"mediatimeline","isLive":true,"name":"timeline","role":"mediatimeline","mimetype":"application/json","depends":["video","audio"]}

• §7.2 requires a “type” of mediatimeline, but msf-01 has no field called type. Both the packaging table (Table 3) and the role table (Table 4) list mediatimeline, so the entry sets both. The MIME type is the application/json §7.2 requires.
• depends (§5.2.14, required by §7.2) lists every track the timeline applies to, as msf-01’s example (§5.6.9) does. It defaults to the video and then the audio track, whichever the catalog has, and every name must be a media track in the same catalog. A location does not say which track it is in, so the entries’ locations point into the first track listed.
• While every track it depends on is withheld (an avc1 entry waiting for its avcC), the timeline entry is withheld too.
• The default track name is timeline (MEDIA_TIMELINE_TRACK).

The Video.js MoQ player (mmcc/moq, checked 2026-10-07) does not subscribe to a media timeline track. Its catalog parser keeps depends and treats any track not packaged as loc as non-renderable, so announcing the track changes nothing for it. Its timeline.ts has helpers for the §7.4 template that nothing calls. Reading the explicit track for seeking is player work.

The origin’s wall-clock time

The broadcast clock records its origin as Unix-epoch microseconds, originEpochUs / origin_epoch_us, and every timestamp it hands out is counted from there. A caller may set it, for example from a capture device’s own wall-clock stamps. Otherwise the clock reads the system clock once, at creation, and subtracts the time between the origin and that moment on its own source:

• TypeScript reads Date.now(). performance.timeOrigin + performance.now() is finer, but it can fall behind the system clock, by however long the page has slept, where performance.now() does not advance while the machine sleeps. Date.now() has millisecond resolution, well inside the error of a system clock synced over the network.
• Python reads time.time_ns().

Since it is read once, timestamps advance at the monotonic clock’s rate from then on, not the system clock’s. The two can drift apart by tens of parts per million, a fraction of a second in an hour, and an NTP step during the broadcast does not move the timeline. A step mid-broadcast would otherwise move every track at once and break the playout clock’s pacing. A long broadcast that has to follow the system clock can be re-anchored at a new generation.

Publishing the track

The rules a publisher follows:

• An entry is added only when a group’s first object was sent: under keyframe grouping, a keyframe that opened a group on the relay. A keyframe the relay had no subscription for, or that was stale or interrupted, adds nothing, since its location would point at a group that does not exist.
• msf-01 §7.3 requires the first object of every timeline group to be an independent document: every record accumulated and still accessible. Each new entry opens a new timeline group whose only object is the independent document of the retained entries. A viewer that joins at the relay’s current group gets the whole history in one object. The incremental objects §7.3 allows after the first are not used.
• How many entries to retain is the publisher’s choice: as far back as its viewers can still fetch, and well under the object size limit. One entry is about 35 bytes, so an hour of the examples’ one-second GOPs is about 126 KB. The accumulator keeps 512 by default, about 18 KB or 8.5 minutes of one-second GOPs, and drops the oldest.
• The timeline is a state-shaped track whose document changes. The document service (createDocumentService / DocumentService) sends the latest document on every relay subscription, as the catalog service does, and on every new entry while the relay is subscribed. Entries made without demand accumulate and go out with the next subscription.
• A supervised publisher’s next generation is a new broadcast with a new namespace, catalog, broadcast clock and origin. It creates a new timeline and service, which start empty. The old entries point at locations in a namespace the new catalog does not list, so they are not carried over. Wall-clock time is what joins two generations, and both stamp their media with it directly.

The WebCodecs example and the Python Video.js streaming example publish the track. The Mediabunny example does not: a file’s media has no capture time to report. Against sjc.relay.mux.global (2026-10-14), every record the Python example’s --verify viewer received named a received keyframe’s group and timestamp, and a subscriber that joined 12 s in got all seven records so far in its first object.

Reading

parseMediaTimeline / parse_media_timeline reads one timeline object’s payload into entries, in document order. It rejects a payload that is not a UTF-8 JSON array. Within the array it is lenient, as msf-01 is for catalogs: a record is read when its first three items are a number, an array whose first two items are numbers, and a number, all non-negative safe integers. Extra items are ignored, and anything else, a §7.4 template array among them, is skipped. Whether the payload is independent or incremental is the caller’s to know from the object’s position in its group. MSF_COMPRESSION payloads are not decompressed.

Spec gaps

Questions the drafts leave open, found while settling #290. Each is worth raising on the MSF or LOC draft. Until then, this repository’s answer is its convention.

• No anchor for a render group. msf-01 §5.2.11 says tracks in a render group “SHOULD be rendered simultaneously”, and §4.2 aligns alternates, but nothing says the tracks of a group share a timestamp anchor, and nothing signals which anchor a track with a declared Timescale uses (LOC §2.3.1.2: “application-defined”). This repository anchors every track at the Unix epoch. draft-frindell-moq-timestamp-00 proposes a clock identifier and an origin, which would let a publisher say so.
• targetLatency needs wall-clock time per frame. msf-01 §5.2.8 defines it from the time a frame was encoded, in UTC. With an application-defined anchor, the only route to that time is the Media Timeline, at millisecond resolution and only at the indexed locations.
• Encoded or captured. msf-01 §5.2.8 and §7.1.1 both define the wall-clock time as when the media was encoded. Stamping capture time departs from that definition by the encoder’s latency. The sentence has no RFC 2119 keyword, but normative text needs none (RFC 8174 §2). Lip sync, syncing across participants and PROGRAM-DATE-TIME all need capture time, which is what this repository stamps and indexes. draft-nandakumar-moq-tempo-00 carries an explicit capture timestamp next to a playout delay.
• Two timescales. msf-01 §5.2.21’s catalog timescale and LOC’s Timescale property (0x08) have no rule for how they relate or which wins. The readers here take the object’s property, then the track’s, and never the catalog’s. LOC has no rule for object versus track precedence either.
• A missing Timescale reads two ways. LOC draft-04 §2.3.1.1 reads a Timestamp without one as wall-clock microseconds since the epoch. draft-lcurley-moq-timestamp-01 §3 says such a track has no timeline and falls back to arrival time for age decisions, though a receiver that implements LOC may read it as LOC does. Neither tells a relay to overwrite the Timestamp. moq-relay’s restamping (below) is its own.
• The relay modifies properties. MOQT-22 §8.4 says a relay must forward properties unmodified and cache them. moq-relay (read at moq-dev/moq 5152ec74) rescales 0x10, writes its own 0x08, restamps tracks without a Timescale, and drops properties it does not know, among them 0x0D and the publisher’s other track properties in SUBSCRIBE_OK. That is a relay bug rather than a draft gap, listed here because it is why the readers scale by whatever Timescale arrives.
• Capture latency has no field. No draft has a place to declare it, so it is folded into the Timestamp (see Capture latency).

Package placement

The media layer has two homes, matching the existing boundary rules.

• A format package per language, @caprid/moq-media and moq-media (module moq_media), holds the format helpers and the readers. It depends on the core only (for KeyValuePair and MoqtObject), has no other runtime dependencies, and knows nothing about Relay. The TypeScript package started as an attributed snapshot of the Video.js media/moq modules with its own UPSTREAM.json, intended to migrate back upstream rather than fork, as the core did. The Python package ports it. The boundary checks hold that the format package imports nothing from the product package and no transport library, and that the core imports nothing from it.
• Media publishing lives in the product package, as @caprid/relay/media (a second package entry point) and caprid_relay.media, because it is built from RelayTrackWriter, demand, and the write deadline, and because Relay-specific observations such as the dropped Video Config property belong there.

The alternative was one home, everything under caprid_relay.media and @caprid/relay/media. It is one package fewer per language, and the code in the samples above is identical apart from the import lines. It gives up the snapshot-and-migrate story, and it puts a catalog builder that any MoQ publisher could use behind a Relay dependency.

Encoders stay out of the product packages, and out of the format packages’ roots. An H264Config record that turns Annex B into an avcC is a format helper. A class that opens libx264 is not. The exception is Python’s Opus encoder and decoder (issue #320), which a voice bot needs both ways: moq_media.opus, on PyAV, behind the opus extra, and never imported by the package root (ADR 0006).

Keeping the languages in lockstep

The media layer uses the mechanisms the rest of the repository uses:

• protocol.json pins the LOC draft (draft-ietf-moq-loc-04), the MSF draft and catalog version (draft-ietf-moq-msf-01, "draft-01"), the microsecond default timescale, the LOC property types, and the default track names, each asserted by a test in every SDK.
• behavior.md has Media helpers, Media publishing and Media subscribing sections: what the catalog builder refuses, how the catalog service treats demand, what the frame writer does without demand and after a failure, and the statement that time-based drop policy is the application’s.
• vectors/media.json holds catalog vectors (track descriptions in, exact JSON bytes out), LOC property vectors (timestamp and config in, key-value pairs out), source URL vectors including the .xx escaping and a token, and avcC vectors (Annex B in, avcC and codec string out), so both SDKs are held to the same bytes.
• The TypeScript snapshot brought parse-catalog.ts with it, so the acceptance test the Video.js builder names for itself, “every emitted document must round-trip through the parser”, runs in this repository too.
• vocabulary.md gets rows before any of these names become public.

Sequencing

Each step leaves the example runnable against the live relay with --verify, which is how the change is checked without a browser, and each step deletes example code. The status at the top of this document says how far each step has gone.

1. Format helpers in both languages, with the vectors, pins, and the behavior.md section. Move the example’s moq_media.py onto them, which removes about 300 lines and fixes #29 for real because the README snippet can then import what it calls.
2. The media publishing handle. The example’s catalog serving and frame writing move into the SDK. Its LiveQueue, encoders, pacing, and finished accounting stay. The example’s tests for catalog serving and track-end handling become the SDK’s tests.
3. The readers, with --verify rewritten on them, and the TypeScript example that the browser sample above sketches.

Open questions

• Names. register_media and MediaTracks are placeholders. The format tier kept the Video.js names buildMsfCatalog and composeMoqSource, with Python spelling them build_msf_catalog and compose_player_url.
• A subscription that arrives mid-GOP. The example waits for the next keyframe. Video.js retains the open group and replays it from object 0 so the viewer decodes at once. Replay is better for viewers and costs one GOP of memory per track. It can be a later option on the frame writer.
• Queue semantics on the frame writer: settled by issue #263. write resolves with the frame’s outcome once the transport accepted it, as write() does, and may be called without awaiting. The frames waiting meanwhile are a bounded backlog that drops rather than queues.
• Codec scope at the start. H.264 is what the example needs and what the relay has been tested with. HEVC (hvcC), AV1 (av1C), and AAC (AudioSpecificConfig) records are the same shape of helper and can follow demand.