MoQ media formats

View .md Open in Claude Open in ChatGPT Source

Pure LOC (draft-ietf-moq-loc-04) and independent MSF (draft-ietf-moq-msf-01) format helpers. The package depends only on the generic MoQ core, and needs no transport library or Relay connection. The one encoder, moq_media.opus, needs the opus extra (see Opus). import moq_media never loads it.

The catalog builder emits compact UTF-8 JSON bytes. It withholds avc1 and avc3 until non-empty init data exists, derives their profile, constraint, and level suffix from avcC (keeping avc3), forces Opus to 48 kHz, and rejects duplicate names and media roles on data tracks. Catalog namespaces use the player’s slash-joined dialect, so fields containing / or empty fields are rejected. Source URLs can escape literal slashes in tuple fields. Omitted fields never become JSON nulls. Empty init data is omitted for every codec, and catalog strings must be valid Unicode (no lone surrogates).

LOC helpers accept nonnegative microsecond timestamps and a positive timescale, both safe integers (up to 2^53−1). A custom timescale rounds to the nearest tick, ties upward. Pass decoder configs only on keyframes, declare the same timescale on the track, and keep init data in the catalog too.

H.264 helpers split Annex B, build avcC from SPS/PPS, and convert access units to four-byte NAL length prefixes. Conversion removes SPS, PPS, and AUD by default, as avc1 requires. Codec-string inspection checks the avcC version and header. It does not validate SPS semantics or decode a bitstream. Construction from SPS/PPS supports Baseline, Main, and Extended profiles. Other profiles require extension fields and are rejected. Callers can still supply an encoder-generated High-profile avcC to the catalog builder.

build_msf_catalog describes a live broadcast and leaves isComplete out: msf-01 §5.1.3 forbids the field while false, and once a catalog carries it, it can never be removed. When a broadcast ends on purpose, close the media tracks first (each close tells subscribers the track ended), then write build_msf_end_catalog(generated_at_ms=...) to the catalog track if it has demand, then close it (msf-01 §11.3). Do not send it after a crash or a lost connection.

parse_msf_catalog reads an independent catalog object’s payload back into an MsfCatalog of MsfCatalogTrack descriptions: namespace tuple, name, packaging, live status, role, codec, dimensions, rates, bitrate, render group, and init data resolved from initRef to bytes. It raises ValueError on unsupported versions and delta updates, ignores unknown fields, and returns non-loc packagings (such as moq-rs cmaf) as declared.

For simulcast layers or a second source, pass media: a list of VideoTrackInfo and AudioTrackInfo, each with a render_group (default 1, tracks played together) and an optional alt_group (alternate versions of one content, of which a viewer plays one). Members of an alternate group share a role and a render group. parse_msf_catalog returns both fields, and group_msf_tracks(catalog.tracks) lists the sets a viewer picks one track from. See several media tracks.

catalog = build_msf_catalog(
    namespace,
    media=[
        VideoTrackInfo(name="video-hi", codec="vp8", width=1280, height=720, alt_group=1),
        VideoTrackInfo(name="video-lo", codec="vp8", width=320, height=180, alt_group=1),
        AudioTrackInfo(name="audio"),
        VideoTrackInfo(name="screen", codec="vp9", render_group=2),
    ],
)

Every media track shares one timeline: LOC timestamps are capture times in microseconds since the Unix epoch, with Timescale 1,000,000 declared on each track. BroadcastClock() pins a moment on the monotonic clock to the system clock (origin_us, origin_epoch_us) and counts from there. Use from_source_us for capture timestamps on that clock. Use source_timeline() for a clock with its own origin and rate, kept in line with calibrate(source_us, at_us=...) pairs. No browser or OS reports a microphone’s input delay, so pass the one you know as capture_latency_us= and set timeline.capture_latency_us again when the device changes (about 30 ms for a laptop microphone, 190 ms for AirPods). Use sample_timeline(48_000) for a sample counter with nothing to calibrate against, such as a file. PlayoutClock() is the player side. Audio is the master: report what is audible with report_audio(timestamp_us), and due(timestamp_us) returns a PlayoutDecision to wait, show, or drop a video frame. With no audio it paces by wall clock. A broadcast with no video never reports a keyframe interval, so its clock would keep the 2 s default target latency. A lasting delay increase would then cost 2 s of silence before a frame sequencer resyncs. Create an audio-only clock with PlayoutClock(target_latency_us=AUDIO_ONLY_TARGET_LATENCY_US) (500 ms), the same constant as TypeScript’s. A clock nothing anchors, as in a bot that only decodes, judges nothing late and needs none. See spec/media-layer.md.

On the subscriber side, loc_timestamp_us(obj.properties, track_properties=ok.track_properties) reads an object’s Timestamp in microseconds, applying whichever Timescale the object or the SUBSCRIBE_OK track properties declare.

msf-01’s Media Timeline track holds one record per video group, mapping media time and the group’s location to wall-clock time, for seeking and playlists. On the epoch timeline the two times are the same, so an entry needs only the timestamp and the group. Announce it with media_timeline=MediaTimelineTrackInfo() in build_msf_catalog. MediaTimeline keeps one broadcast’s records, 512 by default, and returns the document to send each time a keyframe has opened a video group. caprid_relay.media.DocumentService sends it as a new timeline group, and again whenever the relay subscribes. See spec/media-layer.md.

timeline = MediaTimeline()
service = DocumentService(timeline_track)  # from caprid_relay.media; run service.serve() as a task
# Once the frame writer reports the keyframe that opened video group `group_id` as sent:
service.update(timeline.add(timestamp_us, group_id))
read = parse_media_timeline(timeline_object.payload)

Shared vectors live in spec/vectors/media.json and spec/vectors/clock.json. See UPSTREAM.json for Video.js provenance. The publishing handles (FrameWriter, CatalogService, DocumentService) and the object-to-frame reader (FrameReader) need a Relay connection, so they live in caprid_relay.media. FrameSequencer here is the reader’s transport-free engine: it puts one track’s objects in decode order and drops what is too late against a PlayoutClock.

from moq_media import (
    AudioTrackInfo,
    VideoTrackInfo,
    build_msf_catalog,
    loc_properties,
    loc_track_properties,
    parse_msf_catalog,
)
from moq_media.h264 import H264Config

h264 = H264Config.from_annexb(encoder.extradata)
catalog = build_msf_catalog(
    ["room", "live"],
    video=VideoTrackInfo(codec=h264.codec, width=640, height=360, init_data=h264.avcc),
    audio=AudioTrackInfo(),
)
video = publisher.register_track("video", track_properties=loc_track_properties())
# The application opens a group at each keyframe and handles demand and recovery.
await group.write(h264.to_avcc(access_unit), properties=loc_properties(timestamp_us, video_config=h264.avcc))

tracks = parse_msf_catalog(catalog_object.payload, catalog_namespace=["room", "live"]).tracks

Interrupts

A talker that stops mid-utterance, such as a voice bot the user talks over, writes an interrupt so listeners drop the audio they already hold. List the track in the catalog (data=[DataTrackInfo(name=INTERRUPT_TRACK)]), register it, and write build_interrupt(until_us=...) to it, with until_us the end of the last audio sent: the last OpusPacket’s timestamp_us + duration_us. Start the next audio at or after it. parse_interrupt(payload) reads one back as an Interrupt. See spec/media-layer.md.

Transcripts

A talker publishes the text of its audio on the transcript track, so listeners show it as the audio plays. List the track (DataTrackInfo(name=TRANSCRIPT_TRACK)), register it, and write build_transcript_segment(seq=, from_us=, text=) just before the segment’s first audio, then the same seq with until_us= and final=True once its last audio is sent. Send no final version for a segment an interrupt cut. A listener reads each with parse_transcript_segment, keeps them in a Transcript (apply, and cut on each interrupt with the playout position), and shows lines_at(clock.position_us). See spec/media-layer.md.

Opus

pip install "caprid-moq-media[opus]" adds PyAV and moq_media.opus: Opus encoding and decoding for a server-side talker such as a voice bot. OpusEncoder takes interleaved s16 PCM (or a PyAV AudioFrame) at any rate and returns 20 ms OpusPackets stamped on the broadcast timeline. Its defaults are for voice: mono, 32 kb/s, libopus’s voip application. Audio comes in runs. The first encode of a run gives the timestamp of its first sample, later calls continue by sample count, and flush() ends the run (an utterance) by padding the last partial frame with silence. OpusDecoder(rate=16_000) turns packets back into s16 PCM at the rate speech recognition wants. Call reset() at a frame’s discontinuity.

from moq_media import AudioTrackInfo
from moq_media.opus import OpusDecoder, OpusEncoder

encoder = OpusEncoder(input_rate=24_000)  # the TTS engine's rate
audio_info = AudioTrackInfo(channel_config="1", bitrate=encoder.bitrate)  # for build_msf_catalog
# An OpusPacket carries its LOC properties, so caprid_relay.media.FrameWriter takes it as a frame.
for packet in encoder.encode(tts_pcm, timestamp_us=clock.now_us()) + encoder.flush():
    await frame_writer.write(packet)

decoder = OpusDecoder(rate=16_000)
async for frame in frame_reader:
    if frame.discontinuity:
        decoder.reset()
    stt.feed(decoder.decode(frame.payload))

Packet k of a run is stamped at the run’s timestamp plus 20 ms per packet before it, as WebCodecs stamps its output. Decoded audio trails the stamps by libopus’s look-ahead, which is 6.5 ms, or 2.5 ms with application="lowdelay". A flushed run has one packet more than the frames it was given, which carries the look-ahead’s tail. opus_packet_duration_us(payload) reads a packet’s duration from its TOC byte without PyAV, held to the same vectors as TypeScript’s opusPacketDurationUs. See ADR 0006.

The distribution on PyPI is caprid-moq-media, and the import name is moq_media.

Stability

Before 1.0, a minor version can break the API. This package releases together with caprid-relay at the same version. See the changelog.