Version 26.09

Flussonic 26.09: Catena gets DRM and its own storefront, Mcaster takes SDI, ST 2110 and RIST, and Watcher's NVR moves to the new core by a button

Four products, one core

Catena, Mcaster, Agora and Watcher are four different products with four different customers — an OTT operator, a headend engineer, a corporate broadcaster and a security integrator. Underneath they are the same engine: Sapsan, our streaming core written in Rust.

That is why this release note starts with the core. Everything in the first section arrived in all four products at once, without any of them having to build it — and it is also why a fix found on a nine-thousand-camera recorder ends up making a television channel start faster.

Here is what we did for the Sapsan streaming core

Sixty times more WebRTC viewers. WebRTC was the weakest delivery path we had. A node took 200–300 concurrent WHEP sessions and about 0.9 Gbit/s, and then simply stopped accepting viewers: at five hundred sessions, 448 out of 500 fell off on connect timeouts while the process was using 330% of the 7,200% the machine had. It was not short of power — the whole WebRTC subsystem was one task holding every session under one lock, so creating a session queued behind delivering to all the others.

Sessions now live in shards with their own loops, and the numbers changed shape. A single node holds 18,000 concurrent viewers at 32 Gbit/s, or 16,000 heavier ones at 42.9 Gbit/s — a steady state with nobody dropping. Memory per viewer fell from 3.8 MB to 0.6–0.7 MB, and the queue behind it now has a hard limit: six thousand viewers used to take 53 GB and push the machine into swap.

One practical note before you measure your own box: the default fq queueing discipline on Linux costs about 8 Gbit/s of that, and swapping it for pfifo_fast gets them back.

Separately, abandoned WHEP sessions used to live forever: a viewer who closed the tab left a session that kept burning CPU and grew memory at 80 MB/s until the process was killed. Those sessions are now reaped.

Over 100 Gbit/s of HLS from one machine. We set out to pull 100 Gbit/s through the network card and did, at which point the PCIe 3.0 slot of the test machine became the limit rather than the software. Over loopback, where the wire is not in the way, the same server delivered 376 Gbit/s of plain HTTP and 242 Gbit/s over TLS. The practical reading is not the headline number: it is that on ordinary hardware the delivery path is not what you will run out of.

Reading fMP4. Ticket #5350 — "capture MPEG-DASH / HLS fMP4 sources" — was filed in March 2018 and collected about fifty customer requests, the longest such list in our tracker. It asks for one thing, not two: a modern playlist-based source puts fragmented MP4 in its segments, not MPEG-TS, and fMP4 is exactly what we could not read. Every aggregator, every third-party CDN and every contributor who will not open a UDP or SRT port publishes CMAF, and until now we had nothing to answer with.

One rule governs the start, and it is worth knowing. Capture begins with the first segment that appears after you connect — everything already sitting in the playlist window is history. Otherwise a source with a minute-long window would dump a minute of stale media into your channel in a few seconds, and do it again on every reconnect. The same rule covers a dead source: a playlist left behind by a broadcast that has ended never produces a new segment, so capture simply never starts, instead of the stream sitting there looking healthy on a stale window.

Broadcast subtitles, finished. In 26.08 we shipped CEA-608/708 closed captions as WebVTT and promised DVB subtitle recognition next. Here it is, and with it teletext — the second form of subtitles in European broadcast, and the only one where the text arrives as text.

All of it now goes through one design: every producer of text — the teletext decoder, DVB OCR, closed captions, speech recognition — writes into a single canonical layer, and the viewer always receives an ordinary WebVTT track. Whatever the channel carried, the subscriber in your app sees the same subtitle menu as a cable viewer, and accessibility requirements are met without a second input or a separate pipeline per format. Along the way the teletext decoder was fixed in two places that mattered: a Hamming bit-order error that read page 888 as 811, and missing national character sets that turned umlauts and Cyrillic into rubbish.

Hardware transcoding on NVIDIA. The whole video path — decode, scale, encode — moves onto the card, chosen explicitly or automatically, so how many channels a server carries stops being a question of how many CPU cores you bought.

Along the way we finally ran down a bug that had been troubling us since the Flussonic days. It turned out not to be ours: it is a defect in NVIDIA's own driver, and NVIDIA has registered it as Bug 6610402. Until it is fixed at the source, we ship a workaround — so it stops reaching your installation.

A layouter that does the work you would otherwise do by hand. Deciding which node runs which stream is a job somebody has to do — and redo every time a machine is added, drained, overloaded or lost. On a few servers you can hold it in your head. On a hundred you cannot, and the cost of getting it wrong is a channel off the air. Catena's layouter now decides it against 129 rules, written down in a policy document before they are code, with the tests checked against the same document.

What that covers, in the operator's terms. Requirements a placement may never violate: namespace, required labels, a stream that needs a graphics card going only to a node that has one, a stream with an archive going only to a node with archive storage, a per-node limit on the number of streams. Preferences that decide the ties. Reassignment when a node goes offline, is drained or leaves the cluster — while a node you deliberately froze keeps exactly what it has. Overload with a defined order of relief: processor above 90%, network above 95% of the node's limit, archive above its threshold; network is relieved first, archive copies are moved before ingest, the heaviest stream goes first. And the roles beyond ingest — archive copies, leftover archive on machines that no longer run the stream, live delivery on edges, timeshift streams placed near the archive they replay.

One number is worth saying plainly: over the past year 96% of the customer reports we received about the layouter turned out to be a configuration mistake on the customer's side, not a defect in the layouter. The engine was rarely the problem — what was missing was any way to see why it had decided what it decided. That is the half this release closes: every assignment now carries the reason it was made, and the console has a page showing the runs, the assignments and the constraints that could not be satisfied.

Administrators, roles, and a revocation that revokes. Until now management access was a shared password and a static key: you could not add a second administrator, limit them, or cut off one person without changing the secret for everyone. There is now a proper register of administrator accounts with roles. And disabling an account now stops work immediately — previously only login was revoked, while an already-issued token kept creating streams and editing nodes for up to fifteen minutes.

You can see more of what is happening. The stream error graph broke out into separate lines per error kind, and connection-level failures — an SRT session dropping, an HLS source refusing an HTTP request — are counted at all for the first time, so a full source outage is finally visible on the graph instead of only in the log. Source quality telemetry, server statistics, GPU inventory and load, and transcoder counters now reach Retroview. When a node refuses to apply a configuration you saved, the reason reaches the console instead of staying in the node's local log. And on installations with dozens of disks, the used space is broken into current archive, orphaned recordings and everything else, so "where did the space go" has an answer.

Here is what we did for Catena

DRM, and therefore paid broadcasting. This is the single largest gap that kept operators on Flussonic Media Server. Catena could decrypt protected input but could not protect what it delivered, so it did not replace Flussonic on any paid channel. It does now. Streams are encrypted with CMAF cbcs, and one encrypted copy serves HLS/fMP4 for FairPlay and DASH for Widevine and PlayReady — you do not package twice. Keys come either from your own key server over SPEKE v1 or v2, or from configuration for testing.

The cluster design matters here as much as the cryptography. Encryption happens on the way out of the node that captured the source, not on the way to disk: the archive stays clear, so rewind, thumbnails, transcoding and analytics keep working. Edges receive only ciphertext — the key and the key server credentials never leave the capturing node and never appear in an edge's configuration. And when a configuration is imported from Flussonic v3, a stream with a drm section arrives disabled, so a bulk migration cannot quietly start broadcasting a premium channel in the clear.

A storefront with a real programme guide. Catena's viewer application — the one your subscribers open on a TV or in a browser — moved onto the new core together with the backend it needs. A subscriber signs in with their own token, gets the list of channels they are entitled to with ready-to-play addresses, and a playback ticket comes with the list, so concurrent-device limits and revocation work without the storefront knowing anything about issuer keys.

Behind it is a programme guide that did not exist before in any form. Central now owns XMLTV sources, downloads them on a schedule, stores the grid and hands it to the storefront and to the MPEG-TS output as EIT. Two practical notes from building it: the import filters programmes down to the channels you actually bound — a real provider file we tested against was 1.5 million programmes across 4,054 channels, of which nobody would ever read 95% — and the downloaded file is kept, so binding a new channel re-imports without waiting for the next download.

If you already have a middleware. Many of our customers sell one of their own, and Catena is not competing with them. In that arrangement Catena is what your middleware points at: a single playback address, with the balancing redundant inside the cluster — so there is no separate hardware balancer standing in front of it to buy, rack and keep alive. The storefront above is for operators who have no middleware yet and would rather not start by building one.

Timeshift channels. "Channel One +2" as a first-class channel. Rewind has existed for a while, but rewind is a parameter in a player's URL: the shifted version had no name of its own, no line in the channel list, no sessions or statistics, and could not be pushed, protected or handed to a CDN zone. A timeshift channel is now an ordinary stream with an unusual source — so it gets its own viewer URLs, its own sessions, its own DRM and its own pushes for free. One capture and a deep archive cover your time zones instead of a separate regional feed per zone.

The last two format gaps. Catena captures DASH sources now — the other half of the same ticket, running on the same fMP4 reader described above. Between HLS and DASH that covers the CMAF origins, DVB-I gateways and cloud packagers that publish only one of the two. DASH differs in one honest way: a manifest does not list what exists, it gives you a start time and a template, so which segment is already available is computed from the wall clock rather than read off a list. And on the way out, HLS is served in the MPEG-TS container as well as fMP4, at its own address, for the older players and set-top boxes that accept nothing else. Together these remove the last two "Flussonic could do it and Catena could not" answers on the ingest and egress sides.

VOD stopped being manual work. A cluster VOD catalogue could accept a file and play it, but everything that turns an uploaded file into a service — the MBR ladder, the SMIL descriptor, what happens to the original — was done by hand on each node. Now a transcoding template is attached to an asset, the job is created when the upload commits, central picks the node that will run it, and the result comes back into the catalogue as an ordinary file. Templates, jobs and their progress are in the console rather than in curl. And MBR assembled from sibling files plus external subtitle files picked up next to the video mean a migrating customer does not repackage their library.

Edges became real edges. An edge node used to serve archive only: live and rewind were kept away from it, because a stream on an edge would have pulled from the origin forever whether or not anyone was watching. Both reasons are gone. Streams on an edge are raised on demand when a viewer arrives and dropped when they leave, and nodes exchange live over our own inter-node protocol rather than degrading to RTMP or SRT with an extra remux. So the whole viewer load — live included — moves off the capturing node onto the edges.

What you gain by moving from Flussonic Media Server to Catena

Catena is built cluster-first: you add a stream instead of choosing a machine for it, the archive and the VOD catalogue are cluster-wide, and you can see the whole installation from one place. That case we made in 26.08 and it has not changed. What changed in 26.09 is that the list of things Flussonic could do and Catena could not has essentially run out:

Here is what we did for Mcaster

The broadcast inputs, all of them. Mcaster is the headend: the signal arrives from a satellite receiver, a neighbouring network, a partner or your own playout. This release closes the inputs that only the Erlang Flussonic could take.

SDI from DekTec cards runs as a separate process per card port, talking to the streamer over shared memory — so a driver-level fault takes down one capture rather than the media server, which given the history of DTAPI crashes is not a theoretical concern. It deembeds multi-channel audio, extracts VANC and computes aligned timestamps at the source. The console now shows what cards are actually in the machine, how many ports each has, what firmware variant they are in and which streams have taken them — because on a DTA-2178 every channel is assigned by firmware as input or output, and a port that is currently an output answers a capture attempt with "busy" and sends the engineer hunting for a process that does not exist.

SMPTE ST 2110 is received in full: 2110-20 video, 2110-30 audio and 2110-40 ancillary data, with an NMOS node exposing receivers over IS-04 and IS-05 so somebody else's controller can patch them. Reception runs on a dedicated system thread reading packets in batches — interlaced HD at 10 bits is 130 MB/s and about 90,000 packets per second on a single track, and that is not work for a general-purpose task scheduler. MXL, the EBU shared-memory exchange layer, is read too, for installations where the neighbouring media functions hand over signal that way.

RIST arrives as its own implementation — reception and sending, unicast and multicast, with retransmission and the counters you need to prove the link is behaving. It matters commercially rather than technically: RIST is the only reliable-delivery protocol published by an industry body rather than one vendor, so it appears in tender requirements as a line item, and until now we could not answer such a tender at all.

A transport stream you can actually hand over. The outgoing MPEG-TS used to be take-it-or-leave-it: programme number fixed at 1, PIDs auto-assigned, no SDT at all. You can now set the PID map, the programme number and the service name per destination. The immediate case was a customer receiving one channel from us and from a backup provider, whose failover switch requires the PIDs of the two streams to match — but the same control is what a multiplex, an IRD, a NOC monitoring system or a contribution contract specification asks for.

Conditional access. Simulcrypt scrambling on the SPTS output: without a card there is no picture, with a card there is. It was configurable only by hand-editing a document, where one wrong hex field means "the card silently ignores EMM" and costs days on a test bench — so it now has a proper form in the console.

Customers, and what each of them bought. This is the item Mcaster operators asked for most directly. You do not sell a stream, you sell a service: this channel, delivered this way, to this company. Configuration and monitoring only knew about streams, so when a customer phoned there was nothing to look at — what did they buy, which part of it is down, and whose outage is this.

Now the sale is recorded where the delivery lives. A unicast push or an SRT play port carries the identifier of the customer it belongs to; a static per-stream token lets a buyer be given a link once and have their consumption attributed to them. Multicast pushes deliberately do not carry an owner — the switching is done in the network and one field cannot describe that honestly. On top of it sits per-destination delivery statistics: status with a named failure reason, rate, bytes, reconnects, retransmits. Which also fixed a nastier thing — a failing push sometimes reported itself as running with an empty error while its reconnect counter climbed, so the console assured the operator that delivery was alive while not a single byte was leaving.

Who changed what. Central holds the desired state of the installation — stream and node configuration, CDN zones, administrator accounts, the lifecycle of nodes — and until now recorded nothing about who changed it. With more than one operator on shift that makes an incident unresolvable: "who deleted this stream" and "when was this node revoked, and by whom" had no answer at all. They do now: actor, action, time.

Its own package, and a cluster that stays one product. There is now an mcaster-streamer package. Before it, the only node package that existed was Catena's, and the Mcaster console showed a join command wired to Catena paths — so an operator following the instructions installed a node of the wrong product, and nothing along the way complained. A cluster of machines from two products is not exotic, it is a quiet accident: the nodes are licensed under a different key and a different tariff. Joining a node of a foreign product is now refused at registration, with a reason.

What you gain by moving from Flussonic Media Server to Mcaster

A headend is judged on what it can take in and what it can prove it delivered. That is exactly where 26.09 lands:

Here is what we did for Agora

Agora is corporate television: your own internal channels, studio feeds prepared for broadcast, town halls and webinars delivered to employees over the corporate network. In 26.09 it gets its own face on the shared platform — its own console, its own set of modules, its own product identity — while the domain features move onto the new core in the releases ahead.

Two pieces of core work land squarely in Agora's use case.

Low-latency delivery that scales. The WebRTC work described above is what corporate broadcasting actually needs: a town hall is a few thousand people watching the same thing at the same time, at sub-second latency, from inside one network. Going from 200–300 concurrent sessions per node to 18,000 changes what a company-wide broadcast costs in hardware — from a rack to a machine.

Publishing over SRT. A studio, a meeting room or a laptop with OBS can now publish into the cluster over SRT, received on the node it dials, with encryption and loss recovery on the way in. That is the ordinary way an internal event reaches the platform.

Here is what we did for Watcher

The NVR moves to the new core, and you do not need an engineer for it. In 26.08 the media layer got a name of its own — the watcher-streamer package — behind which the engine could be replaced. This release does the replacing, and does it through the only channel a recorder in the field actually has: the update button in the cloud.

Until now, swapping the media server on a recorder meant an engineer over SSH running a migration script that read the disk layout, rendered a configuration and rebuilt the container stack by hand. That never reaches a customer's box. The update path now carries the new media server the same way it carries everything else: press update, the recorder pulls the new stack, and a failure of either step rolls back to the previous one automatically. The per-box values that made this hard — cluster key, admin credentials, streamer name, disk layout — are now assembled on the box rather than baked into a file served to the whole fleet from a public URL.

What you get for it is density. The new core is what records twenty gigabits per second per server, and as the migration proceeds we expect a several-fold reduction in resources for the same installation: fewer machines for the same cameras, less rack space, a smaller power bill.

Camera capture got dramatically more reliable, and we found it by accident. While chasing recording losses on a ten-thousand-stream recorder, we noticed some streams producing fragments ten times smaller than others, went looking at the camera, and found the camera was fine. The problem was in how we assembled fragmented RTP packets into frames on the RTSP path.

The measurements are stark. On the same camera and the same bench, the July build produced 438,453 broken frames out of 4,600,555 — 9.55%. After the fix: 6 errors out of 272,591 frames, 0.002%. On the receiving recorder, running 10,000 streams for an hour, the median archive loss fell from 15,236 to 2 fragments per second, and the number of streams losing anything at all fell from 2,815 to 2.

If you run cameras over RTSP — which is to say, if you run cameras — this is the most valuable line in the release.

Recording stopped throwing data away while memory was free. A closed fragment lived in exactly one place, and the delivery window evicted fragments by count. If the disk writer fell behind by more than a handful, the data was gone forever. On a nine-thousand-camera recorder that was 898,501 lost fragments in ten minutes — while 42 GB of the machine's 125 were sitting unused. Delivery and durability now have separate budgets, the write queue holds its own references and is limited in bytes, and a fragment can only be lost when memory genuinely runs out. Along the way, aligning appends to 4K block boundaries cut losses from 5,524 to 8 on its own.

The recorder owns its cameras. Configuration used to sync in two asymmetric directions, and both hurt real users: a camera created on the recorder did not appear in the cloud until somebody found the "synchronize" button, and a background pull from the cloud every sixty seconds silently overwrote local edits. The architecture is now stated plainly — the recorder is the single source of truth for its own cameras, the cloud is a non-destructive mirror. While the link is up you may edit from the cloud; while it is down, you may not, rather than making edits that will be quietly discarded.

API v2 is switched off. We said in 26.08 that it would be, and it is: v2 endpoints now answer with a warning stub. Because we know some billing systems are still on it, there is a configuration flag to turn it back on for a transition period — but treat that as a deadline, not a reprieve.

Everything v2 offered is in v3 already. What we learned from the customers who did try to move is that the problem is not a missing endpoint but a missing sequence: creating a user, an organization and a folder is a chain of calls where each id comes from the previous answer, and a skipped step yields either a 403 or a subscriber with administrator rights. So we are writing that down as a "how to move to v3" guide for billing integrations. If you have a custom integration on v2, write to us and we will look at your calls with you.

The new player, and the rest. The player got a calendar, quality switching, timeline selection with export of the selected interval, mosaic support and a debug panel separated from the player itself; on recorders it is now the default. Alongside: the full list of a user's organizations is shown rather than a truncated one, the archive information dialog reports real numbers, the mosaic page had its pagination and delete button fixed, a disk-usage forecast that central cannot actually compute was removed rather than left to mislead, and a scheduler that silently stopped doing all periodic work after a Redis restart now notices and restores itself.

Villapass. The settlement access product built on Watcher grew up this release. Residents are now grouped into plots rather than existing as loose individuals, so a restriction — this plot may not issue guest passes — is set where it belongs. Residents sign in with a phone number and an SMS code instead of an email and a password nobody gave them. The gate opener reports back: which car passed, when, in which direction, and on which pass — which is what makes a passage journal possible, and there is now a screen showing it, searchable by plate and by pass. A chairman can switch between their own passes and the whole association's instead of always seeing everyone's. And the operator's admin panel finally shows how each settlement is doing — residents, plots, passes issued this week, and how long ago the gate service last exchanged data — instead of a bare list of names.

Flussonic Media Server

Nothing new, and that is deliberate. As we said in 26.08, the phase of active development of Flussonic Media Server is over: the product stays supported and we keep issuing minimal security updates, but all development goes into the products on the new core.

This release is a good moment to start planning the move, because the reasons to wait are mostly gone. If you do television or OTT, that is Catena — which as of 26.09 has DRM, a programme guide, a middleware of its own if you want one, timeshift channels, fMP4 capture over HLS and DASH, MPEG-TS output for old set-top boxes, and a tool that reads your existing archive in place. If you run a headend, that is Mcaster — SDI, ST 2110, MXL, RIST, full PSI/SI control and conditional access. If you do video surveillance, that is Watcher, whose recorders started moving to the new core in this very release.

A migration is more than a tool, so we take part in it ourselves: tell us about your installation and we will decide together what moves first and how the air is verified along the way.