Interoperate with MXL or shared memory

Shared memory is a very common way of efficiently sharing information between processes: rather than copying data or streaming it over a socket, the processes map the same region of memory and simply read what the others wrote. norsk-ctl supports this with --shared-memory, which mounts shared-memory volumes into Norsk instances.

A very common use case is MXL (Media eXchange Layer), an open-source SDK from the EBU and NABA that moves uncompressed media between processes — between two Norsk instances on the same host, or between Norsk and a third-party application that also supports MXL.

On Linux servers, a common way of creating a shared memory area is a tmpfs (temporary file system — files held in memory). It looks like a normal directory of files, but the contents live in RAM rather than on disk, so access is extremely fast. MXL refers to such a directory as a domain: every participant opens the same directory by path and exchanges media through the flows inside it. The MXL project’s own documentation covers how that works; from norsk-ctl’s point of view, a domain is simply a shared mount.

Setting it up takes three steps, each detailed below:

1. Create the tmpfs on the host — a one-time root step.
2. Launch Norsk with the volume (--shared-memory) and set the domain path in the workflow’s MXL sender/receiver config.
3. Do the same with the other application — another Norsk instance, or a third-party MXL application.

The arrangement is deliberately neutral: the host owns the tmpfs, and each application mounts it its own way, so neither’s restart cycle can break the other. Only the domain directory is shared — every Norsk instance keeps its private /dev/shm for its own working memory (see Shared Memory).

Before you start

Agree three things with the operator of the other system (its documentation should state its side of each; for Norsk-to-Norsk you are both sides):

1. The domain path. This guide uses /mnt/mxl throughout. Each side can technically mount it at a different container path, but using one path everywhere keeps every config and log line comparable — and it is the convention norsk-ctl follows: a mount appears in the container at its host path.
2. The size. The tmpfs must hold the sum of what all participants write — every flow’s ring buffer — plus headroom. A flow’s ring is roughly frame size × frame rate × seconds of history it keeps. An uncompressed v210 frame is ~5.5 MB at 1080p and ~21 MB at UHD, so with one second of history a 1080p50 flow needs ~280 MB while a UHD 130fps flow needs ~2.9 GB — high-end flows are gigabytes each. Audio is negligible by comparison. This guide uses 4g; size yours from the flows you actually plan to carry, and check the ring depth each producer is configured with.
3. File ownership. Note which user each side’s containers run as. The mount below is world-writable with the sticky bit (like /tmp), which works across different UIDs — and is what norsk-ctl verifies at launch. Some systems additionally require a specific UID or group; if so, run the Norsk instance to match with --container-user.

1. Create the shared domain on the host

Creating a shared memory area is two commands:

sudo mkdir -p /mnt/mxl
sudo mount -t tmpfs -o size=4g,mode=1777 tmpfs /mnt/mxl

That’s it — /mnt/mxl is now a 4GB in-memory directory any process on the host can use. Create a dedicated one like this rather than reusing the host’s /dev/shm, which every process on the machine can already see.

Being RAM, it disappears on reboot, so once you’re happy it works, make it permanent. The one-liner is a line in /etc/fstab:

tmpfs /mnt/mxl tmpfs size=4g,mode=1777 0 0

or, tidier on a systemd host, a mount unit at /etc/systemd/system/mnt-mxl.mount:

[Unit]
Description=Shared MXL domain (tmpfs)

[Mount]
What=tmpfs
Where=/mnt/mxl
Type=tmpfs
Options=size=4g,mode=1777

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now mnt-mxl.mount

Either way, the mount is back on reboot before anyone’s containers start, so the host — not either application — is what holds the domain open. This is the one root step in the whole arrangement, once per machine.

Note

mode=1777 is world-writable with the sticky bit: any UID can create flows, only the creator can delete its own files. On an SELinux-enforcing host you will also need the mount labelled so both sides’ containers can use it — sudo chcon -t container_file_t /mnt/mxl (don’t use a :z bind-mount flag here; per-container relabelling fights over a directory two runtimes share).

2. Mount it into the Norsk instance

Name the mount at launch with --shared-memory:

norsk-ctl instance launch-template mixer \
  --template my-product \
  --shared-memory /mnt/mxl

norsk-ctl bind-mounts the path into the media container at the same path, and refuses the launch loudly unless the path exists, is a tmpfs, and is world-writable — each of those failures would otherwise degrade silently (a missing path becomes a root-owned plain directory; a plain directory becomes disk-backed “shared memory” that works, slowly, until it doesn’t). The mount is persisted with the launch config, so instance relaunch keeps it, and it appears on the instance’s detail view. The instance’s private /dev/shm is unaffected — --shm-size still sizes it as its own thing.

The option is repeatable, and that is a feature, not a convenience: an instance bridging two domains mounts both. A ↔ B ↔ C should not imply A ↔ C — give the bridge instance --shared-memory /mnt/mxl-a --shared-memory /mnt/mxl-c, and A and C never see each other’s media or share a budget.

A product template built around a domain can declare the conventional path as a default (advanced.sharedMemory.default), pre-filling the launch form; you still create the mount, and can override or clear the path at launch.

3. Point the workflow at the domain

In Norsk Studio, MXL is spoken by the MXL ingest and MXL egest components from the @norskvideo/norsk-studio-alpha library — make sure the product template you launch includes it in its library list. To read the other system’s flows, add MXL ingest to the workflow and set:

• Domain — the shared path (in this example, /mnt/mxl).
• Match group hint — which flows in the domain to ingest, selected by the NMOS group-hint tag the producing side stamps on them. Agree these names with the other system’s operator the same way you agreed the path. (Explicit flow IDs and format matching are the alternative selectors — see the MXL documentation for details.)

Publishing toward the other system is the mirror image: MXL egest with the same domain, stamping each flow with a group hint the other side will match.

Note

A product built directly on the Norsk SDK rather than Studio configures the same two things wherever its workflow sets up MXL — for example norsk.input.mxl({ domain: "/mnt/mxl", matchGroupHint: "camera-1", ... }).

4. Mount it into the other system

Another Norsk instance under the same norsk-ctl uses the same --shared-memory /mnt/mxl at its launch. Anything else mounts the same host path by whatever mechanism its own orchestration provides — for plain Docker that is -v /mnt/mxl:/mnt/mxl, and any MXL-capable system will document its equivalent. What it must end up with: the same directory visible read-write in its containers, and flows created/opened at that path.

5. Verify

Once the producing side is publishing, the domain is inspectable from the host — flows are just files:

ls -la /mnt/mxl        # the domain's flow structure appears as the producer creates it
df -h /mnt/mxl         # how much of the budget is in use

Then confirm media flows end-to-end: subscribe the Norsk workflow to a flow the other system publishes (or vice versa) and watch the instance’s normal outputs. If nothing arrives, check the three agreements in order — path visible in both containers (docker exec <container> ls /mnt/mxl on each side), sizes not exhausted (df), and file ownership (a flow owned by a UID the reader can’t open).

Operating it

• Watch the budget. A full tmpfs fails every participant’s writes, not just the offender’s. df -h /mnt/mxl is the number to alert on, and sizing generously is cheap — tmpfs pages are only backed by RAM once written.
• Restarts are free. Either side’s containers can stop, upgrade, and return; the domain persists in the host mount. A host reboot clears it (it is RAM), after which the systemd unit remounts it and producers simply recreate their flows.
• Stale flows. If a producer dies without cleaning up, its flow files linger until something removes them. MXL consumers ignore flows nobody is writing, so this is hygiene rather than breakage — but agree whose job cleanup is, since sticky-bit permissions mean only the creating UID (or root) can delete them.
• Trust. Everything in the trust model applies: every participant can read, corrupt, or delete every flow in the domain, and they share one budget. Share a domain only between systems that are allowed to see each other’s media, and keep unrelated tenants in separate domains (separate mounts, not just separate subdirectories, if they must not affect each other’s capacity).