Troubleshooting

Symptoms first, diagnosis second. Most issues fall into one of these categories — start with the matching section.

Daemon won’t start

Symptom: norsk-ctl <anything> errors with a connection refused or “daemon not reachable” message.

Check, in order:

1. Is the config present? ~/.norsk-ctl/config.yaml is read at boot. Missing or unparseable config means the daemon exits immediately.

   ls -l ~/.norsk-ctl/config.yaml
   norsk-ctl config show

   If missing, run norsk-ctl init.
2. Is the port free? Default is 8333 (override via NORSK_CTL_PORT).

   ss -tlnp | grep 8333

3. systemd service running? On --server installs:

   sudo systemctl status norsk-ctl
   sudo journalctl -u norsk-ctl -n 200 --no-pager

4. Docker reachable? The daemon shells out to Docker for every instance op. docker ps should return cleanly as the same user the daemon runs as. On Linux, the daemon user must be in the docker group — norsk-ctl restart refuses to bounce when the current shell isn’t in the group yet (you’d just re-create the problem).

product add rejects the license

Symptom: norsk-ctl product add … exits non-zero (or the Add Product form shows a 400) complaining about the license file. Nothing is registered.

1. “not a V2 license envelope”? The file is readable but isn’t a signed V2 envelope. Two quite different causes share this message, so check the path before assuming the license is at fault:
   • The path points at the wrong thing — an HTML error page from a failed download, a truncated file, the wrong .json. head -c 200 <file> settles it in one look.
   • It really is an older V1 license. Those are no longer accepted when registering a product (V2 is required) and must be reissued — contact Norsk support.
2. “license has no entry for product …”? The file is a valid V2 envelope but doesn’t cover this product — the message lists what it does cover. If it names the product but you still get an error, the entry covers a different image or version than the one you passed to --image — and norsk-ctl product add --license-file <file> with no --image registers exactly what the license entitles, so dropping --image is usually the fix.
3. “license expired at …”? A V2 envelope carries an expiry for the license and one per product entry; either lapsing is rejected with the date. Reissue it.
4. Registration succeeded but warned it “was not validated”? The daemon could not read the file (usually permissions — it runs unprivileged, while Docker mounts the license as root). Registration is allowed to proceed, but nothing was checked: if it turns out to be a V1 license, the engine rejects it at launch instead. Make the file readable by the daemon user and re-add to get a real answer.

Launch hangs or instance never reports healthy

Symptom: instance launch-template returns, but instance describe shows status: starting for minutes.

1. Image pull in progress? First launch on a fresh box pulls the studio (~1.5 GB) and media (~2 GB) images. Watch with docker pull running separately or journalctl -u norsk-ctl -f.
2. License rejected? Studio refuses to start without a valid license. Check the studio container logs (compose names containers <id>-<service>-1):

   docker logs <id>-studio-1 --tail 100

3. Port collision. Ingest ports (SRT/RTMP/etc.) are published directly to the host. Two instances binding the same host port will fail — second one stays stuck.
4. Workflow YAML invalid. A malformed workflow drops Studio into an error state. Visit /instance/<id>/studio/ in the browser; the canvas surfaces the parse error.

Launch refused with a HARDWARE_… code

Symptom: instance launch-template fails immediately (or the launch form blocks) with HARDWARE_NOT_FOUND, HARDWARE_UNAVAILABLE, HARDWARE_PINNED, or HARDWARE_REQUIRED. Nothing was started.

The runner checked the requested accelerator against the host before touching Docker, and the code names why it stopped. Start with norsk-ctl get hardware: accelerators lists what the host actually carries.

1. HARDWARE_NOT_FOUND — quadra was requested but no NETINT card was found. The probe looks for an NVMe controller under /sys/class/nvme with PCI vendor 0x1d82; if cat /sys/class/nvme/*/device/vendor shows nothing of the kind, the card’s driver is not loaded or the card is not seated.
2. HARDWARE_UNAVAILABLE — the product requires an accelerator this host does not have. Launch it elsewhere; if the product only prefers it (a default declaration), pass --hardware none to run unaccelerated.
3. HARDWARE_PINNED — you asked for an accelerator the product does not allow. Drop --hardware; the product’s manifest decides.
4. HARDWARE_REQUIRED — the product leaves the choice to you and none was made. Pass --hardware nvidia, --hardware quadra, or --hardware none.

A launch that passes these checks but fails in compose with could not select device driver "nvidia" is on a CDI-only host the runner classified as legacy; set NORSK_CTL_GPU_MODE=cdi in the daemon’s environment and relaunch. Hardware Acceleration has the full picture.

Proxy returns 502

Symptom: Public URL serves a 502 / “upstream unavailable” from nginx.

• Instance not running? norsk-ctl instance list — does the target instance still exist? If you deleted it, the URL is dead.
• Stale nginx config? norsk-ctl proxy reload regenerates nginx config from the daemon’s current view. Launch/teardown paths trigger it automatically, and the daemon also reloads on Docker start events — so a container restarted behind norsk-ctl’s back (docker restart) self-heals within a second or two. A persistent 502 after that points elsewhere; norsk-ctl proxy status shows the active upstreams.
• Daemon down behind the proxy? Requests to /api/ and / proxy to host.docker.internal:8333. If the daemon stopped, the proxy survives but every backend request 502s.

Instance shows stopped but containers are still running (or vice versa)

norsk-ctl tracks instance state in ~/.norsk-ctl/norsk-ctl.db. Drift happens if you docker rm containers behind its back, or if the daemon was killed mid-operation.

# What does docker think is running? Each instance is a compose
# project named by its instance id.
docker compose ls

# What does norsk-ctl think?
norsk-ctl instance list

If they diverge: norsk-ctl instance restart <id> reconciles by tearing down and re-launching. norsk-ctl instance delete <id> followed by a fresh launch is the heavy-handed reset.

Plugins aren’t discovered

Symptom: the Studio product’s plugin listing (GET /products/norsk-studio/api/plugins?workingDirectory=…) doesn’t show a plugin you’ve dropped under <workingDirectory>/plugins/.

• Missing package.json? Discovery requires a readable package.json with a non-empty name field. Directories without one are skipped with a warning.
• Wrong working directory. Every plugin endpoint takes an explicit workingDirectory — there’s no daemon-side default. Check the path you passed matches where the plugin actually lives.
• Plugin not in the instance? Instances only load plugins baked into the image (pinned by the product template) or listed in the product template’s server.library. Dropping files under plugins/ alone doesn’t affect a launch. See Manage plugins.

Studio loads a blank canvas

• Workflow YAML missing? The workflow save-file ships with the product template. If the product template references a save file that isn’t present in the working directory, Studio silently lands on an empty canvas. Check instance describe and the product template’s bundled save files.
• Library not loaded. Workflows reference nodes from named libraries. If the workflow uses a node from a package the product template’s library list doesn’t include, the canvas loads but the node is broken. See Studio libraries.

TLS browser warning persists after mkcert -install

macOS browsers cache trust decisions per-session. Fully quit and restart the browser (cmd-Q, not just close window). Firefox uses its own cert store — brew install nss before mkcert -install so Firefox is reached too.

Public IP couldn’t be auto-detected at install

Symptom: install.sh --ip auto aborts with couldn't auto-detect a public IP.

--ip auto probes https://api.ipify.org then https://ifconfig.me (each 5-second timeout). If neither responds, pass --ip <host> or --public-host <name> explicitly. On a private network without outbound HTTPS, this is expected.

How to gather diagnostics

When in doubt, capture:

• norsk-ctl status (or status via MCP)
• norsk-ctl config show
• norsk-ctl instance list and instance describe <id> for the affected instance
• journalctl -u norsk-ctl -n 500 --no-pager
• docker compose ls and docker logs <id>-<service>-1 for any wedged containers

The web UI’s Logs view surfaces studio + media + proxy logs without the docker CLI.