app-lb

app-lb is the HWS load balancer, autoscaler and control plane: it routes HTTP traffic to pools of heyvm microVMs, fixed upstreams or static files, and manages those pools through an admin API and dashboard.

What it is

app-lb is one Rust process built on Pingora. It does three jobs:

• Proxy. It terminates HTTP and HTTPS and routes each request to a deployment by host and path.
• Autoscaler. It boots, health-checks, drains and reaps Firecracker or KVM microVMs through the local heyvm daemon (heyvmd), sized to the traffic each deployment is receiving.
• Control plane. It runs an admin HTTP API, a dashboard, a secret store, deploy jobs (image builds, artifact pulls, host updates), security monitoring with block rules, and disk management for the host.

You register a deployment over the admin API. A deployment is routing rules plus exactly one backend:

How it fits with the rest of HWS:

• heyctl is the CLI and client library for the admin API.
• app-lb-auth covers the admin gate, app-tokens, Google and JWT sign-in, federated auth and namespaces.
• artifacts is the store app-lb pulls rootfs images, site bundles, guest mounts and workspace snapshots from.
• app-obs receives app-lb's access log, events, job output and security alerts, and polls /metrics.
• orchestrator can own a static deployment's upstream membership through discovery. multi-region covers regional gateways.
• heyosecret is where canonical service credentials live. You then place them in app-lb's own secret store for deployments to reference.

Process model

The foreground process owns all management state: the state lock, the admin API, discovery, autoscaling, ACME, authentication and request admission. It supervises one forwarding subprocess that owns the HTTP/TLS listeners and streams request and response bodies. The two talk over a Unix socket in a private directory. If the manager dies, the worker dies with it, and new requests depend on the manager being up. Restarting app-lb interrupts in-flight streams. It is not a hot takeover.

Requirements

• Linux with KVM (/dev/kvm).
• A running heyvmd. app-lb routes directly to each sandbox's guest_ip, which the daemon only reports for tap-networked Firecracker and KVM sandboxes on the local host. One app-lb manages one daemon.
• To build from source: cmake (a Pingora dependency) and libssl-dev (the TLS listener uses OpenSSL for per-handshake certificate selection).
• Optional tools on the host, depending on which features you use: heyvm and Docker (image builds), git (git builds and site updates), art (local artifact stores and local workspace stores), aws (DNS-01 wildcard certificates, disk archives, S3 workspace stores), tar.
• Optional: Incus, for the lxc driver (system containers from OCI images).

Install and run

cargo build --release --locked --manifest-path app-lb/Cargo.toml
./app-lb/target/release/app-lb

See installation for release bundles. To run app-lb as a supervised service, use the supervisord program in app-lb/deploy/supervisor/app-lb.conf. It runs as an app-lb user from /var/lib/app-lb, pins APP_LB_STATE_PATH to an absolute path, sends SIGTERM for a graceful drain, and restarts on crashes. Change its placeholder dashboard password before you use it.

app-lb takes no arguments. You configure it with APP_LB_* environment variables.

--forwarding-worker, --apply-host-update and --bootstrap-host-update are internal entry points. You don't run them yourself.

One instance per daemon. At startup app-lb takes a host-wide lock at /run/app-lb/instance.lock. If /run is not writable, it uses the temp directory. A second instance refuses to start and names the process holding the lock. This matters because an app-lb that doesn't know about another instance's sandboxes treats them as orphans. Point APP_LB_INSTANCE_LOCK elsewhere only for instances that talk to different daemons.

Quick check once it is running:

curl localhost:9090/healthz                      # "ok", with an x-heyo-revision header
curl -XPOST localhost:9090/deployments -H 'content-type: application/json' -d '{
  "id": "demo",
  "routes": [{"host": "demo.local"}],
  "vm": {"driver": "firecracker", "image": "nginx", "port": 80},
  "scaling": {"min_replicas": 0, "max_replicas": 4}
}'
curl -H 'Host: demo.local' localhost:6188/

Every proxied response has an x-vm-id header naming the VM (or, for a static deployment, the upstream address) that served it.

Configuration

All configuration comes from the environment. Boolean flags accept 1/true/yes/on.

Listeners, state and identity

heyvm daemon

Admin gate and federated auth

These are summarized here. The details are in app-lb-auth.

TLS and certificates

HTTP-01 validation needs the proxy on port 80. To bind 80/443 as a non-root user, run setcap 'cap_net_bind_service=+ep' /usr/local/bin/app-lb. Let's Encrypt allows 50 new certificates per registered domain per week, so for a fleet of per-sandbox hostnames, use one wildcard plus one wildcard A record. A wildcard covers exactly one label. Failed issuance backs off per host (1 minute doubling to 6 hours), and the backoff persists across restarts.

Builds, images, mounts and site updates

Disks and workspaces

Images

Discovery, fleet views and host updates

Log shipping and security monitoring

LXC driver (Incus)

Where state lives

Everything relative to APP_LB_STATE_PATH sits beside it. With the default name:

A legacy single-file app-lb-state.json is imported on first start and renamed to .migrated. A deployment file whose spec no longer validates is skipped with a warning and left in place. Only one app-lb process may own a state directory.

Deployment spec

The machine-readable schema is app-lb/schema/deployment-spec.json. Ready-to-post examples are in app-lb/examples/.

Top-level fields

Routes

A request reaches a deployment when any of its rules matches. Within one rule, every field that is set must match.

Across all deployments, the single most specific rule wins: an exact host beats any host_suffix, which beats any path-only rule. Within a tier, the longer suffix or prefix wins, and a host plus path outranks the same host alone. Exact ties go to the lexicographically lower deployment id. An empty rule {} is rejected, and a request that matches nothing gets a 404.

"routes": [] is valid only for a managed deployment. It is still autoscaled and reachable through exec and shell, but takes no HTTP traffic. This is the normal shape for an agent sandbox. Static deployments and sites must have at least one route, and so must any deployment with auth.

Hostless deployments

When a base domain is configured (APP_LB_DEPLOY_BASE_DOMAIN, or the first APP_LB_ACME_WILDCARD), a deployment with no host or host_suffix route gets a route to <id>.<base> at registration. Routeless VM deployments are left private.

vm

Changing anything in vm is a template change and recycles the pool. build, artifact, ingress, routes, scaling and health edits do not.

lxc deployments refuse correlated_creates, workspace_archive, image download hints, setup_hooks, open_ports, build, artifact, ingress and (for now) mounts. Their ids must be lowercase letters, digits and dashes, short enough to fit Incus's 63-character instance names.

scaling

Desired replicas is ceil(demand / target_concurrency) + warm_pool, clamped to [min_replicas, max_replicas], where demand counts in-flight requests plus requests waiting on a cold start. A request that arrives at an empty pool is held while a VM boots.

retain preserves only the /workspace data disk. Writes to the rootfs and memory are lost in both modes, because the rootfs is recopied from the image on every cold boot. So retain is only meaningfully stateful with disk_size_gb set and state kept under /workspace. For interchangeable replicas, use destroy.

Consecutive failed boots back off the next create exponentially, from 30 seconds up to one hour, and reset on the first healthy boot.

health

A VM joins the pool only after the daemon reports it running, it has a guest IP, and it passes this probe. Static upstreams are re-probed every autoscaler tick, so a recovered upstream rejoins routing.

build

Builds a new rootfs with heyvm mvm build. On success, app-lb rewrites vm.image to the result and the pool rolls. Set exactly one of repo or store.

The guest's PID 1. heyvm boots every Firecracker guest with init=/init.sh. A Dockerfile written for Docker has none, so its kernel would panic on every boot. If the Dockerfile never mentions /init.sh, app-lb adds its standard guest init (app-lb/src/guest_init.sh). It writes the script into the build context as heyo-guest-init.sh and builds from a copy of the Dockerfile ending in COPY heyo-guest-init.sh /init.sh.

The init mounts /proc, /sys and /dev, finishes network setup and starts sshd if the image has one. It then prints HEYVM_READY and never exits. It does not start the app: vm.start_command does that after boot, and must return (setsid nohup … &).

The image's CMD, ENTRYPOINT and ENV do not survive the rootfs export, so put what they did into start_command. A Dockerfile that mentions /init.sh is built as written.

Install socat in custom images. heyvm forwards the start_command's stdout and stderr to the host over vsock using socat inside the guest, and only when the image has it. Without it, the app's logs stay in /var/log/heyvm-start.log in the guest: they never reach app-obs, heyctl logs, the obs plugin or MCP's deployment_logs, and they are gone when the VM is. heyvm logs guest lacks socat/vsock, workload logs stay in guest files at boot when this happens. The socat must be built with VSOCK support (socat 1.7.4 or newer; the Alpine, Debian and Ubuntu packages are). Add it with apk add socat or apt-get install -y socat. The hub's base images (heyo/alpine, heyo/debian, heyo/ubuntu) include it.

artifact

Pulls bytes that already exist in an artifacts store. Nothing is built.

A deployment with an artifact block boots nothing until its image is on heyvm. Registering one, or scaling one whose image was offloaded, starts the pull (or the thaw) automatically; the first VM is created once it lands.

site

Sites support ETag/304, byte ranges, HEAD, and a 301 from /dir to /dir/. They don't do rewrites, redirects, directory listings or compression. A site gets its files from either update or artifact, not both.

update

For static deployments and sites: commands run on the app-lb host.

Mounts and workspaces

A mount (vm.mounts[]) is data the deployment is given. A workspace (vm.workspace) is data the deployment makes.

Mount trees are fetched by a job (POST /deployments/:id/mounts/pull, started automatically on register or edit), not at VM create. Until the tree exists, the autoscaler refuses to create replicas.

A workspace is seeded into each new replica. When a replica retires, app-lb captures its workspace (it stops the VM and extracts the image), then pushes the snapshot to the store. The replacement boots only after the capture completes. Workspaces require driver: firecracker, max_replicas <= 1 and warm_pool: 0. Files come back owned by app-lb's uid.

POST /deployments/:id/workspace/recoveries explicitly promotes a stopped, retained VM's disk to be the new workspace snapshot. It needs namespace-admin credentials even when CRUD is ungated.

Secret references

build.auth, artifact.auth, mount and workspace auth, update.auth, discovery.source.auth and the auth gate's secrets are all {"secret": "<id>", "key": "token", "username": null}. env_from entries add as. Every reference is bound to the deployment's namespace at registration, whatever the body says. See Secrets.

Auth gate and public_paths

auth puts Google sign-in, app-tokens, a JWT, or a named auth provider in front of everything the deployment serves. It runs in the proxy, so the application needs no changes. auth.public_paths lists path prefixes the sign-in gate doesn't redirect. Each entry carries the scope app-lb requires in the gate's place:

Everything else about gates, tokens, sessions and providers is in app-lb-auth.

Static and managed deployments

Static upstreams. Bare host:port is proxied over plain HTTP, and hostnames are re-resolved per connection. An https://host[:port] origin uses the hostname for SNI and certificate verification and preserves the caller's Host. HTTPS needs a DNS name, not an IP, with no path, credentials or query. To change targets, PUT the spec with a new list.

Cordon and drain. Operator drains are separate from health. A drained upstream stays out of rotation until you remove the drain, even if it passes every probe. The drain survives restarts and spec replays.

heyctl cordon stage us1.internal:8080 --reason 'maintenance'   # stop new traffic, return now
heyctl drain stage us1.internal:8080 --timeout 300             # ...and wait for in_flight = 0
heyctl uncordon stage us1.internal:8080

The raw API is PUT/DELETE /deployments/:id/upstreams/<url-encoded upstream>/drain. A first drain returns 409 unless another upstream is healthy and accepting. {"force": true} overrides this for an intentional outage. A drain timeout never reopens the upstream.

Editing. PUT /deployments/:id replaces the spec in place and keeps the pool unless vm changed. POST /deployments replaces the deployment and tears the pool down. PATCH /deployments/:id/scaling is a partial update that always keeps the pool. DELETE /deployments/:id/vms/:sandbox_id recycles one VM, either gracefully (202, drain) or with ?force=true (200, kill now).

Candidate-first rollouts. For stateless Firecracker services, use POST /deployments/:id/rollouts with {operation_id, expected_revision, spec} (expected_revision is rollout_revision from GET). This avoids tearing the pool down. The target spec must pull an artifact by 64-hex digest and set health.expected_header to x-heyo-revision: <git sha>. Candidates boot unrouted, cut over only when healthy, and the old replicas drain and are stopped rather than destroyed. Poll GET /deployments/:id/rollouts/:operation until status is succeeded. The other terminal states are failed and reconciliation_required. Routes, auth, namespace and maintenance must not change in the same rollout.

Admin API

The admin listener (APP_LB_ADMIN_ADDR) serves the API, the dashboards and /healthz. Routes fall into tiers:

• Open: /healthz, /login, /logout, UI assets.
• View tier: gated by the dashboard password (or a view token) unless APP_LB_DASHBOARD_AUTH=0.
• CRUD tier: gated only when APP_LB_ADMIN_AUTH=1. An ungated CRUD API can run code on the host (builds and updates), so gate it before you expose the listener.
• Always authenticated: fleet, control-plane, recovery and regional routes need credentials even when the gates are off.

A deployment- or namespace-scoped token sees a narrowed view of list and metrics routes. See app-lb-auth.

View tier

CRUD tier

Always-authenticated routes

Capability headers

GET /deployments advertises what this binary supports, so controllers can check before relying on a feature: x-app-lb-create-only, x-app-lb-discovery-source, x-app-lb-discovery-region, x-app-lb-gateway, x-app-lb-regional-admission (each 1). Unknown spec fields are ignored by older binaries, which is why you should check these headers first.

The spec ETag is "<hex>", where the hex is the lowercase SHA-256 of the compact JSON of the response's spec with every object's keys sorted recursively and array order preserved.

Running commands inside a VM

exec and shell are the only way into an unrouted sandbox:

curl -XPOST localhost:9090/deployments/sb-7f3a9c/exec -H 'content-type: application/json' \
  -d '{"command": "ls -la /workspace", "cwd": "/workspace"}'
heyctl shell sb-7f3a9c

Without sandbox_id, the pool picks a VM. With it, you get that VM, a 404 (not in the deployment), or a 409 (not ready or draining). wake defaults to true, so a scaled-to-zero sandbox boots or resumes, bounded by cold_start_timeout_secs. An open command or shell counts as in-flight work, so the VM is not reaped under you. A command that fails returns 200 with a non-zero exit_code. app-lb failing to run the command returns 502.

Dashboards

All pages are served by the admin listener, self-contained, and work over an SSH tunnel.

With fleet views configured, /dashboard defaults to the fleet overview and links to each gateway's ?view=local.

Security monitoring

Security monitoring is on by default (APP_LB_SIEM=0 turns it off). app-lb analyzes every request off the request path and raises alerts with ECS field names:

Repeats with the same rule, source and deployment fold into one alert whose count climbs. Sources are keyed per /32 for IPv4 and per /64 for IPv6, on the socket peer only (X-Forwarded-For is never trusted). Query strings are scanned but never stored: an alert names the parameter, not the value. Credentials are never recorded. Alerts are held in memory and shipped to app-obs as source=security.

The dropped and clients_at_capacity counters on /security and /metrics tell you when detection is sampling. Raise the queue or client capacity if either is non-zero.

Block rules are enforced on the data plane and persisted to APP_LB_GUARD_PATH:

curl -XPOST localhost:9090/security/rules -H 'content-type: application/json' \
  -d '{"action":"block","match":{"client":"203.0.113.9"},"expires_in_secs":3600}'
curl -XPOST localhost:9090/security/rules -H 'content-type: application/json' \
  -d '{"action":"allow","match":{"client":"10.0.0.0/8"},"note":"monitoring"}'
curl -XPATCH localhost:9090/security/rules/5f295e1a86f2 -H 'content-type: application/json' \
  -d '{"expires_in_secs":null}'     # keep indefinitely; the field is required on PATCH

All present fields must match. allow beats block. An empty match is refused. Posting the same conditions again replaces the rule. There are no regular expressions. Rules never apply to the admin API, and ACME challenges are answered before the guard runs. Use APP_LB_GUARD_ENFORCE=0 to try a broad rule in dry-run mode first.

Secrets

app-lb keeps its own write-only secret store. Deployments refer to secrets by name, and no endpoint ever returns a value.

curl -XPOST localhost:9090/secrets -H 'content-type: application/json' \
  -d '{"id": "db", "namespace": "team-a", "data": {"url": "postgres://..."}}'
curl 'localhost:9090/secrets?namespace=team-a'                   # ids and key names only
curl -XPATCH 'localhost:9090/secrets/db?namespace=team-a' -H 'content-type: application/json' \
  -d '{"data": {"url": "postgres://rotated..."}}'               # null removes a key
curl -XDELETE 'localhost:9090/secrets/db?namespace=team-a'       # 409 while referenced

• Secrets are separated by namespace, and references in a spec resolve only within the deployment's namespace.
• env_from is resolved each time a replica is created. Rotation reaches the next replica without re-registering, and a missing secret fails the create.
• At rest, the store is APP_LB_SECRETS_PATH (mode 0600), or AES-256-GCM sealed with APP_LB_SECRET_KEY. Starting without the key that sealed the file is a hard failure, not an empty store.
• For HWS installations, the canonical value of a service credential lives in heyosecret. Load it into app-lb's store under one stable id per role, and never put values in specs, fleet files or config documents.

Metrics

GET /metrics returns JSON (it is not Prometheus text):

Filters narrow deployments only. host, fleet and global always describe the whole visible LB. app-obs scrapes this endpoint. heyctl top summarizes it.

Log shipping

With APP_LB_OBS_URL set, app-lb pushes four record sources to app-obs, tagged source:

• access: one record per request, attributed to the deployment. The query string is never included.
• app-lb: its own events at INFO and above (scaling, slow boots, unhealthy upstreams, ACME, job outcomes).
• job: every line of build and update output.
• security: alerts.

Records that name no deployment go under APP_LB_OBS_DEPLOYMENT. Queues drop records rather than block, so a slow or absent collector costs you telemetry, never traffic.

Disk management

heyvmd leaves per-sandbox disk directories behind in several cases. app-lb inventories them through the daemon (GET /disks, /storage) and reclaims them on a timer. A disk is never reclaimed while:

• its sandbox is running (not even with ?force=1),
• a deployment expects to resume it (idle_action: retain),
• it is pinned (PATCH /disks/:id {"retain": true, "note": "..."}), or
• the daemon is unreachable (the inventory reports complete: false and the sweep does not run).

Unclaimed disks expire after APP_LB_DISK_TTL_SECS (7 days). Disks the daemon has no record of expire after APP_LB_DISK_ORPHAN_TTL_SECS (15 minutes). POST /disks/purge-orphans reclaims all orphans immediately. POST /disks/:id/archive {"purge": true} streams a sparse tar.gz to s3://$APP_LB_DISK_ARCHIVE_BUCKET/<prefix>/<id>/<ts>.tar.gz, then reclaims the disk only if the upload succeeded. Startup logs how much the first sweep will delete. It runs one interval later, so you have time to pin disks or set the TTL to 0.

heyvm prune is complementary. It clears /tmp scratch and, with --images, can delete an image a scaled-to-zero deployment still needs. Re-run pull or build if that happens.

Image management

GET /images (view tier, fleet scope) lists heyvm's image catalog with what holds each image: a deployment's vm.image, a kept rollout operation's spec, a live or inactive sandbox, a running job, or a pin. Records live in app-lb-images.d/. The Images section of /storage and heyctl images show the same.

An image nothing has held for APP_LB_IMAGE_IDLE_SECS is offloaded by a pacer that runs one at a time and stands down while any pool is booting:

• Pulled images: the store they came from is checked to still serve the blob (HEAD /blobs/<digest>, or art stat), the record is marked offloaded, then the image is deleted from heyvm.
• Built images: pushed to APP_LB_IMAGE_OFFLOAD_STORE first; never offloaded without it.
• Images app-lb did not make are listed and never removed.

A failed offload backs off (30 minutes, doubling, up to a day). Above APP_LB_IMAGE_PRESSURE_PCT disk use the idle age is ignored, but references, pins and verification still hold. Nothing is removed when the references cannot be determined. A deployment that needs an offloaded image gets it pulled back under its own name before its next VM is created.

Deleting images needs heyvm's DELETE /images/:name. Against an older heyvm the inventory works and offload reports delete_supported: false.

Discovery and regions

A static deployment can take its upstream list from Orchestrator:

{"id": "cloud", "routes": [{"host": "cloud.example.com"}], "upstreams": [],
 "discovery": {"service_id": "cloud", "region": "eu1",
   "source": {"url": "https://orchestrator.example.com/orchestration/services/cloud/discovery",
              "auth": {"secret": "discovery-reader", "key": "token"}}}}

Each poll atomically replaces the upstream list with healthy, non-draining endpoints and persists the last good set. Failed or stale snapshots leave the list intact. A per-deployment source takes precedence over APP_LB_DISCOVERY_URL/APP_LB_DISCOVERY_TOKEN, and its token is resolved from the secret store on every poll. region requests region-scoped membership and rejects snapshots that don't echo it. Once a deployment has observed an authority, changing it is rejected. Withdrawn upstreams are fenced from new requests and stay counted in discovery-status until their in-flight requests finish.

The gateway block and discovery.regional add authenticated one-hop forwarding between regional app-lbs. That protocol, and how both regions share one discovery authority, is covered in multi-region.

Fleet views are read-only observations of other app-lbs. They are not placement or routing authority. Configure them with PUT /control-plane/config ({"expected_revision": N, "config": {"gateways": [...], "control_plane": [...]}}) or with APP_LB_FLEET_FILE/APP_LB_CONTROL_PLANE_FILE, which make the API read-only. Each gateway entry is:

{"id": "eu1-edge", "region": "eu1", "url": "https://admin.eu1.example.com",
 "auth": {"secret": "fleet-observer", "key": "token"}}

Use "use_caller_auth": true in place of auth to forward the signed-in Heyo user's bearer instead. Only HTTPS origins are accepted. An observer token must be view tier and cover all deployments (heyctl token mint fleet-observer --admin view --all-deployments). GET /fleet/deployments rolls up namespace-by-deployment rows across servers. A server that can't be read appears as an error cell, never as zero capacity.

Host executable rollouts

Replacing the app-lb binary on a host is a separate operation from VM rollouts. It is off unless APP_LB_HOST_UPDATE_CONFIG names an operator-owned, root-owned JSON mapping:

{
  "deployment": "app-lb-host-controller",
  "namespace": "default",
  "executable": "/usr/local/bin/app-lb",
  "process": {"kind": "supervisor", "program": "app-lb"},
  "state_dir": "/var/lib/app-lb/host-updates",
  "artifact_store": "https://artifacts.example.com",
  "health_url": "https://admin.example.com/healthz",
  "config_files": ["/etc/supervisor/conf.d/app-lb.conf"]
}

process may instead be {"kind": "systemd", "unit": "..."}. GET /deployments/:id/update/rollouts reports the current binary_sha256 and config_sha256. POST the same path with {operation_id, expected_binary_sha256, expected_config_sha256, artifact_sha256, binary_sha256, revision}. app-lb downloads the release bundle from the configured store, verifies dist/app-lb, dist/REVISION and dist/SHA256SUMS, keeps the previous binary, and has an independent systemd helper swap the binary and restart only the mapped program. Success requires a new process whose binary hash and compiled revision match, and whose public /healthz returns exactly that x-heyo-revision. Ambiguous outcomes stay fenced for operator reconciliation, with no automatic rollback. Hosts that predate the helper need the one-time --bootstrap-host-update path described in app-lb/README.md.

This restarts the regional ingress. Move traffic away from the region first. See multi-region.

Plugins

Plugins are compiled in and switched on at runtime from /plugins or with heyctl plugins. Configs live in app-lb-plugins.d/, name secrets rather than holding credentials, and expose routes under /api/plugins/<id>/ (reads on the view tier, actions on CRUD).

Installing a plugin in a namespace

Some plugins (today, obs) are installed per namespace. The operator enables and configures the plugin once for the host; then a namespace administrator installs it:

heyctl plugins enable obs   # operator, after `heyctl plugins set obs '{"url": "http://127.0.0.1:9600", "api_token": {"secret": "app-obs", "key": "api_token"}}'`
heyctl plugins install obs -n team-a   # an admin token for all of team-a

obs installs itself by default ("auto_install": true in its fleet configuration). Switching it on installs it in every namespace app-lb knows, and each new namespace gets it when it is declared (POST /namespaces) or when its first deployment is registered, since a namespace created in cloud first reaches app-lb that way. A namespace that uninstalls is remembered as declined and is not reinstalled automatically; installing it again by hand clears that. Set "auto_install": false to make installs explicit only.

An install is a record on the plugin, kept while the plugin is disabled. The namespace's view of the plugin is served at /namespaces/<ns>/plugins/<id>/…, behind the namespace wall: a namespace token reaches its own namespace's pages and no other, a GET needs the view tier there and anything else the admin tier. Installing needs an admin credential for the whole namespace, not a token narrowed to some of its deployments. Before the plugin is installed those routes answer 409 with "code": "plugin_not_installed", and while the operator has it switched off, "code": "plugin_disabled".

For obs, /namespaces/<ns>/plugins/obs/ui is the app-obs dashboard narrowed to the namespace, and …/obs/api/fleet, …/obs/api/deployments/<id>[/logs] and …/obs/api/alerts are its JSON API. app-lb sends them to app-obs's /ns/<ns>/… routes with the token from the plugin's configuration; the caller's own credential never reaches app-obs. A namespace user's dashboard (/dashboard?namespace=<ns>) shows an Observability link once it is installed, or an Install observability button for an admin.

Troubleshooting

A pool stays at zero ready. Start with GET /metrics?deployment=<id> and look at metrics.autoscale:

• vms_created and boot_timeouts both rising: the daemon creates VMs but the guest never passes health. Read the start log inside the guest with heyctl exec <id> -- cat /var/log/heyvm-start.log (and .err.log). It reaches app-obs only when the image has socat (see "Install socat in custom images"); otherwise it lives only as long as that boot. Common causes: a start_command that blocks instead of daemonizing, a server bound to 127.0.0.1 instead of the guest interface, the wrong port, or a health.path behind auth or returning 5xx.
• vms_created flat while desired is above zero: creates fail before a VM exists (create_failures and last_create_error say why). Check for a missing image (for example, after heyvm prune --images), a full disk, host memory admission in heyvmd, a secret in env_from that doesn't exist, or a mount or workspace that isn't ready. The app-lb log names the reason (not creating a replica yet: ...).
• vms_created and vms_drained climbing together with no failures: scale-to-zero churn. Check scale_to_zero_after_secs, but first find out why the replica was retired.

Each booting VM is logged with a waiting_on field every 30 seconds. "the daemon has not reported it Running yet" means wait. "the guest is up but has not answered" means look at the guest.

Health checks never pass but ping works. A host firewall is rejecting forwarded TCP to the tap interface (ufw answers with icmp-host-prohibited, which shows up as No route to host). Allow traffic on the Firecracker taps, for example sudo ufw allow in on tap-fc-+, or scope the rule to the guest subnet.

A workspace deployment will not boot a replica. The log says workspace restore pending: ... or workspace capture of <sb> is queued. For a local-path store, restores and pushes shell out to the art CLI, so it must be installed on the app-lb host or pointed at with APP_LB_ART_BIN. With an http(s):// store, the art server must support GET /tags/<name>. Capture, restore and push run one at a time across deployments, bounded by APP_LB_WORKSPACE_TIMEOUT_SECS, so one stuck transfer delays the others.

Requests hang after a VM is destroyed. A destroyed Firecracker VM takes its tap with it, so no RST ever arrives. app-lb sets TCP keepalive (30 s idle, 10 s interval, 3 probes) and a 60-second TCP_USER_TIMEOUT on every upstream socket, so dead peers are dropped within about a minute. On an older build without this, list the stuck sockets with ss -tno dst <vm-ip> and kill them with ss -K dst <vm-ip> dport = :<port>.

Deploy jobs are slow or failing on a full disk. Builds that time out after APP_LB_BUILD_TIMEOUT_SECS and git failing to write lock files both point to a full disk. A pool whose guest never becomes healthy strands a data disk per failed boot. Set min_replicas: 0 to stop allocating, then reclaim from /storage or with POST /disks/purge-orphans.

Disks are listed but never reclaimed. app-lb's user needs write access to the daemon's data directories (run/, kvm/). A purge that removes nothing returns 500, not success.

401 on everything after a restart. The dashboard password changed and your heyctl context still holds the old one. Log in again.

A certificate never arrives. Check GET /certs and the log for ACME backoff. HTTP-01 needs port 80 and DNS pointing at this host. host_suffix routes need a wildcard (APP_LB_ACME_WILDCARD). Test against the staging directory. Switching directories discards cached state automatically.

A second app-lb refuses to start. That is the instance lock working as intended. Never run a second app-lb against the same daemon (including by passing it unknown arguments on an old build). It would treat the first instance's sandboxes as orphans.