app-obs

app-obs collects logs and metrics for the deployments app-lb manages, stores them as partitioned Parquet, and serves a query API and dashboard over them.

What it is

app-obs is a single host process (not a microVM) that runs next to app-lb. It:

• tails every managed VM's console and start_command output from the heyvm daemon, with no shipper or code inside the guest;
• accepts pushed logs from applications over HTTP (JSON) and syslog (UDP and TCP);
• polls app-lb's GET /metrics for request counts, error counts, latency, pool state, and per-VM and whole-host CPU and memory;
• stores everything as Hive-partitioned Parquet under one data directory, compacts small files, and deletes partitions past a retention window;
• serves a dashboard and JSON API with a fleet view, per-deployment charts, a filterable log viewer, and a live platform status view;
• fires webhook alerts when a deployment's error count crosses a threshold.

It is fronted by app-lb as a static (proxy_pass) deployment, the same way the pg-fc and queue dashboards are.

app-obs is never part of the data plane. If it is down, app-lb keeps serving traffic, and guests that push to a dead ingest port get a connection error rather than blocking.

How data arrives

Native tail from the heyvm daemon

With HEYVM_URL set, app-obs opens a WebSocket to the daemon's GET /sandboxes/{id}/logs/stream for every sandbox the app-lb poll reports. Each line is stored with the deployment and sandbox it came from and a source of stdout, stderr, or console. app-lb is the authority on which sandbox serves which deployment; the daemon only knows sandbox ids. On reconnect, app-obs asks for a small backlog and drops lines it has already stored.

Sandboxes on the host that no deployment manages (made with heyvm, the cloud API, or the desktop app) are reported by app-lb as host_sandboxes. app-obs tails those too and files them under the reserved deployment id _unmanaged, with backend set to the sandbox id.

Unset HEYVM_URL disables native tailing.

Push over HTTP

Every microVM sits on its own /30, with the host at guest_ip - 1. Guests send to their default gateway, which is always this host, so the ingest listener binds 0.0.0.0 by default.

# inside a guest
GW=$(ip route | awk '/^default/ {print $3}')
curl -XPOST "http://$GW:9500/ingest" \
  -H 'content-type: application/json' \
  -H "authorization: Bearer $APP_OBS_INGEST_TOKEN" \
  -d '{
    "deployment": "demo",
    "backend": "sb-abc123",
    "records": [
      {"ts": 1785260096000, "level": "info", "message": "started"},
      {"level": "error", "message": "boom", "fields": {"request_id": "r1"}}
    ]
  }'

Batch fields:

Record fields:

The response is 202 Accepted with {"accepted": n, "dropped": n, "rejected": n}. A record with no resolvable deployment is rejected; retrying it will not help. With APP_OBS_INGEST_TOKEN set, a request without the matching bearer token gets 401.

If an application both prints to stdout and pushes the same lines, and native tailing is on, each line is stored twice under different source values. Pick one path per line.

Push over syslog

logger -n "$GW" -P 9514 -t demo "hello from syslog"

The syslog tag becomes the deployment id when it is a valid id; otherwise records land under syslog. Syslog severities are normalised to error, warning, info, or debug.

Metrics poll

Every APP_OBS_POLL_SECS, app-obs fetches app-lb's GET /metrics and writes one metric row per deployment (plus per-backend rows). Whole-host CPU and memory are stored under the reserved deployment id _host. Don't register deployments named _host or _unmanaged.

When app-lb has admin auth enabled, set APP_LB_PASSWORD; the poller then sends HTTP basic auth with APP_LB_USER (default admin).

Back-pressure

All ingest paths feed one bounded queue (APP_OBS_QUEUE_CAPACITY). When it is full, records are dropped and counted, never blocked, so a slow collector cannot stall an application. GET /stats reports the counters.

Storage

<APP_OBS_DATA_DIR>/
  logs/deployment=<id>/date=YYYY-MM-DD/hour=HH/<ts>-<seq>.parquet
  metrics/deployment=<id>/date=YYYY-MM-DD/<ts>-<seq>.parquet
  alerts.json

• Logs partition hourly; metrics partition daily. deployment, date, and hour live in the path, not in the files, so a query filtered by deployment and time only opens matching directories.
• A partition flushes when it has APP_OBS_FLUSH_ROWS buffered rows or APP_OBS_FLUSH_SECS after its first row, whichever comes first. Until then, the newest rows are in memory and not queryable; /stats reports them as buffered_rows.
• Files are written under a temporary name and renamed into place, so readers never see a partial file.
• A background compactor merges each partition's small files into one every APP_OBS_COMPACT_SECS. Swaps happen with queries paused, and an interrupted merge is rolled back or cleaned up on the next pass.
• Retention deletes whole partition directories older than APP_OBS_RETAIN_DAYS, counting back from and including today (7 keeps seven days). Future-dated partitions are never deleted, so a sender with a skewed clock can grow the data directory beyond what the window suggests.

Log columns

ts, backend, source, level, message, fields, host (plus the deployment/date/hour partition keys).

Metric columns

ts, backend, cpu_percent, memory_bytes, in_flight, ready, pending, draining, requests_total, errors_total, p50_ms, p90_ms, p99_ms, latency_count, latency_sum.

Things to know when reading metrics:

• Percentiles are p50, p90, p99 only, the ones app-lb measures. There is no p95.
• app-lb's latency histogram and request counters are cumulative since app-lb started. The dashboard charts mean latency per interval by differencing latency_count and latency_sum, and rates by differencing the totals. A decrease means app-lb restarted; that interval is reported as null (a gap on the chart), not zero.
• A metric that was not reported is stored as null, not zero.

Dashboard and query API

The API listener (default 127.0.0.1:9600) serves a single self-contained HTML dashboard at /dashboard (/ redirects there). It uses no CDN, so it works on a host with no outbound route. The fleet page lists every deployment with a sparkline and error count, plus the sandboxes on the host outside any deployment. A deployment page shows charts and a log viewer filterable by level, backend, text, and an explicit time range.

"token" means the route requires Authorization: Bearer <APP_OBS_API_TOKEN> when that variable is set, and is open otherwise.

/api/platform-status reads the in-memory result of the last app-lb poll, not Parquet, so a slow historical query cannot hide it. It is marked stale once the last successful poll is older than three poll intervals (minimum 15 seconds).

Windows

window accepts a preset (15m, 1h, 6h, 24h, 7d, 30d) or any relative duration such as 1d, 45m, or 2 weeks. It is clamped to between one minute and 90 days. Anything unparseable falls back to 24h instead of erroring. Bucket width is chosen from a fixed ladder (10s, 30s, 1m, 5m, 15m, 1h, 6h, 1d) based on the window; callers can't set it.

Log query parameters

There is no SQL passthrough. Every query is built by the server, so partition pruning and row caps always apply. Queries run in a bounded pool (APP_OBS_QUERY_CONCURRENCY) with a deadline (APP_OBS_QUERY_TIMEOUT_SECS); a query that arrives while the pool is full gets 503, and one that runs past its deadline gets 504.

Alerts

An alert watches one deployment's error count over the trailing minute. A checker runs every 60 seconds and, when the count is greater than the threshold, POSTs to the webhook:

{"timestamp": 1785260096000, "deployment": "web", "errors": 12}

Create one:

curl -XPOST localhost:9600/api/alerts \
  -H 'content-type: application/json' \
  -d '{"deployment": "web", "threshold": 5, "webhook_url": "https://hooks.example.com/obs"}'

Rules are stored in APP_OBS_ALERTS_FILE (default <APP_OBS_DATA_DIR>/alerts.json) and rewritten atomically on every change. A corrupt file stops app-obs from starting rather than silently dropping every rule. Webhook delivery has a 10-second timeout and is not retried; a failed query never fires an alert.

Configuration

Configuration is environment-only; there is no config file and no CLI flags.

An unparseable numeric value is ignored with a warning and the default is used.

Install and run

The crate builds two binaries: app-obs (the service) and dump (installed as app-obs-dump).

cargo build --release --locked --manifest-path app-obs/Cargo.toml
sudo install -m0755 app-obs/target/release/app-obs /usr/local/bin/app-obs
sudo install -m0755 app-obs/target/release/dump /usr/local/bin/app-obs-dump

sudo useradd --system --no-create-home --shell /usr/sbin/nologin app-obs
sudo install -d -o app-obs -g app-obs /var/lib/app-obs /var/log/app-obs

To run it under supervisord, install app-obs/deploy/supervisor/app-obs.conf:

sudo cp app-obs/deploy/supervisor/app-obs.conf /etc/supervisor/conf.d/
sudo supervisorctl reread && sudo supervisorctl update

Edit its environment= block first. At minimum, decide on APP_OBS_INGEST_TOKEN, and set APP_LB_PASSWORD if app-lb requires admin auth. If the file holds secrets, make it root:root mode 0640.

On SIGTERM, app-obs drains its queue and flushes open partitions, allowing itself 30 seconds. Keep supervisord's stopwaitsecs above 30 (the shipped unit uses 35), or buffered rows are lost.

To try it locally:

APP_OBS_DATA_DIR=/tmp/obs APP_OBS_FLUSH_SECS=2 \
  cargo run --manifest-path app-obs/Cargo.toml --bin app-obs
# then open http://127.0.0.1:9600/dashboard

A short flush interval makes pushed lines queryable within seconds. Metric charts need app-lb reachable at APP_LB_URL; the log side works without it.

Register with app-lb

app-obs/examples/app-obs.json is a static deployment that fronts the API port:

{
  "id": "app-obs",
  "routes": [{ "host": "obs.us2.heyo.work" }],
  "upstreams": ["127.0.0.1:9600"],
  "health": { "path": "/healthz", "timeout_secs": 2 },
  "auth": {
    "client_id": "REPLACE.apps.googleusercontent.com",
    "client_secret": { "secret": "google", "key": "client_secret" },
    "allowed_domains": ["heyo.work"],
    "public_paths": ["/healthz", "/__ui/"],
    "cookie_domain": "us2.heyo.work",
    "forward_identity": true
  }
}

heyctl apply -f app-obs/examples/app-obs.json

Change the host, client id, and allowed domains first, and store the Google client secret (heyctl create secret google --from-stdin client_secret). See app-lb auth for the auth block.

The auth block is required. A proxied route is public unless its spec has one, and without it anyone can read every deployment's logs. Keeping APP_OBS_API_ADDR on loopback does not help, because app-lb proxies the whole hostname.

APP_OBS_API_TOKEN is for machine-to-machine access over a direct route (for example a control plane reading /api/platform-status). Browsers can't attach a bearer header on navigation, so keep using app-lb's sign-in for the interactive dashboard.

The ingest and syslog listeners are deliberately not fronted by app-lb. Guests reach them directly on the tap network.

Common operations

curl -s localhost:9600/healthz
curl -s localhost:9600/stats                       # accepted / dropped / buffered_rows
curl -s 'localhost:9600/api/fleet?window=1h' | jq
curl -s 'localhost:9600/api/deployments/web/logs?window=15m&level=error&limit=50' | jq

Inspect what is on disk without the query engine:

app-obs-dump /var/lib/app-obs/data/logs/deployment=web/date=2026-09-30 | head

app-obs-dump <file-or-directory>... prints Parquet rows as JSON lines, walks directories recursively, skips in-progress temporary files, and reports the row and file count on stderr.

Troubleshooting