Artifacts

The artifacts store (art) is a content-addressed blob store built for ext4: it holds VM images, site bundles, Dockerfile build inputs, and workspace snapshots, and serves them to app-lb and heyvm over the CLI or HTTP.

What it is

A blob's only name is the sha256 of its bytes. Tags, manifests, and labels all point at digests. The project ships as:

• a Rust library (artifacts),
• the art CLI, which owns every destructive operation (rm, untag, gc, heyvm sparsify),
• the art serve HTTP daemon, which exposes content (upload, download, tags, manifests, labels) but never lifecycle.

How it fits into HWS:

• heyvm base images. art heyvm import takes heyvm's Firecracker images into the store, and art heyvm materialize produces a writable rootfs from one while copying only live data.
• app-lb deployments. A deployment's artifact block pulls a rootfs or site bundle from a store; its build.store points at a Dockerfile manifest to build; its vm.workspace block snapshots a workspace into a store. See app-lb.
• heyctl. heyctl artifact (alias heyctl art) logs in to a store and pushes images and Dockerfiles. See heyctl.
• CI. The CI service stores build artifacts in a store. See ci.

The store is Linux-only. It depends on O_TMPFILE, FALLOC_FL_PUNCH_HOLE, copy_file_range, and statx, and fails to build elsewhere.

How it uses ext4

• Hardlinks are references. A read-only materialization is a hardlink, so it costs no bytes or time. The blob's st_nlink is the reference count, maintained by the kernel; the store keeps no counter and needs no crash recovery.
• Sparse storage. ext4 has no reflink, and heyvm images are fully allocated on disk but mostly zeros inside. --squash scans for zero runs and punches holes, so a 20 GiB image with ~600 MiB of real data takes ~600 MiB on disk. The digest still covers the full logical content, so it matches sha256sum of the original file.
• Atomic, deduplicating inserts. Bytes are written to an unnamed O_TMPFILE, synced, and linked into place under their digest. If the name already exists, that is a deduplication hit.
• Free-space guard. Writes that would leave less than ART_MIN_FREE_BYTES free on the filesystem are refused (exit code 3, HTTP 507).
• Immutable blobs. Blobs are mode 0444. A writable materialization is always a private copy.

The store must be owned by the user who reads it. With fs.protected_hardlinks=1 (the Linux default), only the owner can hardlink a 0444 file; a shared store silently falls back to copying for everyone else.

Concepts

A ref is a tag or a digest. When you ask for a blob by a tag that names a manifest, the store steps through it: a manifest with one entry resolves to that entry; a manifest with several is an error that lists them (except art heyvm materialize, which picks the rootfs.ext4 entry).

Manifest kinds used today:

Garbage collection marks every blob reachable from a tag (directly or through a manifest), then deletes blobs that are unreachable, have st_nlink == 1 (no outstanding materializations), and are older than the minimum age. An untagged manifest does not keep its blobs alive. Labels and public markers for deleted content are removed in the same pass.

Install

cargo build --release --locked --manifest-path artifacts/Cargo.toml
sudo install -m0755 artifacts/target/release/art /usr/local/bin/art
art init                    # creates the store under $ART_ROOT or ~/.artifacts

The HTTP daemon is behind the default-on daemon cargo feature. Build with --no-default-features for the library alone.

Configuration

Every value resolves flag, then environment variable, then default.

Store (all commands)

Daemon (art serve)

ART_ADMIN_PASSWORD, ART_DASHBOARD_OPEN, and ART_DASHBOARD_GATE are mutually exclusive; setting two of them is a startup error. With none set, the dashboard is not mounted at all.

CLI reference

Global flags: --root, --min-free-bytes, --json (machine-readable output on every command).

Exit codes: 0 success, 1 failure, 2 bad usage (invalid digest or tag), 3 out of space.

Content

Names and access

Removal

Dockerfiles

A Dockerfile manifest is a build input: the recipe plus the files it copies. The store doesn't build it; app-lb fetches it and runs heyvm mvm build.

art dockerfile put ./Dockerfile --context ./app --tag web-rootfs \
    --image-name web --size-mb 4096

The context is packed deterministically into one blob, so re-pushing an unchanged tree uploads nothing. Nothing is excluded: there is no .dockerignore handling. Point --context at a clean directory, or pack the archive yourself.

heyvm integration

Typical flow on a heyvm host:

art heyvm sparsify            # reclaim space in the existing images first
art heyvm import --all        # take them into the store
art heyvm materialize debian-hermes /tmp/rootfs.ext4
art gc

Run sparsify before import on a nearly full disk: the images are fully allocated until they are sparsified. sparsify refuses to punch holes in a file with more than one link, because that would change a running VM's disk.

HTTP API (art serve)

Uploads and downloads stream, so large images never sit in memory. Small request bodies are capped at 2 MiB; blob uploads are bounded only by the free-space guard.

Blob responses carry ETag (the digest), Cache-Control: public, max-age=31536000, immutable, and x-art-allocated (bytes actually on disk).

Error statuses: 400 bad digest, tag, or label; 401 bad or missing key; 403 read-only mode; 404 not found; 409 digest mismatch or ambiguous manifest; 507 out of space.

There is no gc and no materialize route. Garbage collection stays in the CLI, and a remote materialization is just GET /blobs/{digest}.

Authentication

With ART_API_KEY set, every route except /healthz requires the key as Authorization: Bearer <key> or X-Api-Key: <key>. The comparison is constant-time.

One exception: a request with no credential may GET or HEAD /blobs/{digest} for a blob marked public. Listings, manifests, tags, and all writes still need the key. A request that presents a wrong key is rejected even for a public blob. A blob re-inserted after GC starts private.

art public web-bundle                    # or: curl -XPUT -H "x-api-key: $KEY" $URL/public/web-bundle
curl -O https://art.example.com/blobs/<digest>   # no credential needed

Example: push and tag over HTTP

D=$(sha256sum rootfs.ext4 | cut -d' ' -f1)
curl -T rootfs.ext4 -H "x-api-key: $KEY" "$URL/blobs/$D"
curl -XPUT -H "x-api-key: $KEY" --data "$D" "$URL/tags/web-v2"

heyctl artifact push does this for you and stores the key in a saved registry.

Dashboard

A read-only, server-rendered dashboard (no JavaScript, no external assets) lives at /dashboard: an overview with capacity and usage, a blob list comparing stored and logical size, a manifest list, and detail pages showing which tags and manifests reference a blob. Deleting, tagging, and GC stay in the CLI.

The dashboard credentials are separate from ART_API_KEY. The API key does not open the dashboard, and the dashboard login does not open the API.

Running as a microVM behind app-lb

artifacts/Dockerfile and init.sh package the daemon as a heyvm image. Build from the repository root, because the build needs the shared ui/ directory:

heyvm mvm build --local-only -f artifacts/Dockerfile -n artifacts --size-mb 768
art heyvm sparsify artifacts

In an app-lb build block this is "dockerfile": "artifacts/Dockerfile" with "context": ".". The image build runs a smoke test that art serve starts and answers /healthz.

init.sh brings up networking, mounts /dev/vdb at /workspace, starts sshd, and prints HEYVM_READY. It does not start the daemon; start_command does, so that env_vars (including ART_API_KEY) reach it.

Put ART_ROOT inside /workspace. The rootfs is recopied from the base image on every cold boot, so a store on the rootfs loses every blob on restart.

Example deployments:

Keep a store deployment at exactly one replica (min_replicas: 1, max_replicas: 1). Each VM has its own disk, so multiple replicas are independent stores and round-robin would return 200 or 404 depending on which one answered.

How app-lb uses a store

app-lb addresses a store in one of two forms, and picks the transport from the spelling:

A local store is much faster (holes are skipped, bundles are hardlinked); a URL is what lets one store serve many hosts.

Pulling a rootfs or site (artifact)

"artifact": {
  "store": "http://127.0.0.1:8080",
  "ref": "web-v2",
  "grow_gb": 4,
  "auth": { "secret": "art", "key": "api_key" }
}

The digest is verified before the bytes are used, and the image file is named after it, so a re-pull of content already on disk is skipped. Trigger a pull with heyctl pull <deployment> [--ref REF] [--wait].

Building from a Dockerfile manifest (build.store)

A deployment's build.store (mutually exclusive with build.repo) names a store holding a heyvm.dockerfile.v1 manifest. Push one with art dockerfile put or heyctl artifact push-dockerfile, then point the deployment at it with heyctl set build --store <url> --ref <tag>.

Workspace snapshots (vm.workspace)

A deployment's vm.workspace block keeps a writable directory across VM replacement. When a replica retires, app-lb captures the workspace, stores it as a tar.gz blob, and moves a tag to it; the next replica is seeded from that snapshot.

On the app-lb host the working copies live under /var/lib/app-lb/workspaces/<deployment>/. app-lb refuses to move the tag if the store holds a snapshot this host has never seen, which keeps history linear. Use a dedicated tag per deployment; never share a workspace tag between two deployments. See app-lb for the full workspace lifecycle.

On-disk layout

$ART_ROOT/
  .store.lock              flock: shared for a commit, exclusive for gc
  blobs/<aa>/<64-hex>      mode 0444, immutable
  manifests/<aa>/<64-hex>  canonical JSON, addressed by its own sha256
  labels/<aa>/<64-hex>     name and description for a blob or manifest
  tags/<name>              one digest and a newline
  tmp/                     incoming files on the fallback insert path

<aa> is the first two hex characters of the digest.

Common operations

art usage                              # how much space the store is saving
art ls --tags
art stat debian-hermes                 # links > 1 means VMs are using it
art verify --all                       # re-hash everything
art gc --dry-run                       # see what would be removed
art gc --min-age 2h
curl -s -H "x-api-key: $KEY" "$URL/usage"

Troubleshooting