Build Your Own Serverless Postgres

Performant virtual machines can be excellent for bursty and ephemeral workloads. The concept is to provision a VM on demand, route requests logically while demand is active, and then stop the VM automatically after demand stops. This lets you take advantage of high core count machines and move workloads off of managed services on the hyperscalers.

Vendors like Hetzner and Interserver provide inexpensive machines that can absorb workloads at a much lower cost than a service like RDS, even the "serverless" RDS product.

What you will need:

A server that's KVM capable (most modern linux hardware is)
A pooler to manage the creation and restart of VMs
An image to use as a template
heyvm running as a daemon

Getting the Server Setup

This guide will assume a Debian based host system

Ensure KVM is accessible & Firecracker is installed

Here's a helpful guide from the Firecracker folks themselves: Getting Started with Firecracker

First, we need to ensure that KVM is enabled and we have access:

# check that KVM is enabled
lsmod | grep kvm

# check if your user is in the KVM group:
groups 

# If not, check that the group exists and add your user to it:
getent group kvm

# If the group exists, add your user to it:
[ $(stat -c "%G" /dev/kvm) = kvm ] && sudo usermod -aG kvm ${USER} \
&& echo "Access granted."

# either reboot or run this to activate changes:
newgrp kvm

Great, once KVM is ready to go for your user, let's install firecracker:

# from the official docs, get the latest:
ARCH="$(uname -m)"
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest=$(basename $(curl -fsSLI -o /dev/null -w  %{url_effective} ${release_url}/latest))
curl -L ${release_url}/download/${latest}/firecracker-${latest}-${ARCH}.tgz \
| tar -xz

# Rename the binary to "firecracker"
mv release-${latest}-$(uname -m)/firecracker-${latest}-${ARCH} firecracker

# Now move the binary somewhere in your PATH:
sudo mv firecracker /usr/local/bin/

Before we move on, ensure that heyvm is also installed:

curl -fsSL https://heyo.computer/heyvm/install.sh | sh

# by default its installed to ~/.local/bin/heyvm 
# ensure its in your PATH:

Building an Image

The heyvm CLI has built-in build commands to help create custom KVM and Firecracker images (an "image" actually consists of a kernel and root filesystem). This build step will use Docker to build out the rootfs so ensure its installed and setup properly for linux:

# apt repository:
# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

# install docker:
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Post-install ensure that you have groups configured for docker:
# add your user to the docker group:
sudo usermod -aG docker $USER

# apply the group changes:
newgrp docker

Now let's clone the pooler repository which contains a Dockerfile pre-configured for running Postgres:

git clone https://github.com/Heyo-Computer/pg-fc.git
cd pg-fc

# build the image:
heyvm mvm build --local-only -f Dockerfile --name pg

The Pooler

The pooler itself is based on deadpool and will both manage connections and also create and restart VMs as they are requested. The default behavior is to use the database name in the connection string to either find an existing VM or create a new one.

First we will build the pooler, then we will setup supervisord to manage the service lifecycle (e.g. restart on crash or boot), and then we will configure heyvm similarly to be accessible to the pooler. The pooler requires a rust toolchain so we will start by installing it:

cd pg-fc
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# source the rust environment so that its on the path:
source $HOME/.cargo/env

# build the pooler:
cargo build --release

# now lets install supervisord:
sudo apt install -y supervisor

# copy the supervisor config:
# note that the default config has assumed paths to the pooler binary!
sudo cp deploy/supervisor/pg-fc.conf /etc/supervisor/conf.d/

sudo supervisorctl reread && sudo supervisorctl update
sudo supervisorctl status pg-vm-pool          # start/stop/restart/tail work too

# ensure its listening on the default port:
lsof -i :6432

# now we need `heyvm running on a local port as well:
# for testing purposes we run and detach:
heyvm --api --port 34099 &

# test the e2e by connecting to a new vm:
psql "host=127.0.0.1 port=6432 user=postgres dbname=tenant1" 

# inspect the new vm:
heyvm list

# running heyvm with the "&" will detach it, to kill it, get the process id:
lsof -i :34099
# or
ps aux | grep heyvm

# kill it:
kill -9 <process_id>

Configuring the Pooler

The pg-vm-pool.conf file we used when starting the pooler via supervisord contains environment variables with sane defaults, the full list is here.

The pooler application additionally ships with an optional dashboard that is turned off by default, adding the env var PG_VM_POOL_DASHBOARD_LISTEN with a value e.g. http://127.0.0.1:8080 will boot the dashboard on that port. The dashboard provides a convenient list of active and paused sandboxes, metrics for running sandboxes, and access to logs for the pooler and heyvm daemon. When running in a production environment, make sure to either configure the username and password for the dashboard, manage firewall rules accordingly, or put a different auth gate in front of the dashboard.