Backup, storage and limits

Backup scope#

Asset	Where	Why it matters
Metadata	Postgres (`app-db`)	Users, logs, jobs, settings, dashboards, shares, tokens.
Event data and results	`data/` or the S3 bucket	Parquet logs, module results, retained originals.
Keycloak	the `kc-data` volume	Accounts, realm settings, roles, IdP configuration.
Module sources	`modules/`, `data/uploaded_modules/`	Bundled and user-installed modules.
Environment	`.env`, `scripts/deploy.env`	Secrets and deployment configuration — keep them in a secret manager.

Backup, in four commands

cd ~/mate
docker compose exec -T app-db pg_dump -U mate -Fc mate > mate-db-$(date +%F).dump
tar czf mate-data-$(date +%F).tgz data
docker run --rm -v kc-data:/v -v "$PWD":/b alpine tar czf /b/kc-data-$(date +%F).tgz -C /v .
cp .env mate-env-$(date +%F).bak        # encrypt this

In S3 mode the bucket already holds the authoritative event data; the platform also writes an hourly metadata snapshot to _system/metadata.dump (or metadata.db on SQLite deployments) and archives uploaded module sources to _system/modules/.

Restoring a backup#

Stop the stack so nothing writes during the restore: docker compose down.
Restore the files: tar xzf mate-data-<date>.tgz into the repository root.
Restore identities: unpack the kc-data archive into the volume.

Restore metadata into an empty database, before the API starts:

Terminal

docker compose up -d app-db
docker compose run --rm api python -m mate.api.storage.db_backup restore

Start the stack, then verify: sign in, open a process, run one module, check the storage gauge.

db_backup restore refuses to write into a database that already has tables — deliberately, because restoring over a live schema produces an inconsistent platform that is harder to diagnose than an empty one. From S3, the same command restores on a fresh VM before the first boot, after which logs hydrate on demand as users open them.

S3 mode#

Aspect	Behaviour
Authoritative store	The bucket holds event logs, module results, the metadata snapshot, and uploaded module sources.
Local disk	A working cache. `LOCAL_CACHE_MAX_BYTES` bounds it; the reaper trims least-recently-used directories to 90 % of the budget.
Safety	A directory is deleted locally only after its bucket prefix is confirmed non-empty, and never while a job holds a lease on it.
Never synced	Staging, module virtual environments and bundles, cached interpreters — all reproducible.

Switching an existing deployment to S3

# 1. set STORAGE_MODE=s3 plus the STORAGE_S3_* variables, then recreate the API
docker compose exec api python -m mate.api.storage.migration check     # probe + usage report
docker compose exec api python -m mate.api.storage.migration to_s3     # copy-only, re-runnable

# 2. set LOCAL_CACHE_MAX_BYTES, run with CACHE_EVICT_DRY_RUN=true, then enable deletes
# to go back: run `to_local` while the mode still says s3, then flip the mode

Both migration directions are copy-only: they never delete the source.

Resource limits on a shared host#

By default the platform behaves as if it owned the machine. On a shared VM, bound it:

Setting	Suggested
`WORKER_CONCURRENCY`	2–3
`MODULE_PROCESS_POOL_SIZE`	2–4
`MAX_OFFLOADS_PER_USER`	At least one below the pool size, so no account can hold every slot
`DUCKDB_THREADS` / `DUCKDB_MEMORY_LIMIT`	Half the cores, half the RAM
Container memory limits	The shipped overlay: `api` 8 GB, `web` 768 MB, Keycloak 1 GB, databases 512 MB, proxy 128 MB

Capacity rules of thumb: RAM = 2 GB baseline + the largest expected module working set + 1 GB headroom; disk = total log size × 3 for caches and working copies, plus 20 %; and keep MAX_OFFLOADS_PER_USER × expected users below MODULE_PROCESS_POOL_SIZE, or the cap never bites.