Architecture and data

Services#

Service	Technology	Role	Port
`api`	Python 3.12, FastAPI	Routes, module loader, jobs, ingest, MATE AI proxy, MCP server	`8000`
`web`	Next.js 15, React 19	Interface, plus esbuild bundling of module frontends	`3000`
`keycloak`	Keycloak 26	OIDC login, realms, roles	`8080`
`app-db`	Postgres 16	Application metadata — the only database the platform reads	internal
`keycloak-db`	Postgres 16	Keycloak's own store	internal
`neo4j`	Neo4j 5	Sidecar for graph-backed modules, behind the `graph` compose profile	loopback

A host development API runs on SQLite instead, so make dev and the test suite need no database server.

Browser

Next.js server componentscookie-backed sessionDirect API callsNEXT_PUBLIC_API_URL, CORS, Bearer token

webrenders the interface, bundles module frontends

apiroutes, modules, jobs, ingest, MCP

app-dbusers, logs, jobs, settings, sharesParquetevent logs and module results under data/DuckDBin-process SQL over Parquet

Outside the host, only when configured

Keycloaklogin and JWKS token validationLLM providersMATE AI and module AI featuresS3-compatible bucketSTORAGE_MODE=s3

How a request travels. Two client paths into one API. The API owns the stores, and reaches outside the host only for what you configured.

Repository layout#

Repository

mate/
├── apps/api/                # FastAPI backend (src/mate/api)
├── apps/web/                # Next.js frontend (app/, components/, lib/)
├── modules/                 # bundled modules – discovered at startup
├── packages/
│   ├── module-sdk-py/       # Python SDK (mate.sdk)
│   ├── module-sdk-ts/       # frontend SDK for panels and widgets
│   ├── module-sdk-jvm/      # Java SDK
│   └── shared-types/        # generated TypeScript types
├── data/                    # bind-mounted state
├── docs/                    # design spec, deploy runbook, MCP reference
├── infra/                   # Caddyfile, Keycloak realm and scripts
└── landing/                 # public site and this manual

modules/ sits outside apps/ on purpose: it is the extension point, and a module is authored and shipped on its own. Folder names are arbitrary — the manifest's id is authoritative.

Technology choices#

Concern	Choice	Why
Metadata	Postgres (SQLite in dev and tests)	Transactions and concurrency where they matter, zero setup where they do not.
Event data	Parquet	Columnar, compressed, 5–20× smaller than XES, readable by every Python data tool.
Queries	DuckDB	Embedded SQL over Parquet; no service to run, sub-second aggregation.
Jobs	asyncio queue + job rows	No broker, no extra container, state that survives a restart.
Live updates	Server-Sent Events	The production proxy chain carries HTTP streaming but drops WebSocket upgrades.
Types	Pydantic → OpenAPI → `openapi-typescript`	One schema, generated types, no drift.

Metadata tables#

Group	Tables
Identity and access	`users` (a mirror of the Keycloak subject), `api_tokens`, `user_settings`, `system_settings`, `control_policies`
Logs	`process_logs`, `process_folders`, `watched_folders`, `watched_folder_files`, `event_edits`
Work	`jobs` (status, progress, payload, parent job)
Modules	`module_configs`, `module_installs`, `module_layouts`
Sharing	`dashboards`, `teams`, `team_members`, `dashboard_shares`
Usage (optional)	`analytics_sessions`, `analytics_events`, plus the OCEL-shaped `analytics_objects`, `analytics_object_relations`, `analytics_event_objects`

Logs, folders, watched folders, and teams use soft deletes: the row stays for audit and cascade bookkeeping while the data is removed.

On-disk layout#

data/

data/
├── users/{user_id}/
│   ├── event_logs/{log_id}/
│   │   ├── meta.json         # source format, ingest stats, detected schema, roles
│   │   ├── events.parquet    # event table, sorted by (case, timestamp)
│   │   ├── cases.parquet     # cached per-case aggregates
│   │   ├── original.*        # the untouched upload, for re-import and export
│   │   └── ocel/             # events, objects, relations, o2o (OCEL only)
│   └── module_results/{log_id}/{module_id}/
├── staging/{user_id}/{token}/     # uploads awaiting confirmation (swept after 2 h)
├── uploaded_modules/{module_id}/  # modules installed by upload
└── uv-python/                     # cached interpreters and wheels

Log ids and job ids are UUID v7: time-ordered, so they sort and index well without coordination. Display names never appear in URLs, so renaming never breaks a link.

The import pipeline#

Stage	Module	Does
Detect	`ingest/detect.py`	Format family from the extension, refined by content sniffing (including OCEL vs case-centric).
Decompress	`ingest/compression.py`	gzip, bzip2, xz, zip.
Stage	`ingest/staging.py`	Bytes land under `data/staging/{uid}/{token}/` so probing and confirming never re-upload.
Probe	`ingest/probe.py`	A bounded sample proposes column roles.
Parse	`ingest/{xes,csv_parser,xml_parser,json_parser,ocel}.py`	Format-specific parsing into one frame; XES streams.
Map · coerce · aggregate	`mapping.py`, `parquet_coerce.py`, `aggregation.py`	Apply roles, type the columns, normalise timestamps, compute case aggregates.
Store · dispatch	`storage.py`, `dispatch.py`	Write the Parquet set, bump the log's data revision, flip the status to `ready`, emit `log.imported`.

Automatic repairs (duplicate columns, text that should be numeric, mixed timestamp formats) are recorded and reported rather than rejected.

Storage modes#

Mode	Behaviour
`local` (default)	One copy of everything under `data/`; the directory is the backup.
`s3`	An S3-compatible bucket is authoritative and local disk is a bounded cache with an eviction reaper.

Details, including migration and quota: Backup, storage and limits.