Jobs, modules and isolation

The job model#

queuedrunningcompleteddependents released

runningfailedthe error is kept on the row

runningcancelledcooperative, then SIGKILL for isolated workers

queuedpausedthe queue was paused; running jobs continue

The queue is an in-process asyncio queue sized by WORKER_CONCURRENCY (default 2, admin-changeable live). Heavy CPU work moves to a process pool through ctx.run_in_process(...), bounded by MODULE_PROCESS_POOL_SIZE and, per user, by MAX_OFFLOADS_PER_USER.

The event bus#

In-process pub/sub, with a bounded queue per subscriber: when one fills, the oldest entry is dropped — acceptable for a progress tick, not for a terminal event.

GET/api/v1/events?topic=… is the platform stream, GET/api/v1/jobs/{id}/stream the per-job one; both are SSE with a Bearer header, and the client reconnects with backoff.

The module system#

Precompute is declared by stacking @on_event("log.imported") and @job on one handler, and it is lazy: a log is ready once parsed, and a module computes for it only when something asks — opening the module page, a dashboard card, or an MCP read — or at import for a manifest that says precompute: eager (an admin can override that per module). One row per (log, module) in log_module_runs records the state (queued, running, ready, failed, cancelled); idle and stale are derived by comparing the row with the log's data revision and the module's version, so a data change never has to touch a row. Requests are deduplicated per (log, module).

A committed filter, a reimport or a cell edit bumps the data revision and re-runs only the modules already computed on the log, with their result cache emptied first (cell edits wait for a pause). Ordering is pull-based: asking for a module first runs what it depends on — the modules it chains off via <module_id>.completed, and its hard requirements.modules — and parks it until they finish. A module whose upstream fails or is cancelled is failed with that reason, never left waiting. A run lost to an API restart is resubmitted at boot.

Install is a job: an uploaded archive (POST/api/v1/modules/install) lands in data/uploaded_modules/{id}/, the manifest is validated, dependencies materialised, the frontend bundled, and the module mounted — a failure rolls back. Git-URL and registry installs are specified in the design docs but not implemented; discovery separately picks up installed packages that expose the mate.modules entry point. Ownership is reference-counted per user, and shared artifacts are deleted at zero owners.

Isolation modes#

In-process modules run on the platform's interpreter (currently 3.12) and are ABI-locked to it: requires-python is a validation gate there, and an interpreter selector for subprocess modules. Each module's .venv is private; inherit names libraries the platform already ships (pandas, numpy, pm4py, duckdb) — an in-process module resolves them from the platform's interpreter, while a subprocess module installs them into its own environment, because no interpreter is shared across the process boundary.

The subprocess bridge is a Unix socket with newline-delimited JSON: a 30-second handshake, ctx.* calls proxied back to the host, DataFrames handed over as Parquet, a three-second soft-cancel window followed by SIGKILL, and automatic respawn with exponential backoff (capped at 30 s, five consecutive attempts, ladder reset after 60 s of stable uptime). The JVM runtime speaks the same protocol, so the SDK surface (eventLog(), cache(), bus(), progress(), …) is equivalent.

Sidecars are optional compose profiles the operator starts; the module only connects to a configured address, probes health, wipes its scratch space before and after a run, and keeps results in ctx.cache.

MATE AI internals#

Prompt assembly per call: base system prompt → navigation note → optionally the process list → optionally activity and variant aggregates (≤40 activities, top 15 variants; skipped for OCEL) → cached module outputs within a character budget → the user's own prompt. Raw rows are never part of any of it: the module context built for an AI call has its event-log accessors replaced by objects that raise on access — the same mechanism the MCP server uses.

Guidance is cached per module and log together with a hash of the payload it was generated from, so it invalidates itself when the underlying result changes.

Observability#

Logs are structured, namespaced per module, and mirrored into a bounded in-memory ring buffer that the admin Jobs page tails. Container logs are capped by the compose log driver so a noisy module cannot fill the disk.