docs architecture.md

Architecture

LEMC keeps selection at the edges and execution in one server-owned job path. Web, API, scheduler, and LEMCSSH callers converge before a recipe reaches a runner.

System shape

human browser ── web session ─┐
LEMCSSH / agent ─ SSH session ├──► LEMC web/API server
scheduler ───── server state ─┘          │
                                         ├─ authenticate + authorize
                                         ├─ validate action + inputs
                                         ├─ preflight managed image
                                         ├─ acquire queue lock
                                         └─ enqueue task
                                                │
                                                ▼
                                     hosted runner coordinator
                                                │
                              fresh jailed Firecracker guest / job
                                                │
                    ┌───────────────────────────┴───────────────────────────┐
                    ▼                                                       ▼
          structured recipe events                              logs + artifacts
                    │                                                       │
                    ▼                                                       ▼
             LEMC server store                                    durable storage
              │            │
              ▼            ▼
           /web/ws       LEMCSSH stream
           browser       LEMCSSH

Server authority

The server owns decisions that must not vary by client:

  • authentication, account scope, authorization, and action visibility;
  • form validation, idempotency, queue admission, and scheduling;
  • execution mode, runner/pool selection, isolation, and resource limits;
  • image authorization and digest-pinned managed registry references (a stored pin keeps a host token that the server expands, so a host rename needs no data edit);
  • guest size: a recipe may declare cpu, memory_mb, and disk_mb, and the server refuses a size above the install cap before any job is queued;
  • Provider admission: approved Profile version; one call pinned to each exact job, callback, or scheduled occurrence; current access rechecked at execution; Custom Provider-only plaintext and bounded context; bounded YAML result; and managed-output redaction;
  • event persistence, filtering, websocket fanout, and terminal mapping;
  • durable log/artifact/cache sync and terminal cleanup proof.

The browser, API payload, and LEMCSSH command do not carry a per-request “run this on Docker/Firecracker/runner X” switch.

Hosted job lifecycle

  1. Job started: record target, caller, queue type, task ID, and policy.
  2. Image preflight: require an authorized digest from the managed image path.
  3. Provision and boot: create one fresh jailed Firecracker guest for this job.
  4. Prepare brokers: expose only job-scoped DNS, registry, storage, events, proof, and allowed egress surfaces.
  5. Verify denials: prove host, private, metadata, and Docker-bridge targets are unreachable.
  6. Prepare runtime and load image: ready the guest runtime without host Docker as a server fallback.
  7. Run recipe: execute ordered steps and send structured output back to the server.
  8. Finalize: persist terminal state and verify durable logs/artifacts/cache.
  9. Cleanup: destroy guest resources and release the queue lock on success, failure, cancel, timeout, or lost heartbeat.

Queue shape

Queue typeAdmissionRuntime shape
NOWRun as soon as capacity/policy allow.The due job uses the same system steps, recipe execution, event path, storage finalization, and cleanup proof.
INBecome due after a delay.
EVERYCreate recurring due instances.

Queue type changes timing and lock semantics; it does not create a weaker execution path. A late duplicate delivery of an occurrence that LEMC already claimed starts no second occurrence.

Events and fanout

Runners and guests post job events to the server. The server validates and persists them before authorized clients consume the task-scoped stream.

  • /web/ws authenticates through the browser's web session.
  • LEMCSSH authenticates with an enrolled SSH key and receives the restricted gateway stream. A login opens a mock shell that takes only lemcli lines, and a script pipes many lines to one connection. It is not an OS shell, and a one-shot ssh host command is refused. A web-password login opens only the interactive shell.
  • Both receive matching LEMC verbs, system steps, recipe steps, terminal status, and terminal reason.
  • Raw stdout/stderr diagnostics remain durable and bounded rather than flooding the user-visible stream.

Storage and cleanup

Local files are scratch. Rendered output, logs, artifacts, cache, and workspace state synchronize to durable object storage before local scratch is removed. A later browser refresh or LEMCSSH read hydrates from durable state first. A reviewer with output-review access browses one user's files for an App or Cookbook through the same durable state.

A job is not clean merely because the recipe process exited. Finalization must prove durable sync, guest teardown, process cleanup, bounded disk growth, and lock release.

Network boundary

Hosted-untrusted guests receive narrow job-scoped broker endpoints, not the host network or primary LEMC web/API listener. Public egress is policy-controlled and tested separately from denial of host, loopback, private, metadata, and Docker-bridge targets.

Continue with LEMC verbs to see how recipe output crosses this boundary.