# Operations: how S13 is built

What S13 built (plan CORE §3.14; TuteDesk's readiness, `app:check`, `app:deploy` and heartbeat, with Atrium's own
checks, page and measurements; 24 October 2026). The rules: **every readiness check has a label and a test for each
state it can report**, and **every deploy step runs in order and stops on failure**. The server's set-up for a
release is at the end: it is the deployment notes until S19 writes the deployment itself.

## The pieces

| Piece | What it does |
|---|---|
| `Readiness` | The thirteen checks. `STATES` lists each check's icon and every `state.say` it can report (`ok`, `warn`, `off`, `fail`); `run(?keys)` times each and turns a throw into `fail` / `threw`; `record(scheduled)` keeps a `readiness_run` and raises `readiness.failed` for checks that weren't failing on the previous run (once a streak); `words(line)` reads a line as `operations.check.<key>.<say>` |
| `readiness_run` / `ReadinessRun` | Platform data: `lines` (jsonb: key, state, say, params, ms), `failing`, `warning`, `scheduled`, `ran_at`. The hero, the uptime strip (share of runs with none failing, by UTC day) and the readiness list read the latest |
| `app:check {--record}` | One line a check (✓ ! – ✗); fails only on a ✗. The schedule runs it `--record` every 10 minutes, `withoutOverlapping()->onOneServer()` |
| The heartbeat | `routes/console.php`: every minute `Cache::forever(Readiness::HEARTBEAT, now)`, `onOneServer()`, pinging `operations.heartbeat.url` when set. The scheduler check fails when it's missing or older than `stale_minutes` |
| `readiness.failed` (`ReadinessFailed`) | A locked platform event to the Service (`recipients: service`): `checks.failing`, `checks.at`, `operations.url` |
| `JobMetrics` | Queue events, registered in `AppServiceProvider`: each finished or failed job a `job_run` row (queue, short name, `tenant_id` read from the payload without unserialising it, wait on the first try, run time); `Looping` upserts `worker_beat` at most every 15 s. A failure to measure is swallowed |
| `job_run`, `worker_beat` | Platform data (`TenantGuardTest`'s platform list); the tiles, the queues |
| `OldMeasurements` | A `Sweep` on the night's purge (`measurement`): `job_run` after 48 hours, `readiness_run` after 31 days, `worker_beat` a day after it stopped |
| `Dashboard` | What the page shows, read fresh each poll: `hero`, `tiles` (24 hourly values each, `date_trunc`, `percentile_cont` for the p95), `readiness`, `queues`, `scheduled` (the schedule's events, overdue while the heartbeat is stale), `feed` (8 latest audit rows), `space` (the vault's disk, `pg_database_size`, backups kept), `purges` |
| `FailedJob` | Laravel's `failed_jobs` (route key `uuid`; `payload` and `exception` hidden): `name()` (short class), `tenantId()`, `locale()` (read from the payload), `attempts()` (the payload's count, else the tries it was allowed), `why()` (the error's first line without where it was thrown), `trace(15)` (paths from the app's folder) |
| `FailedJobList` | The list engine's (`failed_job`, `operations.view`) |
| `SecurityHeaders` | Appended first to the `web` group: `X-Content-Type-Options: nosniff`, `X-Frame-Options: SAMEORIGIN`, `Referrer-Policy: strict-origin-when-cross-origin`, HSTS a year in production over https; `Permissions-Policy` (camera, microphone, geolocation, payment, USB and the like off; passkeys this origin's) and `Cross-Origin-Opener-Policy: same-origin` (plan SEC, SEC-5). A response that brought its own policy (a file's sandbox) keeps it; Laravel's debug error page gets none |
| `ContentSecurityPolicy` (plan SEC, SEC-1 to SEC-4) | Built per response with a fresh nonce (`Vite::useCspNonce`): `script-src 'self' 'nonce-…'`, `style-src-elem` likewise, `style-src-attr 'unsafe-inline'` (Vue's static trees write `style` attributes), pictures and fonts from Atrium's own hosts, `object-src 'none'`, `base-uri 'none'`, `frame-ancestors 'self'`, no `form-action` (SEC-2). The shell's inline script and styles carry the nonce, `<meta name="csp-nonce">` hands it to Inertia's own styles (`app.ts`); SweetAlert2 is imported without its style injection (`dialog.ts`). Turnstile's origin only on the bot check's pages (`login`, `join*`, `invitation.show`, `me.signin`, `me.claim`) and only with a site key. `CSP_MODE` = `enforce` (default) or `report` (`config/security.php`) |
| `CspReportController` (`POST /csp-report`, limiter `csp-report`) | What the browser reports (either shape), logged as `csp.violation` with the directive, the blocked origin and the page's path only; no session or CSRF token |
| Proxies (`TRUSTED_PROXIES`, `AppServiceProvider::configureProxies`) | Only the addresses named are believed about the client's address, scheme and port, never the host (SEC4 F10); none by default |
| `app:deploy` | `config/operations.php` `deploy`, in order through the console kernel, each one's output kept back unless it fails (or `-v`); the first failure stops it and the command fails |

## The checks

| Check | Passing | Otherwise |
|---|---|---|
| Queue worker | the database queue; nothing waiting longer than 10 minutes, none failed in 24 h | `warn` failed today; `fail` stuck (oldest waiting > `queue.wait_minutes`); `fail` not the database queue in production, `off` elsewhere |
| Scheduler | a beat within 15 minutes | `fail` never beat; `fail` stale |
| Mail | the mailer and sender | `fail` in production with `log`/`array`, or no sender, or a `.test` / `@example.com` sender |
| File vault | the key set, a test object put and read back | `fail` no key; `fail` the probe |
| Malware scanner | the scanner answers its version | `fail` silent; the fake scanner: `fail` in production, `off` elsewhere |
| Debug mode | off | `fail` on (production); `off` elsewhere |
| Cookies and headers | secure cookies, https `APP_URL`, an encrypted session, `SecurityHeaders` in `web`, the policy enforced | `fail` cookies; `fail` session (`SESSION_ENCRYPT`); `fail` headers; `warn` proxies_any (`TRUSTED_PROXIES=*`); `warn` csp_report (`CSP_MODE=report`) (production); `off` elsewhere |
| Product servers’ keys (`server_keys`; plan SEC, SEC4 F4) | no live key ends within a month | `warn` expiring, naming the server and the day |
| Backups | the newest backup within 26 hours, copied off the server (S19) | `warn` older, or one copy only; `fail` none, or not set up in production; `off` not set up |
| Languages | every switched-on language complete | `fail` the incomplete ones |
| Migrations | none waiting | `fail` how many wait |
| Option lists | `lookups:check` has no problem | `fail` how many |
| Notifications | `notifications:check` has no problem | `fail` how many |
| The brand | `brand:check` passes | `fail` |

## Pages

| Page | Route | Permission |
|---|---|---|
| Operations | `/console/operations`; `POST /console/operations/check` (`key` for one) | `operations.view`; checking `operations.manage` |
| Failed jobs | `/console/failed-job`, `/console/failed-job/{uuid}` | `operations.view` |
| Retry, retry all, forget | `POST …/{uuid}/retry`, `POST /console/failed-job/retry`, `DELETE …/{uuid}` | `operations.manage` |

Both permissions are platform ones (the Service; `operations.view` reads). The page reloads its props every 10 seconds
as a background request (`X-Background`), so it never keeps an idle session awake; the feed can be paused.

## Adding a check

1. A private method on `Readiness` named after the key (camel case), returning `[state, say, params]`.
2. Its line in `STATES` (icon and every `state.say`), its name in `operations.name.<key>`, its words in
   `operations.check.<key>.<say>`.
3. A case for each state in `ReadinessTest::readinessCases()`: the guard fails until every state has one.

## Adding a deploy step

Add the command to `config/operations.php` `deploy` where it must run. It must be safe to run twice and fail (non-zero)
when it can't do its job; `OperationsTest` checks every step names a command that exists.

## The server's set-up (deployment notes)

The whole of it is `../DEPLOYMENT.md` (a new server, a release, the commands) and `../BACKUP.md` (S19). In short: a
release runs `deploy/server/release.sh` (the build, a backup, `app:deploy`), then `queue:restart` and `app:check`; the
cron line runs `schedule:run` every minute; a worker runs `queue:work database` under a supervisor.

| Setting (`config/operations.php`) | Shipped |
|---|---|
| `heartbeat.url` (`SCHEDULE_HEARTBEAT_URL`) | none |
| `heartbeat.stale_minutes` | 15 |
| `backup.*` | see `deployment.md` (S19): off until `BACKUP_PATH` and `BACKUP_PUBLIC_KEY` are set |
| `backup.stale_hours` | 26 |
| `queue.wait_minutes` | 10 |
| `queue.wait_alert_seconds` | 5 |

These are the server's set-up, not settings a person changes on a page.

## The guards

| Test | Proves |
|---|---|
| `Operations/ReadinessTest` | the done-when: each `state.say` of each check reported in a case of its own, and the cases are exactly `STATES` with a name for each check (the guard); a throw fails with its reason; a run kept, the Service told once a failing streak; `app:check` fails on a ✗ |
| `Operations/PagePolicyTest` | a fresh nonce on every page carried by every inline script and style; report-only by `CSP_MODE`; Turnstile on the bot check's pages alone; `Permissions-Policy` and COOP; reports logged by directive, origin and path; no proxy believed by default |
| `Operations/OperationsTest` | the page's props for the Service, 403 for a workspace's people; check again (all, one, an unknown key refused); job metrics and the worker's beat; failed jobs listed with workspace and language, their input never in the page; retry (back on its queue with its workspace) and forget, audited; the measurements' sweep; `app:deploy` in order, stopping at the first failure; every deploy step a command; the security headers |

## Following a request (plan AF slice C4)

| Piece | What it does |
|---|---|
| `AssignRequestId` | Every request gets an ID, or keeps a caller's well-formed `X-Request-Id`, and answers with it. Laravel's Context carries it into each log line (`extra.request_id`) and each queued job |
| `Http::globalRequestMiddleware` | Outbound calls send it, when there is one; scheduled work has none |
| The `json` log channel | One JSON object a line, in `storage/logs/app.json-<date>.log`. Production uses it (`LOG_STACK=json`); development keeps plain lines |
| `ProveHealth` | `/up` answers 200 only if the database answers and the queue can be reached, otherwise 500. It says nothing about either: the workers being alive and jobs moving are the readiness check's |