# Architecture

AegisFlow keeps the reasoning agent separate from the authority to release. It recovers the preserved go-httpbin application through a real Duo-authored MR, with the controller and protected pipeline enforcing bounded decisions.

```mermaid
flowchart LR
  U[Real HTTP request] --> A[go-httpbin on Cloud Run]
  A --> L[Structured Cloud Logging failure]
  L --> M[Cloud Monitoring alert]
  M --> P[Pub/Sub authenticated push]
  P --> C[AegisFlow controller]
  C <--> F[(Firestore ledger)]
  C --> I[GitLab incident issue]
  C --> API[Explicit Duo Flows API]
  API --> D[Registered Duo diagnose and repair flow]
  D --> MC[Official MCP stdio client and server]
  MC -->|Read-only scoped token| C
  D --> MR[Incident-linked repair MR]
  MR --> V[GitLab tests and security gates]
  V --> C
  C -->|Exact verified SHA merge| B[Protected main pipeline]
  B --> CB[Cloud Build and Artifact Registry]
  CB --> SB[SBOM and immutable digest]
  SB --> TF[Bounded app Terraform plan]
  TF --> CAN[10% Cloud Run canary]
  CAN --> H[120 probes and independent revision metrics]
  H --> G{Mode and outcome gate}
  O[Separate operator approval] --> G
  G --> PROM[Promote or restore stable traffic]
  PROM --> C
  C --> UI[Public read-only dashboard]
```

## Nine-stage loop

Monitor authenticates Google's Pub/Sub JWT, checks subscription, policy label, project, service and revision, and fetches actual Cloud Logging evidence. Plan creates a GitLab incident with a root-cause hypothesis. Govern reserves a limited attempt and captures the trusted policy hash. Create starts the enabled catalog consumer through the documented Flows API only after verifying the caller's `/api/v4/user` ID equals the configured licensed human; privilege fields are omitted so GitLab applies the flow's default full set. Duo uses an official MCP SDK handshake to read scoped evidence. The executor supplies its short-lived composite GitLab OAuth token, and the controller verifies the caller with GitLab's MCP `get_user` tool. Duo then reproduces the bug, writes a general repair and new regression test, and opens a related MR.

Verify and Secure check canonical GitLab APIs, exact MR SHA, successful MR pipeline, required jobs, bounded diff and a real Duo/MCP execution trace. The MR must come from the enabled Duo service account and target protected `main`. The controller merges that SHA once. The issue remains open. Package verifies incident release eligibility before building, scans the image, generates a CycloneDX SBOM and records the digest. Configure applies only permissible resource updates to the separate app state. Release pins stable traffic, creates an immutable canary revision, tests that revision, independently checks request counts and errors, and either promotes or rolls back. Only confirmed promotion closes the incident.

### Release remediation

A merged repair can still be blocked at Package, for example by advisories in the container base image. The controller reads the failed `package` job from GitLab, records `Package · blocked`, and classifies the job's image scan. When every blocking finding is an OS package, it is a base-image blocker. If a separate remediation flow is registered (`DUO_REMEDIATION_CONSUMER_ID`), the controller reserves one remediation attempt and starts it. That Duo agent may change only `FROM` lines in `Dockerfile.app`, and its MR carries `aegisflow-remediation::KEY`, so the repair evaluator never counts it as a repair attempt.

A remediation commit inherits the incident's release authorization only at Package time, inside `/api/release/SHA`, and only if all of these hold against canonical GitLab APIs ([policies/release-remediation.json](../policies/release-remediation.json)):

- the verified merge SHA is an ancestor of the commit, and the commit is on the protected release branch;
- `httpbin/`, `cmd/aegis-app/`, `go.mod`, `go.sum`, `infra/tuning.json`, both licences and `policies/recovery.json` are byte-identical (only HTTP 404 counts as absent, so transport errors fail closed);
- `Dockerfile.app` differs only in `FROM` lines, each matching an allowlisted image;
- every required recovery job plus `base_image_scan` succeeded in that commit's protected pipeline;
- the incident has adopted fewer than two remediation commits.

The ledger then records `Package · remediated` with a compare link. Package, Configure, canary and supervised approval all run again for the new SHA; the recovery policy hash is unchanged, so `scripts/release.py` still verifies it. `base_image_scan` runs the Package image gate on the runtime base image in every MR pipeline, so the same blocker stops an MR before merge.

## Authorities and credentials

| Identity | Authority | Excluded authority |
| --- | --- | --- |
| Application runtime SA | Execute stateless public HTTP application | No GCP IAM roles, secrets or deployment access |
| Google Monitoring publisher | Publish to the incident topic | No GitLab or Cloud Run deployment access |
| Pub/Sub push SA | JWT identifying push origin | No controller mutation credential beyond authenticated alert route |
| Scheduler SA | JWT for reconciliation route | No GitLab/Cloud Run administration |
| Controller SA | Firestore, logging/monitoring reads, app-only Cloud Run viewer, six specific secrets | No Cloud Run deploy or project IAM administration |
| Controller GitLab identity | Project API, incidents, canonical reads, gated merge; flow initiation only if API identity equals configured licensed user | A bot token is rejected for flow initiation; no token is passed to the agent workspace |
| Licensed GitLab user | Starts the Duo flow; owns the human side of composite identity for MCP reads | User token stays in Secret Manager and is never passed to the agent workspace |
| Duo composite service account | Developer role: diagnose, push repair branch, open MR | No protected-main push/merge, operator token, CI report token or GCP release credential |
| MCP reader | Read incident evidence and policy | No approval, merge, release or file-write MCP tools |
| Protected CI release SA | WIF, app-only Run developer, required build/storage/registry capabilities | No foundation state or project IAM administration |
| Cloud Build SA | Build bucket, registry writes, build logs | No Cloud Run deployment or secrets |
| Operator | Approve exact validated SHA using separate token | Approval does not bypass tests, scope, image or traffic gates |

Use a dedicated hackathon GCP project: project-level log/metric reads and Cloud Build permissions are broader than per-service roles. Foundation state and app state are separate; CI accesses only app state. Resource changes cannot alter identities, secrets, policies, traffic, image ownership or other services.

Scope release CI variables to `aegisflow/*` environments, protect them and keep them out of ambient agent jobs. WIF additionally requires the exact namespace/project, protected main, a release environment, and a user different from the Duo service account. This is essential: a protected variable with scope `*` could be inherited by an ambient flow started on the default branch. The custom executor image does not ship GitLab's SRT sandbox; runner isolation and narrow credentials are necessary and no SRT enforcement is claimed.

## State, retries and evidence

Firestore transactions reserve attempts and deduplicate concurrent alert deliveries. Re-notifications for the same alert, or another active alert on the same revision, reuse the incident. At most three nonterminal incidents and one release slot are allowed. Failed MR pipelines can consume a second attempt once; repeated webhooks for the failed MR cannot consume it again. At most 32 component cycles per attempt and 30 minutes of release-eligible agent time are allowed. Late MRs pause for operator reconciliation; this does not assert an unsupported vendor stop API.

A lost issue/flow/merge write response stops automatic retries because the server may already have acted. Polling canonical APIs repairs missing webhook delivery; webhook payloads never authorize code by themselves. A failed evidence lookup pauses rather than launching an uninformed agent. Missing native scanner reports, insufficient metrics, changed MR heads and out-of-policy plans fail closed.

Each event has sequence, UTC time, stage, status, links, previous hash and SHA256. Issue notes carry hash checkpoints. Hash chaining detects changes but is **not immutable storage**: an administrator able to replace the whole ledger could recompute it. GitLab issue notes, protected artifacts, Cloud Audit Logs and bucket versioning provide independent corroboration. No WORM retention claim is made.

Terminal release acknowledgements are replay-safe. The controller verifies actual Run traffic and revision image rather than trusting the CI report alone. A rollback leaves the incident open and records the restored revision. HTTP liveness is distinct from functional health; 4xx expected by invalid-input tests are not treated as 5xx failures.

## Public and private evidence

The dashboard exposes real state, safe stage descriptions, revision/digest identifiers and evidence links. It never embeds raw failure logs or credentials. Private incident evidence is served only to the scoped MCP reader. Scanner artifacts should stay visible to authorized reviewers where needed; public artifact publishing must be reviewed for diagnostic input. Both services can be public, but mutation routes require independent JWT/token authentication.

## Operational limits

This is a bounded hackathon design, not a general production incident commander. Arbitrary GitLab Developer actions cannot be eliminated by prompt text; server-side branch/environment protections and project isolation must be configured. Uncertain mutations, exhausted budgets, provider/API incompatibility, stale plans, missing telemetry and crash-at-merge cases intentionally require reconciliation. The one-document ledger has an 850 KB cap and events per incident are bounded; archive resolved incidents before reaching the limit. Keep operator approval timely: a validated canary remains at 10% while supervised promotion waits, and promotion always revalidates health.

Regression files are new per incident: `httpbin/aegisflow_regression_INCIDENT_KEY_test.go`. The MR regression job executes those specific new test functions against the merge base and repaired head; skipped tests and compilation failures do not count as before/after proof. This is enforced separately from the preserved upstream suite.

If promotion health and traffic are confirmed but the final controller acknowledgement is lost, CI saves `promoted_pending_ack` and fails for reconciliation. It does not blindly undo a possibly recorded resolution. Evidence POSTs have bounded replay retries; exact terminal acknowledgements are idempotent.