# Deployment and recovery runbook

The public GitLab project and initial Cloud Run services now exist. The current deployment evidence is recorded in `evidence/cloud-deployment.json`. The controller dashboard is public, but recovery must stay disabled until its health-route discrepancy and real Duo execution are resolved.

## 1. Access and prerequisites

Install Git, Python 3.12+, Go 1.27.3 and the [official Google Cloud CLI](https://cloud.google.com/sdk/docs/install). Create a private local secret directory outside the repository, with directory permissions 0700. Authenticate with `gcloud auth login` and `gcloud auth application-default login`. Supply a Google Cloud project ID. The human bootstrap identity needs service enablement, IAM, storage, Artifact Registry, Cloud Build, Cloud Run, Firestore, Monitoring, Pub/Sub, Scheduler and Secret Manager administration in that dedicated project. Runtime identities are narrowed by Terraform afterwards.

Supply the GitLab hackathon namespace/project URL. The target must have runners, AI Catalog/custom flow enablement, sufficient credits/plan, identity verification and an eligible composite service account. A project-scoped controller bot with `api` scope and Maintainer role is needed for gated merges; the enabled Duo service account has Developer role. Set the protected recovery branch so the Duo account cannot push or merge. Human CODEOWNERS can manage automation, CI, IAM and policy changes.

Put the GitLab API credential in a private local file and set `GITLAB_TOKEN_FILE` to its path. The controller checks `/api/v4/user` against `DUO_USER_ID` before using that credential to start Duo; a project access token or other bot identity fails closed. The live token currently resolves to the project owner's human account. The credential remains in Secret Manager and is never passed to the agent workspace. Never paste token contents or put them in Git remotes. The public project is `circuitsai/aegisflow`. Its custom Duo flow is enabled and has completed a real workflow. Consumer ID `13154292` and group service-account ID `43726501` are available to the controller.

```sh
python3 -m venv .venv
.venv/bin/pip install --require-hashes -r requirements.txt -r requirements-dev.txt
.venv/bin/python -m scripts.install_tools terraform gitleaks syft trivy
```

The installer pins versions, verifies official SHA256 checksums and writes only `.tools/`. Both Terraform provider lock files are supplied. Commit the reviewed project before container builds; do not commit credentials, state or plans.

## 2. Publish and configure GitLab

The delivery includes a Git checkout with its original baseline and upstream remote. For a new public project in an explicitly selected namespace, run:

```sh
export GITLAB_TOKEN_FILE=/absolute/private/path/gitlab-token
.venv/bin/python -m scripts.publish --create-in-namespace NAMESPACE_ID
```

Alternatively provide an existing **empty** target using `--project PROJECT_ID`. The helper refuses a nonempty project and never force-pushes. Preserve the original baseline in GitLab history. Set the default branch explicitly; this deployment uses `aegisflow-source`. Set `RECOVERY_BRANCH` and Terraform `gitlab_default_branch` to that same branch. Replace `@aegisflow-maintainers` in CODEOWNERS with the real maintainers group. Protect the actual default branch, require successful pipelines, disallow force pushes, and restrict protected-branch push/merge to Maintainers/controller bot. Do not give the Duo service account a protected-branch exception. Configure Code Owner approvals if supported by the selected plan.

Run GitLab CI Lint against the target so native includes are expanded, then a baseline pipeline with `AEGIS_RECOVERY_ENABLED=false`. Confirm `semgrep-sast`, `secret_detection`, independent scanner jobs, `security_gate`, unit, controller, integration and regression jobs actually ran successfully. A local YAML parse cannot prove remote CI compatibility. Native dependency scanning may be enabled only after supported-language/entitlement checks; independent dependency scans are always mandatory.

Use `.venv/bin/python -m scripts.run_pipeline --ref aegisflow-source` to start the real baseline pipeline after publishing. No cloud release job is authorized for an arbitrary baseline commit: the controller must recognize a verified incident merge SHA. Initial application/controller deployment is a distinct operator bootstrap.

## 3. Foundation and secrets

Use nonsecret IDs from the target project and parent namespace. All bootstrap invocations must use the same project, namespace, region and mode. The foundation creates the Artifact Registry repository, bounded runtime identities, Firestore, state/build buckets, secret containers, alert resources and WIF provider. It first creates Google's Pub/Sub service identity; the Monitoring notification channel creates its own publisher identity before its IAM binding.

```sh
.venv/bin/python -m scripts.bootstrap foundation \
  --project GCP_PROJECT_ID --gitlab-project GITLAB_PROJECT_ID \
  --gitlab-namespace GITLAB_NAMESPACE_ID --region us-central1
```

Foundation state is in `GCP_PROJECT_ID-aegisflow-bootstrap/foundation`, with bucket versioning. App state is in the separate bucket output by Terraform. Existing Firestore default database conflicts must be reconciled explicitly; do not destroy an existing database to satisfy this bootstrap.

Create these six files in the private directory, each 0600:

| Filename | Value and scope |
| --- | --- |
| `gitlab-token` | GitLab API credential for incident operations and Duo start; human identity must match `DUO_USER_ID` |
| `gitlab-duo-user-token` | Licensed human user's GitLab API token; used only by the controller to start the real Duo flow |
| `gitlab-webhook` | Random independent GitLab webhook secret |
| `mcp-read` | Random independent read-only credential for local CLI/verification; never sent to a Duo workload |
| `ci-report` | Random independent CI evidence credential |
| `operator` | Random independent exact-SHA approval credential |

Use a password manager or secure local generation. Secret versions are uploaded through file paths. Secret values never enter Terraform variables/state. Do not reuse credentials across roles. Rotate and revoke them after the hackathon.

## 4. Build and activate services

```sh
.venv/bin/python -m scripts.bootstrap services \
  --project GCP_PROJECT_ID --gitlab-project GITLAB_PROJECT_ID \
  --gitlab-namespace GITLAB_NAMESPACE_ID --region us-central1 \
  --secrets-dir /absolute/private/secret-directory
```

This Cloud Build operation uses the dedicated build identity and excludes secrets/state from source upload. It deploys the healthy application by immutable digest, applies app-only state, and activates the controller/notification subscription/scheduler. `reports/bootstrap.json` contains actual immutable images, build IDs and app/dashboard URLs; no placeholders are reported as URLs. Run functional HTTPS checks on the app and review `/api/dashboard` on the controller. Recovery CI remains disabled until the early Duo proof is complete.

The controller is public to serve the dashboard; its mutations require Google JWTs or independent tokens. The dashboard and `/health` return HTTP 200; `/health` reports whether integrations are configured. Cloud Run reserves some paths ending in `z`, so public probes use `/health`. This was a reserved-path issue rather than a source/image mismatch. The application is a bounded public HTTP testing service with a zero-role runtime identity. Configure Google budget alerts separately; instance limits bound capacity, not total spend.

### Redeploy only the controller

Use this for controller or dashboard changes after they merge. It never rebuilds or redeploys the app, so a merged repair still reaches production only through its gated release pipeline.

```sh
git checkout aegisflow-source && git pull
.venv/bin/python -m scripts.install_tools terraform
.venv/bin/python -m scripts.bootstrap controller \
  --project GCP_PROJECT_ID --gitlab-project GITLAB_PROJECT_ID \
  --gitlab-namespace GITLAB_NAMESPACE_ID --branch aegisflow-source --mode supervised \
  --consumer CONSUMER_ID --duo-actor SERVICE_ACCOUNT_ID --duo-user DUO_USER_ID
```

It builds `Dockerfile.control` at the committed HEAD, plans only `google_cloud_run_v2_service.controller[0]`, and refuses the plan if it touches another resource or changes anything but the controller image. Pass the same IDs and mode the controller already runs with; a mismatch shows up as a refused plan, not a silent change. The receipt is `reports/controller-deploy.json`.

## 5. Register the genuine Duo flow

The live custom flow is published at https://gitlab.com/explore/ai-catalog/flows/1019723 and enabled for the project. Its consumer is `13154292`; the group service account is `43726501`. A live start created workflow `10023097` and workload `3994756`, and both agents finished. This test made no source changes. The controller still needs its configuration revision to serve those IDs; verify `/api/dashboard` reports `configured: true` after deployment.

Add the exact deployed controller hostname to `.gitlab/duo/agent-config.yml` allowed domains, and commit it to the protected default branch through a reviewed change. Register `.gitlab/duo/flows/recovery.yaml` as a custom flow in AI Catalog and enable it for this project. Keep the custom flow definition aligned with the committed version. Assign the composite service account Developer, with no protected-branch bypass. Record its numeric actor ID.

The ambient workload receives GitLab's short-lived composite OAuth token as `DUO_WORKFLOW_GIT_HTTP_PASSWORD`; the bundled stdio MCP client uses it for its authenticated evidence read. The controller validates it through GitLab's authenticated `/api/v4/user` endpoint and accepts only the pinned initiating human or configured composite service-account identity. This avoids passing an AegisFlow secret or CI variable to the agent. The flow sets the public controller URL inline and uses `.agent-venv/bin/python -m automation.mcp_client KEY`, which performs real MCP initialize/list/call steps. Workspace MCP configuration remains for supported CLI/IDE clients; ambient auto-loading is not assumed. Session 10053574 performed a successful MCP read for real Monitoring incident 304cc0cbdc423fcf and authored MR !1. The repair component made a redundant MCP call without the controller URL and failed; the diagnostic component’s successful tool result supplies the real evidence. Keep the URL explicit in client commands.

Retrieve the enabled consumer through the documented GraphQL query:

```graphql
query {
  aiCatalogConfiguredItems(projectId: "gid://gitlab/Project/PROJECT_ID") {
    nodes { id item { name } }
  }
}
```

Use the numeric suffix of `gid://gitlab/AiCatalogItemConsumer/ID`, not the catalog item or version ID. Configure the controller:

```sh
.venv/bin/python -m scripts.bootstrap flow \
  --project GCP_PROJECT_ID --gitlab-project GITLAB_PROJECT_ID \
  --gitlab-namespace GITLAB_NAMESPACE_ID --consumer CONSUMER_ID \
  --duo-actor DUO_SERVICE_ACCOUNT_ID --duo-user 11911697
```

Set local `GITLAB_PROJECT_ID`, `GOOGLE_CLOUD_PROJECT`, `DUO_CONSUMER_ID`, `DUO_SERVICE_ACCOUNT_ID`, `DUO_USER_ID`, `GITLAB_TOKEN_FILE`, `AEGIS_CONTROLLER_URL` and `AEGIS_MCP_TOKEN_FILE`. The flow-start API omits optional privilege filters so the registered flow receives GitLab's default full privilege set; the project policy, flow prompts, protected CI and exact-SHA release gates provide the bounds. Run `python -m scripts.prove_duo` and require a real session/workload plus successful MCP trace before recovery is enabled.

### Optional: release-remediation flow

Register `.gitlab/duo/flows/release-remediation.yaml` as a second custom flow and enable it for the project. Run the `flow` phase with the usual IDs plus `--remediation-consumer CONSUMER_ID`, and review its plan. Adding the environment variable is a deliberate configuration change, which the image-only controller phase refuses by design. Without it, a base-image blocker is recorded and classified, and an operator can open the same bounded remediation MR. See "Release remediation" in ARCHITECTURE.md for the checks a remediation commit must pass.

## 6. Prove the real alert-to-Duo path early

Set up the GitLab webhook at `CONTROLLER_URL/events/gitlab`, enabling MR and pipeline events with the independent `gitlab-webhook` secret, SSL verification on. Scheduler also polls canonical APIs, so webhook delivery loss cannot approve a stale MR.

Create an isolated fault checkout from the reviewed AegisFlow source. Run `python -m scripts.inject_fault /absolute/fault-checkout`; existing tests must fail. Commit the disclosed demonstration regression as a controlled operator setup on the protected default branch through the project's authorized process, with recovery release jobs disabled. Do not weaken tests to make this commit green. Build its `Dockerfile.app` using Cloud Build and deploy the resulting digest to a tagged controlled fault revision with zero production traffic. Trigger the reproducible request against the tagged URL; retain 100% healthy production traffic during the exercise. Record the source commit, build, digest and traffic. This intentional incident setup is distinct from the agent-authored repair. AegisFlow has no predefined fix.

Send a real `GET /base64/decode/!!!` to the deployed app. The adapter should return 500 and emit `application_failure`; liveness can still be healthy. Inspect the actual log, Monitoring incident and Pub/Sub push. Confirm one GitLab incident and one genuine Duo session/workload. Then run:

```sh
.venv/bin/python -m scripts.prove_duo --incident REAL_INCIDENT_KEY
```

Do not proceed until the actual AI trace shows a successful official MCP read for the matching incident, not just a command suggestion. Inspect the issued session URL and agent comment/diagnosis. Check actual Monitoring notification labels against the supported schema; if revision scoping is absent, leave recovery blocked and adapt the integration with real evidence.

## 7. Configure protected release jobs

Set public/nonsecret CI variables: `GOOGLE_CLOUD_PROJECT`, `GCP_REGION`, `APP_SERVICE`, `AEGIS_CONTROLLER_URL`, `WIF_PROVIDER`, `RELEASE_SERVICE_ACCOUNT`, `BUILDER_SERVICE_ACCOUNT`, `RUNTIME_SERVICE_ACCOUNT`, `APP_STATE_BUCKET`, and `AEGIS_MODE`. Use actual Terraform outputs, not invented service-account names.

Set `CI_REPORT_TOKEN` masked/hidden and protected, with **environment scope `aegisflow/*`**. The cloud jobs have matching environments; ambient agent jobs do not. Do not scope this credential to `*`. Do not store controller GitLab or operator credentials in project CI variables. Do not permit arbitrary pipeline-variable overrides from Developers. Disable CI debug tracing. WIF requires project/namespace/the protected recovery branch/release environment and excludes the Duo actor.

Protect the `aegisflow/production` environment and authorize only the intended release identity/operators where supported. Confirm the chosen plan supports the necessary protections. Set `AEGIS_RECOVERY_ENABLED=true` only after early Duo proof and after checking the trusted CI configuration. Select `supervised` consistently in controller Terraform and CI for the first run.

If the early proof run has already merged its verified repair while recovery jobs were disabled, start a new pipeline with `.venv/bin/python -m scripts.run_pipeline --ref aegisflow-source --sha EXACT_VERIFIED_MERGE_SHA` after enabling them. Do not replay against a changed default-branch head.

The genuine repair MR must preserve all existing tests, add a new `TestAegisRegression*`, demonstrate red/green behavior, and link `Related to #INCIDENT_IID`. “Closes” is intentionally avoided: merge is not recovery. A successful MR pipeline and real Duo/MCP trace authorize an exact-SHA merge; the default-branch pipeline then packages and deploys the canary.

## 8. Canary and outcome approval

The default 10% canary receives at least 120 functional probes across healthy and malformed-input cases over at least 60 seconds, with revision headers checked. The controller independently reads Cloud Monitoring counts/error rate and actual Run revision image/traffic. Insufficient telemetry causes rollback. Tagged probes exercise the exact revision while 10% allocation exposes it to service traffic. p95 is the measured functional-probe latency, not a claim about all-user latency.

In supervised mode, inspect the MR, package digest/SBOM, Terraform gate and canary evidence. Then, from a trusted local shell with the separate operator file:

```sh
export AEGIS_OPERATOR_TOKEN_FILE=/absolute/private/path/operator
.venv/bin/python -m scripts.approve REAL_INCIDENT_KEY EXACT_MERGE_SHA
```

The controller approves only the validated canary's exact SHA and plays the manual promotion job when available; reconciliation handles a job that becomes manual later. A manual GitLab click alone does not bypass controller approval. Promotion re-runs health/metric validation, confirms 100% actual traffic and performs final functional probes before recording resolution and closing the incident.

Hands-off mode uses `--mode hands_off` in Terraform and matching CI `AEGIS_MODE=hands_off`. It automatically promotes only after the same gates. Never enable it by relaxing policies, security reports or protected branches.

## Reconciliation and rollback

If canary health, metrics, image or traffic validation fails, the release script restores the recorded stable revision to 100%, checks actual allocation and records rollback. If rollback itself fails, the CI job fails and the incident requires intervention; never mark it successful. The incident remains open on rollback/exhaustion.

For uncertain issue creation, Duo starts or merges: inspect canonical GitLab resources and the private ledger first. Do not reset status or rerun writes blindly. An API response can be lost after a successful action. Late MRs, flow deadlines, exhausted attempts and blocked release slots require a reviewed operator reconciliation that appends evidence rather than erasing history. A lost terminal release acknowledgement can safely replay the exact same report.

Do not destroy infrastructure to reset a demo. Keep state snapshots private. Foundation/app deletion protection and Firestore abandon semantics preserve audit data. Before eventual teardown, revoke tokens, disable alert/scheduler/subscription, export required evidence and deliberately review any deletion plan.

Regression files are new per incident: `httpbin/aegisflow_regression_INCIDENT_KEY_test.go`. The MR regression job executes those specific new test functions against the merge base and repaired head; skipped tests and compilation failures do not count as before/after proof. This is enforced separately from the preserved upstream suite.


## Integration corrections verified on 2026-10-26

Secret files and environment-mounted values are trimmed identically. Upload normalized values to Secret Manager; formatting newlines must not enter HTTP headers or shared-token comparisons. No credential values belong in receipts.

GitLab composite identities attribute the MR to the initiating human while the agent-created pipeline identifies the service account. The controller checks both identities, exact commit provenance, and successful MCP/MR tool results from the reserved session. It never accepts an agent's narrative claim as execution proof. Native scanners run in MR pipelines with `AST_ENABLE_MR_PIPELINES=true`.

For a blocked, unmerged MR after an integration correction, an authenticated operator can POST `/api/reconcile-mr` with `key`, numeric `iid`, and exact `sha`. The controller verifies the ledger chain, unchanged policy, canonical GitLab resources, and session proof before appending a bounded review window. It retains all prior failures and does not reset attempts or start another agent. This is separate from supervised production approval.