11. Rollout
11. Rollout
← Previous: 10. Audit and Evidence | Index | Next: 12. Rationale →
11.1 Principle
Additive until the last step. Every phase leaves the existing login path working; each phase is verified before the next; the shared key is removed only at cutover, with a break-glass drill proven first. The team keeps shipping throughout (G-10).
11.2 Phases
| Phase | What happens | Gate to proceed | Rollback |
|---|---|---|---|
| 0 — Review | This RFC reviewed by the Platform Lead | Approval | — |
| 1 — Azure prerequisites (no host changes) | Terraform: pn-host-* Entra groups; pn_class tags; all 16 Linux hosts in arc_machines (production pair + collector receive the extension and SSH endpoint); class → group → role map; PIM policy on pn-host-admin; Graph read permission per host identity. Sentinel: DCRs for auth / sudo / auditd / pn-access / pn-deploy; the Section 10 alerts. Existing grants untouched. | Plan reviewed; apply clean | destroy the new resources; nothing old changed |
| 2 — uat pilot | Ansible roles (pn-access, pn-deploy + dc, host access policy) applied to uat only. The Editors and the Deployer use it for real against the acceptance list (11.3). Old path still works. | Acceptance list passes; pilot users say yes | remove the roles |
| 3 — All other classes, Arc-only from here | Same roles to prod, erp, data, k8s, pve, infra (non-deploy classes receive Operator/Admin only). Same day: retire the RG-wide POL-Developer grant and the direct user grants; fold POL-DataServers into the class groups — matrix-checked so nobody loses access they need. | Matrix check passes | re-apply legacy grants via Terraform |
| 4 — Cutover (point of no return) | Remove the shared mfaizan key and all personal keys from every host; remove humans from docker; enable the su lock; confirm every break-glass key is in Key Vault; run a deliberate break-glass drill and prove the alert fires. | Phases 1–3 accepted; Security owner and Platform Lead go | break-glass key from Key Vault restores any host |
| 5 — Steady state | Monthly access review; quarterly break-glass drill; 60-day decision on Level-2 recording; Evolution items | — | — |
Wall-clock is driven by review and pilot feedback, not by soak time: Phase 1 is hours; Phase 2 is a real working session by the pilot users; Phase 3 follows their yes; Phase 4 one to two days later.
11.3 uat Acceptance List
| Check | Expected |
|---|---|
Operator sudo apt install … | runs, no prompt, I/O-logged in Sentinel |
Operator su -, sudo -u <other>, sudo bash | refused; denial visible in Sentinel |
Editor dc up -d --build <svc> in /srv/pnats/<project> | deploys; attributed to the UPN; #infra note posted |
Deployer dc up <svc> | deploys |
Deployer write to /srv/pnats, read .env, docker ps | all refused |
PIM activation on pn-host-admin | requires MFA + reason; next login is Admin; expires at TTL |
| Remove a pilot user from a tier group | session terminated and groups stripped within one cycle (≤5 min); heartbeat shows the action |
| Lock a test account 30 days (simulated clock) | home archived, files re-owned, user removed |
| Break-glass login on uat | alert fires |
| Legacy key login still works during the pilot | yes (additive) |
11.4 Risks and Their Answers
| Risk | Answer |
|---|---|
| Lockout | staged rollout; break-glass per host tested before cutover |
| Graph outage | agent changes nothing (Invariant 10); Azure still gates logins |
PIM approver unavailable during a prod incident | second approver defined; break-glass as last resort; both logged |
pn-deploy defect blocks a deploy | Editor activates PIM and uses raw Docker for the window; wrapper fixed by PR |
| A host missing from the Terraform inventory | it has no human login (fail-closed) — visible immediately, fixed by PR |
11.5 Ownership
| Role | Owner |
|---|---|
| Azure / Terraform / PIM / Sentinel | Security owner (Faizan) |
| Ansible roles, review, CODEOWNER approval | Platform Lead (Noorullah) |
| Pilot users | Saif (Editor), Mujahid (Deployer) |
11.6 Definition of Done
All success criteria in Section 2.5 met; no host edited by hand; the access-map and break-glass runbooks required by STD-IDENT-001 exist; RFC status moved to Implemented.
Document Navigation
| Previous | Index | Next |
|---|---|---|
| ← 10. Audit and Evidence | Table of Contents | 12. Rationale → |
End of Section 11