An Azure platform for running a service on AKS with Terraform, Argo CD, CI, observability and incident handling.
View the Project on GitHub Franklindot04/production-platform-Azure
Status: planned. None of the work below is done, except where an issue is marked closed.
The plan covers the Kubernetes platform, delivery, observability, security, incident handling and reliability work. It is built around two limits: the cluster exists only during a live session, and the target subscription has a regional vCPU quota of 4 in swedencentral (ADR 0003).
docs/sessions/session-1-results.md):
kind, helm lint, helm template and kubeconform. kind does not show the AKS load balancer, Azure CNI Overlay behaviour or whether network policy is enforced there.main, so a pull request that depends on another waits for it to merge.| Area | Decision | Record |
|---|---|---|
| Registry | ghcr.io, public package |
ADR 0013 |
| Sample service | Go, health endpoints, settable latency and error rate | ADR 0014 |
| Network policy | Azure’s own engine, set in the session Terraform | ADR 0015 |
| Ingress | The lighter of the AKS add-on or one controller, chosen from the documentation. HTTP on the plain IP, no domain, no certificate | ADR 0016 |
| Bootstrap | A bootstrap step in session.sh installs Argo CD with Helm and a root Application. No third Terraform root |
ADR 0017 |
| Argo CD | Part of Epic 3. Epic 4 covers CI. The admin credential is generated, read at run time and never stored. The UI is reached by port-forward only | ADR 0017 |
| First image | Published by a GitHub Actions workflow | ADR 0018 |
| Image tag update | A workflow opens a pull request that changes the tag | ADR 0018 |
| Traces and dashboards | No paid managed Grafana. Dashboards are queries in the Azure Monitor workspace. The trace backend is chosen in the ADR | ADR 0019 |
| Logs | No cluster-wide log collection. Logs are read with kubectl logs |
ADR 0019 |
| Policy and scanning | Tools chosen in the ADR | ADR 0020 |
| Cluster sign-in | Entra ID and Azure RBAC, local accounts kept as a fallback | ADR 0021 |
| Load generator | k6 from the maintainer’s machine | Issue #49 |
| Alert email | A run-time input, never committed | Issue #37 |
Epic 3, Kubernetes Platform (#3). Issues #28 to #31 and #36 are closed.
| Issue | Title |
|---|---|
| #28 | Decision records for the Kubernetes platform |
| #29 | Sample service in Go |
| #30 | Chart, cluster rules and GitOps layout |
| #36 | Network policy engine and ingress in the session Terraform |
| #31 | Argo CD bootstrap in session.sh |
| #32 | Live session 1: converge and measure |
Epic 4, CI/CD and GitOps (#4):
| Issue | Title |
|---|---|
| #33 | CI for the service, chart and manifests |
| #34 | Image publish and tag update |
| #35 | OIDC identity for pull request plans |
Epic 5, Observability and SRE (#5):
| Issue | Title |
|---|---|
| #37 | Persistent monitoring layer |
| #38 | Metrics |
| #39 | Traces |
| #40 | Dashboards as code |
| #41 | SLIs, SLOs and alerts |
Epic 6, Security and Governance (#6):
| Issue | Title |
|---|---|
| #42 | Key Vault and workload identity |
| #43 | Least-privilege roles and secret rules |
| #44 | Policy and scanning checks |
| #45 | Entra ID access to the cluster |
Epic 7, Incident Management and Recovery (#7):
| Issue | Title |
|---|---|
| #46 | Runbooks |
| #47 | Incident process and postmortem template |
| #48 | Measured recovery |
Epic 8, Reliability and Optimization (#8):
| Issue | Title |
|---|---|
| #49 | Load tests |
| #50 | Failure experiments and live session 2 |
| #52 | Cost review |
| #51 | Documentation site and releases |
Numbering continues from ADR 0012.
| ADR | Title |
|---|---|
| 0013 | Use GitHub Container Registry for images |
| 0014 | Write the sample service in Go |
| 0015 | Use Azure network policy |
| 0016 | Choose the ingress |
| 0017 | Bootstrap Argo CD from the session script |
| 0018 | Update the image tag with a pull request |
| 0019 | Choose the trace backend and Grafana hosting |
| 0020 | Choose policy and scanning tools |
| 0021 | Sign in to the cluster with Entra ID |
| PR | Content | Issues | Needs |
|---|---|---|---|
| 1 | This plan and the run sheets | none | none |
| 2 | ADR 0013 to 0021 and the architecture document corrections | #28 | 1 |
| 3 | Sample service, chart, cluster rules, GitOps layout, kind checks |
#29, #30 | 2 |
| 4 | CI, image publish, tag update workflow | #33, #34 | 3 |
| 5 | Session Terraform (policy engine, ingress) and the bootstrap step |
#36, #31 | 3 |
| 6 | Persistent layer and identities: workspaces, Key Vault, OIDC | #37, #42, #35 | 2 |
| 7 | Observability code: scrape configuration, OpenTelemetry, dashboards, SLOs, alerts | #38, #39, #40, #41 | 3, 6 |
| 8 | Security: scans, role table, secret rules, Entra ID access | #43, #44, #45 | 4, 6 |
| 9 | Session 1 record, screenshots, fixes | #32 | 3 to 8 |
| 10 | Runbooks and incident process | #46, #47 | 9 |
| 11 | Load tests, experiments, Session 2 record, recovery time, cost review | #48, #49, #50, #52 | 9, 10 |
| 12 | Documentation site, changelog, release notes | #51 | 11 |
If Session 1 fails a gate or time runs out, the work is cut in this order:
Cluster-wide log collection and a managed Grafana are already out of scope.
A release is created when the maintainer approves it.
| Tag | After |
|---|---|
| v0.1.0 | Foundation and Azure infrastructure (Epics 1 and 2, done) |
| v0.2.0 | PR 3 to 5, checked on kind |
| v0.3.0 | Session 1: the GitOps flow and the first observability data |
| v0.4.0 | Security and identity (PR 8) |
| v0.5.0 | SLOs and alerts proven |
| v0.6.0 | Runbooks and incident process (PR 10) |
| v0.7.0 | Load tests and failure experiments (Session 2) |
| v0.9.0 | Cost review and the documentation site |
| v1.0.0 | All epics closed and the documents consistent |