An Azure platform for running a service on AKS with Terraform, Argo CD, CI, observability and incident handling.
View the Project on GitHub Franklindot04/production-platform-Azure
This document describes the intended design of the platform. It is the first version.
Built today: the repository setup (templates, contribution rules, license, security policy, decision records), the infrastructure/ folders with two Terraform roots for dev (the persistent root is applied and defines the resource group, the Azure Monitor and Log Analytics workspaces, Application Insights, the action group, the four burn-rate alerts, the Key Vault and the identities for pull request plans and for the sample service; the session root defines a virtual network, a node subnet and an AKS cluster, and was applied once in the first create; the session layer was destroyed afterwards; the Azure network policy and the ingress add-on were added to the session root afterwards and are planned, not applied), the bootstrap script and the state backend it created in Azure (a resource group, a storage account and the tfstate-dev container), workflows that check Terraform, plan Terraform, check the service, chart and manifests, run the security scans, publish the image and open the pull request for its tag (all have run on GitHub; the Terraform plan workflow plans the two dev roots on pull requests from this repository), and the code of the sample service, its Helm chart, the cluster rules, the GitOps layout, a script that checks them in a local kind cluster, and the bootstrap step of the session script, which installs Argo CD (run against kind, not yet on AKS). Everything else in this document is planned. Of the dev roots, the persistent layer exists in Azure. Azure creates two things around it: the resource group MA_pp-dev-amw_swedencentral_managed for the Monitor workspace, and an action group named “Application Insights Smart Detection” in the persistent resource group.
Where a choice is not made, the text says so. Where a choice is made but not yet tested, it is marked as an assumption. Decisions are recorded as ADRs in docs/adr/ and listed in section 14.
Tracking: this document is issue #15 under Epic 1 (#1).
The design covers:
Design limit:
swedencentral. Section 7 describes its effects.Assumptions:
swedencentral. The state storage account is in northeurope. Resource prefix is pp. (ADR 0003)Unknowns:
Section 7 gives the status of each component. Nothing is running now.
The platform has two Terraform layers:
Both layers exist once per environment. The state storage is shared by all environments and sits outside both layers. The diagram shows one environment. See section 9 and ADR 0010.
flowchart LR
maint["Maintainer"] -->|"push and pull request"| repo["GitHub repository"]
repo --> actions["GitHub Actions"]
actions -->|"OIDC token"| entra["Microsoft Entra ID"]
actions -->|"build and push image"| acr["Container registry (ghcr.io)"]
maint -->|"terraform apply and destroy"| tf["Terraform"]
state["State storage (shared by all environments)"]
subgraph persistent ["Persistent layer"]
kv["Key Vault"]
mon["Azure Monitor and Log Analytics workspaces"]
end
subgraph session ["Session layer: created and destroyed each session"]
vnet["Virtual network"]
aks["AKS cluster"]
argo["Argo CD"]
ingress["Ingress"]
svc["Sample service"]
otel["OpenTelemetry Collector and metrics agent"]
end
tf --> state
tf --> persistent
tf --> session
aks --- vnet
argo -->|"pulls desired state"| repo
argo -->|"applies"| svc
aks -->|"pulls images"| acr
svc -->|"reads secrets with workload identity"| kv
ingress --> svc
svc --> otel
otel --> mon
mon -->|"alert rules"| notify["Email notification"]
The status column is built for a component that exists, written for a component whose code is in the repository and has not run on AKS, and planned for the rest. A built component of the session layer exists only while a session runs.
| Component | Role | Layer | Epic | Status |
|---|---|---|---|---|
| Terraform | Creates all Azure resources | n/a | 2 (#2) | built |
| State storage | Holds Terraform state | shared | 2 (#2) | built |
| Virtual network | Network for nodes | session | 2 (#2) | built |
| Container registry | Stores images | outside Azure (ghcr.io) |
4 (#4) | planned |
| Key Vault | Holds secrets | persistent | 6 (#6) | built |
| AKS cluster | Runs workloads | session | 3 (#3) | built |
| Ingress | Routes outside traffic to the service | session | 3 (#3) | written |
| Namespaces, quotas, network policy | Cluster rules | session | 3 (#3) | written |
| Sample service | The workload measured by SLOs | session | 3 (#3) | written |
| GitHub Actions | Build, test, scan, publish | n/a | 4 (#4) | built |
| Argo CD | Syncs cluster state from Git | session | 3 (#3) | written |
| Azure Monitor and Log Analytics | Metric and log storage, alert rules | persistent | 5 (#5) | built (workspaces, Application Insights, action group and the four burn-rate alerts) |
| OpenTelemetry Collector | Collects traces and metrics | session | 5 (#5) | planned |
| Dashboards | Saved queries on the workspaces, no Grafana | persistent | 5 (#5) | planned |
| Pull request plan identity | Reads Azure and state so a pull request can run terraform plan |
persistent | 6 (#6) | built |
| Policy checks | Block unsafe changes | n/a | 6 (#6) | built (Trivy scans in the Security scans workflow) |
| Runbooks and postmortems | Incident handling | n/a | 7 (#7) | runbooks written, three exercised in Session 2 (docs/runbooks/); incident process and postmortem template written, one postmortem written (docs/incident-process.md, docs/postmortems/) |
The target subscription has a regional vCPU quota of 4 in swedencentral. This is a design limit. It limits the plan:
The first AKS plan and the result of the first create are in ADR 0012. AKS accepted two Standard_D2s_v5 nodes. The allocatable capacity is 3.8 vCPU and about 11.3 GiB, and the tools fit (session 1 results). The saved plans are in docs/plans/.
The folders docs/, infrastructure/, applications/, kubernetes/, helm/, gitops/ and tests/ exist today. monitoring/ and security/ are planned, and names may change. The rule for the three folders that overlap:
kubernetes/ holds plain manifests for cluster-level resources, such as namespaces and quotas.helm/ holds charts written in this repository.gitops/ holds Argo CD Application definitions and per-environment values. It points to the other two.Other folders: infrastructure/ for Terraform, applications/ for the sample service, tests/ for checks that need no Azure, monitoring/ for dashboards, alert rules and SLO definitions, security/ for policies and scanning configuration.
Planned flow from a code change to a running pod. Section 6 shows where each piece sits.
flowchart TD
pr["Pull request"] --> checks["Checks: format, validate, lint, tests, scans"]
checks --> review["Review and merge to main"]
review --> build["Build image"]
build --> push["Push image to registry"]
push --> bump["Update image tag in gitops/"]
bump --> sync["Argo CD detects the change and syncs"]
sync --> health["Health checks"]
health -->|"healthy"| done["Running"]
health -->|"unhealthy"| revert["Revert the commit in Git"]
revert --> sync
Notes:
Infrastructure changes follow a separate path: pull request, plan, review, merge, then apply from the maintainer’s machine (assumption, ADR 0007).
Promotion between environments is described in section 9.
The environments are dev, staging and prod. They are separated by resource group inside one subscription. The decision is ADR 0010, which is proposed. All of it is planned. Only dev is created first. staging and prod are defined here and are not created until they are needed.
| Resource group | Holds | Created |
|---|---|---|
pp-shared-state-rg |
State storage account in northeurope, by a manual bootstrap step |
once |
pp-<env>-persistent-rg |
Key Vault, monitoring workspaces | per environment |
pp-<env>-session-rg |
Virtual network, AKS cluster | per environment, each session |
pp-<env>-aks-nodes-rg |
AKS node resources, named by Terraform | with the cluster |
Names follow ADR 0003. shared takes the place of <env> for the state storage only. The environment resource groups are in swedencentral.
tfstate-<env>.persistent.tfstate and session.tfstate. (ADR 0004).tf files, because .gitignore excludes *.tfvars.env tag, so cost can be filtered by environment. The dev environment has a budget with alerts. It is set in Azure and is not defined in Terraform. Budgets for staging and prod are planned. (ADR 0009)dev, then staging, then prod. Each step waits until the previous environment is checked.gitops/<env>/, in the same order. A workflow opens the pull request that changes the image tag (ADR 0018).The end-of-session teardown in ADR 0009 applies to every environment, including prod. Keeping prod running between sessions means revisiting ADR 0009.
prod is a name for the most protected environment. It is not a service with uptime.The access to the Terraform state, the Key Vault and the identities of the persistent layer are built. The session settings are written and not applied.
infrastructure/environments/dev/persistent/identity.tf and is applied. Its federated credential subject is repo:<owner>@<owner-id>/<name>@<repo-id>:pull_request, and it has Reader on the persistent resource group and Storage Blob Data Reader on the tfstate-dev container. (ADR 0022)pp-<env>-sample-service is defined in infrastructure/environments/dev/persistent/workload-identity.tf with Key Vault Secrets User on the vault, and is applied. The cluster settings (OIDC issuer, workload identity, the Key Vault secrets provider add-on) and the federated credential for the service account apps/sample-service are in the session root and are applied with the session layer. The chart mounts one secret as a file when vaultSecret.enabled is true. The bootstrap part identity turns it on at run time and no ID is written in Git. In Session 1 the chart mount and the bootstrap part ran, and both sample service pods mounted the secret. (ADR 0029)sample-service-secret is set in the vault. (ADR 0022, ADR 0029) They are not committed. .gitignore covers common secret files, and GitHub secret scanning and push protection are on for this repository.Argo CD generates its admin credential when it starts. It is read at run time and is not stored (ADR 0017).
The persistent layer (workspaces, Application Insights, action group, alerts) is applied. The rest is planned. See ADR 0008.
monitoring/. There is no Grafana. (ADR 0019)The budget of dev is set in Azure. The rest is planned. No cost has been measured. See ADR 0009.
Plan:
dev is set in Azure and is not defined in Terraform.staging and prod and cost views per environment are planned. (ADR 0010)Main cost drivers to check in the Azure pricing calculator, at the time of use:
The target subscription has a regional vCPU quota of 4, which limits node sizes and counts (section 7).
| Risk | Effect | Mitigation |
|---|---|---|
| Cluster left running after a session | Unplanned cost. Quota stays in use, so the next cluster cannot be created | End-of-session resource check. The budget of dev has alerts, set in Azure. |
| Budget exceeded | Unplanned cost | A budget with alerts is set for dev in Azure and is not defined in Terraform. Budgets for staging and prod are planned. Resources are tagged. The session layer is destroyed after each session. (ADR 0009) |
| Regional vCPU quota of 4 | A cluster over 4 vCPU cannot be created. Upgrades and node replacement need spare quota | Size the cluster within 4 vCPU. Do not upgrade in place. The system pool in ADR 0012 breaks an AKS guideline. AKS accepted it in the first create. An upgrade or a node replacement is not tested. |
| State lost or corrupted | Infrastructure cannot be managed from code | Versioning and soft delete on the state container. |
| Key Vault name held after deletion | Cannot re-create it | Keep Key Vault in the persistent layer. |
| Public API server | Larger attack surface | Authorized IP range, Entra ID authentication. |
| Shared subscription: quota, budget, subscription-level failure | All environments affected together | Run one environment at a time. Revisit separate subscriptions if the project outgrows this model. (ADR 0010) |
| Shared state storage account | State of every environment lost or exposed | Versioning and soft delete, a container per environment, Entra ID access only. (ADR 0004, ADR 0010) |
One maintainer, and admin bypass on main |
No second reviewer | Follow the pull request flow in CONTRIBUTING.md anyway. |
| Leaked credential in a public repository | Account access | Secret scanning, push protection, no stored secrets. Report as in SECURITY.md. |
| Tool versions drift | Builds break | Pin versions in Terraform and in pipelines when they exist. |
| ADR | Title | Status |
|---|---|---|
| 0002 | Record decisions as ADRs | accepted |
| 0003 | Choose Azure region and resource naming convention | proposed |
| 0004 | Use Azure Storage as the Terraform state backend | proposed |
| 0005 | Choose the AKS networking model | proposed |
| 0006 | Choose the GitOps tool | proposed |
| 0007 | Authenticate GitHub Actions to Azure with OIDC | proposed |
| 0008 | Choose the metrics and dashboard stack | proposed |
| 0009 | Set cost guardrails and teardown policy | proposed |
| 0010 | Separate environments by resource group | proposed |
| 0011 | Design the session network | proposed |
| 0012 | Plan the first AKS cluster | proposed |
| 0013 | Use GitHub Container Registry for images | proposed |
| 0014 | Write the sample service in Go | proposed |
| 0015 | Use Azure network policy | proposed |
| 0016 | Choose the ingress | proposed |
| 0017 | Bootstrap Argo CD from the session script | proposed |
| 0018 | Update the image tag with a pull request | proposed |
| 0019 | Choose the trace backend and Grafana hosting | proposed |
| 0020 | Choose policy and scanning tools | proposed |
| 0021 | Sign in to the cluster with Entra ID | proposed |
| 0022 | Define the persistent layer and the pull request plan identity | proposed |
| 0023 | Serve metrics on the admin port | proposed |
| 0024 | Enable managed Prometheus on the cluster | proposed |
| 0025 | Send traces with the Azure Monitor exporter and a connection string | proposed |
| 0026 | Trace the sample service with the OpenTelemetry SDK | proposed |
| 0027 | Keep dashboard queries as files under monitoring/ | proposed |
| 0028 | Set SLOs for the sample service and alert on burn rate | proposed |
| 0029 | Give the sample service access to Key Vault with workload identity | proposed |
| 0030 | Set the incident process and write postmortems | proposed |
Other open decisions are listed in section 15.
staging created, and what check must dev pass before a change moves on? (ADR 0010)staging and prod use? (ADR 0009)