Infrastructure as Code with Terraform — Deploy AWS Resources Like a Pro

The $4,200 Drift That Tanked the API Gateway

On a Tuesday at 03:14 UTC, the api-prod-internal endpoint started throwing 502 Bad Gateway errors across all zones in us-east-1. Our monitoring suite showed a total loss of connectivity between the application load balancer (ALB) and the private EC2 instances. Historically, this felt like a network ACL change or a security group rotation gone wrong.

The post-mortem revealed something more insidious: manual intervention. A senior engineer, responding to a perceived "urgent" ticket regarding database latency, had manually detached a subnet from the production VPC's routing table to "test a shortcut." They forgot to reattach it. Because our Terraform state was not being continuously enforced at the time, the infrastructure sat in a degraded, non-declarative state for 48 hours until a routine scaling event triggered a re-evaluation of the network path, leading to a total blackout.

This incident highlighted the "Terraform Gap"—the space between writing HCL (HashiCorp Configuration Language) and actually running an automated, immutable infrastructure lifecycle. If you are manually running terraform apply from your laptop, you aren't doing Infrastructure as Code; you’re doing "Infrastructure as Scripts," and you’re one local environment variable away from a disaster.

Anatomy of the Failure: State Deserialization

The root cause was rooted in the fragility of our terraform.tfstate. In this incident, the manual change created a delta that terraform plan would have caught, but since the pipeline only ran on PR merges, the drift remained invisible.

[Manual Change] -> [Terraform State Out of Sync] -> [Scaling Event] -> [Total Outage]
       |                      |                         |                |
   Dev wipes              Remote file              ASG tries to       Route not
   route table            remains old              spawn node         found

We found that the standard_vmdk module we used for EC2 instances lacked a lifecycle block to prevent accidental deletion of critical network interfaces. More importantly, we were not using S3 with DynamoDB for state locking, leading to a "split-brain" scenario where two different CI runners attempted to modify the same resource simultaneously during the recovery attempt.

To fix this, we had to move from "Click-Ops" debt to a hardened, modular architecture.

Architecting for Persistence: S3 Backends and DynamoDB Locking

The first remediation step was moving state out of the local filesystem. If your state file is on a disk that can be formatted, or in a Git repo where it can be leaked, you have already failed. A production-grade Terraform setup requires an encrypted S3 bucket with versioning enabled and a DynamoDB table for state locking.

```hcl
terraform {
  backend "s3" {
    bucket         = "corp-terraform-state-us-east-1"
    key            = "environments/prod/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-lock-table"
    encrypt        = true
  }

required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } }

provider "aws" { region = "us-east-1" default_tags { tags = { Project = "Phoenix" ManagedBy = "Terraform" Owner = "Platform-Team" } } } ```

By enforcing default_tags, we ensure that any resource created via this provider is immediately identifiable. During the incident, we wasted 20 minutes just trying to identify who "owned" a specific rogue subnet because it had no tags. In a professional environment, untagged resources should be treated as technical debt to be reaped by an automated script.

The Module Pattern: Encapsulating the VPC and RDS

The second contributing factor was our "monolith" Terraform file. We had 2,000 lines of HCL in a single main.tf. This made terraform plan operations take upwards of 4 minutes, leading to engineer fatigue and "skipped" plans.

We broke the infrastructure into discrete modules. We now treat the Network (VPC, Subnets, IGW) as a separate lifecycle from the Compute layer (EC2, ASG) and the Data layer (RDS, S3).

Hardened VPC Module A production VPC must be multi-AZ. If you are running in one Availability Zone to save on NAT Gateway costs, you are accepting a p99 latency spike or total outage during AWS zone-level brownouts.

```hcl
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.1.0"

name = "prod-vpc" cidr = "10.0.0.0/16"

azs = ["us-east-1a", "us-east-1b", "us-east-1c"] private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"] public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

enable_nat_gateway = true single_nat_gateway = false # Higher cost, but prevents single point of failure one_nat_gateway_per_az = true

manage_default_network_acl = true public_dedicated_network_acl = true } ```

The Database Barrier During the incident, we realized our RDS instance was accessible via the public internet due to a misconfigured `publicly_accessible = true` flag in a dev-to-prod copy-paste error. We mitigated this by enforcing `internal` security groups that only accept traffic from the application tier security group.

CI/CD Enforcement with GitHub Actions

The remediation for manual drift was to remove AdministratorAccess from human users and delegate it to a service principal used by GitHub Actions.

We implemented a workflow that requires a terraform plan output to be commented on every Pull Request. No one can merge unless the plan is clean and reviewed. This prevents the "I'll just fix it in the console" mentality because the OIDC-linked runner will simply overwrite manual changes on the next push.

We use OIDC (OpenID Connect) to authenticate to AWS, avoiding the need for long-lived AWS_ACCESS_KEY_ID secrets.

```yaml
name: 'Terraform Plan/Apply'

on: push: branches: [ "main" ] pull_request:

permissions: id-token: write # Required for OIDC contents: read pull-requests: write

jobs: terraform: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3

- name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v2 with: role-to-assume: arn:aws:iam::123456789012:role/github-actions-role aws-region: us-east-1

- name: Terraform Plan run: terraform plan -input=false -out=tfplan - name: Terraform Apply if: github.ref == 'refs/heads/main' && github.event_name == 'push' run: terraform apply -auto-approve tfplan ```

Policy as Code: Stopping the "Urgent" Manual Fix

To prevent a repeat of the "shortcut" that caused our 502 errors, we integrated trivy and checkov into the pipeline. These tools scan the HCL for security anti-patterns (e.g., S3 buckets without versioning, security groups with 0.0.0.0/0 on port 22).

However, the most important technical shift was moving to Atomic Commits for Infrastructure. We found that during the recovery, trying to fix the VPC and the RDS instance in one PR caused a circular dependency error (cycle detected).

The Hard-Won Lesson: Keep your state files small. We split our global state into five smaller state files: global (IAM/S3 buckets), network (VPC/Route53), shared-services (EKS clusters/MSK), app-tier (EC2/ALB), and data (RDS/ElastiCache). After this split, our terraform plan time dropped from over 4 minutes to a p99 of 22 seconds. This speed encouraged developers to run plans more frequently, catching drift hours before it became an outage.

Remediating the IAM "Wildcard" Problem

During the post-mortem, we discovered that the engineer who broke the VPC was able to do so because they had ec2:* permissions. In a professional environment, this is unacceptable. We moved to a "Least Privilege" model using Terraform to manage IAM Roles for sessions.

We implemented Permission Boundaries. Even if an engineer manages to escalate their local privileges, the Permission Boundary attached to their role prevents them from modifying critical networking components unless they are authenticated via the specific "Network-Admin" role, which requires Multi-Factor Authentication (MFA).

The Production Reality of "Destroy"

The most terrifying command is terraform destroy. In our remediation, we added prevent_destroy = true to the lifecycle block of every RDS instance and production S3 bucket.

resource "aws_db_instance" "prod_db" {
  # ... configuration ...
  lifecycle {
    prevent_destroy = true
  }
}

This acts as a final circuit breaker. If a bug in a script or a tired engineer tries to tear down the data layer, the AWS API call will never be sent because Terraform will error out locally. We learned this the hard way when a "cleanup script" targeted the wrong workspace and nearly nuked our customer metadata table.

Final Post-Mortem Insight

The Tuesday outage wasn't a failure of AWS, nor was it a failure of Terraform as a tool. It was a failure of workflow integrity.

Infrastructure as Code is only as strong as the human process surrounding it. By moving to a model of remote state locking, modularized resources, and CI/CD-enforced plans, we eliminated the possibility of "invisible" manual drift. We moved from a 120-minute Mean Time to Recovery (MTTR) down to a 5-minute automated rollback capability.

The sharpest takeaway: Treat your infrastructure code with more rigor than your application code. An application bug might crash a service, but an infrastructure bug can delete the company’s entire presence on the internet. If it isn't in a PR, it doesn't exist. If it was done in the console, it is a bug that must be paved over.