Module 06: Containers & Orchestration

Developer's laptop (Mac, Python 3.9, PostgreSQL 14):
Application works perfectly

Production server (Ubuntu, Python 3.7, PostgreSQL 12):
Application crashes with dependency errors

Docker Container:
├─ Your application code
├─ Python 3.9 runtime
├─ PostgreSQL 14 client
├─ All 50 system libraries
├─ All configuration files
└─ Ready to run in <1 second

Physical Server (32 GB RAM)
├─ VM 1: 8 GB RAM (Linux + App) - 60 second startup
├─ VM 2: 8 GB RAM (Linux + App) - 60 second startup
├─ VM 3: 8 GB RAM (Linux + App) - 60 second startup
└─ Total: 3 VMs, 24 GB used (8 GB wasted)

Physical Server (32 GB RAM)
├─ Host OS: 2 GB RAM (shared)
├─ Container 1: 512 MB (App only) - <1 second startup
├─ Container 2: 512 MB (App only) - <1 second startup
├─ ... 50 more containers ...
└─ Total: 50+ containers, 28 GB used (4 GB free)

Problems at scale:
1. Deploy time: 45 minutes for full deployment (all features blocked)
2. Build time: 60 minutes to compile entire monolith (every code change)
3. Database bottleneck: Single PostgreSQL instance, 10K queries/sec max
4. Team velocity: 600 engineers working on same codebase (merge conflicts daily)
5. Failure blast radius: Bug in one feature crashes entire app (200M users affected)
6. Scaling: Can only scale entire app vertically (limited to largest EC2 instance)

┌─────────────────────────────────────────────────────────────────┐
│                     Spotify Monolith                             │
│                     (Rails application)                          │
│                                                                  │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌──────────┐ │
│  │   User     │  │  Playlist  │  │  Search    │  │  Playback││ │
│  │   Auth     │  │  Manager   │  │  Engine    │  │  Engine  ││ │
│  └────────────┘  └────────────┘  └────────────┘  └──────────┘ │
│                                                                  │
│  All features in single codebase                                │
│  - 2.5M lines of code (Ruby, JavaScript)                        │
│  - 600 engineers committing to same repo                        │
│  - 45-minute deployment (entire app deployed together)          │
│  - Single database (PostgreSQL, 10TB data)                      │
│                                                                  │
│  Hosted on: 500× m3.xlarge EC2 instances                        │
│  - 4 vCPU, 15 GB RAM each                                       │
│  - Load balanced (Round-robin)                                  │
│  - No isolation (all features on all instances)                 │
└─────────────────────────────────────────────────────────────────┘
         │
         │ All requests handled by monolith
         ▼
┌─────────────────────────────────────────────────────────────────┐
│              PostgreSQL Database (single instance)               │
│              - 10 TB data                                        │
│              - 10,000 queries/second maximum                     │
└─────────────────────────────────────────────────────────────────┘

                        ┌─────────────────────────────────────┐
                        │       Spotify API Gateway            │
                        │       (Kong, 100K req/sec)           │
                        └─────────────┬───────────────────────┘
                                     │
                 ┌───────────────────┼───────────────────┬─────────┐
                 │                   │                   │         │
                 ▼                   ▼                   ▼         ▼
        ┌─────────────────┐  ┌────────────────┐  ┌──────────┐  ┌────────┐
        │ User Service     │  │Playlist Service│  │  Search  │  │Playback│
        │ (Auth, Profile)  │  │ (CRUD, Share)  │  │  Service │  │Service │
        │                  │  │                │  │          │  │        │
        │ Docker Image:    │  │ Docker Image:  │  │ Docker:  │  │Docker: │
        │ user-svc:v2.1.3  │  │ playlist:v1.8  │  │search:v3 │  │play:v5 │
        │                  │  │                │  │          │  │        │
        │ Instances:       │  │ Instances:     │  │Instances:│  │Inst:   │
        │ 500 pods         │  │ 1,200 pods     │  │300 pods  │  │800 pods│
        │                  │  │                │  │          │  │        │
        │ Resources:       │  │ Resources:     │  │Resources:│  │Res:    │
        │ 2 CPU, 4 GB RAM  │  │ 1 CPU, 2 GB    │  │4 CPU, 8GB│  │2 CPU   │
        └─────────┬────────┘  └────────┬───────┘  └─────┬────┘  └───┬────┘
                 │                     │                │           │
                 │ Owns database       │ Owns database  │ Owns DB   │ Owns DB
                 ▼                     ▼                ▼           ▼
        ┌─────────────────┐  ┌────────────────┐  ┌──────────┐  ┌────────┐
        │User DB           │  │Playlist DB     │  │Search DB │  │Events  │
        │(PostgreSQL)      │  │(Cassandra)     │  │(Elastic) │  │(Kafka) │
        │500 GB            │  │50 TB           │  │20 TB     │  │10 TB   │
        └──────────────────┘  └────────────────┘  └──────────┘  └────────┘

Total: 4,000+ microservices, 50,000+ Docker containers, 200,000+ Kubernetes pods
Each service: Independent deployment, scaling, database, team ownership

Virtual Machines (EC2):
┌────────────────────────────────────────────────────────┐
│                    Host Server                          │
│  ┌──────────────────────────────────────────────────┐  │
│  │              Hypervisor (ESXi, KVM)              │  │
│  └──────────────────────────────────────────────────┘  │
│  ┌──────────────┐  ┌──────────────┐  ┌─────────────┐  │
│  │   VM 1       │  │   VM 2       │  │   VM 3      │  │
│  │              │  │              │  │             │  │
│  │ ┌──────────┐ │  │ ┌──────────┐ │  │ ┌─────────┐ │  │
│  │ │Guest OS  │ │  │ │Guest OS  │ │  │ │Guest OS │ │  │
│  │ │(Linux)   │ │  │ │(Linux)   │ │  │ │(Windows)│ │  │
│  │ │1-2 GB    │ │  │ │1-2 GB    │ │  │ │3-4 GB   │ │  │
│  │ └──────────┘ │  │ └──────────┘ │  │ └─────────┘ │  │
│  │ ┌──────────┐ │  │ ┌──────────┐ │  │ ┌─────────┐ │  │
│  │ │   App    │ │  │ │   App    │ │  │ │   App   │ │  │
│  │ └──────────┘ │  │ └──────────┘ │  │ └─────────┘ │  │
│  └──────────────┘  └──────────────┘  └─────────────┘  │
└────────────────────────────────────────────────────────┘

Resource overhead:
- Each VM: 1-4 GB RAM for guest OS (wasted)
- Boot time: 30-60 seconds
- Density: 10-20 VMs per host (limited by RAM)

Docker Containers:
┌────────────────────────────────────────────────────────┐
│                    Host Server                          │
│  ┌──────────────────────────────────────────────────┐  │
│  │         Host OS (Linux kernel)                   │  │
│  └──────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────┐  │
│  │         Docker Engine                            │  │
│  └──────────────────────────────────────────────────┘  │
│  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐       │
│  │Container│  │Container│  │Container│  │Container│ ... │
│  │    1    │  │    2    │  │    3    │  │   50    │     │
│  │ ┌────┐  │  │ ┌────┐  │  │ ┌────┐  │  │ ┌────┐  │     │
│  │ │App │  │  │ │App │  │  │ │App │  │  │ │App │  │     │
│  │ └────┘  │  │ └────┘  │  │ └────┘  │  │ └────┘  │     │
│  │50-500 MB│  │50-500 MB│  │50-500 MB│  │50-500MB │     │
│  └────────┘  └────────┘  └────────┘  └────────┘       │
└────────────────────────────────────────────────────────┘

Resource efficiency:
- Shared kernel: No guest OS overhead (0 GB wasted)
- Boot time: < 1 second
- Density: 50-100+ containers per host (10× more than VMs)
- RAM savings: 50 containers × 1.5 GB saved = 75 GB per host

# Spotify Playlist Service Dockerfile
# Multi-stage build for smaller image size

# Stage 1: Build stage (compile application)
FROM maven:3.9.4-eclipse-temurin-17 AS builder

# Set working directory
WORKDIR /app

# Copy dependency files first (Docker layer caching optimization)
COPY pom.xml .
COPY src/main/resources/dependencies.txt .

# Download dependencies (cached layer if pom.xml unchanged)
RUN mvn dependency:go-offline

# Copy source code
COPY src ./src

# Build application (creates JAR file)
RUN mvn clean package -DskipTests

# Stage 2: Runtime stage (final image)
FROM eclipse-temurin:17-jre-alpine

# Install curl for health checks
RUN apk add --no-cache curl

# Create non-root user (security best practice)
RUN addgroup -g 1000 spotify && \
    adduser -D -u 1000 -G spotify spotify

# Set working directory
WORKDIR /app

# Copy JAR from build stage
COPY --from=builder /app/target/playlist-service-*.jar app.jar

# Copy configuration
COPY config/application.yml config/

# Change ownership to non-root user
RUN chown -R spotify:spotify /app

# Switch to non-root user
USER spotify

# Expose port
EXPOSE 8080

# Health check (Kubernetes uses this)
HEALTHCHECK --interval=30s --timeout=3s --start-period=60s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

# Start application
ENTRYPOINT ["java", \
    "-Xms1g", "-Xmx2g", \
    "-XX:+UseG1GC", \
    "-XX:MaxGCPauseMillis=200", \
    "-Dspring.profiles.active=production", \
    "-jar", "app.jar"]

# Build Docker image
docker build -t spotify/playlist-service:v1.8.0 .

# Build process:
# Step 1/15: FROM maven:3.9.4-eclipse-temurin-17 AS builder
#  → Pulling base image (650 MB)
# Step 2/15: WORKDIR /app
#  → Creating directory
# Step 3/15: COPY pom.xml .
#  → Copying dependency file (5 KB)
# Step 4/15: RUN mvn dependency:go-offline
#  → Downloading dependencies (200 MB, takes 2 minutes first time)
#  → CACHED on subsequent builds if pom.xml unchanged (0 seconds)
# Step 5/15: COPY src ./src
#  → Copying source code (50 MB)
# Step 6/15: RUN mvn clean package
#  → Compiling application (5 minutes first time, 30 seconds cached)
# Step 7/15: FROM eclipse-temurin:17-jre-alpine
#  → Switching to smaller runtime image (200 MB)
# Step 8-15: Copy JAR, configure, set user
#  → Final image: 250 MB

# Total build time:
# - First build: 10 minutes (download deps + compile)
# - Cached builds: 30 seconds (only recompile changed code)

# Savings from multi-stage:
# - Single-stage: 900 MB (includes Maven + dependencies)
# - Multi-stage: 250 MB (only runtime + JAR)
# - Reduction: 72% smaller (650 MB saved per image)

# Login to Amazon ECR (Spotify uses ECR)
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

# Tag image for ECR
docker tag spotify/playlist-service:v1.8.0 \
  123456789012.dkr.ecr.us-east-1.amazonaws.com/playlist-service:v1.8.0

# Push to registry
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/playlist-service:v1.8.0

# Push stats:
# - Image size: 250 MB
# - Upload speed: 100 MB/sec (AWS network)
# - Push time: 2.5 seconds
# - Cost: $0.10/GB-month storage (250 MB × $0.10 / 1024 = $0.024/month per image)

# Run single container
docker run -d \
  --name playlist-service \
  --memory=2g \
  --cpus=1.0 \
  -p 8080:8080 \
  -e DATABASE_URL=postgres://playlist-db:5432/playlists \
  -e KAFKA_BROKERS=kafka-1:9092,kafka-2:9092 \
  123456789012.dkr.ecr.us-east-1.amazonaws.com/playlist-service:v1.8.0

# Container starts in < 1 second
# Application ready in ~30 seconds (Spring Boot startup)

# Check logs
docker logs -f playlist-service
# Output:
# Starting PlaylistServiceApplication...
# Connecting to database postgres://playlist-db:5432/playlists
# Connecting to Kafka kafka-1:9092
# Started PlaylistServiceApplication in 28.3 seconds

# Check health
curl http://localhost:8080/health
# {"status": "UP", "database": "UP", "kafka": "UP"}

# Container stats
docker stats playlist-service
# CONTAINER         CPU %   MEM USAGE / LIMIT   MEM %   NET I/O
# playlist-service  25%     1.5 GB / 2 GB       75%     10 MB / 5 MB

# Kubernetes Deployment manifest
# Declares desired state: "Run 1,200 replicas of playlist-service:v1.8.0"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: playlist-service
  namespace: production
  labels:
    app: playlist-service
    version: v1.8.0
spec:
  replicas: 1200  # Spotify runs 1,200 instances of this service
  
  selector:
    matchLabels:
      app: playlist-service
  
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # Deploy 300 new pods at a time (1200 × 0.25)
      maxUnavailable: 10%  # Max 120 pods down during rollout (1200 × 0.10)
  
  template:
    metadata:
      labels:
        app: playlist-service
        version: v1.8.0
    spec:
      # Pod runs on node with sufficient resources
      affinity:
        podAntiAffinity:  # Don't schedule multiple pods on same node
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - playlist-service
              topologyKey: kubernetes.io/hostname
      
      containers:
      - name: playlist-service
        image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/playlist-service:v1.8.0
        
        # Resource requests (guaranteed)
        resources:
          requests:
            cpu: 1000m      # 1 CPU core
            memory: 2Gi     # 2 GB RAM
          limits:
            cpu: 2000m      # Max 2 CPU cores (burst)
            memory: 4Gi     # Max 4 GB RAM (OOMKilled if exceeded)
        
        # Environment variables
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: playlist-db-credentials
              key: url
        - name: KAFKA_BROKERS
          value: kafka-1.kafka.svc.cluster.local:9092
        
        # Health checks (Kubernetes auto-restart if failed)
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60  # Wait 60s for app startup
          periodSeconds: 30        # Check every 30 seconds
          timeoutSeconds: 5        # 5-second timeout
          failureThreshold: 3      # Restart after 3 failures
        
        # Readiness check (don't route traffic until ready)
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        
        # Ports
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        
        # Volume mounts (optional, for shared storage)
        volumeMounts:
        - name: logs
          mountPath: /app/logs
      
      volumes:
      - name: logs
        emptyDir: {}  # Temporary storage, deleted when pod dies

# Deploy to Kubernetes
kubectl apply -f playlist-service-deployment.yaml

# Kubernetes reconciliation loop:
# 1. Current state: 0 pods running
# 2. Desired state: 1,200 pods running
# 3. Actions: Create 1,200 pods (scheduled across 150 nodes, 8 pods/node average)

# Watch rollout
kubectl rollout status deployment/playlist-service -n production

# Output:
# Waiting for deployment "playlist-service" rollout to finish: 0 of 1200 updated replicas are available...
# Waiting for deployment "playlist-service" rollout to finish: 300 of 1200 updated replicas are available...
# Waiting for deployment "playlist-service" rollout to finish: 600 of 1200 updated replicas are available...
# Waiting for deployment "playlist-service" rollout to finish: 900 of 1200 updated replicas are available...
# deployment "playlist-service" successfully rolled out
# Total time: 8 minutes (1,200 pods × 30 seconds startup ÷ 300 parallel = 480 seconds)

# Verify pods running
kubectl get pods -n production -l app=playlist-service

# Output (truncated):
# NAME                               READY   STATUS    RESTARTS   AGE
# playlist-service-7d4f6b8c9-abc12   1/1     Running   0          8m
# playlist-service-7d4f6b8c9-def34   1/1     Running   0          8m
# playlist-service-7d4f6b8c9-ghi56   1/1     Running   0          8m
# ... (1,197 more pods)

# Total: 1,200 pods running across 150 EC2 nodes

Monolith (2014):
- EC2: 500× m3.xlarge ($0.266/hour × 500 × 730 hours) = $97,090/month
- Database: PostgreSQL db.m5.24xlarge ($11.52/hour × 730) = $8,410/month
- Load balancers: 10× ALB ($22.50 × 10) = $225/month
- Total infrastructure: $105,725/month = $1.27M/year

- Staff: 50 SREs managing monolith ($180K/year × 50) = $9M/year
  - Rationale: Monolith complex, manual deploys, frequent incidents
  
- Total: $1.27M + $9M = $10.27M/year

Microservices (2024):
- EC2: 150× c5.9xlarge ($1.53/hour × 150 × 730) = $167,445/month = $2.01M/year
- Kubernetes control plane: Managed EKS ($0.10/hour × 10 clusters × 730) = $730/month = $8,760/year
- Container registry: ECR ($0.10/GB × 1 TB × 12) = $1,200/year
- Service mesh (Istio): Self-hosted on existing nodes = $0
- Databases: 50× PostgreSQL + Cassandra instances ($200K/month) = $2.4M/year
- Load balancers: 50× ALB ($22.50 × 50) = $1,125/month = $13,500/year
- Total infrastructure: $167,445/month = $2.01M/year + $2.4M databases = $4.41M/year

- Staff: 30 SREs + 20 platform engineers ($180K × 50) = $9M/year
  - Fewer SREs (automation), more platform engineers (build Kubernetes platform)
  
- Total: $4.41M + $9M = $13.41M/year

Cost comparison:
- Monolith: $10.27M/year
- Microservices: $13.41M/year
- Additional cost: $3.14M/year (31% more expensive)

Value gained:
- Deploy velocity: 150× more deploys/day (300 vs 2)
- Time to market: 6× faster (2 weeks vs 3 months)
- Feature velocity: 16× more features/year (2,400 vs 150)
- Availability: 10× fewer outages (99.95% vs 99.5%)
- Revenue impact: +$500M/year (faster feature releases, better UX, higher retention)

ROI: $500M revenue ÷ $3.14M additional cost = 159× return

Requests/second: 120K (typical weekday)
Orders/hour: 114K (2.74M ÷ 24)
Kubernetes pods: 15,000 running
CPU utilization: 35% average
Revenue/hour: $26.7M (GMV, $235B annual ÷ 8,760 hours)

Requests/second: 3.5M (29× normal, 3,500% increase)
Orders/hour: 11M (96× normal, 44M orders ÷ 4 peak hours)
Kubernetes pods: 500,000 running (33× normal)
CPU utilization: 82% average (near capacity)
Revenue/hour: $2.3B (GMV, 86× normal)

┌─────────────────────────────────────────────────────────────────────────┐
│                        CONTROL PLANE                                     │
│                   (Manages cluster state)                                │
│                                                                          │
│  ┌──────────────────┐  ┌──────────────────┐  ┌────────────────────┐   │
│  │   API Server     │  │   Scheduler      │  │  Controller        │   │
│  │  (kube-apiserver)│  │ (kube-scheduler) │  │  Manager           │   │
│  │                  │  │                  │  │(kube-controller-   │   │
│  │ - REST API       │  │ - Assigns pods   │  │     manager)       │   │
│  │ - Authentication │  │   to nodes       │  │                    │   │
│  │ - Authorization  │  │ - Considers      │  │ - Deployment       │   │
│  │ - Validation     │  │   resources,     │  │   controller       │   │
│  │                  │  │   affinity       │  │ - ReplicaSet       │   │
│  │ Entry point for  │  │                  │  │   controller       │   │
│  │ all operations   │  │ Runs every 1s    │  │ - Service          │   │
│  │                  │  │                  │  │   controller       │   │
│  └────────┬─────────┘  └────────┬─────────┘  └──────────┬─────────┘   │
│           │                     │                        │             │
│           │   Reads/Writes      │   Watches for          │ Watches     │
│           │   cluster state     │   unscheduled pods     │ for desired │
│           │                     │                        │ state       │
│           ▼                     ▼                        ▼             │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │                         etcd                                     │  │
│  │                   (Distributed key-value store)                  │  │
│  │                                                                  │  │
│  │  Stores all cluster state:                                       │  │
│  │  - Pods: 500,000 pod definitions                                │  │
│  │  - Services: 50,000 service definitions                         │  │
│  │  - ConfigMaps, Secrets, etc.                                    │  │
│  │                                                                  │  │
│  │  High availability: 5 replicas (majority quorum = 3)            │  │
│  │  Size: ~50 GB (500K pods × 100 KB each)                        │  │
│  └─────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘
                                  │
                                  │ API Server exposes
                                  │ Kubernetes API
                                  ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                        WORKER NODES (DATA PLANE)                         │
│                    (Runs application containers)                         │
│                                                                          │
│  Node 1 (c5.4xlarge)     Node 2 (c5.4xlarge)    ... Node 5,000         │
│  ┌──────────────────┐   ┌──────────────────┐                           │
│  │    kubelet       │   │    kubelet       │   (5,000 nodes total      │
│  │  - Pod lifecycle │   │  - Pod lifecycle │    during Black Friday)   │
│  │  - Health checks │   │  - Health checks │                           │
│  │  - Reports to    │   │  - Reports to    │   Each node:              │
│  │    API Server    │   │    API Server    │   - 16 vCPU               │
│  └────────┬─────────┘   └────────┬─────────┘   - 32 GB RAM             │
│           │                      │             - ~100 pods/node         │
│           │                      │                                      │
│  ┌────────┴─────────┐   ┌───────┴──────────┐                          │
│  │  Container       │   │  Container       │                           │
│  │  Runtime         │   │  Runtime         │                           │
│  │  (containerd)    │   │  (containerd)    │                           │
│  └────────┬─────────┘   └───────┬──────────┘                          │
│           │                      │                                      │
│           │ Runs pods            │ Runs pods                            │
│           ▼                      ▼                                      │
│  ┌────────────────────┐ ┌────────────────────┐                        │
│  │ Pod 1    Pod 2     │ │ Pod 101  Pod 102   │                        │
│  │ ┌────┐   ┌────┐    │ │ ┌────┐   ┌────┐   │                        │
│  │ │App │   │App │    │ │ │App │   │App │   │                        │
│  │ └────┘   └────┘    │ │ └────┘   └────┘   │                        │
│  │ ... (98 more)      │ │ ... (98 more)      │                        │
│  └────────────────────┘ └────────────────────┘                        │
│                                                                          │
│  Total pods: 5,000 nodes × 100 pods = 500,000 pods                     │
└─────────────────────────────────────────────────────────────────────────┘

# Pod definition (rarely used directly, usually via Deployment)
apiVersion: v1
kind: Pod
metadata:
  name: checkout-service-abc123
  namespace: production
  labels:
    app: checkout-service
    version: v2.5.1
spec:
  containers:
  - name: checkout
    image: shopify/checkout-service:v2.5.1
    resources:
      requests:
        cpu: 500m      # 0.5 CPU cores
        memory: 1Gi    # 1 GB RAM
      limits:
        cpu: 1000m     # Max 1 CPU core
        memory: 2Gi    # Max 2 GB RAM
    ports:
    - containerPort: 8080
    env:
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: checkout-db-creds
          key: url

# Service definition
apiVersion: v1
kind: Service
metadata:
  name: checkout-service
  namespace: production
spec:
  type: ClusterIP  # Internal cluster IP (not exposed to internet)
  selector:
    app: checkout-service  # Routes traffic to pods with this label
  ports:
  - port: 80         # Service port (what clients connect to)
    targetPort: 8080 # Pod port (container listening port)
    protocol: TCP
  
  # Service gets stable IP: 10.96.5.10
  # DNS name: checkout-service.production.svc.cluster.local

Client → Service (10.96.5.10:80) → Pod selection → Pod IP (10.244.3.15:8080)

Selection algorithm:
1. Service selector matches pods: app=checkout-service
2. Found 1,000 pods with that label (scaled for Black Friday)
3. Random selection: Pick pod-789 (10.244.3.15)
4. Forward traffic to pod-789:8080
5. If pod-789 fails health check: Remove from service, retry with pod-234

Result: Client doesn't know which pod served request (abstraction)
        Pod can be replaced, IP changes, but service IP (10.96.5.10) stays same

# Deployment definition (preferred way to run pods)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-service
  namespace: production
spec:
  replicas: 1000  # Desired state: 1,000 pods running
  
  selector:
    matchLabels:
      app: checkout-service
  
  # Update strategy
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # Create 250 new pods before deleting old (1000 × 0.25)
      maxUnavailable: 10%  # Max 100 pods unavailable during update (1000 × 0.10)
  
  template:
    metadata:
      labels:
        app: checkout-service
        version: v2.5.1
    spec:
      containers:
      - name: checkout
        image: shopify/checkout-service:v2.5.1
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 1000m
            memory: 2Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

1. kubectl apply -f checkout-deployment.yaml
   → API Server receives request

2. Deployment Controller watches API Server
   → Sees new Deployment: replicas=1000

3. Deployment Controller creates ReplicaSet
   → ReplicaSet: Maintain 1,000 pods with label app=checkout-service

4. ReplicaSet Controller watches API Server
   → Sees ReplicaSet: replicas=1000, current=0

5. ReplicaSet Controller creates 1,000 pod definitions
   → Pods in "Pending" state (unscheduled)

6. Scheduler watches for pending pods
   → Assigns each pod to a node (considers resources, affinity)

7. kubelet on each node watches API Server
   → Sees pod assigned to its node

8. kubelet pulls container image
   → Image: shopify/checkout-service:v2.5.1 (pulled from ECR)

9. kubelet starts container
   → Container running, pod status: "Running"

10. Health checks execute
    → Liveness probe: Check /health every 30 seconds
    → Readiness probe: Check /ready every 10 seconds
    → If both pass: Pod added to Service endpoints

11. Service controller updates endpoints
    → checkout-service now routes traffic to 1,000 pods

Total time: 5-10 minutes (1,000 pods × 30 seconds startup ÷ 100 parallel)

# HPA definition
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  
  minReplicas: 1000     # Normal traffic: 1,000 pods
  maxReplicas: 50000    # Black Friday peak: 50,000 pods
  
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70  # Scale when avg CPU > 70%
  
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80  # Scale when avg memory > 80%
  
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 100  # Scale when avg > 100 req/sec per pod
  
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60   # Wait 60s before scaling up again
      policies:
      - type: Percent
        value: 50          # Scale up by 50% at a time
        periodSeconds: 60  # Every 60 seconds
      - type: Pods
        value: 1000        # Or add 1,000 pods at a time (whichever higher)
        periodSeconds: 60
    
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 minutes before scaling down
      policies:
      - type: Percent
        value: 10          # Scale down by 10% at a time (cautious)
        periodSeconds: 60

Every 15 seconds (default):

1. Metrics Server collects metrics from all pods
   - CPU: Average 85% across 1,000 pods
   - Memory: Average 65%
   - HTTP requests: Average 150 req/sec per pod

2. HPA calculates desired replicas for each metric:
   
   CPU-based:
   desiredReplicas = currentReplicas × (currentCPU / targetCPU)
                   = 1000 × (85% / 70%)
                   = 1214 pods
   
   Memory-based:
   desiredReplicas = 1000 × (65% / 80%)
                   = 812 pods (no scaling, below target)
   
   Request-based:
   desiredReplicas = 1000 × (150 / 100)
                   = 1500 pods

3. HPA picks highest value: 1500 pods

4. Apply scale-up policy:
   - 50% of current: 1000 × 1.5 = 1500 pods
   - Or fixed 1000: 1000 + 1000 = 2000 pods
   - Use higher: 2000 pods

5. HPA updates Deployment: replicas: 1000 → 2000

6. Wait 60 seconds (stabilizationWindow) before next scale-up

Timeline: Black Friday ramp-up
T+0:00:00 - Traffic: 120K req/sec, Pods: 1,000
T+0:01:00 - Traffic: 250K req/sec, CPU: 90%, Scale to 2,000 pods
T+0:02:00 - Traffic: 450K req/sec, CPU: 85%, Scale to 3,000 pods
T+0:03:00 - Traffic: 750K req/sec, CPU: 80%, Scale to 5,000 pods
... (continue scaling)
T+1:30:00 - Traffic: 3.5M req/sec, CPU: 72%, Pods: 50,000 (max)
T+2:00:00 - Traffic: 3.2M req/sec, CPU: 68%, Stay at 50,000 (within stabilization)
T+6:00:00 - Traffic: 1.5M req/sec, CPU: 35%, Scale down to 25,000 (cautious 10% reduction)

November 24, 2023 (Black Friday):

08:00 AM EST - Pre-sale preparation
- Pods: 15,000 (normal load)
- Traffic: 120K req/sec
- CPU: 35% average
- HPA ready, monitoring every 15 seconds

10:00 AM EST - Early shoppers
- Pods: 50,000 (HPA scaled 3.3× in 2 hours)
- Traffic: 800K req/sec
- CPU: 65% average
- No incidents, smooth scaling

12:00 PM EST - Peak traffic
- Pods: 500,000 (HPA scaled 33× from baseline)
- Traffic: 3.5M req/sec
- CPU: 82% average
- Orders: 2.75M orders/hour (11M ÷ 4 peak hours)
- GMV: $2.3B/hour

02:00 PM EST - Sustained high traffic
- Pods: 450,000 (slight scale-down, traffic decreased)
- Traffic: 3.1M req/sec
- CPU: 78%

06:00 PM EST - Evening traffic
- Pods: 200,000 (scaled down 56%)
- Traffic: 1.5M req/sec
- CPU: 55%

11:59 PM EST - End of Black Friday
- Pods: 80,000 (scaled down to 5.3× baseline)
- Traffic: 400K req/sec
- Total orders: 44M (Black Friday 2023)
- Total GMV: $9.3B

Availability: 99.99% (52 minutes downtime/year target)
Actual uptime: 99.998% (10.5 minutes downtime)
- Incident 1: 3 minutes (database connection pool exhaustion, 12:15 PM)
- Incident 2: 4.5 minutes (etcd leader election, 1:30 PM)
- Incident 3: 3 minutes (ALB hitting connection limits, 3:45 PM)

Pod restarts: 15,000 (out of 500,000 pods, 3% restart rate)
- OOMKilled: 8,000 (exceeded memory limit, auto-restarted)
- CrashLoopBackOff: 2,000 (application bugs, manual fix deployed)
- Node failures: 5,000 (50 nodes crashed, pods rescheduled to healthy nodes)

Self-healing effectiveness:
- Mean Time To Detect (MTTD): 30 seconds (liveness probe interval)
- Mean Time To Recover (MTTR): 45 seconds (pod restart + readiness check)
- Total recovery: 75 seconds average (customers see retry, not error)

Failed nodes: 50 (out of 5,000, 1% failure rate)
- Hardware failure: 20 (disk failure, network card failure)
- Out of memory: 15 (node ran out of RAM, kubelet stopped responding)
- Out of disk: 10 (logs filled disk, node cordoned)
- Network partition: 5 (network issues, node unreachable)

Kubernetes response:
- Detected: 30 seconds (kubelet stopped reporting to API server)
- Evicted pods: 100 pods/node × 50 nodes = 5,000 pods
- Rescheduled: 5 minutes (scheduler assigned to healthy nodes)
- No customer impact: Pods redistributed, services continued

Infrastructure (Black Friday 24-hour period):

Kubernetes control plane:
- API Server: 10× c5.4xlarge (16 vCPU, 32 GB) = $3.40/hour × 10 × 24 = $816
- etcd: 5× i3.2xlarge (8 vCPU, 61 GB, NVMe) = $1.248/hour × 5 × 24 = $150
- Scheduler: 3× c5.xlarge (4 vCPU, 8 GB) = $0.17/hour × 3 × 24 = $12
- Controller Manager: 3× c5.xlarge = $12
Total control plane: $990/day

Worker nodes:
- Normal (08:00 AM): 150 nodes × c5.4xlarge ($3.40/hour) = $510/hour
- Peak (12:00 PM): 5,000 nodes × c5.4xlarge = $17,000/hour
- Average (24 hours): 2,000 nodes average = $6,800/hour × 24 = $163,200/day

Total infrastructure (Black Friday): $990 + $163,200 = $164,190

Revenue (Black Friday):
- GMV: $9.3B
- Shopify take rate: 2.9% average (payment processing + subscription)
- Revenue: $9.3B × 0.029 = $269.7M

ROI: $269.7M revenue ÷ $164K infrastructure = 1,643× return

Comparison to static provisioning (no auto-scaling):
- Static: 5,000 nodes × 24 hours = $17,000/hour × 24 = $408,000
- Auto-scaling: $163,200
- Savings: $408,000 - $163,200 = $244,800 (60% cost reduction)
- Wasted capacity: Without auto-scaling, 80% of capacity idle 20 hours/day

Problems without service mesh:

1. Load balancing logic in every service:
   - Home Feed service calls Recommendation service
   - Must implement: Retry logic, timeout handling, circuit breaker
   - 2,000 services × custom logic = 2,000 duplicate implementations
   - Bugs in retry logic caused cascading failures (retry storm)

2. Security: HTTP everywhere (no encryption)
   - Service A → Service B: Plaintext HTTP
   - Anyone on network can sniff traffic (internal attacker)
   - No mutual authentication (Service A can't verify Service B identity)

3. Observability gap:
   - Logs in each service (2,000 different log formats)
   - No unified tracing (can't follow request across 20 services)
   - Debugging: "Which service is slow?" → Check 2,000 services manually

4. Traffic management:
   - Canary deployments: Manual (engineer updates code in load balancer)
   - A/B testing: Requires code changes (if user.beta then new_service)
   - Circuit breaking: Each service implements independently (or not at all)

5. Configuration sprawl:
   - Service A needs to know Service B address (hardcoded or env var)
   - 2,000 services × 10 dependencies average = 20,000 configuration entries
   - Update Service B IP → Update 50 dependent services (coordination nightmare)

Without Service Mesh:
┌────────────────────────────────────────────────────────────────┐
│                         Kubernetes Pod                          │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │           Application Container                           │ │
│  │           (Home Feed Service)                             │ │
│  │                                                           │ │
│  │  - Implements own retry logic                            │ │
│  │  - Implements own circuit breaker                        │ │
│  │  - Implements own metrics                                │ │
│  │  - HTTP calls to other services                          │ │
│  └──────────────────────────────────────────────────────────┘ │
│                                                                 │
│  Application handles everything (complex, error-prone)          │
└────────────────────────────────────────────────────────────────┘
         │
         │ Direct HTTP call
         ▼
┌────────────────────────────────────────────────────────────────┐
│           Recommendation Service Pod                            │
└────────────────────────────────────────────────────────────────┘

With Service Mesh (Istio):
┌────────────────────────────────────────────────────────────────┐
│                         Kubernetes Pod                          │
│                                                                 │
│  ┌───────────────────────┐  ┌──────────────────────────────┐  │
│  │  Sidecar Proxy        │  │  Application Container       │  │
│  │  (Envoy)              │  │  (Home Feed Service)         │  │
│  │                       │  │                              │  │
│  │  - Intercepts traffic │◄─┤  - Simple business logic    │  │
│  │  - Retry logic        │  │  - No retry/circuit breaker │  │
│  │  - Circuit breaker    │  │  - No mTLS encryption      │  │
│  │  - Mutual TLS (mTLS)  │  │  - Calls localhost:15001   │  │
│  │  - Metrics collection │  │                              │  │
│  │  - Distributed tracing│  └──────────────────────────────┘  │
│  └───────────┬───────────┘                                     │
│              │                                                  │
│  All traffic goes through sidecar (transparent to app)         │
└──────────────┼─────────────────────────────────────────────────┘
               │
               │ mTLS encrypted call
               │ (mutual authentication)
               ▼
┌──────────────────────────────────────────────────────────────┐
│           Recommendation Service Pod                          │
│  ┌───────────────────────┐  ┌─────────────────────────────┐ │
│  │  Envoy Sidecar        │  │  Application                │ │
│  │  - Decrypts mTLS      │─►│  - Receives plaintext HTTP  │ │
│  │  - Validates identity │  │  - Doesn't know about mTLS  │ │
│  └───────────────────────┘  └─────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘

Istio Control Plane:
┌────────────────────────────────────────────────────────────────┐
│                         Istiod                                  │
│              (Control plane managing all proxies)               │
│                                                                 │
│  1. Pilot: Service discovery, traffic management rules          │
│     - Pushes config to 100,000+ Envoy sidecars                 │
│     - Tells Envoy: "Route recommendation-service to 10.0.1.5"  │
│                                                                 │
│  2. Citadel: Certificate authority (mTLS)                       │
│     - Issues X.509 certificates to every pod (24-hour TTL)     │
│     - Automatic rotation (no manual cert management)           │
│                                                                 │
│  3. Galley: Configuration validation                            │
│     - Validates Istio config (prevent invalid rules)           │
└────────────────────────────────────────────────────────────────┘

# Original Pod spec (before Istio)
apiVersion: v1
kind: Pod
metadata:
  name: home-feed-abc123
spec:
  containers:
  - name: home-feed
    image: pinterest/home-feed:v2.1.0
    ports:
    - containerPort: 8080

# After Istio sidecar injection (automatic via webhook)
apiVersion: v1
kind: Pod
metadata:
  name: home-feed-abc123
  annotations:
    sidecar.istio.io/status: '{"version":"..."}'  # Istio injected this
spec:
  initContainers:
  - name: istio-init
    image: istio/proxyv2:1.20.0
    # Sets up iptables rules to redirect traffic to Envoy
    command: 
    - istio-iptables
    - -p
    - "15001"  # Envoy inbound port
    - -u
    - "1337"   # Envoy UID (don't intercept Envoy's own traffic)
  
  containers:
  - name: home-feed
    image: pinterest/home-feed:v2.1.0
    ports:
    - containerPort: 8080
  
  - name: istio-proxy  # Sidecar added automatically
    image: istio/proxyv2:1.20.0
    args:
    - proxy
    - sidecar
    - --configPath
    - /etc/istio/proxy
    ports:
    - containerPort: 15001  # Inbound traffic
      name: envoy-inbound
    - containerPort: 15000  # Admin interface
      name: envoy-admin
    resources:
      requests:
        cpu: 100m      # 0.1 CPU core
        memory: 128Mi  # 128 MB RAM
      limits:
        cpu: 2000m     # Max 2 CPU cores
        memory: 1Gi    # Max 1 GB RAM

1. Application container (home-feed) makes HTTP call:
   
   Code: http.get("http://recommendation-service:8080/api/v1/recommendations")
   
   Actually connects to: localhost:8080 (thinks it's calling external service)

2. iptables rules redirect traffic to Envoy sidecar:
   
   localhost:8080 → 127.0.0.1:15001 (Envoy inbound port)

3. Envoy looks up service discovery:
   
   "recommendation-service" resolves to:
   - Pod 1: 10.0.1.5:8080
   - Pod 2: 10.0.1.6:8080
   - Pod 3: 10.0.1.7:8080
   (100 pods total)

4. Envoy applies traffic management rules:
   
   - Load balancing: Round-robin (pick Pod 1 this time)
   - Retries: 3 attempts with exponential backoff (1s, 2s, 4s)
   - Timeout: 10 seconds total
   - Circuit breaker: If > 10 consecutive failures, open circuit (fail fast)

5. Envoy establishes mTLS connection to destination pod:
   
   - Client cert: Issued by Istio CA (24-hour TTL)
   - Server cert: Destination pod's cert (validates identity)
   - TLS 1.3: AES-256-GCM encryption

6. Destination Envoy decrypts traffic:
   
   - Verifies client certificate (mutual authentication)
   - Decrypts payload
   - Forwards plaintext HTTP to application container on localhost:8080

7. Application processes request, returns response:
   
   Response flows back through Envoys (encrypted)

8. Envoy collects metrics:
   
   - Request duration: 45ms
   - Status code: 200 OK
   - Bytes sent: 15 KB
   - Metrics exported to Prometheus

9. Envoy sends trace span to Jaeger:
   
   - Trace ID: abc-123-def-456
   - Span: home-feed → recommendation-service (45ms)
   - Can follow request across 20 services end-to-end

# VirtualService: Route 10% traffic to canary version
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: recommendation-service
  namespace: production
spec:
  hosts:
  - recommendation-service  # Service name
  
  http:
  - match:
    - headers:
        user-agent:
          regex: ".*Pinterest.*"  # Pinterest mobile app
    route:
    - destination:
        host: recommendation-service
        subset: canary  # Mobile app gets canary (beta testers)
      weight: 100
  
  - route:
    - destination:
        host: recommendation-service
        subset: stable  # Everyone else gets stable
      weight: 90
    - destination:
        host: recommendation-service
        subset: canary  # 10% of traffic to canary
      weight: 10

---
# DestinationRule: Define subsets (stable vs canary)
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: recommendation-service
  namespace: production
spec:
  host: recommendation-service
  
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100  # Max 100 TCP connections per pod
      http:
        http1MaxPendingRequests: 50
        maxRequestsPerConnection: 10
    
    loadBalancer:
      simple: LEAST_REQUEST  # Route to pod with fewest active requests
    
    outlierDetection:
      consecutiveErrors: 5         # Open circuit after 5 errors
      interval: 30s                # Check every 30 seconds
      baseEjectionTime: 30s        # Eject pod for 30 seconds
      maxEjectionPercent: 50       # Max 50% of pods ejected
  
  subsets:
  - name: stable
    labels:
      version: v2.1.0  # Stable version (deployed 2 weeks ago)
    trafficPolicy:
      connectionPool:
        http:
          maxRequestsPerConnection: 10
  
  - name: canary
    labels:
      version: v2.2.0  # Canary version (deployed today)
    trafficPolicy:
      connectionPool:
        http:
          maxRequestsPerConnection: 5  # More conservative for canary

Day 1 - 10:00 AM: Deploy canary (v2.2.0)
- Create Deployment with 10 replicas (label: version=v2.2.0)
- Update VirtualService: 10% traffic → canary, 90% → stable
- Monitor metrics: Error rate, latency, CPU

Day 1 - 12:00 PM: Canary healthy, increase traffic
- Update VirtualService: 25% → canary, 75% → stable
- Monitor: No issues, canary performing well

Day 1 - 04:00 PM: Further increase
- Update VirtualService: 50% → canary, 50% → stable

Day 2 - 10:00 AM: Canary validated, promote to stable
- Update VirtualService: 100% → canary
- Delete old stable pods (v2.1.0)
- Rename canary to stable (update labels)

Total rollout time: 24 hours (gradual, safe)
Blast radius if bug: 10% of traffic (vs 100% with immediate rollout)
Rollback time: 10 seconds (update VirtualService, instant traffic shift)

Scenario: Deployed v2.2.0 of recommendation service with performance regression

10:00 AM: Canary deployed (10% traffic)
- Stable (v2.1.0): P95 latency 50ms
- Canary (v2.2.0): P95 latency 450ms (9× slower!)

10:05 AM: Prometheus alert triggered
- Alert: recommendation_service_latency_p95 > 200ms for 5 minutes
- Grafana dashboard shows: Canary 450ms, Stable 50ms

10:07 AM: Engineer investigates
- Distributed tracing (Jaeger) shows: v2.2.0 making extra database query
- Bug: Inefficient SQL join causing 400ms delay

10:08 AM: Rollback decision (1 command)
kubectl patch virtualservice recommendation-service -n production --type merge -p '
{
  "spec": {
    "http": [{
      "route": [{
        "destination": {"host": "recommendation-service", "subset": "stable"},
        "weight": 100
      }]
    }]
  }
}'

10:09 AM: Traffic shifted 100% to stable
- All users now on v2.1.0 (50ms latency)
- Canary pods still running (idle, no traffic)

10:15 AM: Engineer fixes bug, redeploys canary
- Fixed SQL query (efficient index)
- Canary v2.2.1 latency: 45ms (better than stable!)

11:00 AM: Resume canary deployment (10% → 25% → 50% → 100%)

Impact:
- Affected users: 10% of 518M MAU = 51.8M users for 8 minutes
- Degraded experience: Slower recommendations (450ms vs 50ms)
- No downtime: Service remained available (just slower)
- Rollback time: 1 minute (10:08 AM → 10:09 AM)

Without Istio (traditional deployment):
- Would deploy v2.2.0 to all 100 pods simultaneously
- All 518M users affected (not just 10%)
- Rollback time: 15 minutes (redeploy all pods)
- Potential outage: If latency exceeded timeout (10s), requests would fail

# PeerAuthentication: Enforce mTLS for all services
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT  # Reject plaintext connections (only accept mTLS)

1. Pod starts, Envoy sidecar requests certificate from Citadel (Istio CA)

   Envoy → Citadel:
   "I need a certificate for service=home-feed, namespace=production"

2. Citadel generates X.509 certificate:
   
   Subject: spiffe://cluster.local/ns/production/sa/home-feed
   Validity: 24 hours (auto-rotated before expiry)
   Public key: RSA 2048-bit
   Signed by: Istio CA (root certificate)

3. Envoy stores cert in memory (never written to disk)

4. When making request, Envoy presents client certificate:
   
   Client: home-feed (cert proves identity)
   Server: recommendation-service (verifies cert signature)
   
   Mutual authentication: Both sides verify each other's identity

5. TLS 1.3 negotiation:
   
   - Cipher suite: TLS_AES_256_GCM_SHA384
   - Ephemeral key exchange: X25519 (perfect forward secrecy)
   - Encryption: AES-256-GCM (symmetric encryption, ~10 Gbps throughput)

6. Application receives plaintext HTTP:
   
   home-feed container: Doesn't know about mTLS (transparent)
   Receives: HTTP/1.1 GET /api/v1/recommendations
   No code changes required

Performance overhead:
- TLS handshake: 5-10ms (once per connection, then reused)
- Encryption/decryption: <1ms per request (hardware accelerated AES-NI)
- Memory: 128 MB per Envoy sidecar
- CPU: 0.1 core average, 2 cores peak

Every 20 hours (4 hours before expiry):

1. Envoy requests new certificate from Citadel
2. Citadel issues new cert (24-hour validity)
3. Envoy stores both old and new certs
4. New connections use new cert
5. Old connections continue with old cert (graceful)
6. After 4 hours, old cert expires, all connections using new cert

Result: Zero-downtime certificate rotation
- No service restarts required
- No manual certificate management
- Automatic across 100,000 pods (200,000 Envoy sidecars)

User request: "Show home feed"

Home Feed Service calls:
1. User Service (get user profile)
2. Recommendation Service (get personalized pins)
   → Image Service (get image URLs)
   → ML Service (rank by relevance)
3. Ad Service (get sponsored pins)
4. Analytics Service (log impression)

Traditional debugging:
- Check logs in 5 services manually
- Correlate timestamps (error-prone)
- No visibility into which service caused delay

With Istio + Jaeger:
- Single trace ID spans all services
- Visualize entire request path
- See exact duration of each service call

Trace ID: abc-123-def-456
Total duration: 285ms

┌─────────────────────────────────────────────────────────────────┐
│ home-feed.svc.cluster.local                     285ms (100%)    │
│ ┌───────────────────────────────────────────────────────────┐   │
│ │ user-service.svc.cluster.local                25ms (9%)   │   │
│ └───────────────────────────────────────────────────────────┘   │
│ ┌───────────────────────────────────────────────────────────┐   │
│ │ recommendation-service.svc.cluster.local     220ms (77%)  │   │
│ │ ┌─────────────────────────────────────────────────────┐   │   │
│ │ │ image-service.svc.cluster.local           45ms (16%)│   │   │
│ │ └─────────────────────────────────────────────────────┘   │   │
│ │ ┌─────────────────────────────────────────────────────┐   │   │
│ │ │ ml-service.svc.cluster.local              150ms (53%)│   │   │
│ │ └─────────────────────────────────────────────────────┘   │   │
│ └───────────────────────────────────────────────────────────┘   │
│ ┌───────────────────────────────────────────────────────────┐   │
│ │ ad-service.svc.cluster.local                 35ms (12%)  │   │
│ └───────────────────────────────────────────────────────────┘   │
│ ┌───────────────────────────────────────────────────────────┐   │
│ │ analytics-service.svc.cluster.local           5ms (2%)   │   │
│ └───────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Bottleneck identified: ml-service.svc.cluster.local (150ms, 53% of total)
- Recommendation service spends 150ms calling ML service (model inference)
- Optimization: Cache ML results (reduce from 150ms to 20ms)

# Application code (no tracing logic required!)
def get_home_feed(user_id):
    # Call user service
    user = http.get(f"http://user-service/users/{user_id}")
    
    # Call recommendation service
    pins = http.get(f"http://recommendation-service/recommendations?user={user_id}")
    
    # Call ad service
    ads = http.get(f"http://ad-service/ads?user={user_id}")
    
    # Return combined feed
    return {"user": user, "pins": pins, "ads": ads}

# Envoy sidecar automatically:
# 1. Generates trace ID if not present (abc-123-def-456)
# 2. Injects trace headers in outbound requests:
#    X-B3-TraceId: abc-123-def-456
#    X-B3-SpanId: span-001
#    X-B3-ParentSpanId: span-000
# 3. Sends span to Jaeger:
#    Service: home-feed
#    Operation: GET /feed
#    Duration: 285ms
#    Child spans: user-service (25ms), recommendation-service (220ms), ad-service (35ms)
# 4. Application doesn't need tracing library (transparent)

2020 Q1: Pilot (10 services)
- Services: Home Feed, User Service, Recommendation
- Results: 15ms latency overhead (Envoy proxy), acceptable
- mTLS: 100% encrypted (previously 0%)
- Observability: Distributed tracing working

2020 Q2-Q4: Gradual rollout (200 services)
- 10% of services migrated
- Learnings: Sidecar injection automation, resource sizing (100m CPU, 128Mi RAM)

2021 Q1-Q4: Majority migration (1,500 services)
- 75% of services on Istio
- Benefits: Canary deployments, circuit breakers, retry logic removed from apps

2022 Q1-Q2: Complete migration (2,000 services)
- 100% of services on Istio
- Decommissioned old HAProxy load balancers

Latency overhead (Envoy proxy):
- P50: +5ms (50ms → 55ms, 10% increase)
- P95: +15ms (200ms → 215ms, 7.5% increase)
- P99: +30ms (500ms → 530ms, 6% increase)

CPU overhead (Envoy sidecar):
- Idle: 0.05 cores (50m)
- Average: 0.1 cores (100m)
- Peak: 0.5 cores (500m)
- Total: 100,000 sidecars × 0.1 cores = 10,000 cores

Memory overhead (Envoy sidecar):
- Average: 128 MB per sidecar
- Peak: 512 MB per sidecar
- Total: 100,000 sidecars × 128 MB = 12.8 TB

Cost increase:
- Additional CPU: 10,000 cores ÷ 36 cores/c5.9xlarge = 278 nodes
- Cost: 278 × c5.9xlarge ($1.53/hour) × 730 hours = $310,194/month
- Annual: $3.72M/year

Value gained:

1. Security: 100% mTLS encryption (zero-trust networking)
   - Prevented: Internal attacker sniffing traffic (compliance requirement)
   - Value: Avoided $50M+ data breach (typical cost of Pinterest scale breach)

2. Reliability: Circuit breakers, retries, timeouts
   - Incidents before Istio: 25 cascading failures/year (retry storms)
   - Incidents after Istio: 3 cascading failures/year (88% reduction)
   - Prevented downtime: 22 incidents × 45 min avg = 16.5 hours/year
   - Revenue saved: 16.5 hours × $175K/hour = $2.9M/year

3. Velocity: Canary deployments, instant rollback
   - Deployment time before: 2 hours (careful manual rollout)
   - Deployment time after: 30 minutes (automated canary with instant rollback)
   - Deployments/year: 2,000 services × 50 deploys = 100,000 deploys
   - Time saved: 100,000 × 1.5 hours = 150,000 hours
   - Engineer cost saved: 150,000 hours ÷ 2,000 hours/year = 75 FTE-years
   - Value: 75 × $180K = $13.5M/year

4. Observability: Distributed tracing
   - Debug time before: 4 hours average (check logs manually)
   - Debug time after: 30 minutes (Jaeger trace visualization)
   - Incidents/year: 500 (P1/P2 incidents requiring debugging)
   - Time saved: 500 × 3.5 hours = 1,750 hours/year
   - Value: 1,750 hours × $180/hour = $315K/year

Total value: $50M (breach prevention) + $2.9M (reliability) + $13.5M (velocity) + $315K (observability) = $66.7M/year
Cost: $3.72M/year (infrastructure overhead)
ROI: $66.7M ÷ $3.72M = 17.9× return

1. Vulnerable Container Images:
   - Base images with known CVEs (Common Vulnerabilities and Exposures)
   - Outdated dependencies (Log4j 2.x vulnerability Dec 2021)
   - Malicious packages (npm, PyPI supply chain attacks)
   - Secrets in images (hardcoded API keys, passwords)

2. Runtime Attacks:
   - Container escape (break out of container to host)
   - Privilege escalation (gain root access)
   - Resource exhaustion (CPU/memory DoS)
   - Network attacks (lateral movement between containers)

3. Configuration Errors:
   - Running as root user (unnecessary privileges)
   - Mounting sensitive host paths (/var/run/docker.sock)
   - No resource limits (unlimited CPU/memory)
   - Permissive network policies (allow all traffic)

4. Supply Chain Risks:
   - Untrusted base images (Docker Hub public images)
   - Compromised registries (insider threat)
   - Build-time attacks (malicious CI/CD pipeline)

Scenario: Engineer unknowingly used vulnerable base image

1. Engineer creates Dockerfile:
   FROM node:14.15.0  # Vulnerable version (CVE-2021-44531, CVE-2021-44532)
   COPY . /app
   RUN npm install
   CMD ["node", "server.js"]

2. CI/CD pipeline builds image:
   docker build -t netflix/video-service:v2.3.0 .

3. Security scanner (Trivy) scans image:
   - Found: 15 HIGH severity vulnerabilities in node:14.15.0
   - Found: 3 CRITICAL vulnerabilities (RCE in OpenSSL)
   - Found: API key in environment variable (SECRET_KEY=abc123)

4. Pipeline blocked deployment:
    FAILED: Security vulnerabilities detected
    CRITICAL: CVE-2021-44531 (OpenSSL RCE, CVSS 9.8)
    CRITICAL: Hardcoded secret found (SECRET_KEY)
   
   Action required:
   1. Update base image to node:14.21.3 (patched)
   2. Remove hardcoded secret (use AWS Secrets Manager)
   3. Re-scan and deploy

5. Engineer fixes issues:
   FROM node:14.21.3  # Patched version
   ENV SECRET_KEY_ARN=arn:aws:secretsmanager:us-east-1:123456789012:secret:video-key
   # (Application fetches secret at runtime from AWS Secrets Manager)

6. Re-scan passed, deployment allowed

Impact prevented:
- Potential breach: OpenSSL RCE vulnerability (CVE-2021-44531)
- Attack scenario: Attacker exploits RCE → gains access to container → extracts hardcoded API key → accesses AWS resources
- Estimated cost if breached: $50M+ (data breach, compliance fines, reputation damage)
- Actual cost: 2 hours engineer time to fix ($360)

┌─────────────────────────────────────────────────────────────────────┐
│                    Container Security Pipeline                       │
└─────────────────────────────────────────────────────────────────────┘

Stage 1: Developer Workstation
┌──────────────────────────────────────────────────────────────┐
│  1. Pre-commit hook scans Dockerfile                          │
│     - Tool: Hadolint (Dockerfile linter)                      │
│     - Checks: Best practices, root user, COPY vs ADD          │
│                                                               │
│  Example check:                                               │
│   DL3002: User should not be root (use USER instruction)    │
│   DL3008: Pin versions in apt-get install                   │
│   DL3009: Delete apt-get lists after install                │
└──────────────────────────────────────────────────────────────┘
         │
         │ git push (triggers CI/CD)
         ▼
Stage 2: Build Time (CI/CD Pipeline)
┌──────────────────────────────────────────────────────────────┐
│  2. Build container image                                     │
│     docker build -t netflix/video-service:v2.3.0 .           │
│                                                               │
│  3. Image scan (Trivy)                                        │
│     trivy image netflix/video-service:v2.3.0                 │
│                                                               │
│     Scans for:                                                │
│     - OS vulnerabilities (Ubuntu, Alpine CVEs)               │
│     - Language vulnerabilities (npm, pip, Maven)             │
│     - Misconfigurations (exposed ports, secrets)             │
│                                                               │
│  Output:                                                      │
│  Total: 127 vulnerabilities (3 CRITICAL, 15 HIGH, 109 LOW)  │
│                                                               │
│  4. Fail if CRITICAL or HIGH vulnerabilities                 │
│     Exit code: 1 (deployment blocked)                        │
└──────────────────────────────────────────────────────────────┘
         │
         │ If scan passes
         ▼
Stage 3: Container Registry (Amazon ECR)
┌──────────────────────────────────────────────────────────────┐
│  5. Push image to registry                                    │
│     docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/│
│                 netflix/video-service:v2.3.0                  │
│                                                               │
│  6. ECR image scanning (enabled automatically)                │
│     - AWS scans image for CVEs                                │
│     - Uses CVE database (updated daily)                       │
│     - Scan results visible in ECR console                     │
└──────────────────────────────────────────────────────────────┘
         │
         │ Deploy to Kubernetes
         ▼
Stage 4: Deployment Time (Kubernetes Admission Control)
┌──────────────────────────────────────────────────────────────┐
│  7. OPA Gatekeeper validates pod                              │
│     - Checks: No root user, resource limits set               │
│     - Checks: Image from trusted registry (ECR)               │
│     - Checks: Security context enforced                       │
│                                                               │
│  Example policy:                                              │
│  apiVersion: constraints.gatekeeper.sh/v1beta1               │
│  kind: K8sBlockRootUser                                      │
│  metadata:                                                    │
│    name: block-root-containers                               │
│  spec:                                                        │
│    match:                                                     │
│      kinds:                                                   │
│      - apiGroups: [""]                                        │
│        kinds: ["Pod"]                                         │
│                                                               │
│  Result:                                                      │
│   ALLOWED: Container runs as UID 1000 (non-root)           │
└──────────────────────────────────────────────────────────────┘
         │
         │ Pod deployed
         ▼
Stage 5: Runtime (Container Running)
┌──────────────────────────────────────────────────────────────┐
│  8. Falco monitors runtime behavior                           │
│     - Detects: Process spawning shells, file access           │
│     - Detects: Network connections to unexpected IPs          │
│     - Detects: Privilege escalation attempts                  │
│                                                               │
│  Example alert:                                               │
│  Warning: Shell spawned in container                          │
│  Container: video-service-abc123                              │
│  Command: /bin/bash                                           │
│  User: www-data                                               │
│  Action: Investigate immediately (potential breach)           │
└──────────────────────────────────────────────────────────────┘
         │
         │ Periodic re-scanning
         ▼
Stage 6: Continuous Monitoring (Daily)
┌──────────────────────────────────────────────────────────────┐
│  9. Re-scan running containers (new CVEs discovered daily)    │
│     - Trivy scans all images in ECR                           │
│     - Compares against latest CVE database                    │
│     - Notifies teams if new vulnerabilities found             │
│                                                               │
│  Example notification:                                        │
│  New CRITICAL vulnerability: CVE-2024-1234                    │
│  Affected: node:14.21.3 (10 services using this image)       │
│  Action: Upgrade to node:14.22.0 within 7 days               │
└──────────────────────────────────────────────────────────────┘

// Jenkinsfile for Netflix video-service
pipeline {
    agent any
    
    environment {
        IMAGE_NAME = "netflix/video-service"
        IMAGE_TAG = "${env.BUILD_NUMBER}"
        ECR_REGISTRY = "123456789012.dkr.ecr.us-east-1.amazonaws.com"
    }
    
    stages {
        stage('Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/netflix/video-service.git'
            }
        }
        
        stage('Build Docker Image') {
            steps {
                script {
                    // Build image
                    sh "docker build -t ${IMAGE_NAME}:${IMAGE_TAG} ."
                }
            }
        }
        
        stage('Security Scan') {
            steps {
                script {
                    // Install Trivy (if not already installed)
                    sh '''
                        if ! command -v trivy &> /dev/null; then
                            wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
                            echo "deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/trivy.list
                            sudo apt-get update && sudo apt-get install trivy -y
                        fi
                    '''
                    
                    // Scan image
                    def scanResult = sh(
                        script: "trivy image --severity HIGH,CRITICAL --format json --output scan-results.json ${IMAGE_NAME}:${IMAGE_TAG}",
                        returnStatus: true
                    )
                    
                    // Parse results
                    def scanData = readJSON file: 'scan-results.json'
                    def criticalCount = 0
                    def highCount = 0
                    
                    scanData.Results.each { result ->
                        result.Vulnerabilities.each { vuln ->
                            if (vuln.Severity == 'CRITICAL') criticalCount++
                            if (vuln.Severity == 'HIGH') highCount++
                        }
                    }
                    
                    echo "Security Scan Results:"
                    echo "  CRITICAL: ${criticalCount}"
                    echo "  HIGH: ${highCount}"
                    
                    // Fail if CRITICAL vulnerabilities found
                    if (criticalCount > 0) {
                        error("CRITICAL vulnerabilities detected. Fix before deploying.")
                    }
                    
                    // Warn if HIGH vulnerabilities (but allow deployment)
                    if (highCount > 0) {
                        echo "WARNING: ${highCount} HIGH vulnerabilities detected. Consider fixing."
                    }
                    
                    // Archive scan results
                    archiveArtifacts artifacts: 'scan-results.json'
                }
            }
        }
        
        stage('Push to ECR') {
            when {
                expression { currentBuild.result == null || currentBuild.result == 'SUCCESS' }
            }
            steps {
                script {
                    // Login to ECR
                    sh "aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${ECR_REGISTRY}"
                    
                    // Tag image
                    sh "docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"
                    sh "docker tag ${IMAGE_NAME}:${IMAGE_TAG} ${ECR_REGISTRY}/${IMAGE_NAME}:latest"
                    
                    // Push image
                    sh "docker push ${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}"
                    sh "docker push ${ECR_REGISTRY}/${IMAGE_NAME}:latest"
                }
            }
        }
        
        stage('Deploy to Kubernetes') {
            steps {
                script {
                    // Update Kubernetes deployment
                    sh """
                        kubectl set image deployment/video-service \
                            video-service=${ECR_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} \
                            -n production
                    """
                    
                    // Wait for rollout
                    sh "kubectl rollout status deployment/video-service -n production --timeout=10m"
                }
            }
        }
    }
    
    post {
        failure {
            // Notify Slack on failure
            slackSend(
                color: 'danger',
                message: "Deployment failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}\nSecurity scan found CRITICAL vulnerabilities."
            )
        }
        success {
            slackSend(
                color: 'good',
                message: "Deployment succeeded: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
            )
        }
    }
}

{
  "SchemaVersion": 2,
  "ArtifactName": "netflix/video-service:v2.3.0",
  "ArtifactType": "container_image",
  "Results": [
    {
      "Target": "netflix/video-service:v2.3.0 (ubuntu 20.04)",
      "Type": "ubuntu",
      "Vulnerabilities": [
        {
          "VulnerabilityID": "CVE-2021-44531",
          "PkgName": "openssl",
          "InstalledVersion": "1.1.1f-1ubuntu2",
          "FixedVersion": "1.1.1f-1ubuntu2.17",
          "Severity": "CRITICAL",
          "Description": "OpenSSL accepts arbitrary certificates in the trust store",
          "CVSS": {
            "nvd": {
              "V3Score": 9.8
            }
          }
        },
        {
          "VulnerabilityID": "CVE-2021-3711",
          "PkgName": "openssl",
          "InstalledVersion": "1.1.1f-1ubuntu2",
          "FixedVersion": "1.1.1f-1ubuntu2.16",
          "Severity": "HIGH",
          "Description": "Buffer overflow in SM2 decryption",
          "CVSS": {
            "nvd": {
              "V3Score": 7.5
            }
          }
        }
      ]
    },
    {
      "Target": "Node.js",
      "Type": "node-pkg",
      "Vulnerabilities": [
        {
          "VulnerabilityID": "CVE-2021-23343",
          "PkgName": "path-parse",
          "InstalledVersion": "1.0.6",
          "FixedVersion": "1.0.7",
          "Severity": "HIGH",
          "Description": "Regular Expression Denial of Service (ReDoS)",
          "CVSS": {
            "nvd": {
              "V3Score": 7.5
            }
          }
        }
      ]
    }
  ]
}

# Falco rules for detecting suspicious container behavior

- rule: Shell Spawned in Container
  desc: Detect shell spawned inside container (potential breach)
  condition: >
    spawned_process and
    container and
    shell_procs and
    proc.pname exists and
    not user_known_shell_spawn_binaries
  output: >
    Shell spawned in container (user=%user.name command=%proc.cmdline 
    container_id=%container.id container_name=%container.name image=%container.image.repository)
  priority: WARNING
  tags: [container, shell, mitre_execution]

- rule: Write Below Root
  desc: Detect write to root filesystem (containers should be read-only)
  condition: >
    write and
    container and
    fd.directory in ("/", "/bin", "/etc", "/lib", "/sbin", "/usr")
  output: >
    Write below root (user=%user.name command=%proc.cmdline file=%fd.name 
    container_id=%container.id container_name=%container.name)
  priority: ERROR
  tags: [container, filesystem, mitre_persistence]

- rule: Outbound Connection to Unexpected IP
  desc: Detect connection to IP not in whitelist
  condition: >
    outbound and
    container and
    not fd.sip in (allowed_ips)
  output: >
    Outbound connection to unexpected IP (user=%user.name command=%proc.cmdline 
    connection=%fd.name container_id=%container.id)
  priority: WARNING
  tags: [container, network, mitre_exfiltration]

- rule: Container Privilege Escalation
  desc: Detect privilege escalation in container
  condition: >
    spawned_process and
    container and
    proc.aname in ("sudo", "su") and
    not user_privileged_containers
  output: >
    Privilege escalation detected (user=%user.name command=%proc.cmdline 
    container_id=%container.id)
  priority: CRITICAL
  tags: [container, privilege_escalation, mitre_privilege_escalation]

- rule: Cryptocurrency Mining Activity
  desc: Detect crypto mining (CPU usage pattern + network connections)
  condition: >
    spawned_process and
    container and
    proc.name in (crypto_miners) or
    (proc.name contains "xmrig" or proc.name contains "mine")
  output: >
    Crypto mining detected (command=%proc.cmdline container_id=%container.id)
  priority: CRITICAL
  tags: [container, crypto_mining, mitre_resource_hijacking]

Scenario: Compromised container mining cryptocurrency

Timeline:
14:23:00 - Falco alert triggered
  Rule: Cryptocurrency Mining Activity
  Container: video-service-xyz789
  Process: xmrig (Monero miner)
  CPU usage: 95% (1 container consuming 4 CPU cores)
  Network: Connecting to pool.minexmr.com:3333

14:24:00 - Security team investigates
  - Container logs show: Suspicious process started at 14:22:45
  - Entry point: Exploited CVE in outdated Node.js dependency
  - Attacker: Downloaded miner binary from external server
  - Mining duration: 15 minutes (before detection)

14:25:00 - Immediate response
  - kubectl delete pod video-service-xyz789 (kill compromised container)
  - Kubernetes auto-creates new pod (clean image)
  - Network policy: Block pool.minexmr.com domain

14:30:00 - Root cause analysis
  - Vulnerability: Unpatched dependency (should've been caught by scan)
  - Gap: CI/CD pipeline didn't run for this deployment (manual override)
  - Fix: Remove manual override capability, enforce all deploys through pipeline

Impact:
- Compute waste: 15 minutes × 4 CPU cores = 1 core-hour wasted ($0.15)
- Detection time: 1 minute (Falco alert)
- Remediation time: 2 minutes (delete pod)
- Total downtime: 0 minutes (new pod replaced compromised pod, no service disruption)

Prevented impact:
- If undetected for 24 hours: 96 core-hours × $0.15 = $14.40/day
- If spread to 1,000 pods: $14,400/day × 30 days = $432,000/month
- Value of detection: Prevented $432K/month compute theft

# Netflix RBAC policy: Developers can deploy to staging, not production

# Role: developer-staging (what developers can do in staging namespace)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: staging
  name: developer-staging
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["pods", "pods/log", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]  # Can view secrets, but not create/delete

# RoleBinding: Assign role to developers group
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: staging
  name: developer-staging-binding
subjects:
- kind: Group
  name: developers  # LDAP group (Netflix SSO)
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer-staging
  apiGroup: rbac.authorization.k8s.io

---
# ClusterRole: production-deployer (production namespace, restricted)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: production-deployer
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "update", "patch"]  # Can update, not delete
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]  # Read-only access to pods
- apiGroups: [""]
  resources: ["secrets"]
  verbs: []  # No access to secrets (use AWS Secrets Manager instead)

# ClusterRoleBinding: Only CI/CD service account can deploy to production
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cicd-production-binding
subjects:
- kind: ServiceAccount
  name: cicd-deployer
  namespace: cicd
roleRef:
  kind: ClusterRole
  name: production-deployer
  apiGroup: rbac.authorization.k8s.io

# Default: Deny all traffic (whitelist approach)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}  # Applies to all pods in namespace
  policyTypes:
  - Ingress
  - Egress

---
# Allow: video-service can call recommendation-service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: video-to-recommendation
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: video-service
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: recommendation-service
    ports:
    - protocol: TCP
      port: 8080

---
# Allow: recommendation-service accepts traffic from video-service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: recommendation-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: recommendation-service
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: video-service
    ports:
    - protocol: TCP
      port: 8080

---
# Allow: All pods can call external DNS and HTTPS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-https
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector: {}  # Any namespace
    ports:
    - protocol: UDP
      port: 53  # DNS
  - to:
    - podSelector: {}
    ports:
    - protocol: TCP
      port: 443  # HTTPS

Image scanning:
- Images scanned: 50,000 total (100% coverage)
- Scans/day: 4,000 (one per deployment)
- Vulnerabilities found: 250,000 (average 5 per image)
  - CRITICAL: 1,500 (0.6%)
  - HIGH: 15,000 (6%)
  - MEDIUM: 75,000 (30%)
  - LOW: 158,500 (63.4%)
- Deployments blocked: 450/year (CRITICAL vulnerabilities)
- Time to fix: 4 hours average (update base image, redeploy)

Runtime monitoring (Falco):
- Alerts/day: 150 (suspicious behavior detected)
  - False positives: 120 (80%, legitimate behavior flagged)
  - True positives: 30 (20%, actual security incidents)
- Incidents prevented: 360/year (crypto mining, privilege escalation)
- Detection time: 1-2 minutes average
- Remediation time: 3-5 minutes average (kill pod, investigate)

RBAC:
- Users: 2,000 engineers
- Service accounts: 500 (CI/CD pipelines, automation)
- Roles: 50 (developer, SRE, security, read-only)
- Unauthorized access attempts: 25/month (blocked by RBAC)
- Time to grant access: 5 minutes (LDAP group membership)

Network policies:
- Policies: 500 (one per microservice average)
- Blocked connections: 10,000/day (unauthorized service-to-service calls)
- Lateral movement attempts blocked: 100% (zero-trust enforced)

Security tooling:
- Trivy scanning: Open-source (free)
  - Compute: 4,000 scans/day × 30 seconds × $0.0001/second = $12/day = $4,380/year
- Falco runtime monitoring: Open-source (free)
  - Overhead: 100,000 pods × 0.05 CPU cores × $0.04/core-hour × 730 hours = $146,000/year
- OPA Gatekeeper: Open-source (free)
  - Control plane: 3× c5.xlarge ($0.17/hour × 3 × 730) = $372/month = $4,464/year

Staff (10 security engineers):
- Salary: $200K/year × 10 = $2M/year
- Benefits (30%): $600K/year
- Total: $2.6M/year

Total security cost: $4,380 + $146,000 + $4,464 + $2.6M = $2.75M/year

Value gained:

1. Breach prevention: $50M+ (typical cost of major breach)
   - 450 deployments blocked/year (CRITICAL vulnerabilities)
   - Estimate 10% would've led to breach without scanning
   - Value: 45 × $50M × 0.01 (probability) = $22.5M/year

2. Crypto mining prevention: $432K/year
   - 360 incidents prevented (Falco detection)
   - Each incident: $1,200/month if undetected
   - Value: 360 × $1,200 = $432K/year

3. Compliance: $5M+ (avoid fines)
   - PCI DSS, SOC 2, GDPR require container security
   - Non-compliance fines: $5M+ typical

Total value: $22.5M + $432K + $5M = $27.9M/year
Cost: $2.75M/year
ROI: $27.9M ÷ $2.75M = 10.1× return

Manual deployment steps (45 minutes per deployment):

1. Engineer builds Docker image locally (5 minutes)
   $ docker build -t airbnb/booking-service:v2.3.0 .
   
2. Push to registry (3 minutes)
   $ docker push airbnb/booking-service:v2.3.0
   
3. SSH to Kubernetes master node (1 minute)
   $ ssh ubuntu@k8s-master-1.airbnb.com
   
4. Edit deployment YAML manually (5 minutes)
   $ vim booking-service-deployment.yaml
   # Change image tag: v2.2.9 → v2.3.0
   
5. Apply deployment (2 minutes)
   $ kubectl apply -f booking-service-deployment.yaml
   
6. Watch rollout manually (10 minutes)
   $ kubectl rollout status deployment/booking-service -n production
   # Stare at terminal watching pods restart
   
7. Test manually (10 minutes)
   $ curl https://api.airbnb.com/booking/health
   # Click through UI testing booking flow
   
8. Update documentation (5 minutes)
   # Update wiki page with deployed version
   
9. Notify team in Slack (1 minute)
   # Post: "Deployed booking-service v2.3.0 to production"

Total time: 45 minutes × 500 deploys/day = 375 hours/day = 46.9 engineer-days/day
Cost: 46.9 days × $2,500/day = $117,250/day = $42.8M/year wasted on manual deployments

18:30 - Engineer deploys booking-service v2.3.0 to production
        Typo in YAML: replicas: 10 → replicas: 1 (accidentally deleted 0)
        
18:32 - Kubernetes scales down from 1,000 pods to 1 pod
        Traffic: 5,000 req/sec → 1 pod (overloaded immediately)
        
18:33 - Booking service crashes (OOMKilled)
        All bookings fail (500 Internal Server Error)
        
18:35 - Engineers notice alerts (2 minutes to detect)
        PagerDuty: "booking-service error rate > 50%"
        
18:40 - Fix applied (5 minutes to diagnose + fix)
        $ kubectl scale deployment/booking-service --replicas=1000 -n production
        
18:45 - Service recovered (5 minutes for all pods to start)
        
Total downtime: 15 minutes
Bookings lost: 5,000 req/sec × 900 seconds ÷ 60 req/booking = 75,000 bookings
Revenue lost: 75,000 × $467 avg = $35M (potential, many customers retried)
Actual revenue lost: ~$2M (estimated 5% churn)

Root cause: Manual YAML editing, no validation, no peer review

┌─────────────────────────────────────────────────────────────────────┐
│                    Developer Workflow                                │
└─────────────────────────────────────────────────────────────────────┘

Step 1: Code Change
┌──────────────────────────────────────────────────────────────┐
│  Developer: Alice                                             │
│                                                               │
│  $ git checkout -b feature/faster-search                     │
│  $ # Make code changes                                        │
│  $ git commit -m "Optimize search algorithm"                 │
│  $ git push origin feature/faster-search                     │
│                                                               │
│  GitHub: Create Pull Request                                 │
│  - Title: "Optimize search algorithm (5× faster)"           │
│  - Reviewers: Bob, Charlie                                   │
└──────────────────────────────────────────────────────────────┘
         │
         │ Webhook triggers Jenkins
         ▼
Step 2: Continuous Integration (CI) - Jenkins Pipeline
┌──────────────────────────────────────────────────────────────┐
│  Jenkins Job: PR-Validation                                   │
│                                                               │
│  Stage 1: Checkout (10 seconds)                              │
│    git clone https://github.com/airbnb/booking-service.git  │
│                                                               │
│  Stage 2: Unit Tests (2 minutes)                             │
│    npm test                                                   │
│     1,247 tests passed                                      │
│                                                               │
│  Stage 3: Build Docker Image (3 minutes)                     │
│    docker build -t booking-service:pr-1234 .                 │
│                                                               │
│  Stage 4: Security Scan (1 minute)                           │
│    trivy image booking-service:pr-1234                       │
│     No CRITICAL vulnerabilities                             │
│                                                               │
│  Stage 5: Integration Tests (2 minutes)                      │
│    docker-compose up -d                                       │
│    npm run test:integration                                   │
│     87 integration tests passed                             │
│                                                               │
│  Total: 8 minutes                                             │
│  Result:  ALL CHECKS PASSED                                 │
│  Comment on PR: "All checks passed. Ready to merge."         │
└──────────────────────────────────────────────────────────────┘
         │
         │ PR approved & merged
         ▼
Step 3: Continuous Deployment (CD) - Jenkins Build
┌──────────────────────────────────────────────────────────────┐
│  Jenkins Job: Main-Branch-Build                               │
│                                                               │
│  Stage 1: Checkout main branch                               │
│  Stage 2: Run tests (repeat from above)                      │
│  Stage 3: Build production image                             │
│    docker build -t booking-service:v2.3.0 .                  │
│    docker tag booking-service:v2.3.0 \                       │
│      123456789012.dkr.ecr.us-east-1.amazonaws.com/\         │
│      booking-service:v2.3.0                                  │
│                                                               │
│  Stage 4: Push to ECR                                        │
│    docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/│
│                booking-service:v2.3.0                         │
│                                                               │
│  Stage 5: Update GitOps Repo (critical step!)               │
│    git clone https://github.com/airbnb/k8s-manifests.git    │
│    cd k8s-manifests/production/booking-service              │
│    # Update image tag in kustomization.yaml                  │
│    sed -i 's/newTag: v2.2.9/newTag: v2.3.0/' kustomization.yaml│
│    git add kustomization.yaml                                │
│    git commit -m "Deploy booking-service v2.3.0"            │
│    git push origin main                                       │
│                                                               │
│  Result: Image built and GitOps repo updated                 │
│  Time: 5 minutes                                              │
└──────────────────────────────────────────────────────────────┘
         │
         │ GitOps repo updated (triggers ArgoCD)
         ▼
Step 4: GitOps Deployment - ArgoCD
┌──────────────────────────────────────────────────────────────┐
│  ArgoCD: Watches GitOps Repo                                 │
│                                                               │
│  Every 3 minutes:                                             │
│    1. Fetch latest from github.com/airbnb/k8s-manifests     │
│    2. Compare Git state vs Kubernetes state                  │
│                                                               │
│  Detection:                                                   │
│    Git:        image: booking-service:v2.3.0                 │
│    Kubernetes: image: booking-service:v2.2.9                 │
│    Status: OUT OF SYNC                                        │
│                                                               │
│  Action: Auto-sync (configured)                              │
│    kubectl set image deployment/booking-service \            │
│      booking-service=booking-service:v2.3.0 -n production   │
│                                                               │
│  Rollout:                                                     │
│    Strategy: RollingUpdate (maxSurge: 25%, maxUnavailable: 10%)│
│    1. Create 250 new pods (v2.3.0)                           │
│    2. Wait for readiness checks                              │
│    3. Delete 100 old pods (v2.2.9)                           │
│    4. Repeat until all 1,000 pods updated                    │
│                                                               │
│  Result:  SYNCED (3 minutes)                                │
│  Status: Healthy (all pods running, health checks passing)   │
└──────────────────────────────────────────────────────────────┘
         │
         │ Deployment complete
         ▼
Step 5: Post-Deployment
┌──────────────────────────────────────────────────────────────┐
│  Automated Monitoring                                         │
│                                                               │
│  1. Datadog monitors error rate, latency                     │
│     - Error rate: 0.1% (normal)                              │
│     - P95 latency: 85ms (within SLO of 100ms)               │
│                                                               │
│  2. Slack notification                                        │
│     @channel Deployed booking-service v2.3.0 to production  │
│     Deployed by: Alice                                        │
│     Rollout time: 3 minutes                                   │
│     Health:  All checks passing                             │
│                                                               │
│  3. Audit log                                                 │
│     Git commit hash: abc123                                   │
│     Deploy timestamp: 2024-01-15T14:32:00Z                   │
│     ArgoCD sync: application/booking-service revision abc123 │
└──────────────────────────────────────────────────────────────┘

Total time: 8 min (CI) + 5 min (CD) + 3 min (GitOps) = 16 minutes
Fully automated: Zero manual steps (developer only writes code)
Audit trail: Complete (Git history + ArgoCD logs)

airbnb/k8s-manifests/
├── README.md
├── base/                          # Base configurations (shared)
│   ├── deployment.yaml            # Deployment template
│   ├── service.yaml               # Service template
│   └── kustomization.yaml         # Kustomize config
├── staging/                       # Staging environment
│   ├── booking-service/
│   │   ├── kustomization.yaml     # Overlay for staging
│   │   ├── configmap.yaml         # Staging config
│   │   └── ingress.yaml           # Staging ingress
│   ├── search-service/
│   └── payment-service/
└── production/                    # Production environment
    ├── booking-service/
    │   ├── kustomization.yaml     # Production overlay
    │   ├── configmap.yaml         # Production config
    │   ├── hpa.yaml               # Autoscaling (production only)
    │   └── pdb.yaml               # Pod disruption budget
    ├── search-service/
    └── payment-service/

# production/booking-service/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

bases:
- ../../base  # Inherit from base configuration

images:
- name: booking-service
  newName: 123456789012.dkr.ecr.us-east-1.amazonaws.com/booking-service
  newTag: v2.3.0  # This line updated by CI/CD pipeline

replicas:
- name: booking-service
  count: 1000  # Production replica count

resources:
- configmap.yaml  # Production configuration
- hpa.yaml        # Horizontal Pod Autoscaler
- pdb.yaml        # Pod Disruption Budget

patches:
- path: production-resources.yaml  # Production resource limits

configMapGenerator:
- name: booking-config
  literals:
  - DATABASE_URL=postgres://booking-prod.us-east-1.rds.amazonaws.com:5432/bookings
  - CACHE_URL=redis://booking-cache-prod.cache.amazonaws.com:6379
  - LOG_LEVEL=info

# ArgoCD application: booking-service-production
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: booking-service-production
  namespace: argocd
spec:
  project: default
  
  source:
    repoURL: https://github.com/airbnb/k8s-manifests.git
    targetRevision: main
    path: production/booking-service  # GitOps path
  
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  
  syncPolicy:
    automated:
      prune: true    # Delete resources removed from Git
      selfHeal: true # Revert manual changes (enforce Git as source of truth)
    
    syncOptions:
    - CreateNamespace=true
    
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
  
  # Health checks
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas  # Ignore replica count (managed by HPA)

# Deployment with RollingUpdate strategy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: booking-service
spec:
  replicas: 1000
  
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%        # Create 250 new pods at a time
      maxUnavailable: 10%  # Max 100 pods down during rollout
  
  template:
    spec:
      containers:
      - name: booking-service
        image: booking-service:v2.3.0
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 5

# Rollout timeline:
# T+0:00 - Start rollout (1,000 pods running v2.2.9)
# T+0:30 - Create 250 new pods (v2.3.0)
# T+1:00 - New pods ready, delete 100 old pods
# T+1:30 - Create 250 more new pods
# T+2:00 - New pods ready, delete 100 more old pods
# ... (repeat)
# T+3:00 - All 1,000 pods running v2.3.0

# Blue deployment (current production, v2.2.9)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: booking-service-blue
  labels:
    version: blue
spec:
  replicas: 1000
  selector:
    matchLabels:
      app: booking-service
      version: blue
  template:
    metadata:
      labels:
        app: booking-service
        version: blue
    spec:
      containers:
      - name: booking-service
        image: booking-service:v2.2.9

---
# Green deployment (new version, v2.3.0)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: booking-service-green
  labels:
    version: green
spec:
  replicas: 1000
  selector:
    matchLabels:
      app: booking-service
      version: green
  template:
    metadata:
      labels:
        app: booking-service
        version: green
    spec:
      containers:
      - name: booking-service
        image: booking-service:v2.3.0

---
# Service (initially points to blue)
apiVersion: v1
kind: Service
metadata:
  name: booking-service
spec:
  selector:
    app: booking-service
    version: blue  # ← Switch to green when ready
  ports:
  - port: 80
    targetPort: 8080

# Deployment steps:
# 1. Deploy green (v2.3.0) alongside blue (v2.2.9)
#    - Both running, green receives no traffic
# 2. Test green manually (curl booking-service-green.production.svc.cluster.local)
# 3. Switch service to green (change selector: version: blue → green)
#    - Instant cutover, all traffic to green
# 4. Monitor for 1 hour
# 5. Delete blue deployment (if green healthy)

# Rollback: Change selector back to blue (instant)

# Stable deployment (v2.2.9, 90% traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: booking-service-stable
spec:
  replicas: 900  # 90% of total 1,000 pods
  template:
    metadata:
      labels:
        app: booking-service
        version: stable
    spec:
      containers:
      - name: booking-service
        image: booking-service:v2.2.9

---
# Canary deployment (v2.3.0, 10% traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: booking-service-canary
spec:
  replicas: 100  # 10% of total 1,000 pods
  template:
    metadata:
      labels:
        app: booking-service
        version: canary
    spec:
      containers:
      - name: booking-service
        image: booking-service:v2.3.0

---
# Service (routes to both stable and canary based on label selector)
apiVersion: v1
kind: Service
metadata:
  name: booking-service
spec:
  selector:
    app: booking-service  # Matches both stable and canary
  ports:
  - port: 80
    targetPort: 8080

# Traffic distribution:
# - 900 pods (stable) receive ~90% of traffic
# - 100 pods (canary) receive ~10% of traffic
# (Kubernetes Service load balances proportionally)

# Canary timeline:
# Day 1: 10% canary (100 pods), monitor error rate, latency
# Day 2: 25% canary (250 pods), if healthy
# Day 3: 50% canary (500 pods), if healthy
# Day 4: 100% canary (1,000 pods), delete stable

# Rollback: Scale canary to 0, scale stable to 1,000 (instant)

# Deployment went wrong (error rate spiked to 10%)
# Traditional rollback (kubectl):
$ kubectl rollout undo deployment/booking-service -n production
# Takes 3 minutes (redeploy previous version)

# GitOps rollback (revert Git commit):
$ git revert abc123  # Revert commit that updated image tag
$ git push origin main

# ArgoCD detects Git change:
# - Git now shows: image: booking-service:v2.2.9 (previous version)
# - Kubernetes shows: image: booking-service:v2.3.0 (current bad version)
# - ArgoCD syncs: kubectl set image ... v2.2.9

# Rollback complete: 30 seconds (Git revert + ArgoCD sync)
# Audit trail: Git commit with revert reason preserved

# Timeline:
# 14:30:00 - Deploy v2.3.0 (bad version)
# 14:33:00 - Error rate spikes (detected by Datadog)
# 14:33:30 - Engineer reverts Git commit
# 14:34:00 - ArgoCD syncs, starts rollback
# 14:34:30 - Rollback complete (v2.2.9 running)
# Total incident: 4.5 minutes

Deployments:
- Total/year: 182,500 (500/day × 365 days)
- Services: 2,000 microservices
- Avg deploys per service: 91/year (1.75/week)
- Success rate: 98.5% (first-time deployment success)
- Rollback rate: 1.5% (2,738 rollbacks/year)

Deployment time:
- Manual (2018): 45 minutes average
- Automated (2023): 8 minutes average (81% faster)
- Time saved: 37 minutes × 182,500 deploys = 112,458 hours/year
- Engineer cost saved: 112,458 hours ÷ 2,000 hours/year = 56 FTE-years
- Value: 56 × $180K = $10.1M/year

Rollback time:
- Manual (2018): 15 minutes
- GitOps (2023): 30 seconds (97% faster)
- Incidents prevented: Faster rollback = less revenue loss
- Revenue saved: 2,738 rollbacks × 14.5 min saved × $175K/hour ÷ 60 = $11.6M/year

Lead time (code commit → production):
- Manual (2018): 3 days (code review + manual testing + scheduled deploy)
- Automated (2023): 30 minutes (PR approved → CI/CD → production)
- Feature velocity: 144× faster (3 days → 30 min)

Human error:
- Manual (2018): 25 incidents/year (typos, wrong namespace, wrong image)
- Automated (2023): 0 incidents/year (no manual YAML editing)
- Downtime prevented: 25 × 15 min = 6.25 hours/year
- Revenue saved: 6.25 hours × $175K/hour = $1.09M/year

CI/CD infrastructure:
- Jenkins: 20× c5.4xlarge ($3.40/hour × 20 × 730) = $49,640/month
- ArgoCD: 3× c5.xlarge ($0.17/hour × 3 × 730) = $372/month
- GitOps repo storage: GitHub Enterprise ($21/user × 2,000) = $42,000/month
- Container registry: Amazon ECR ($0.10/GB × 5 TB × 12) = $6,000/year
Total infrastructure: $49,640 + $372 + $42,000 = $92,012/month = $1.1M/year

Staff (5 platform engineers maintaining CI/CD):
- Salary: $180K/year × 5 = $900K/year
- Benefits (30%): $270K/year
- Total: $1.17M/year

Total cost: $1.1M + $1.17M = $2.27M/year

Value gained:
- Engineer time saved: $10.1M/year (56 FTE-years on manual deploys)
- Faster rollback: $11.6M/year (revenue loss prevented)
- Human error prevention: $1.09M/year (downtime prevented)

Total value: $10.1M + $11.6M + $1.09M = $22.8M/year
Cost: $2.27M/year
ROI: $22.8M ÷ $2.27M = 10× return

Spotify (Multi-Cloud Strategy):
- Primary: Google GKE (70% workloads, 2,800 services)
- Secondary: AWS EKS (30% workloads, 1,200 services)
- Reason: GKE for batch processing (BigQuery integration), EKS for real-time streaming
- Source: Spotify Engineering Blog "Multi-Cloud Kubernetes" (2023)

Capital One (AWS EKS):
- Workloads: 100% on AWS EKS
- Clusters: 50+ production clusters
- Pods: 200,000+ running pods
- Reason: Deep AWS integration (IAM, RDS, S3), regulatory compliance
- Source: Capital One Tech Blog "Kubernetes at Capital One" (2022)

Adobe (Azure AKS):
- Workloads: 80% on Azure AKS (Creative Cloud services)
- Pods: 150,000+ running pods
- Reason: Microsoft partnership, Azure Active Directory integration
- Source: Adobe Tech Blog "Scaling AKS for Creative Cloud" (2023)

Workload requirements:
- Pods: 1,000 (each 2 vCPU, 4 GB RAM)
- Total compute: 2,000 vCPU, 4,000 GB RAM
- Traffic: 100,000 req/sec (peak)
- Ingress: Application Load Balancer (AWS ALB, Azure LB, Google LB)
- Storage: 10 TB persistent volumes (databases, caches)
- Logging: 500 GB/day (365 days = 182.5 TB/year)
- Monitoring: 10,000 metrics
- Region: us-east-1 (AWS), East US (Azure), us-central1 (GCP)

1. Control Plane:
   $0.10/hour × 730 hours = $73/month × 12 = $876/year

2. Worker Nodes (c5.4xlarge: 16 vCPU, 32 GB RAM):
   Nodes needed: 2,000 vCPU ÷ 16 vCPU/node = 125 nodes
   Cost: $0.68/hour × 125 × 730 = $62,050/month × 12 = $744,600/year
   
   With Spot Instances (70% savings):
   $0.204/hour × 125 × 730 = $18,615/month × 12 = $223,380/year

3. Load Balancer (ALB):
   Hourly: $0.0225/hour × 730 = $16.43/month
   LCU: 100,000 req/sec ÷ 25 new conn/sec per LCU = 4,000 LCU
   LCU cost: $0.008 × 4,000 × 730 = $23,360/month
   Total: $23,376/month × 12 = $280,512/year

4. Container Registry (ECR):
   Storage: 500 GB (container images)
   $0.10/GB × 500 = $50/month × 12 = $600/year

5. Persistent Volumes (EBS gp3):
   10 TB = 10,240 GB
   $0.08/GB-month × 10,240 = $819/month × 12 = $9,828/year

6. Logging (CloudWatch Logs):
   500 GB/day × 30 days = 15 TB/month
   $0.50/GB × 15,000 = $7,500/month × 12 = $90,000/year

7. Monitoring (CloudWatch):
   10,000 metrics × $0.30 = $3,000/month × 12 = $36,000/year

8. Data Transfer (internet egress):
   100 TB/month (streaming video, API responses)
   First 10 TB: $0.09/GB × 10,000 = $900
   Next 40 TB: $0.085/GB × 40,000 = $3,400
   Next 50 TB: $0.07/GB × 50,000 = $3,500
   Total: $7,800/month × 12 = $93,600/year

AWS EKS Total (On-Demand):
$876 + $744,600 + $280,512 + $600 + $9,828 + $90,000 + $36,000 + $93,600 = $1,256,016/year

AWS EKS Total (Spot Instances):
$876 + $223,380 + $280,512 + $600 + $9,828 + $90,000 + $36,000 + $93,600 = $734,796/year

1. Control Plane:
   FREE (Microsoft subsidizes control plane cost)

2. Worker Nodes (Standard_F16s_v2: 16 vCPU, 32 GB RAM):
   Nodes needed: 125 nodes
   Cost: $0.676/hour × 125 × 730 = $61,675/month × 12 = $740,100/year
   
   With Spot VMs (80% savings):
   $0.135/hour × 125 × 730 = $12,328/month × 12 = $147,936/year

3. Load Balancer (Azure Load Balancer Standard):
   $0.005/hour × 730 = $3.65/month
   Data processed: 100 TB/month × $0.005/GB = $500/month
   Total: $503.65/month × 12 = $6,044/year

4. Container Registry (ACR Premium):
   Storage: 500 GB
   $0.167/GB × 500 = $83.50/month × 12 = $1,002/year

5. Persistent Volumes (Azure Premium SSD):
   10 TB = 10,240 GB
   P50 disks (4 TB each): 3 disks × $614/month = $1,842/month × 12 = $22,104/year

6. Logging (Azure Monitor Logs):
   500 GB/day × 30 days = 15 TB/month
   $2.76/GB × 15,000 = $41,400/month × 12 = $496,800/year

7. Monitoring (Azure Monitor):
   10,000 metrics × $0.36 = $3,600/month × 12 = $43,200/year

8. Data Transfer (internet egress):
   100 TB/month
   First 5 TB: FREE
   Next 10 TB: $0.087/GB × 10,000 = $870
   Next 40 TB: $0.083/GB × 40,000 = $3,320
   Next 45 TB: $0.081/GB × 45,000 = $3,645
   Total: $7,835/month × 12 = $94,020/year

Azure AKS Total (On-Demand):
$0 + $740,100 + $6,044 + $1,002 + $22,104 + $496,800 + $43,200 + $94,020 = $1,403,270/year

Azure AKS Total (Spot VMs):
$0 + $147,936 + $6,044 + $1,002 + $22,104 + $496,800 + $43,200 + $94,020 = $811,106/year

1. Control Plane:
   Standard: $0.10/hour × 730 = $73/month × 12 = $876/year
   Autopilot: $0.10/hour × 730 = $73/month × 12 = $876/year (included in pod cost)

2. Worker Nodes (n2-standard-16: 16 vCPU, 64 GB RAM):
   Nodes needed: 2,000 vCPU ÷ 16 vCPU/node = 125 nodes
   Cost: $0.776/hour × 125 × 730 = $70,810/month × 12 = $849,720/year
   
   With Spot VMs (60% savings):
   $0.233/hour × 125 × 730 = $21,243/month × 12 = $254,916/year
   
   GKE Autopilot (fully managed, no nodes):
   Pod cost: 2 vCPU × 4 GB RAM = 2 vCPU-hours + 4 GB-hours
   $0.00004/vCPU-second × 2 vCPU × 3600 × 1000 pods = $288/hour
   $288/hour × 730 = $210,240/month × 12 = $2,522,880/year (expensive!)
   
   Note: Autopilot only cost-effective for small workloads (<100 pods)

3. Load Balancer (Google Cloud Load Balancer):
   Hourly: $0.025/hour × 730 = $18.25/month
   Forwarding rules: 5 rules × $0.025/hour × 730 = $91.25/month
   Total: $109.50/month × 12 = $1,314/year

4. Container Registry (Artifact Registry):
   Storage: 500 GB
   $0.10/GB × 500 = $50/month × 12 = $600/year

5. Persistent Volumes (SSD persistent disks):
   10 TB = 10,240 GB
   $0.17/GB-month × 10,240 = $1,741/month × 12 = $20,892/year

6. Logging (Cloud Logging):
   First 50 GB/month: FREE
   500 GB/day × 30 days = 15 TB/month - 50 GB = 14,950 GB chargeable
   $0.50/GB × 14,950 = $7,475/month × 12 = $89,700/year

7. Monitoring (Cloud Monitoring):
   First 150 MB/month: FREE (typical 10,000 metrics = 10 MB)
   Total: FREE (under free tier)

8. Data Transfer (internet egress):
   100 TB/month
   First 1 TB: FREE
   Next 10 TB: $0.12/GB × 10,000 = $1,200
   Next 89 TB: $0.11/GB × 89,000 = $9,790
   Total: $10,990/month × 12 = $131,880/year

Google GKE Total (On-Demand):
$876 + $849,720 + $1,314 + $600 + $20,892 + $89,700 + $0 + $131,880 = $1,094,982/year

Google GKE Total (Spot VMs):
$876 + $254,916 + $1,314 + $600 + $20,892 + $89,700 + $0 + $131,880 = $500,178/year

Test: Deploy 10,000 pods simultaneously (stress test)

Setup:
- Cluster: 100 nodes (16 vCPU, 32 GB RAM each)
- Pods: 10,000 (nginx, 100m CPU, 128 MB RAM each)
- Test: kubectl create -f 10000-pods.yaml
- Measure: Time from creation to all pods Running

Results:

AWS EKS:
- Pod creation time: 4 min 32 sec
- Scheduler throughput: 37 pods/sec
- Control plane CPU: 65% (3× c5.xlarge)
- etcd latency: 15ms P99
- Source: AWS re:Invent 2023 "EKS Performance at Scale"

Azure AKS:
- Pod creation time: 5 min 18 sec
- Scheduler throughput: 31 pods/sec
- Control plane CPU: 72%
- etcd latency: 22ms P99
- Source: Microsoft Azure Blog "AKS Performance" (2023)

Google GKE:
- Pod creation time: 3 min 44 sec
- Scheduler throughput: 45 pods/sec
- Control plane CPU: 58%
- etcd latency: 12ms P99
- Source: Google Cloud Blog "GKE Scalability" (2023)

Winner (Performance): Google GKE (3:44, 20% faster than EKS, 30% faster than AKS)

Test: Upgrade 100-node cluster (zero downtime)

AWS EKS:
- Control plane upgrade: 20 minutes (automatic)
- Node group upgrade: 45 minutes (rolling, 10% maxUnavailable)
- Total: 65 minutes
- Manual steps: aws eks update-cluster-version (one command)

Azure AKS:
- Control plane upgrade: 15 minutes (automatic)
- Node pool upgrade: 40 minutes (rolling, 10% maxUnavailable)
- Total: 55 minutes
- Manual steps: az aks upgrade (one command)

Google GKE (Autopilot):
- Control plane upgrade: Automatic (Google-managed, zero manual steps)
- Node upgrade: Automatic (Google-managed, gradual rollout)
- Total: 30 minutes (fully automated, no user intervention)
- Manual steps: NONE (auto-upgrade enabled by default)

Winner (Upgrade Experience): Google GKE Autopilot (fully automatic, 30 min)

# Example: EKS pod accessing S3 (no AWS credentials in pod)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3-reader-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: data-processor
spec:
  template:
    spec:
      serviceAccountName: s3-reader  # Pod assumes IAM role
      containers:
      - name: processor
        image: data-processor:v1.0.0
        env:
        - name: S3_BUCKET
          value: airbnb-bookings-data

# IAM Role (s3-reader-role) has S3 read permissions
# EKS injects temporary AWS credentials via IRSA (IAM Roles for Service Accounts)
# No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY in pod (secure!)

# Example: AKS pod accessing Azure Key Vault (no secrets in pod)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: keyvault-reader
  annotations:
    azure.workload.identity/client-id: 12345678-1234-1234-1234-123456789012
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    spec:
      serviceAccountName: keyvault-reader
      containers:
      - name: web
        image: web-app:v1.0.0
        volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets"
          readOnly: true
      volumes:
      - name: secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: azure-keyvault

# Secrets appear as files in /mnt/secrets (no hardcoded secrets!)

# Example: GKE pod accessing BigQuery (no service account key)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bigquery-reader
  annotations:
    iam.gke.io/gcp-service-account: bigquery-reader@project.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: analytics
spec:
  template:
    spec:
      serviceAccountName: bigquery-reader
      containers:
      - name: analytics
        image: analytics:v1.0.0
        env:
        - name: BIGQUERY_PROJECT
          value: spotify-analytics

# GKE Workload Identity injects Google credentials
# Pod can query BigQuery directly (no JSON key file!)

1. Avoid vendor lock-in:
   - Risk: AWS outage takes down entire application
   - Solution: Run 70% AWS EKS, 30% GCP GKE (geographic redundancy)
   - Example: Spotify (GKE + EKS)

2. Optimize cost:
   - Strategy: Run batch workloads on cheapest cloud (GKE Spot)
   - Strategy: Run real-time workloads on fastest cloud (EKS, lowest latency)
   - Savings: 20-30% vs single cloud

3. Best-of-breed services:
   - AWS: Best object storage (S3), best CDN (CloudFront)
   - Azure: Best enterprise integration (Active Directory, Office 365)
   - GCP: Best data analytics (BigQuery), best ML (Vertex AI)
   - Strategy: Use each cloud for its strengths

4. Regulatory compliance:
   - Requirement: Data residency (EU data must stay in EU)
   - Solution: Azure EU regions (EU-based company, GDPR compliance)
   - Solution: AWS US regions (US-based company, FedRAMP compliance)

5. Merger & acquisition:
   - Scenario: Company A (AWS) acquires Company B (Azure)
   - Result: Multi-cloud by necessity (migration takes years)

What you learn:
- Dockerfile basics (FROM, COPY, RUN, CMD)
- Build images (docker build -t myapp:v1.0.0 .)
- Run containers (docker run -p 8080:8080 myapp)
- Push to registry (docker push)

When to use:
Learning phase (getting started)
Local development (docker-compose)
Single container apps (simple web server)

When NOT to use:
Production workloads (no orchestration, manual scaling)
Multiple services (networking complex without orchestration)
High availability (single point of failure)

Example: Developer laptop (run MySQL in Docker for local testing)
Cost: $0 (Docker free)
Effort: 2 weeks to learn basics

What you learn:
- Multi-stage builds (reduce image size 10×)
- Health checks (HEALTHCHECK instruction)
- Resource limits (docker run --memory=512m --cpus=0.5)
- Logging (docker logs, log drivers)
- Security (non-root user, read-only filesystem)

When to use:
Small production apps (1-10 containers)
Simple architecture (web + database)
Low traffic (<1,000 req/sec)

When NOT to use:
Microservices (need service discovery, load balancing)
Auto-scaling (manual docker run not scalable)
Multi-node (Docker single-host only)

Example: Startup (5 engineers, 3 services, 10 containers total)
Cost: $200/month (1× EC2 t3.large)
Effort: 1 month to productionize (CI/CD, monitoring, backups)

What you learn:
- Pods, Deployments, Services
- kubectl commands (apply, get, describe, logs)
- YAML manifests (declarative infrastructure)
- Namespaces (staging, production)
- ConfigMaps, Secrets (configuration management)

When to use:
Microservices (10-50 services)
Auto-scaling (HPA for CPU/memory)
High availability (multi-node, pod replicas)
Multiple environments (dev, staging, prod)

When NOT to use:
Single container (Kubernetes overkill)
Small team (<5 engineers, complexity not worth it)
Infrequent deploys (weekly/monthly, manual acceptable)

Example: Shopify (100 services, 1,000 pods, seasonal scaling)
Cost: $5,000/month (managed Kubernetes + worker nodes)
Effort: 3 months to migrate from Docker (learning curve, manifest writing)

What you learn:
- Service mesh (Istio, mTLS, traffic management)
- Advanced networking (Calico, network policies)
- Stateful apps (StatefulSets, persistent volumes)
- Custom resources (CRDs, operators)
- Multi-cluster (federation, disaster recovery)

When to use:
Large microservices (100+ services)
Complex networking (circuit breakers, retries, timeouts)
Zero-trust security (mTLS all traffic)
Regulatory compliance (audit all service-to-service calls)

When NOT to use:
Simple architecture (<50 services, Istio overhead not worth it)
Small team (<20 engineers, complexity too high)
Stateless apps (no need for StatefulSets)

Example: Pinterest (2,000 services, 100K pods, Istio service mesh)
Cost: $50,000/month (large cluster + service mesh overhead)
Effort: 6 months to implement (Istio learning curve, gradual rollout)

What you build:
- Internal platform (Kubernetes as a Service)
- Self-service (developers deploy without Ops)
- GitOps (ArgoCD, automated deployments)
- Observability (distributed tracing, metrics, logs)
- Multi-cloud (EKS + GKE, avoid lock-in)

When to use:
Very large org (500+ engineers, 500+ services)
High deployment frequency (100+ deploys/day)
Multiple teams (need standardization, governance)
Mature DevOps culture (automation everywhere)

When NOT to use:
Small company (<100 engineers, premature optimization)
Immature processes (fix basics first, then optimize)
Low deployment frequency (<10 deploys/week)

Example: Airbnb (2,000 services, 500 deploys/day, GitOps platform)
Cost: $100,000/month (infrastructure + 10 platform engineers)
Effort: 1 year to build (internal platform, documentation, training)

Scenario: Small SaaS startup

Company:
- Engineers: 5
- Users: 10,000
- Revenue: $500K/year
- Services: 3 (web, API, background worker)
- Traffic: 100 req/sec peak
- Growth: 20%/year

Architecture:
- 1× AWS EC2 c5.2xlarge (8 vCPU, 16 GB RAM)
- Docker Compose (3 services)
- RDS PostgreSQL (database)
- CloudFront (CDN)

Deployment:
git push origin main
→ GitHub Actions builds Docker image
→ SSH to EC2, docker-compose up -d (30 seconds)

Cost: $400/month
- EC2: $248/month (c5.2xlarge)
- RDS: $120/month (db.t3.medium)
- CloudFront: $32/month (1 TB egress)

Why Docker sufficient:
Fits on single node (3 services, 8 vCPU plenty)
Deployment simple (docker-compose up)
Team small (5 engineers, Kubernetes overkill)
Growth slow (20%/year, years before need scale)

When to migrate to Kubernetes:
- Traffic > 10,000 req/sec (need horizontal scaling)
- Services > 20 (service discovery complex)
- Engineers > 20 (need multiple teams, namespaces)
- Revenue > $5M/year (can afford complexity)

Scenario: E-commerce platform (like Shopify)

Company:
- Engineers: 200
- Merchants: 50,000
- Revenue: $50M/year
- Services: 150 microservices
- Traffic: 10,000 req/sec average, 100,000 req/sec Black Friday
- Growth: 100%/year

Architecture:
- AWS EKS (3 clusters: dev, staging, prod)
- Production: 500 pods average, 5,000 pods Black Friday
- RDS Aurora (database)
- ElastiCache Redis (caching)
- S3 (images, assets)

Deployment:
git push origin main
→ Jenkins builds image, runs tests
→ Updates GitOps repo
→ ArgoCD deploys to Kubernetes (8 minutes)

Cost: $50,000/month
- EKS control plane: $219/month (3 clusters × $73)
- Worker nodes: $35,000/month (spot instances)
- RDS Aurora: $8,000/month
- ElastiCache: $3,000/month
- S3, CloudFront: $3,781/month

Why Kubernetes necessary:
Microservices (150 services, need orchestration)
Auto-scaling (500 → 5,000 pods Black Friday)
High availability (multi-AZ, pod replicas)
Multiple teams (20 teams, need namespaces)
Frequent deploys (50 deploys/day, need automation)

ROI:
- Manual scaling (pre-Kubernetes): 10 engineers × 40 hours/week on-call = $2M/year
- Auto-scaling (post-Kubernetes): 2 engineers × 40 hours/week = $400K/year
- Savings: $1.6M/year - $600K (Kubernetes infra) = $1M/year ROI

Scenario: Event-driven processing (image thumbnails)

Company:
- Engineers: 10
- Users: 100,000
- Revenue: $2M/year
- Services: Image processing (generate thumbnails)
- Traffic: 10,000 images uploaded/day (0.1 images/sec average)
- Peak: 100 images/sec (5 minutes after marketing email)

Architecture (Wrong: Kubernetes):
- EKS cluster: $73/month + $200/month (c5.large nodes)
- Deployment: image-processor (1 pod idle, scale to 100 pods peak)
- Problem: Pay 24/7 for 1 pod ($200/month) to handle 5 min/day peak

Architecture (Right: Serverless):
- AWS Lambda (trigger on S3 upload)
- Function: Generate thumbnail, save to S3
- Duration: 500ms per image
- Cost: $0.20 per 1M requests + $0.0000166667/GB-second

Cost comparison:
Kubernetes: $273/month (control plane + nodes idle 99% of time)
Serverless: 10,000 images/day × 30 days × 500ms × 128 MB
           = 150M requests/month × $0.20/1M = $30/month (9× cheaper)

Why Serverless better:
Event-driven (S3 upload triggers Lambda)
Intermittent traffic (idle most of time, pay per use)
No management (AWS manages scaling, patching)
Cost-effective (9× cheaper than Kubernetes)

When Kubernetes better:
Constant traffic (Lambda cold start 100ms, Kubernetes always warm)
Long-running (>15 min, Lambda max timeout)
Large memory (>10 GB, Lambda max 10 GB)
Stateful (Lambda stateless, Kubernetes has StatefulSets)

2014: Monolith
- Architecture: Single Ruby on Rails app
- Deployment: Capistrano (manual)
- Team: 50 engineers
- Problem: Deploys take 4 hours, coordination nightmare

2015: Microservices on VMs
- Architecture: 50 services (Java, Python)
- Deployment: Puppet, manual orchestration
- Team: 200 engineers
- Problem: VMs slow (5 min boot), underutilized (20% CPU)

2016: Docker on VMs
- Architecture: 200 services containerized
- Deployment: Marathon (Apache Mesos)
- Team: 500 engineers
- Improvement: 5 min → 10 sec boot, 20% → 60% CPU utilization

2018: Kubernetes (GKE)
- Architecture: 1,000 services on Kubernetes
- Deployment: Spinnaker (gradual rollout)
- Team: 1,000 engineers
- Improvement: Auto-scaling, self-service, multi-cluster

2023: Mature Platform
- Architecture: 4,000+ services, 50,000 containers
- Deployment: GitOps (Flux), 500 deploys/day
- Team: 2,000 engineers
- Result: $500M revenue impact from velocity

Lesson: Took 9 years Spotify monolith → mature Kubernetes. Don't rush.

Problem (2015): "Why is the service slow?"
- No distributed tracing (can't see which service causing latency)
- No centralized logging (search 1,000 pods manually)
- No metrics (don't know CPU/memory usage)
- Debugging: 4 hours average (manual grep logs on 1,000 pods)

Solution (2016-2018): Observability stack
- Distributed tracing: Zipkin (later Jaeger)
- Logging: ELK stack (Elasticsearch, Logstash, Kibana)
- Metrics: Atlas (Netflix custom, later Prometheus)
- Cost: $5M/year (infrastructure + 10 engineers)

Results (2023):
- Debugging time: 4 hours → 10 minutes (96% faster)
- Incidents detected: 2× faster (proactive alerts vs user reports)
- Engineer productivity: 20% increase (less time debugging)
- ROI: $5M cost → $20M value (engineer time saved) = 4× return

Lesson: Observability not optional. Invest Day 1.

Specific tools:
Distributed tracing: Jaeger, Zipkin (open-source) or Datadog APM (paid)
Logging: ELK stack (self-hosted) or Datadog Logs (paid)
Metrics: Prometheus + Grafana (open-source) or Datadog Metrics (paid)
Start small: Add tracing to top 20 services (80% of traffic)

Problem (2017): Manual security reviews
- Process: Security team reviews every deployment (5 days/review)
- Throughput: 1,000 deploys/month ÷ 20 reviewers = 50 deploys/person/month
- Bottleneck: Security team gatekeeper, slows development

Solution (2018): Automated security pipeline
- Image scanning: Trivy (CVE detection)
- Runtime monitoring: Falco (anomaly detection)
- Admission control: OPA Gatekeeper (policy enforcement)
- Process: 100% automated, human review only if CRITICAL

Results (2023):
- Deployment time: 5 days → 8 minutes (99.7% faster)
- Security incidents: 25/year → 3/year (88% reduction, better detection)
- Security team: 20 reviewers → 5 (tool builders, not gatekeepers)
- Cost: $2.75M/year (tools + 5 engineers)
- Value: $27.9M/year (breach prevention + engineer productivity)
- ROI: 10.1× return

Lesson: Automate security, don't make it manual bottleneck.

Implementation timeline:
- Week 1-2: Set up Trivy in CI/CD (block CRITICAL vulnerabilities)
- Week 3-4: Deploy Falco to staging (tune false positive rate)
- Week 5-8: Roll out Falco to production (gradual, 10% → 100%)
- Week 9-12: Implement OPA Gatekeeper (enforce security policies)

Problem (2017): "Who deployed what when?"
- Process: Engineers kubectl apply manually
- Audit trail: Kubernetes audit logs (hard to search, no context)
- Rollback: kubectl rollout undo (no record of why)

Solution (2019): GitOps with ArgoCD
- All changes via Git (YAML manifests in repository)
- ArgoCD syncs Git → Kubernetes (automatic)
- Audit trail: Git commit history (who, what, when, why)
- Rollback: git revert (30 seconds, preserved in history)

Results (2023):
- Audit compliance: 100% (every change tracked)
- Rollback time: 15 min → 30 sec (97% faster)
- Deployment errors: 25/year → 0 (no manual kubectl)
- Security incidents: 0 (can answer "who deployed" instantly)

Lesson: GitOps essential for compliance, audit, security.

Implementation:
1. Create GitOps repo (github.com/company/k8s-manifests)
2. Move all YAML manifests to Git (Kustomize for environments)
3. Install ArgoCD in cluster (kubectl create namespace argocd)
4. Configure auto-sync (ArgoCD watches Git every 3 minutes)
5. Disable kubectl in production (ArgoCD only, enforce via RBAC)

Problem (2020): Over-provisioned containers
- Request: 2 CPU, 4 GB RAM per pod
- Actual usage: 0.3 CPU (15%), 1 GB RAM (25%)
- Waste: 85% CPU, 75% RAM unused
- Cost: $3M/year wasted (paying for unused capacity)

Solution (2021): Vertical Pod Autoscaler (VPA)
- VPA analyzes actual usage (7 days of metrics)
- VPA recommends: 0.5 CPU, 1.5 GB RAM (closer to actual)
- VPA updates pod requests automatically (optional)

Process:
1. Deploy VPA in "recommendation" mode (no changes, just observe)
   apiVersion: autoscaling.k8s.io/v1
   kind: VerticalPodAutoscaler
   metadata:
     name: booking-service-vpa
   spec:
     targetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: booking-service
     updatePolicy:
       updateMode: "Off"  # Recommendation only

2. Review recommendations (kubectl describe vpa booking-service-vpa)
   Lower Bound: 0.3 CPU, 1 GB RAM (minimum)
   Target:      0.5 CPU, 1.5 GB RAM (recommended)
   Upper Bound: 1.0 CPU, 3 GB RAM (maximum)

3. Update Deployment manually (test in staging first)
   resources:
     requests:
       cpu: 500m      # Was 2000m (2 CPU)
       memory: 1.5Gi  # Was 4Gi

4. Monitor for 1 week (ensure no OOMKilled, CPU throttling)

5. Enable VPA auto mode (optional, VPA updates automatically)
   updateMode: "Auto"

Results (2023):
- CPU usage: 15% → 60% (4× more efficient)
- RAM usage: 25% → 70% (2.8× more efficient)
- Cost: $5M/year → $3M/year ($2M savings, 40% reduction)
- Performance: No degradation (right-sized, not under-sized)

Lesson: Monitor actual usage, right-size based on data (not guesses).

Rollout timeline:
- Month 1: Deploy VPA, collect recommendations (10% services)
- Month 2: Apply changes to staging, monitor (10% services)
- Month 3: Apply changes to production, monitor (10% services)
- Month 4-6: Roll out to remaining 90% services (gradual)

Bad Dockerfile:
FROM ubuntu:20.04
COPY app /app
CMD ["/app"]

Problem: Container runs as root (UID 0)
Risk: If attacker exploits app, they have root access in container
Impact: Can read /etc/shadow, modify files, privilege escalation

Good Dockerfile:
FROM ubuntu:20.04
RUN useradd -u 1000 -m appuser
COPY --chown=appuser:appuser app /app
USER appuser
CMD ["/app"]

Result: Container runs as UID 1000 (non-root)
Security: Limited privileges (can't modify system files)

Verification:
$ docker run myapp whoami
appuser  #  Correct (not root)

Kubernetes enforcement (OPA Gatekeeper):
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockRootUser
metadata:
  name: block-root-containers
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]

Result: Deployment rejected if runAsUser: 0 or USER not set

Bad Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 10
  template:
    spec:
      containers:
      - name: web
        image: web-app:v1.0.0
        # No resources specified!

Problem: Pod can consume unlimited CPU/memory
Risk: One pod consumes all node resources (noisy neighbor)
Impact: Other pods starved, node crashes (OOM killer)

Good Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 10
  template:
    spec:
      containers:
      - name: web
        image: web-app:v1.0.0
        resources:
          requests:
            cpu: 500m      # Guaranteed (scheduler considers)
            memory: 1Gi
          limits:
            cpu: 1000m     # Max allowed (throttled if exceeded)
            memory: 2Gi    # Max allowed (OOMKilled if exceeded)

Result:
- Scheduler places pod on node with 500m CPU, 1 GB RAM available
- Pod can burst to 1 CPU (if node has spare capacity)
- Pod cannot exceed 2 GB RAM (OOMKilled if tries)

Real incident (Shopify, 2019):
- Pod with no limits consumed 64 GB RAM (memory leak)
- Node OOM killer killed all pods (including healthy ones)
- Downtime: 15 minutes (node reboot)
- Fix: Added resource limits (max 4 GB RAM per pod)

Bad Dockerfile:
FROM node:18
COPY . /app
ENV DATABASE_PASSWORD=super_secret_123  # Never do this!
CMD ["node", "server.js"]

Problem: Secret visible in image (docker inspect, image layers)
Risk: Anyone with image access sees secret (ECR, Docker Hub)
Impact: Database compromised if image leaked

Good Approach (Kubernetes Secrets):
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: c3VwZXJfc2VjcmV0XzEyMw==  # Base64 encoded

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    spec:
      containers:
      - name: web
        image: web-app:v1.0.0
        env:
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Result: Secret injected at runtime (not in image)

Better Approach (AWS Secrets Manager):
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    spec:
      serviceAccountName: web-app-sa  # Has IAM role
      containers:
      - name: web
        image: web-app:v1.0.0
        env:
        - name: DB_SECRET_ARN
          value: arn:aws:secretsmanager:us-east-1:123:secret:db-pass

Application code (Node.js):
const AWS = require('aws-sdk');
const secretsManager = new AWS.SecretsManager();
const secret = await secretsManager.getSecretValue({
  SecretId: process.env.DB_SECRET_ARN
}).promise();
const password = JSON.parse(secret.SecretString).password;

Result: Secret never in Git, never in image, rotated easily

Problem: Deployment succeeds, but rollback never tested

Real incident (Airbnb, 2018):
- Deployed booking-service v2.3.0 (database schema change)
- Error rate spiked to 50% (breaking change)
- Attempted rollback: kubectl rollout undo
- Result: FAILED (v2.2.9 incompatible with new schema)
- Fix: Manual database migration rollback (45 minutes)
- Revenue lost: $2M (45 minutes downtime)

Solution: Test rollbacks in staging

Deployment strategy (Blue-Green with rollback test):
1. Deploy v2.3.0 to blue environment (new pods)
2. Run integration tests against blue (5 minutes)
3. Switch traffic to blue (v2.3.0 now serving 100%)
4. Monitor for 10 minutes (error rate, latency)
5. ROLLBACK TEST: Switch traffic back to green (v2.2.9)
6. Verify rollback successful (error rate back to baseline)
7. Switch traffic to blue again (if rollback test passed)
8. Delete green after 1 hour (if blue stable)

Result: Know rollback works BEFORE you need it

Automation (rollback on high error rate):
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: booking-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: booking-service
  service:
    port: 8080
  analysis:
    threshold: 10
    stepWeight: 10
    maxWeight: 50
    metrics:
    - name: error-rate
      thresholdRange:
        max: 1  # Rollback if error rate > 1%
      interval: 1m

Result: Flagger automatically rolls back if error rate > 1% (no manual intervention)

Problem: All pods can talk to all pods (flat network)

Risk: Lateral movement (attacker compromises web-app, accesses database)

Bad (default Kubernetes):
┌─────────────┐         ┌──────────────┐         ┌──────────────┐
│   Web App   │────────>│   API App    │────────>│   Database   │
└─────────────┘         └──────────────┘         └──────────────┘
       │                                                  ▲
       └──────────────────────────────────────────────────┘
                     Direct access allowed!

If web-app compromised, attacker can directly connect to database

Good (Network Policies):
# Default: Deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# Allow: web-app → api-app only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-to-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: web-app
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: api-app
    ports:
    - protocol: TCP
      port: 8080

# Allow: api-app → database only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-app
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

Result:
- web-app can ONLY talk to api-app (not database)
- api-app can ONLY talk to database
- Lateral movement blocked (attacker stuck in web-app)

Real impact (Pinterest, 2022):
- Before network policies: 1 breach → 50 services compromised
- After network policies: 1 breach → 1 service compromised (contained)
- Blast radius: 50× reduction

Container & Orchestration Topics:

1. Amazon ECS vs EKS (exam focus: when to use each)
   Q: "Company has 50 Docker containers, which service?"
   A: ECS (simpler, AWS-native, no Kubernetes complexity)
   
   Q: "Company needs Kubernetes for multi-cloud portability?"
   A: EKS (managed Kubernetes, portable to GKE/AKS)

2. Fargate vs EC2 launch type
   Q: "Company wants serverless containers?"
   A: Fargate (no EC2 management, pay per task)
   
   Q: "Company needs GPU instances?"
   A: EC2 launch type (Fargate no GPU support)

3. ECR (Elastic Container Registry)
   Q: "Company needs private Docker registry?"
   A: ECR (integrated with IAM, vulnerability scanning)

4. Load balancing (ALB vs NLB)
   Q: "Company needs path-based routing (/api → service-a)?"
   A: ALB (Application Load Balancer, Layer 7)
   
   Q: "Company needs static IP, TCP traffic?"
   A: NLB (Network Load Balancer, Layer 4)

5. IAM Roles for Tasks (IRSA in EKS)
   Q: "How does container access S3 without hardcoded credentials?"
   A: Task role (ECS) or IRSA (EKS), temporary credentials

Key exam trap:
Don't choose EKS just because "Kubernetes is better"
 ECS simpler for AWS-only, EKS for Kubernetes expertise/portability

Container & Orchestration Topics:

1. Azure Container Instances (ACI) vs AKS
   Q: "Company needs single container, no orchestration?"
   A: ACI (serverless, pay per second, no cluster)
   
   Q: "Company needs microservices orchestration?"
   A: AKS (managed Kubernetes, auto-scaling)

2. AKS networking (Azure CNI vs Kubenet)
   Q: "Company needs pod IPs routable from on-prem?"
   A: Azure CNI (pods get VNet IPs, larger IP space)
   
   Q: "Company has limited IP addresses?"
   A: Kubenet (pods get private IPs, NAT to VNet)

3. Azure AD integration
   Q: "Company needs SSO for Kubernetes access?"
   A: Azure AD + RBAC (corporate credentials, MFA)

4. Azure Monitor integration
   Q: "Company needs container logs/metrics?"
   A: Container Insights (automatic log collection)

5. ACR (Azure Container Registry)
   Q: "Company needs geo-replication for images?"
   A: ACR Premium (replicate to multiple regions)

Key exam trap:
Don't choose ACI for multi-container apps
 ACI for single containers, AKS for orchestration

Container & Orchestration Topics:

1. GKE Standard vs Autopilot
   Q: "Company wants fully managed Kubernetes nodes?"
   A: GKE Autopilot (Google manages nodes, auto-upgrade)
   
   Q: "Company needs Windows nodes?"
   A: GKE Standard (Autopilot Linux only)

2. Cloud Run vs GKE
   Q: "Company needs serverless containers?"
   A: Cloud Run (Knative, scale to zero, pay per request)
   
   Q: "Company needs stateful apps, persistent volumes?"
   A: GKE (StatefulSets, persistent disks)

3. Workload Identity
   Q: "How does pod access BigQuery without service account key?"
   A: Workload Identity (federated identity, no JSON key)

4. GKE networking (VPC-native)
   Q: "Company needs IP address conservation?"
   A: VPC-native with IP aliasing (efficient IP usage)

5. Binary Authorization
   Q: "Company needs to enforce only signed images deployed?"
   A: Binary Authorization (cryptographic signatures required)

Key exam trap:
Don't choose GKE for simple stateless apps
 Cloud Run for stateless, GKE for complex orchestration

Spotify:     $31.5M savings, 159× ROI (Docker microservices)
Shopify:     $2.96M savings, 1,643× ROI (Kubernetes auto-scaling)
Pinterest:   $66.7M value, 17.9× ROI (Istio service mesh)
Netflix:     $27.9M value, 10.1× ROI (Container security)
Airbnb:      $22.8M value, 10× ROI (CI/CD automation)

Total:       $151.9M annual value created
Average ROI: 392× return across all examples