Shared Cloud Network (pre-VPC):
├─ Company A's database: Public IP 54.23.45.67
├─ Company B's database: Public IP 54.23.45.68
├─ Anyone can try to connect to any IP
└─ Security relies solely on passwords (risky!)

Your Private VPC (10.0.0.0/16):
├─ Private subnet: 10.0.1.0/24 (databases, no internet access)
│  └─ Your database: 10.0.1.50 (only accessible from your VPC)
├─ Public subnet: 10.0.2.0/24 (web servers, internet accessible)
│  └─ Your web server: 10.0.2.10 (can access database privately)
└─ Firewall rules: You control every connection

Your Database → Public Internet → Your Application
├─ Risk: Hackers can try to connect
├─ Latency: 20-100ms (routing through internet)
├─ Cost: $0.09/GB data transfer
└─ Security: Must rely on passwords/encryption

Your Database → Private VPC → Your Application
├─ Risk: Only your apps can connect (firewall enforced)
├─ Latency: <1ms (direct private connection)
├─ Cost: $0.01/GB data transfer (10× cheaper)
└─ Security: Network-level isolation + passwords

Global Architecture (Simplified):

Region us-east-1 (N. Virginia) - Primary
├── VPC 10.0.0.0/16 (65,536 IPs)
│   ├── Public Subnet: 10.0.1.0/24 (256 IPs) - Load balancers, NAT gateways
│   ├── Private Subnet: 10.0.10.0/23 (512 IPs) - API services (Zuul gateway)
│   ├── Private Subnet: 10.0.20.0/22 (1,024 IPs) - Application tier (microservices)
│   ├── Private Subnet: 10.0.30.0/22 (1,024 IPs) - Data tier (Cassandra, EVCache)
│   └── Private Subnet: 10.0.50.0/23 (512 IPs) - Internal tools, monitoring

Region eu-west-1 (Ireland) - GDPR compliance
├── VPC 10.1.0.0/16
│   └── [Similar subnet structure]

Region ap-southeast-1 (Singapore) - APAC users
├── VPC 10.2.0.0/16
│   └── [Similar subnet structure]

... (Total: 3 regions for core services, 21 regions for edge caching)

Why This Design:

1. Public Subnets (10.0.1.0/24):
   - Only load balancers and NAT gateways
   - Internet-facing (attached to Internet Gateway)
   - Small CIDR (/24 = 256 IPs) because few resources need public IPs
   - Security: Locked down by security groups (only ports 80, 443 open)

2. Private Subnets (10.0.10.0/22, etc.):
   - No direct internet access (route through NAT gateway)
   - Larger CIDR blocks (/22 = 1,024 IPs, /23 = 512 IPs)
   - Security: Defense in depth (even if service compromised, no internet egress)
   - Segmented by tier (API, application, data isolation)

CIDR Allocation Math:
   /16 = 65,536 IPs (VPC level)
   /22 = 1,024 IPs (larger subnets for microservices)
   /23 = 512 IPs (medium subnets for APIs)
   /24 = 256 IPs (small subnets for edge resources)

Reserved IPs (AWS):
   Every subnet loses 5 IPs:
   - 10.0.1.0 = Network address (reserved)
   - 10.0.1.1 = VPC router (reserved)
   - 10.0.1.2 = DNS server (reserved)
   - 10.0.1.3 = Future use (reserved)
   - 10.0.1.255 = Broadcast (reserved, even though VPC doesn't support broadcast)
   
   Example: /24 = 256 IPs - 5 reserved = 251 usable IPs

Public Route Table (attached to public subnets):
Destination         Target              Purpose
10.0.0.0/16         local               VPC-internal traffic (stays in VPC)
0.0.0.0/0           igw-12345           Internet traffic (via Internet Gateway)

Private Route Table (attached to private subnets):
Destination         Target              Purpose
10.0.0.0/16         local               VPC-internal traffic
0.0.0.0/0           nat-67890           Outbound internet (via NAT Gateway)
10.1.0.0/16         pcx-abc123          Inter-region traffic (VPC peering to eu-west-1)
192.168.0.0/16      vgw-def456          Corporate access (VPN Gateway to Netflix offices)

Why NAT Gateway:
- Private instances need outbound internet (download patches, call external APIs)
- NAT Gateway in public subnet provides one-way internet access
- Stateful firewall (responses to outbound requests allowed back)
- Cost: $0.045/hour + $0.045/GB data processed = ~$35/month + data transfer

Netflix Scale:
- NAT Gateways: 3 per region (one per AZ for high availability)
- Data processed: 1+ PB/month outbound (microservice API calls)
- Cost: $35 × 3 × 3 regions = $315/month + $45K/month data = ~$45.3K/month

Security Groups (Stateful Firewall at Instance Level):

SG-LoadBalancer (ELB):
Inbound:
    Port 80 (HTTP):     0.0.0.0/0 (any IP)        → Allow all internet traffic
    Port 443 (HTTPS):   0.0.0.0/0 (any IP)        → Allow all internet traffic
Outbound:
    All traffic:        0.0.0.0/0                 → Return responses (stateful)

SG-API-Tier (Zuul API Gateway):
Inbound:
    Port 8080:          SG-LoadBalancer           → Only from load balancers
    Port 7001:          SG-Application-Tier       → Eureka service discovery
Outbound:
    Port 443:           0.0.0.0/0                 → Call external APIs (AWS services)
    Port 7002:          SG-Application-Tier       → Call downstream microservices

SG-Application-Tier (Microservices):
Inbound:
    Port 7002:          SG-API-Tier               → Requests from API gateway
    Port 7002:          SG-Application-Tier       → Inter-microservice communication
    Port 7001:          SG-Application-Tier       → Eureka heartbeats
Outbound:
    Port 7199:          SG-Data-Tier              → Cassandra queries
    Port 11211:         SG-Data-Tier              → EVCache (memcached protocol)

SG-Data-Tier (Cassandra, EVCache):
Inbound:
    Port 7199:          SG-Application-Tier       → Cassandra native protocol
    Port 11211:         SG-Application-Tier       → EVCache get/set operations
    Port 7000:          SG-Data-Tier              → Cassandra inter-node gossip
Outbound:
    Port 7000:          SG-Data-Tier              → Cassandra replication

Key Design Principles:
1. Least privilege: Only allow required ports between specific security groups
2. No 0.0.0.0/0 in private tiers: Application tier can't be accessed from internet
3. Security group chaining: SG-API → SG-Application → SG-Data (defense in depth)
4. Stateful: Responses automatically allowed (no need for outbound rules for responses)

VPC Peering: Connect VPCs across regions without internet

us-east-1 VPC (10.0.0.0/16) ←→ eu-west-1 VPC (10.1.0.0/16)

Use Cases:
1. Data replication: Cassandra multi-region clusters
2. Disaster recovery: Failover between regions
3. Global services: User authentication (shared DynamoDB tables)

Configuration:
# Create peering connection
aws ec2 create-vpc-peering-connection \
    --vpc-id vpc-us-east \
    --peer-vpc-id vpc-eu-west \
    --peer-region eu-west-1

# Accept peering connection (in eu-west-1)
aws ec2 accept-vpc-peering-connection \
    --vpc-peering-connection-id pcx-abc123

# Update route tables (both sides)
aws ec2 create-route \
    --route-table-id rtb-us-east \
    --destination-cidr-block 10.1.0.0/16 \
    --vpc-peering-connection-id pcx-abc123

Cost:
- VPC peering: FREE (no hourly charge)
- Data transfer: $0.01/GB (same region), $0.02/GB (cross-region)
- Netflix scale: 100 TB/month cross-region = $2,000/month

Why Not VPN:
- VPN: $0.05/hour + $0.09/GB = $36/month + $9,000/month data = $9K/month
- VPC peering: $0 + $2,000/month data = $2K/month
- Savings: $7K/month = $84K/year per peering connection

VPC Flow Logs: Capture all IP traffic in/out of VPC

Configuration:
aws ec2 create-flow-logs \
    --resource-type VPC \
    --resource-id vpc-12345 \
    --traffic-type ALL \
    --log-destination-type s3 \
    --log-destination arn:aws:s3:::netflix-flowlogs

Flow Log Record Format:
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

Example:
2 123456789 eni-abc123 10.0.1.50 203.0.113.25 43923 443 6 10 5000 1620000000 1620000060 ACCEPT OK

Decoded:
- Source: 10.0.1.50 (private instance)
- Destination: 203.0.113.25 (external IP)
- Port: 443 (HTTPS)
- Protocol: 6 (TCP)
- Packets: 10
- Bytes: 5,000
- Action: ACCEPT (allowed by security group)

Netflix Use Cases:
1. Security analysis: Detect unauthorized access attempts
2. Troubleshooting: Why is service X not reaching service Y?
3. Cost optimization: Identify chatty services (high data transfer)
4. Compliance: Audit all network traffic (PCI DSS requirement)

Scale:
- Flow logs: 10+ TB/day across all VPCs
- Storage: S3 Standard-IA ($0.0125/GB) = $125K/month
- Analysis: Athena queries (SQL on S3) = $5/TB scanned = $50/day

Query Example (Athena):
SELECT srcaddr, dstaddr, SUM(bytes) as total_bytes
FROM flow_logs
WHERE dstaddr LIKE '10.0.%'  -- Internal traffic only
GROUP BY srcaddr, dstaddr
ORDER BY total_bytes DESC
LIMIT 100;

Result: Top 100 chattiest service pairs (optimize these first)

Monthly VPC Costs (us-east-1 region):

VPC itself: FREE (no charge for VPC or subnets)

Internet Gateway: FREE (no charge for gateway itself)

NAT Gateways: 3 (one per AZ for HA)
    Hourly: 3 × $0.045 × 730 hours = $98.55/month
    Data: 500 GB/day × 30 days × $0.045/GB = $675/month
    Subtotal: $773.55/month

VPC Flow Logs:
    Storage: 300 GB/day × 30 days × $0.0125/GB = $112.50/month
    
VPC Peering (to eu-west-1 + ap-southeast-1):
    Peering: FREE
    Data: 3 TB/month × $0.02/GB = $60/month

Total per region: ~$946/month = $11.4K/year
All 3 regions: $946 × 3 = $2,838/month = $34K/year

Additional Networking Costs (outside VPC):
- CloudFront CDN: $50M+/year (15% of internet bandwidth!)
- Route 53 DNS: $500K/year (billions of queries)
- Direct Connect: $10K/month (1 Gbps × 10 locations)
- Load Balancers: $5K/month (100+ ALBs across regions)

Total Netflix Network Infrastructure: ~$60M+/year estimated

Before AWS (2009-2016, own datacenters):
    Locations: 2 primary datacenters (Virginia, California)
    Capacity planning: 18-month lead time (hardware procurement)
    Scaling: Manual (add physical servers)
    Global expansion: Slow (build datacenters in each country)
    Disaster recovery: Complex (replicate between 2 datacenters)
    Cost: $100M+/year (hardware, power, cooling, staff)

After AWS (2016-present, cloud-native):
    Regions: 21 AWS regions for edge, 3 for core services
    Capacity: Auto-scaling (spin up 1,000s of instances in minutes)
    Scaling: Automated (based on traffic patterns)
    Global expansion: Fast (launch in new AWS region in weeks)
    Disaster recovery: Built-in (multi-AZ, multi-region)
    Cost: $60M+/year networking (but $1B/year total AWS spend)

Business Impact:
    Subscribers: 60M (2016) → 230M (2024) = 3.8× growth
    Availability: 99.9% (2016) → 99.99% (2024)
    Global reach: 50 countries → 190 countries
    Innovation speed: Quarterly releases → Daily deployments
    Cost per subscriber: $1.67/month → $0.26/month (infrastructure)

Key Learning: VPC architecture enabled Netflix's global scale
- Multi-region VPCs: Serve users locally (low latency)
- Security groups: Defense in depth (API → App → Data isolation)
- VPC peering: Cross-region replication without internet exposure
- NAT gateways: Private services access external APIs securely
- Flow logs: Network visibility for security and optimization

Global Traffic Flow:

1. User makes payment on e-commerce site
   ↓
2. HTTPS request to api.stripe.com
   ↓
3. Route 53 GeoDNS routes to nearest region
   ↓
4. CloudFront CDN (static content, API Gateway caching)
   ↓
5. Application Load Balancer (ALB) - Layer 7
   ↓
6. Target: API servers (EC2 Auto Scaling Group)

Regional Architecture (us-east-1):

Internet
  ↓
CloudFront Edge (optional, API caching)
  ↓
Application Load Balancer (ALB)
  ├─ Listener: Port 443 (HTTPS)
  ├─ SSL/TLS Certificate: ACM (AWS Certificate Manager)
  ├─ Target Groups:
  │   ├─ TG-Payments-v1 (80% traffic) - Current API version
  │   │   ├─ EC2: 10.0.10.101:8080 (healthy)
  │   │   ├─ EC2: 10.0.10.102:8080 (healthy)
  │   │   ├─ EC2: 10.0.10.103:8080 (healthy)
  │   │   └─ ... (100+ instances total)
  │   │
  │   └─ TG-Payments-v2 (20% traffic) - Canary deployment
  │       ├─ EC2: 10.0.10.201:8080 (healthy)
  │       └─ EC2: 10.0.10.202:8080 (healthy)
  │
  └─ Rules:
      ├─ /v1/* → TG-Payments-v1 (80%)
      ├─ /v1/* → TG-Payments-v2 (20% weighted)
      ├─ /v2/* → TG-Payments-v2 (100%)
      ├─ /health → TG-Health-Check
      └─ Default → 503 Service Unavailable

# Application Load Balancer
resource "aws_lb" "stripe_api" {
  name               = "stripe-api-alb"
  internal           = false  # Internet-facing
  load_balancer_type = "application"
  
  security_groups    = [aws_security_group.alb.id]
  subnets            = [
    aws_subnet.public_us_east_1a.id,
    aws_subnet.public_us_east_1b.id,
    aws_subnet.public_us_east_1c.id
  ]
  
  # Enable deletion protection (prevent accidental deletion)
  enable_deletion_protection = true
  
  # Enable access logs (S3)
  access_logs {
    bucket  = "stripe-alb-logs"
    prefix  = "api-alb"
    enabled = true
  }
  
  # Enable HTTP/2 (faster for mobile clients)
  enable_http2 = true
  
  # Enable cross-zone load balancing (distribute evenly across AZs)
  enable_cross_zone_load_balancing = true
  
  tags = {
    Name        = "stripe-api-alb"
    Environment = "production"
    Team        = "platform"
  }
}

# HTTPS Listener (Port 443)
resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.stripe_api.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS-1-2-2017-01"  # TLS 1.2+
  certificate_arn   = aws_acm_certificate.stripe_api.arn
  
  # Default action: Return 503 (force explicit rules)
  default_action {
    type = "fixed-response"
    fixed_response {
      content_type = "application/json"
      message_body = jsonencode({
        error = {
          type    = "api_error"
          message = "No route found for this path"
        }
      })
      status_code = "503"
    }
  }
}

# HTTP Listener (Port 80) - Redirect to HTTPS
resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.stripe_api.arn
  port              = "80"
  protocol          = "HTTP"
  
  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"  # Permanent redirect
    }
  }
}

# Target Group: Payments API v1 (Current)
resource "aws_lb_target_group" "payments_v1" {
  name     = "stripe-payments-v1"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = aws_vpc.stripe.id
  
  # Health check configuration
  health_check {
    enabled             = true
    path                = "/v1/health"
    port                = "8080"
    protocol            = "HTTP"
    healthy_threshold   = 2      # 2 consecutive successes = healthy
    unhealthy_threshold = 3      # 3 consecutive failures = unhealthy
    timeout             = 5      # 5 seconds timeout
    interval            = 30     # Check every 30 seconds
    matcher             = "200"  # Expect HTTP 200
  }
  
  # Deregistration delay (drain connections before removing instance)
  deregistration_delay = 30  # 30 seconds (complete in-flight requests)
  
  # Stickiness (NOT used by Stripe, APIs stateless)
  stickiness {
    enabled = false
  }
  
  # Connection draining
  connection_draining = true
  connection_draining_timeout = 30
  
  tags = {
    Name    = "stripe-payments-v1"
    Version = "v1"
  }
}

# Target Group: Payments API v2 (Canary)
resource "aws_lb_target_group" "payments_v2" {
  name     = "stripe-payments-v2"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = aws_vpc.stripe.id
  
  # Same health check config as v1
  health_check {
    enabled             = true
    path                = "/v2/health"
    port                = "8080"
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    timeout             = 5
    interval            = 30
    matcher             = "200"
  }
  
  deregistration_delay = 30
  
  stickiness {
    enabled = false
  }
  
  tags = {
    Name    = "stripe-payments-v2"
    Version = "v2"
  }
}

# Listener Rule: Route /v1/* to v1 (80%) and v2 (20%)
resource "aws_lb_listener_rule" "payments_v1_weighted" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 100
  
  action {
    type = "forward"
    forward {
      # Weighted target groups (canary deployment)
      target_group {
        arn    = aws_lb_target_group.payments_v1.arn
        weight = 80  # 80% traffic to v1
      }
      
      target_group {
        arn    = aws_lb_target_group.payments_v2.arn
        weight = 20  # 20% traffic to v2 (canary)
      }
      
      # Stickiness duration (not used, but configurable)
      stickiness {
        enabled  = false
        duration = 3600
      }
    }
  }
  
  condition {
    path_pattern {
      values = ["/v1/charges*", "/v1/customers*", "/v1/payments*"]
    }
  }
}

# Listener Rule: Route /v2/* to v2 only (100%)
resource "aws_lb_listener_rule" "payments_v2_full" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 90
  
  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.payments_v2.arn
  }
  
  condition {
    path_pattern {
      values = ["/v2/*"]
    }
  }
}

# Listener Rule: Route based on custom header (internal testing)
resource "aws_lb_listener_rule" "canary_header" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 80
  
  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.payments_v2.arn
  }
  
  condition {
    http_header {
      http_header_name = "X-Stripe-Canary"
      values           = ["true"]
    }
  }
}

# Auto Scaling Group (register with target group)
resource "aws_autoscaling_attachment" "payments_v1" {
  autoscaling_group_name = aws_autoscaling_group.payments_v1.name
  lb_target_group_arn    = aws_lb_target_group.payments_v1.arn
}

Canary Deployment: Gradually roll out new version to reduce risk

Stage 1: Internal Testing (X-Stripe-Canary: true header)
    - Engineers test new API version
    - Internal traffic only (1% of total)
    - Duration: 2-3 days
    - Rollback: Instant (remove header routing rule)

Stage 2: External Canary (5% traffic)
    - Update weighted target groups: v1=95%, v2=5%
    - Monitor error rates, latency, success rates
    - Duration: 1 week
    - Rollback: Change weights back to v1=100%

Stage 3: Ramp Up (20% → 50% → 80%)
    - Increase v2 traffic gradually
    - Each stage: 3-7 days
    - Monitor metrics at each stage
    - Rollback: Decrease weight instantly

Stage 4: Full Rollout (100%)
    - v1 decommissioned
    - All traffic on v2
    - v1 target group kept for emergency rollback (0% weight)

Monitoring During Rollout:
    - Error rate: <0.01% (below SLA)
    - Latency P95: <200ms (below SLA)
    - Latency P99: <500ms (below SLA)
    - Success rate: >99.99% (above SLA)
    
    If ANY metric exceeds threshold:
        → Automatic rollback (CloudWatch Alarm → Lambda → Update weights)

Benefits:
    - Risk mitigation: Catch issues before 100% rollout
    - Fast rollback: Change weights (instant, no deployment)
    - A/B testing: Compare v1 vs v2 performance
    - Zero downtime: Both versions running simultaneously

Metrics (Production, us-east-1):

Throughput:
    - Requests: 1 billion/day = 11,574 req/sec average
    - Peak: 50,000 req/sec (Black Friday, Cyber Monday)
    - Steady state: 10,000-20,000 req/sec

Latency (P95):
    - ALB processing: 5ms (SSL termination + routing)
    - Network: 20ms (user → ALB)
    - Application: 100ms (API processing)
    - Total: 125ms (well below 200ms SLA )

Availability:
    - ALB uptime: 99.99%+ (AWS SLA)
    - Multi-AZ: 3 availability zones (us-east-1a, 1b, 1c)
    - Health checks: 30-second interval, 3 failures = unhealthy
    - Auto-healing: Remove unhealthy instances automatically

Connection Handling:
    - Concurrent connections: 50,000+ sustained
    - New connections/sec: 5,000+
    - Keep-alive: Enabled (reuse TCP connections)
    - HTTP/2: Enabled (multiplexing, reduces connections)

SSL/TLS Performance:
    - Cipher: ECDHE-RSA-AES256-GCM-SHA384 (strong encryption)
    - Handshake: 50ms average (ALB handles)
    - Session resumption: Enabled (reuse session, skip handshake)
    - OCSP stapling: Enabled (certificate validation cached)

Monthly Cost (us-east-1):

ALB Itself:
    Hourly: $0.0225/hour × 730 hours = $16.43/month
    
LCU (Load Balancer Capacity Units):
    Metrics:
        - New connections: 5,000/sec = 200 LCUs
        - Active connections: 50,000 = 16.67 LCUs
        - Bandwidth: 100 GB/hour = 10 LCUs
        - Rule evaluations: 11,574 req/sec × 5 rules = 0.58 LCUs
    
    Max LCU: 200 (new connections is bottleneck)
    Cost: 200 LCU × $0.008 × 730 hours = $1,168/month
    
Data Processing:
    Included in LCU pricing (no separate charge)

SSL Certificates (ACM):
    Cost: FREE (AWS Certificate Manager public certs)
    Renewal: Automatic (zero operations)

Total per region: $16.43 + $1,168 = $1,184.43/month = $14.2K/year

All regions (5): $14.2K × 5 = $71K/year

Alternative (NLB):
    Hourly: $0.0225/hour × 730 = $16.43/month
    NLCU: $0.006 per NLCU-hour
    
    Cheaper per LCU but loses Layer 7 features (not suitable)

Alternative (NGINX on EC2):
    Instances: 3 × c5.2xlarge (8 vCPU) = $700/month
    Staff: 0.25 engineer × $200K/year = $4,167/month
    Operations: Patching, certificates, monitoring
    Total: $700 + $4,167 = $4,867/month = $58K/year
    
    Savings with ALB: $58K - $14K = $44K/year
    Plus: Managed service (zero operations, auto-scaling, AWS SLA)

Scenario 1: Single instance fails
    Detection: Health check fails 3 times (90 seconds)
    Action: ALB removes instance from target group
    Impact: Zero (remaining instances handle load)
    Recovery: Auto Scaling launches replacement (5 minutes)

Scenario 2: Entire AZ fails (us-east-1a down)
    Detection: All health checks fail in AZ (90 seconds)
    Action: ALB routes all traffic to us-east-1b and 1c
    Impact: Minimal (33% capacity reduction, handled by remaining AZs)
    Recovery: AWS restores AZ (hours), Auto Scaling replaces instances

Scenario 3: Application bug (high error rate on v2 canary)
    Detection: CloudWatch Alarm (error rate >0.01%)
    Action: Lambda function updates weights (v2=0%, v1=100%)
    Impact: 20% of users saw errors briefly (rollback in 60 seconds)
    Recovery: Fix bug in v2, redeploy, retry canary

Scenario 4: DDoS attack (100K req/sec)
    Detection: AWS Shield (DDoS protection) detects attack
    Action: WAF rate limiting + ALB auto-scaling
    Impact: Legitimate traffic slightly slower (P95 200ms → 300ms)
    Cost: ALB LCU increases (more capacity units)
    Recovery: Attack mitigated by Shield, normal traffic resumes

Scenario 5: SSL certificate expiration
    Prevention: ACM auto-renewal (60 days before expiry)
    Fallback: CloudWatch Alarm 30 days before expiry
    Impact: Zero (ACM handles renewal automatically)
    Stripe action: None required (fully automated)

Zoom Requirements Analysis:

Requirement 1: UDP protocol support
    - Real-time video requires UDP (not TCP)
    - UDP: No retransmission, lower latency
    - TCP: Retransmission causes stuttering video
    - Verdict: MUST support UDP → NLB 

Requirement 2: Ultra-low latency (<150ms end-to-end)
    - ALB: 5-10ms overhead (Layer 7 processing)
    - NLB: <100µs overhead (Layer 4 passthrough)
    - Every millisecond matters for video quality
    - Verdict: NLB provides 50-100× lower latency 

Requirement 3: Millions of concurrent connections
    - ALB: ~1M requests/sec capacity
    - NLB: ~10M connections/sec capacity
    - Zoom scale: 30M concurrent meetings
    - Verdict: NLB handles 10× more connections 

Requirement 4: Static IP addresses (enterprise firewalls)
    - Enterprise IT: Whitelist Zoom IPs in corporate firewalls
    - ALB: Dynamic IPs (change over time)
    - NLB: Static IPs (via Elastic IP addresses)
    - Verdict: NLB provides static IPs required by enterprises 

Why NOT ALB:
     Layer 7 only (HTTP/HTTPS, no UDP support)
     Higher latency (5-10ms vs <100µs)
     Lower throughput (1M vs 10M connections/sec)
     Dynamic IPs (enterprise firewall rules impossible)

Network Load Balancer Perfect Fit:
    Layer 4 (TCP/UDP support)
    Ultra-low latency (<100µs)
    Massive scale (10M+ connections/sec)
    Static IPs (Elastic IP addresses)
    Connection tracking (stateful firewall)

# AWS Network Load Balancer Configuration
# Zoom Media Server Cluster (us-west-2)

resource "aws_lb" "zoom_media" {
  name               = "zoom-media-nlb"
  internal           = false  # Internet-facing
  load_balancer_type = "network"
  
  # Subnet mapping with static IPs (Elastic IPs)
  subnet_mapping {
    subnet_id     = aws_subnet.public_us_west_2a.id
    allocation_id = aws_eip.zoom_media_2a.id  # Static IP: 54.200.1.1
  }
  
  subnet_mapping {
    subnet_id     = aws_subnet.public_us_west_2b.id
    allocation_id = aws_eip.zoom_media_2b.id  # Static IP: 54.200.1.2
  }
  
  subnet_mapping {
    subnet_id     = aws_subnet.public_us_west_2c.id
    allocation_id = aws_eip.zoom_media_2c.id  # Static IP: 54.200.1.3
  }
  
  # Enable cross-zone load balancing (even distribution)
  enable_cross_zone_load_balancing = true
  
  # Enable deletion protection
  enable_deletion_protection = true
  
  tags = {
    Name        = "zoom-media-nlb"
    Environment = "production"
    Service     = "video-conferencing"
  }
}

# UDP Listener: Video/Audio Streams (Port 8801)
resource "aws_lb_listener" "media_udp_8801" {
  load_balancer_arn = aws_lb.zoom_media.arn
  port              = "8801"
  protocol          = "UDP"
  
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.media_servers.arn
  }
}

# TCP Listener: WebSocket Fallback (Port 443)
resource "aws_lb_listener" "media_tcp_443" {
  load_balancer_arn = aws_lb.zoom_media.arn
  port              = "443"
  protocol          = "TCP"
  
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.media_servers.arn
  }
}

# Target Group: Media Servers
resource "aws_lb_target_group" "media_servers" {
  name     = "zoom-media-servers"
  port     = 8801
  protocol = "UDP"
  vpc_id   = aws_vpc.zoom.id
  
  # Health check (TCP on management port)
  health_check {
    enabled             = true
    protocol            = "TCP"
    port                = 8080  # Health check endpoint
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 10
    interval            = 30
  }
  
  # Connection termination (drain UDP streams gracefully)
  deregistration_delay = 60  # Allow 60s to finish ongoing calls
  
  # Preserve source IP (important for rate limiting)
  preserve_client_ip = true
  
  tags = {
    Name = "zoom-media-servers"
  }
}

# Register media servers (bare metal + EC2)
resource "aws_lb_target_group_attachment" "media_server_1" {
  target_group_arn = aws_lb_target_group.media_servers.arn
  target_id        = "10.0.10.101"  # Media server IP
  port             = 8801
}

# ... (100+ media servers registered)

Security Group Strategy: Defense in Depth

Layer 1: NLB Security Group (Internet-facing)
SG-NLB-Media:
Inbound:
    UDP 8801-8810:  0.0.0.0/0           → Video/audio streams (any client)
    TCP 443:        0.0.0.0/0           → WebSocket fallback (restrictive NATs)
Outbound:
    All traffic:    10.0.0.0/16         → Forward to media servers only (VPC CIDR)

Layer 2: Media Server Security Group
SG-Media-Server:
Inbound:
    UDP 8801-8810:  SG-NLB-Media        → Only from NLB (not directly from internet)
    TCP 8080:       SG-NLB-Media        → Health checks
    TCP 22:         SG-Bastion          → SSH access (bastion hosts only)
    TCP 9090:       SG-Monitoring       → Prometheus metrics
Outbound:
    UDP 8801-8810:  0.0.0.0/0           → Send video/audio to clients
    TCP 443:        0.0.0.0/0           → API calls (AWS services, Zoom APIs)
    TCP 5432:       SG-Database         → PostgreSQL (meeting metadata)

Layer 3: Database Security Group
SG-Database:
Inbound:
    TCP 5432:       SG-Media-Server     → Only from media servers
    TCP 5432:       SG-API-Server       → Only from API servers
Outbound:
    None                                 → Database never initiates connections

Layer 4: Bastion Host Security Group (SSH access)
SG-Bastion:
Inbound:
    TCP 22:         203.0.113.0/24      → Zoom office IPs only (not 0.0.0.0/0)
Outbound:
    TCP 22:         10.0.0.0/16         → SSH to any VPC resource

Key Security Principles:
1. Least privilege: Only allow required ports between specific SGs
2. No direct internet access: Media servers behind NLB (no public IPs)
3. Bastion hosts: SSH access through hardened jump boxes (not direct)
4. Source IP restriction: Bastion only from Zoom office IPs (not anywhere)
5. Security group chaining: Internet → NLB → Media → Database (isolation)

Zoom NACL Strategy: Redundant Layer (Belt and Suspenders)

Public Subnet NACL (NLB tier):
Inbound:
    Rule 100: UDP 8801-8810   0.0.0.0/0        ALLOW   → Video/audio clients
    Rule 110: TCP 443         0.0.0.0/0        ALLOW   → WebSocket fallback
    Rule 120: TCP 32768-65535 0.0.0.0/0        ALLOW   → Ephemeral ports (return traffic)
    Rule *:   ALL             0.0.0.0/0        DENY    → Deny all else (default)

Outbound:
    Rule 100: UDP 8801-8810   0.0.0.0/0        ALLOW   → Forward to clients
    Rule 110: TCP 443         0.0.0.0/0        ALLOW   → HTTPS to AWS APIs
    Rule 120: TCP 32768-65535 10.0.0.0/16      ALLOW   → Ephemeral to media servers
    Rule *:   ALL             0.0.0.0/0        DENY    → Deny all else

Private Subnet NACL (Media server tier):
Inbound:
    Rule 100: UDP 8801-8810   10.0.1.0/24      ALLOW   → From NLB subnet only
    Rule 110: TCP 8080        10.0.1.0/24      ALLOW   → Health checks from NLB
    Rule 120: TCP 22          10.0.5.0/24      ALLOW   → SSH from bastion subnet
    Rule *:   ALL             0.0.0.0/0        DENY    → Deny all else

Outbound:
    Rule 100: UDP 8801-8810   0.0.0.0/0        ALLOW   → Video streams to clients
    Rule 110: TCP 443         0.0.0.0/0        ALLOW   → AWS APIs
    Rule 120: TCP 5432        10.0.30.0/23     ALLOW   → Database queries
    Rule *:   ALL             0.0.0.0/0        DENY    → Deny all else

AWS WAF: Protect web/API layer from common attacks

WAF Rules (Applied to CloudFront distribution):

Rule 1: Rate Limiting (Prevent Brute Force)
    Condition: >100 requests per 5 minutes from same IP
    Action: BLOCK
    Why: Prevent password guessing, API abuse
    Example: Attacker trying 1,000 passwords/minute → Blocked after 100

Rule 2: Geo-Blocking (Compliance)
    Condition: IP from sanctioned countries (North Korea, Iran, Syria)
    Action: BLOCK
    Why: US export control regulations (OFAC compliance)
    Note: Zoom blocked in China (Great Firewall), separate China service

Rule 3: SQL Injection Prevention
    Condition: Request contains SQL keywords (SELECT, UNION, DROP)
    Action: BLOCK
    Why: Prevent database attacks via web forms
    Example: Username field: "admin' OR '1'='1" → Blocked

Rule 4: XSS (Cross-Site Scripting) Prevention
    Condition: Request contains <script>, javascript:, onerror=
    Action: BLOCK
    Why: Prevent code injection in meeting names, chat messages
    Example: Meeting name: "&lt;script>alert('XSS')</script>" → Blocked

Rule 5: Known Bot Protection (AWS Managed Rule)
    Condition: IP matches known bot signatures
    Action: CHALLENGE (CAPTCHA)
    Why: Prevent automated meeting creation (resource exhaustion)
    Example: 1,000 meetings created by bot → CAPTCHA challenge

Rule 6: Anonymous IP List (AWS Managed Rule)
    Condition: IP from VPN, proxy, or Tor network
    Action: ALLOW (but log for analysis)
    Why: Many legitimate users use VPNs (privacy), don't block
    Note: Some competitors block VPNs (Zoom allows for privacy)

WAF Pricing:
    Base: $5/month per WebACL
    Rules: $1/month per rule × 6 = $6/month
    Requests: $0.60 per million requests
    
    Zoom scale:
        Requests: 10 billion/month (web + API)
        Cost: $5 + $6 + (10,000 × $0.60) = $6,011/month = $72K/year
    
    Value: Prevents attacks that could cost millions (reputation, lawsuits)

Incident 1: "Zoombombing" (2020)
Problem: Unauthorized users join meetings and share inappropriate content
Root cause: Default meeting settings (no password, waiting room off)

Zoom's Response:
1. Changed defaults (April 2020):
   - Passwords required by default (not optional)
   - Waiting room enabled by default (host approves participants)
   - Screen sharing limited to host only (not everyone)

2. Network security improvements:
   - Meeting IDs randomized (not sequential, harder to guess)
   - Rate limiting on meeting ID attempts (prevent brute force)
   - WAF rules block automated meeting ID scanning

Results:
   - Zoombombing incidents: 90% reduction within 2 months
   - User satisfaction: Increased (safer platform)
   - Enterprise adoption: Accelerated (trust restored)

Incident 2: End-to-End Encryption Controversy (2020)
Problem: Zoom claimed "end-to-end encryption" but used TLS only
Reality: TLS encrypts client → server, but Zoom servers could decrypt

Zoom's Response (October 2020):
1. Implemented true E2E encryption:
   - Encryption keys generated on client devices
   - Zoom servers cannot decrypt (zero knowledge)
   - Only meeting participants have keys
   
2. Architecture change:
   Before: Client → [TLS] → Zoom Server [decrypt] → [TLS] → Client
   After: Client → [E2E] → Zoom Server [no decrypt] → [E2E] → Client

3. Trade-offs disclosed:
   - E2E enabled: No cloud recording, no phone participants
   - E2E disabled: Full features but server can decrypt
   - User choice: Host decides per meeting

Results:
   - E2E meetings: 4+ billion (as of 2023)
   - Trust restored: Transparency about encryption model
   - Compliance: HIPAA, GDPR, FedRAMP certified

Incident 3: China Server Routing (2020)
Problem: Some non-China users routed through China servers
Root cause: Network load balancing bug (incorrect GeoIP routing)

Zoom's Response (June 2020):
1. Geo-fencing implemented:
   - Non-China users NEVER route through China
   - China users ONLY route through China servers
   - Separate infrastructure (data residency compliance)

2. Network architecture change:
   - Route 53 GeoDNS: Strict geographic routing
   - No fallback to other regions (even during outage)
   - Prefer service degradation over wrong region

3. Audit and monitoring:
   - Real-time GeoIP verification (every connection)
   - Alerts if unexpected country routing detected
   - Weekly audits of routing tables

Results:
   - Zero cross-region routing incidents (2020-2024)
   - GDPR compliant (data residency enforced)
   - US government approval (FedRAMP authorization)

Production Metrics (Global):

Latency (End-to-End):
    Video: P50 = 80ms, P95 = 150ms, P99 = 300ms
    Audio: P50 = 50ms, P95 = 100ms, P99 = 200ms
    Target: <150ms P95 (imperceptible to humans)
    
    Breakdown:
        User → ISP: 10-30ms
        ISP → Zoom edge: 20-50ms (via peering agreements)
        Zoom processing: 10-20ms (NLB + media server)
        Zoom edge → recipient ISP: 20-50ms
        Recipient ISP → user: 10-30ms
        Total: 70-180ms (typically <150ms )

Packet Loss:
    Target: <1% (video quality acceptable)
    Typical: 0.1-0.3% (excellent quality)
    Mitigation: Forward Error Correction (FEC)
        - Send redundant data (1.2× bandwidth)
        - Reconstruct lost packets without retransmission
        - Trade-off: 20% more bandwidth for 99%+ reliability

Jitter (Variation in Latency):
    Target: <30ms (smooth video)
    Typical: 10-20ms
    Mitigation: Jitter buffer (200ms client-side)
        - Buffer incoming packets to smooth timing
        - Trade-off: Adds 200ms latency but eliminates stuttering

Bandwidth Usage:
    Video (720p): 1.5-2 Mbps per participant
    Video (1080p): 2.5-3 Mbps per participant
    Audio: 50-80 Kbps (Opus codec)
    Screen sharing: 1-2 Mbps (optimized for text)
    
    25-participant meeting (Gallery View):
        Download: 25 × 1.5 Mbps = 37.5 Mbps
        Upload: 1.5 Mbps (only your video)
        
    Zoom optimization: Send only 4-6 active speaker videos
        Download: 6 × 1.5 Mbps = 9 Mbps (75% reduction!)

Concurrent Connections:
    Peak (2020 pandemic): 300 million daily participants
    Current (2024): 300 million+ (maintained post-pandemic)
    Simultaneous meetings: 30+ million
    Media servers: 10,000+ worldwide (bare metal + cloud)

Availability:
    Target SLA: 99.9% (enterprise plan)
    Actual: 99.95%+ (better than SLA)
    Downtime: <4.4 hours/year (vs 8.7 hours allowed)
    
    Outage breakdown:
        Planned maintenance: 2 hours/year (announced 2 weeks ahead)
        Unplanned: 2 hours/year (DDoS, AWS outages)
        Per user: Often zero (multi-region redundancy)

Monthly Network Infrastructure Costs (Estimated):

Network Load Balancers (50 regions × 3 per region):
    NLB: 150 × $0.0225/hour × 730 hours = $2,462/month
    NLCU: Connection-heavy workload
        - 30M concurrent meetings = 60M connections (2 per meeting)
        - Per NLB: 400K connections average
        - NLCU: 400K connections ÷ 3,000 = 133 NLCU
        - Cost: 150 NLBs × 133 NLCU × $0.006 × 730 = $87K/month
    Subtotal: $89.5K/month = $1.07M/year

Data Transfer:
    Inbound: FREE (AWS doesn't charge incoming)
    Outbound: $0.09/GB (internet)
        - 300M participants × 1.5 Mbps average × 45 min average meeting
        - 300M × 1.5 Mbps × 2,700 seconds = 1.2 PB/day
        - 1.2 PB × 30 days = 36 PB/month
        - 36 PB × $0.09/GB = $3.24M/month (HUGE!)
    
    Optimization strategies:
        1. Peering agreements with ISPs (CDN-style)
           - Direct connections to Comcast, AT&T, Verizon
           - Reduced cost: $0.09 → $0.02/GB (78% reduction)
           - New cost: 36 PB × $0.02/GB = $720K/month
        
        2. P2P for 1-on-1 meetings (bypass Zoom servers)
           - Direct connection: User A ↔ User B (no server)
           - Zoom servers: Only for 3+ participants
           - Savings: 40% of meetings are 1-on-1 = 40% × $720K = $288K saved
           - New cost: $432K/month
        
        3. Regional caching (colocation datacenters)
           - 19 colocations worldwide (near major ISPs)
           - Edge caching reduces core network traffic 60%
           - New cost: $432K × 0.4 = $173K/month

Security (WAF, Shield, GuardDuty):
    WAF: 50 WebACLs × $6K/year = $300K/year = $25K/month
    Shield Advanced: $3,000/month × 50 distributions = $150K/month
    GuardDuty: $5/million events × 100B/month = $500K/month
    Subtotal: $675K/month = $8.1M/year

VPC & Networking:
    NAT Gateways: 150 (3 per region × 50) × $100/month = $15K/month
    VPC Flow Logs: 1 PB/month × $0.0125/GB = $12.5K/month
    VPC Peering: FREE (but data transfer charged above)
    Direct Connect: 10 Gbps × 10 locations × $2,500/month = $25K/month
    Subtotal: $52.5K/month = $630K/year

Total Monthly Network Cost: 
    NLB: $89.5K
    Data transfer: $173K (optimized)
    Security: $675K
    VPC: $52.5K
    Total: $990K/month = $11.88M/year

Zoom Total Revenue: $4.3B (2023)
Network cost: $11.88M / $4.3B = 0.28% of revenue (efficient!)

Cost Per Meeting Minute:
    11.88M/year ÷ 3 trillion minutes = $0.00396 per 1,000 minutes
    = $0.24 per 100 hours of meetings (incredibly low)

Multi-Layer Defense Strategy:

Layer 1: Anycast Network (Traffic Distribution)
┌─────────────────────────────────────────────────────────┐
│ Attacker Traffic (100 Tbps DDoS)                        │
│ ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓     │
│                                                           │
│ Anycast: Same IP (1.1.1.1) announced from 310+ locations│
│ ├─ San Francisco: Receives 30 Tbps (nearest to attacker)│
│ ├─ London: Receives 25 Tbps                             │
│ ├─ Singapore: Receives 20 Tbps                          │
│ ├─ ... 307 other locations share remaining 25 Tbps     │
│                                                           │
│ Result: 100 Tbps ÷ 310 locations = 322 Gbps per location│
│         (Manageable vs 100 Tbps to single location!)    │
└─────────────────────────────────────────────────────────┘

Layer 2: Edge-Based Filtering (Block at Entry)
┌─────────────────────────────────────────────────────────┐
│ Each Cloudflare Data Center (310+ worldwide)            │
│                                                           │
│ 1. IP Reputation Check                                  │
│    - Known botnet IPs → DROP (no processing)           │
│    - Tor exit nodes → CHALLENGE (CAPTCHA)               │
│    - VPN/Proxy → ALLOW (but monitor)                    │
│                                                           │
│ 2. Rate Limiting (Per-Source IP)                        │
│    - Normal: 10-100 req/sec per IP                      │
│    - Suspicious: >1,000 req/sec → CHALLENGE             │
│    - Attack: >10,000 req/sec → BLOCK temporarily        │
│                                                           │
│ 3. Behavioral Analysis (Machine Learning)               │
│    - Human patterns: Mouse movements, scroll behavior   │
│    - Bot patterns: Perfect timing, no JS execution      │
│    - Decision: <10ms (real-time classification)         │
│                                                           │
│ 4. Geographic Filtering (Optional)                      │
│    - Customer choice: Block specific countries          │
│    - Example: US-only site blocks traffic from Russia   │
│    - Trade-off: False positives (legit users blocked)   │
└─────────────────────────────────────────────────────────┘

Layer 3: Application-Level Protection (WAF)
┌─────────────────────────────────────────────────────────┐
│ Web Application Firewall (Layer 7 Protection)           │
│                                                           │
│ 1. HTTP Flood Detection                                 │
│    - Normal: 1-10 page views per minute                 │
│    - Attack: 1,000+ requests per second                 │
│    - Action: Rate limit, CAPTCHA challenge              │
│                                                           │
│ 2. Slowloris Mitigation                                 │
│    - Attack: Open many connections, send data slowly    │
│    - Cloudflare: Connection timeout 100 seconds         │
│    - Result: Attacker can't exhaust server connections  │
│                                                           │
│ 3. Application-Specific Rules                           │
│    - WordPress: Block wp-admin bruteforce               │
│    - API: Rate limit per API key (not just IP)          │
│    - Custom: Customer-defined rules (regex patterns)    │
└─────────────────────────────────────────────────────────┘

Layer 4: Origin Protection (Hiding Real Server)
┌─────────────────────────────────────────────────────────┐
│ Origin Server Protection                                 │
│                                                           │
│ 1. Origin IP Cloaking                                   │
│    - Real server IP: 203.0.113.50 (not public)          │
│    - Public IP: Cloudflare's IPs only                   │
│    - Attackers can't bypass Cloudflare (don't know IP)  │
│                                                           │
│ 2. Authenticated Origin Pulls                           │
│    - Cloudflare → Origin: mTLS certificate required     │
│    - Direct Origin access: Rejected (no valid cert)     │
│    - Result: Even if IP leaked, can't connect directly  │
│                                                           │
│ 3. Origin Rate Limiting                                 │
│    - Cloudflare → Origin: Max 10K req/sec               │
│    - Cache hit rate: 80% (only 20% hit origin)          │
│    - Protection: Origin never overwhelmed               │
└─────────────────────────────────────────────────────────┘

Attack Details:

Target: Cloudflare customer (financial services company)
Attack type: HTTP/2 Rapid Reset (CVE-2023-44487)
Attack volume: 71 million HTTP requests per second
Previous record: 46 million rps (August 2021, also Cloudflare)
Attack duration: Less than 30 seconds (extremely brief but intense)
Attacker: Mirai botnet variant (30,000+ compromised IoT devices)

Attack Method: HTTP/2 Rapid Reset Exploit
1. Attacker opens HTTP/2 connection
2. Sends thousands of requests with RST_STREAM immediately
3. Server allocates resources but attacker cancels before response
4. Repeat at massive scale (requests per connection)
5. Result: Server CPU/memory exhausted (denial of service)

Normal HTTP/2 usage:
    Client → Server: 1-10 concurrent requests per connection
    Client waits for responses before sending more

Attack abuse:
    Attacker → Server: 1,000+ requests per connection
    Attacker cancels (RST_STREAM) before response
    Server wasted CPU preparing responses that were cancelled
    Multiply by 30,000 bots = 71 million rps

Cloudflare's Defense (Automatic):

Detection (<10 seconds):
    - Anomaly detection: RST_STREAM rate 100× normal
    - Pattern recognition: Same user-agent, no cookies
    - Source clustering: 30,000 IPs from same ASN range
    - Decision: DDoS attack confirmed

Mitigation (Automatic):
    1. Rate limiting: Max 100 RST_STREAM per connection
    2. Connection limits: Max 1,000 requests per connection
    3. IP blocking: Block top 10,000 attacking IPs (botnet)
    4. Challenge: CAPTCHA for suspicious sources
    5. Geographic: No filter needed (global attack)

Results:
    - Attack blocked: 99.9% of malicious traffic dropped at edge
    - Customer impact: ZERO downtime (completely transparent)
    - Legitimate users: No degradation (normal access)
    - Cloudflare cost: $0 extra (absorbed by network capacity)
    - Time to mitigate: <60 seconds (fully automated)

Customer Response:
    "We didn't even know we were under attack until Cloudflare
     sent us the incident report. Our monitoring showed zero
     anomalies. That's how good the protection is."
     - CTO, Financial Services Company (anonymous)

Network Infrastructure:

Total Capacity: 310+ Tbps (yes, 310 terabits per second!)
    - 310 data centers × 1 Tbps average = 310 Tbps
    - Some large locations: 5-10 Tbps (NYC, London, San Francisco)
    - Smaller locations: 100-500 Gbps (regional cities)

For Context:
    - Total internet traffic: ~5,000 Tbps globally (2024 estimate)
    - Cloudflare capacity: 6% of entire internet!
    - Largest DDoS attack ever: 3.47 Tbps (Microsoft Azure, 2021)
    - Cloudflare can absorb: 100× the largest attack recorded

How They Built This Capacity:

1. Peering Agreements (Cost Optimization)
    - Direct connections to 10,000+ ISPs
    - Settlement-free peering (no money exchanged)
    - Benefit: Zero bandwidth cost for 95% of traffic
    
    Without peering:
        50 million rps × 1 KB avg = 50 GB/sec × 86,400 = 4.3 PB/day
        Cost: 4.3 PB × $0.09/GB = $387K/day = $11.6M/month = $141M/year!
    
    With peering:
        95% free (peering) + 5% paid ($0.02/GB negotiated)
        Cost: 4.3 PB × 0.05 × $0.02/GB = $4.3K/day = $1.57M/year
        Savings: $139M/year (99% reduction!)

2. Anycast (Load Distribution)
    - Same IP address announced from all locations
    - BGP routing sends traffic to nearest data center
    - Automatic failover (if one location fails, others take over)
    - DDoS distribution: 100 Tbps ÷ 310 = 322 Gbps per location
    
    Without Anycast (single location):
        100 Tbps attack → Single data center (1 Tbps capacity)
        Result: Overwhelmed, service down 
    
    With Anycast:
        100 Tbps attack → 310 data centers (310 Tbps total)
        Each receives: 322 Gbps (well within capacity )

3. Edge Computing (Distributed Defense)
    - Filtering at edge (not centralized)
    - Each data center: Independent mitigation
    - No backhaul to central location (would create bottleneck)
    - Result: Scale linearly (add data center = add capacity)

4. Hardware Acceleration (Custom Silicon)
    - Gen 12 servers (2023): Custom FPGA for packet filtering
    - FPGA: 10× faster than CPU for pattern matching
    - Throughput: 1 Tbps per server (vs 100 Gbps CPU-only)
    - Cost: $50K per server (amortized over 5 years = $10K/year)

1. Volumetric Attacks (Bandwidth Exhaustion)

Attack: UDP Flood
    Method: Send billions of UDP packets
    Volume: 1-100 Tbps (massive bandwidth)
    Goal: Saturate network link
    
    Example: DNS Amplification
        1. Attacker sends small DNS query (60 bytes)
        2. Spoofs source IP (victim's IP)
        3. DNS server responds to victim (4,000 bytes)
        4. Amplification: 4,000 ÷ 60 = 67× larger response!
        5. Multiply by 1M DNS servers = 1 Tbps+ attack

Cloudflare Mitigation:
    - Anycast: Distribute across 310 locations (divide volume)
    - Scrubbing: Drop invalid UDP packets at edge (no forwarding)
    - Rate limiting: Limit UDP per source IP (prevent amplification)
    - BCP38: Block spoofed IPs (validate source address)
    
    Result: 100 Tbps attack becomes 322 Gbps per location (manageable)

2. Protocol Attacks (State Exhaustion)

Attack: SYN Flood
    Method: Send millions of TCP SYN packets (connection requests)
    Goal: Exhaust server connection table (max 65,535 connections)
    No ACK: Attacker never completes handshake (half-open connections)
    
    Example:
        Server: 65,535 max connections
        Attacker: 100K SYN/sec for 10 seconds = 1M half-open connections
        Result: Connection table full, legitimate users rejected 

Cloudflare Mitigation:
    - SYN Cookies: Don't allocate state until ACK received
        1. Client sends SYN
        2. Cloudflare responds SYN-ACK (no state allocated yet)
        3. Client sends ACK → Only then allocate connection
        4. Result: Infinite half-open connections (no memory used)
    
    - Challenge ACK: Require client to prove legitimacy
        1. Send SYN-ACK with special sequence number
        2. Valid client responds with correct ACK
        3. Bot/attacker: Wrong ACK (connection rejected)
    
    Result: SYN floods completely ineffective (zero state exhaustion)

3. Application Attacks (Resource Exhaustion)

Attack: HTTP Flood
    Method: Send millions of valid HTTP requests
    Goal: Exhaust server CPU/memory (generate dynamic pages)
    Challenge: Looks like legitimate traffic (hard to distinguish)
    
    Example:
        Endpoint: /search?q=expensive-query (database search)
        Normal: 10 searches/sec
        Attack: 100K searches/sec (10,000× normal)
        Result: Database overload, site down 

Cloudflare Mitigation:
    - Caching: Serve from cache (don't hit origin)
        Cache hit rate: 80% (only 20% hit origin server)
        100K req/sec × 0.2 = 20K req/sec to origin (manageable)
    
    - Rate limiting: Limit requests per IP
        Normal user: 10 req/sec
        Suspicious: >100 req/sec → CAPTCHA challenge
        Attack: >1,000 req/sec → Block temporarily (5 minutes)
    
    - JavaScript challenge: Require browser execution
        1. First request: Return JS challenge (compute hash)
        2. Browser executes JS (5 seconds)
        3. Submit result: Valid → Allow access
        4. Bots: Can't execute JS → Blocked
    
    - Machine learning: Behavioral analysis
        Human: Mouse movements, scroll behavior, varied timing
        Bot: Perfect timing, no mouse, no scroll, predictable
        Accuracy: 99.9% (false positives extremely rare)
    
    Result: 99% of attack traffic blocked, origin receives <1% (normal load)

4. Layer 7 Sophisticated Attacks

Attack: Slowloris
    Method: Open many connections, send HTTP headers slowly
    Goal: Keep connections open (exhaust server connection limit)
    Stealth: Low bandwidth (hard to detect by volume)
    
    Example:
        1. Open 10,000 connections to server
        2. Send HTTP header slowly (1 byte every 10 seconds)
        3. Server waits for complete header (default timeout: 300s)
        4. Result: 10,000 connections occupied for 5 minutes each

Cloudflare Mitigation:
    - Aggressive timeouts: 100-second connection timeout
        Slowloris: 1 byte/10 seconds = 10 bytes in 100 seconds
        Cloudflare: Timeout after 100 seconds (close connection)
        Result: Attacker must send faster (becomes noisy, easy to detect)
    
    - Connection limits per IP: Max 100 concurrent connections
        Slowloris: Needs 10,000 connections (100 IPs required)
        Cloudflare: Rate limit kicks in (CAPTCHA challenge)
        Result: Attack ineffective (can't establish enough connections)

Cloudflare Magic Transit: Enterprise DDoS Protection

Pricing Model: Capacity-based (not usage-based)
    - Purchase committed capacity (e.g., 10 Gbps)
    - Unlimited attacks within capacity (no overage charges)
    - Auto-scaling: Burst to 310+ Tbps if needed (included!)

Pricing Tiers:

Tier 1: Up to 10 Gbps
    Cost: $5,000/month = $60K/year
    Includes: Unlimited DDoS mitigation, WAF, bot management
    Best for: Medium businesses (e-commerce, SaaS)

Tier 2: Up to 100 Gbps
    Cost: $30,000/month = $360K/year
    Includes: Same as Tier 1 + dedicated support
    Best for: Large enterprises (banks, gaming, media)

Tier 3: Custom (100 Gbps+)
    Cost: Negotiated (typically $100K-500K/month)
    Includes: Same as Tier 2 + custom engineering
    Best for: Hyperscalers (cloud providers, ISPs)

Example: E-commerce Site (10 Gbps tier)

Without Cloudflare:
    Normal traffic: 2 Gbps (handled by origin)
    DDoS attack: 50 Gbps (origin down, revenue lost)
    Downtime: 6 hours/year (99.93% uptime)
    Revenue impact: $10M/year × 0.07% = $70K lost annually
    
    Mitigation options:
        1. Buy 50 Gbps capacity: $500K/year (expensive)
        2. Scrubbing service: $100K/year + $10/Gbps during attack
        3. Self-defense: $200K/year staff + $300K infrastructure

With Cloudflare Magic Transit ($60K/year):
    Normal traffic: Routed through Cloudflare (80% cache hit)
    DDoS attack: Absorbed by 310 Tbps network (origin unaffected)
    Downtime: 0 hours/year (99.99%+ uptime)
    Revenue impact: $0 lost (attacks blocked transparently)
    
    ROI: $70K revenue saved - $60K cost = $10K net benefit
    Plus: No staff needed (fully managed), no infrastructure (cloud)

Magic Transit vs Traditional DDoS Protection:

| Feature | Cloudflare Magic Transit | Traditional Scrubbing | Self-Defense |
|---------|-------------------------|----------------------|--------------|
| Capacity | 310+ Tbps | 1-10 Tbps | 1-10 Gbps |
| Mitigation time | <10 seconds | 5-30 minutes | Manual (hours) |
| Attack distribution | 310 locations | 1-5 locations | 1 location |
| Cost (10 Gbps) | $60K/year | $100K/year | $500K/year |
| Staff required | 0 (fully managed) | 1-2 engineers | 5+ engineers |
| On-call burden | None (24/7 SOC) | Yes (escalations) | Yes (24/7 team) |

Customer 1: Gaming Company (10M concurrent players)

Before Cloudflare:
    DDoS attacks: 50+ per month (competitors, angry players)
    Attack size: 10-50 Gbps typical, 200 Gbps peak
    Downtime: 10-20 hours/month (during attacks)
    Revenue loss: $500K/month (players churn during downtime)
    Infrastructure: $200K/year (DDoS mitigation hardware)
    Staff: 3 network engineers × $150K = $450K/year

After Cloudflare:
    Cost: $360K/year (100 Gbps Magic Transit tier)
    DDoS attacks: Same 50+ per month (still targeted)
    Attack size: Up to 500 Gbps (larger than before!)
    Downtime: 0 hours/month (all attacks blocked)
    Revenue loss: $0/month (zero downtime)
    Infrastructure: $0 (Cloudflare handles)
    Staff: 0 dedicated (network engineers redeployed)

ROI:
    Savings: $6M/year revenue + $200K infrastructure + $450K staff
    Cost: $360K/year Cloudflare
    Net benefit: $6.29M/year (1,746% ROI!)

Customer 2: Financial Services (Banking Platform)

Before Cloudflare:
    Compliance: PCI DSS Level 1 (strictest)
    DDoS risk: High (financial incentive for attacks)
    Previous solution: AWS Shield Advanced ($3,000/month)
    Problem: 26 Gbps attack overwhelmed AWS Shield (September 2021)
    Downtime: 4 hours (peak banking hours)
    Cost: $2M penalties (SLA violations with enterprise customers)

After Cloudflare:
    Cost: $360K/year (100 Gbps tier + compliance features)
    Largest attack: 1.2 Tbps (February 2023)
    Impact: Zero (completely transparent to users)
    Compliance: Passed PCI DSS audit (Cloudflare AOC included)
    Downtime: 0 hours/year (99.999% uptime achieved)

Quote:
    "The 1.2 Tbps attack would have destroyed our infrastructure.
     Cloudflare absorbed it like it was nothing. We didn't even
     see a blip in our monitoring. Worth every penny."
     - CISO, Financial Services Company

Customer 3: Political Campaign Website (High-Profile Target)

Before Cloudflare:
    Hosting: AWS (ALB + EC2 + WAF)
    Attack frequency: Daily during election season
    Attack motivation: Political (hacktivist groups)
    Largest attack: 15 Gbps (ALB overwhelmed)
    Downtime: 30 hours total during 6-month campaign
    Impact: Bad press ("Can't even keep website up")

After Cloudflare (Free tier!):
    Cost: $0/month (Cloudflare Project Galileo - free for NGOs)
    Attack frequency: 10-20 per day (even more frequent)
    Largest attack: 400 Gbps (July 2023, election week)
    Downtime: 0 hours (perfect uptime)
    Impact: Positive press ("Technical excellence")

Note: Project Galileo provides $360K/year enterprise protection free
      to NGOs, political campaigns, artistic groups, and journalists.
      Cloudflare's way of protecting free speech and democracy.

Market Disruption:

Traditional DDoS Market (Before Cloudflare):
    Providers: Arbor Networks, Akamai, Radware, F5
    Pricing: $100K-$1M/year for 10-100 Gbps protection
    Model: Capacity-based + overage charges during attacks
    Problem: Expensive, manual mitigation, limited capacity

Cloudflare Impact (2010-present):
    Entry price: $20/month (Pro plan with basic DDoS)
    Enterprise: $60K-$360K/year (10-100 Gbps)
    Model: Flat rate, unlimited attacks, automatic mitigation
    Capacity: 310+ Tbps (100× larger than competitors)
    
    Result: 90% price reduction, 100× capacity increase

Market Share (2024):
    Cloudflare: 20% of DDoS protection market
    Akamai: 18% (forced to lower prices)
    AWS Shield: 15% (bundled with AWS)
    Google Cloud Armor: 10% (bundled with GCP)
    Others: 37% (traditional vendors struggling)

Industry Transformation:
    Before: DDoS protection = luxury (only large enterprises)
    After: DDoS protection = commodity (even small sites protected)
    
    Websites protected:
        2010: ~100K enterprise sites (expensive traditional solutions)
        2024: 30M+ sites (Cloudflare alone), 100M+ globally
    
    Average attack size:
        2010: 1-10 Gbps (rarely >20 Gbps)
        2024: 10-100 Gbps common, 1+ Tbps recorded (arms race)
    
    Mitigation time:
        2010: 30-120 minutes (manual intervention)
        2024: <10 seconds (fully automated)

Cloudflare's Financial Model (Why They Can Offer This):

Revenue streams:
    1. DDoS protection: $360K/year per enterprise customer
    2. CDN services: Bundled (additional value)
    3. Zero Trust: $7/user/month (SASE platform)
    4. Workers: $5/month per 10M requests (edge computing)
    5. Stream: $1/month per 1,000 minutes (video platform)
    
    Average enterprise customer: $500K/year (multiple products)

Cost structure (per customer):
    DDoS mitigation: $5K/year (shared infrastructure, marginal cost)
    CDN bandwidth: $50K/year (peering reduces cost 95%)
    Support: $20K/year (1 CSM per 50 customers)
    Total cost: $75K/year per customer
    
    Gross margin: $500K - $75K = $425K/year (85% margin!)

Scale economics:
    Customers: 200K+ businesses (growing 30%/year)
    Revenue: $1B+/year (2023)
    Network: $500M capex amortized over 30M+ sites
    Per-site cost: $500M ÷ 30M = $16.67 per site per year
    
    Result: Massive scale enables low prices + high profit margins

Phase 1: Hybrid Cloud (2013-2016)
├── On-Premises Datacenters (Primary)
│   ├── Core Banking: Mainframes (IBM Z Systems)
│   ├── Credit Cards: Oracle databases
│   ├── Mobile Banking: Java/Spring applications
│   └── ATM Network: 15,000+ ATMs nationwide
│
└── AWS Cloud (Experimental)
    ├── Dev/Test Environments: EC2 instances
    ├── Marketing Websites: Static content on S3
    └── Analytics: Hadoop clusters on EMR

Phase 2: Cloud-First (2017-2019)
├── On-Premises (Legacy)
│   └── Core Banking: 30% (migrating)
│
└── AWS Cloud (Growing)
    ├── Mobile Banking: 100% (fully migrated)
    ├── Credit Card Processing: 70% (in progress)
    ├── Fraud Detection: 100% (machine learning on SageMaker)
    └── Customer Data: DynamoDB + Aurora

Phase 3: Cloud-Only (2020-Present)
└── AWS Cloud (Everything)
    ├── All applications: 100% on AWS
    ├── Datacenters: Closed (all 8 facilities shut down)
    ├── Cost savings: $2B+ over 5 years
    └── Innovation speed: 4× faster feature delivery

2017 Hybrid Architecture:

On-Premises Datacenter (Ashburn, VA)
├── Core Banking: Mainframe (DB2)
├── Credit Card DB: Oracle RAC (40 TB)
├── User Directory: Active Directory (50M users)
└── ATM Network: Real-time transaction processing

Connection to AWS:
1. AWS Direct Connect (Primary)
   ├── Location: Equinix DC2 (Ashburn, VA)
   ├── Speed: 10 Gbps × 4 connections = 40 Gbps total
   ├── Redundancy: 2 physical routers (active-active)
   └── Cost: $1,440/month per 10 Gbps = $5,760/month total

2. Site-to-Site VPN (Backup)
   ├── IPsec tunnels over internet
   ├── Speed: 1.25 Gbps max per tunnel
   ├── Redundancy: 2 tunnels (active-standby)
   └── Cost: $36/month per VPN connection = $72/month

AWS VPC (us-east-1)
├── Private Subnets:
│   ├── Mobile API: EC2 Auto Scaling (100-500 instances)
│   ├── Web Tier: ALB + ECS containers
│   └── Data Tier: Aurora PostgreSQL + DynamoDB
│
└── Connectivity:
    ├── Virtual Private Gateway (VGW): Terminates Direct Connect
    ├── Customer Gateway: On-premises router (Cisco ASR 1000)
    └── BGP Routing: Dynamic route propagation (autonomous system)

Capital One Requirements Analysis:

Requirement 1: High Bandwidth (Mainframe ↔ AWS)
    Data transfer: 10 TB/day (database replication)
    
    VPN Capacity:
        Speed: 1.25 Gbps max per tunnel
        Transfer time: 10 TB × 8 = 80 Tb ÷ 1.25 Gbps = 64,000 seconds = 17.7 hours 
        Problem: Can't complete daily sync in 24 hours!
    
    Direct Connect Capacity:
        Speed: 10 Gbps per connection
        Transfer time: 80 Tb ÷ 10 Gbps = 8,000 seconds = 2.2 hours 
        With 4 connections: 2.2 hours ÷ 4 = 33 minutes (plenty of headroom)
    
    Verdict: Need Direct Connect for bandwidth 

Requirement 2: Consistent Latency (Real-time Transactions)
    ATM transaction: User → AWS → On-prem DB → AWS → User
    Target: <500ms total (acceptable for ATM)
    
    VPN Latency:
        Internet routing: Variable (50-200ms P95)
        Peak hours: 300ms+ (congested)
        Jitter: 50-100ms (inconsistent)
        Result: 200ms + 100ms jitter = 300ms P95 ️
    
    Direct Connect Latency:
        Dedicated fiber: Consistent (<10ms P95)
        No internet congestion: Always optimal path
        Jitter: <1ms (stable)
        Result: 10ms + 1ms jitter = 11ms P95 
    
    Verdict: Direct Connect provides predictable latency 

Requirement 3: Security (PCI DSS Compliance)
    Requirement: Payment Card Industry Data Security Standard Level 1
    
    VPN Security:
        Encryption: IPsec (AES-256-GCM)
        Key exchange: IKEv2 (perfect forward secrecy)
        Transit: Over public internet (potential eavesdropping)
        Compliance: Acceptable but requires additional controls
    
    Direct Connect Security:
        Encryption: Optional (private connection, not internet)
        MACsec: Layer 2 encryption (10 Gbps line-rate)
        Transit: Dedicated fiber (physically separate from internet)
        Compliance: Preferred by auditors (air-gap from internet)
    
    Verdict: Both compliant, Direct Connect preferred 

Requirement 4: Cost (5-Year TCO)
    Transfer: 10 TB/day × 30 days = 300 TB/month
    
    VPN Cost:
        VPN Connection: $36/month × 2 = $72/month
        Data transfer: 300 TB × $0.09/GB = $27,000/month
        Total: $27,072/month = $324,864/year
        5-year TCO: $1,624,320
    
    Direct Connect Cost:
        Port fee: $1,440/month × 4 = $5,760/month
        Data transfer: 300 TB × $0.02/GB = $6,000/month
        Total: $11,760/month = $141,120/year
        5-year TCO: $705,600
    
    Savings: $1,624,320 - $705,600 = $918,720 over 5 years! 
    
    Verdict: Direct Connect 56% cheaper despite higher port fees 

Decision: Use Direct Connect (primary) + VPN (backup)
    - Direct Connect: Handle all normal traffic (40 Gbps)
    - VPN: Failover only (if Direct Connect fails)
    - Benefit: High performance + disaster recovery

Physical Topology:

Capital One Datacenter (Ashburn, VA)
  ↓ Single-mode fiber (10 km)
Equinix DC2 (Colocation Facility)
  ├─ Router 1: Cisco ASR 1002-HX (Direct Connect 1 + 2)
  ├─ Router 2: Cisco ASR 1002-HX (Direct Connect 3 + 4)
  ↓ AWS Direct Connect
AWS Direct Connect Location (Ashburn)
  ↓ Cross-connect
AWS VPC (us-east-1)

Direct Connect Configuration:

# Connection 1 (10 Gbps)
aws directconnect create-connection \
    --location EqDC2 \
    --bandwidth 10Gbps \
    --connection-name "CapitalOne-Primary-1"

# Virtual Interface (VIF) - Private connectivity
aws directconnect create-private-virtual-interface \
    --connection-id dxcon-abc123 \
    --new-private-virtual-interface \
        virtualInterfaceName=CapitalOne-VIF-1,\
        vlan=100,\
        asn=65001,\                     # Capital One ASN
        amazonAddress=169.254.1.1/30,\  # AWS router IP
        customerAddress=169.254.1.2/30  # Capital One router IP

BGP Configuration (Cisco ASR 1002-HX):
router bgp 65001
  neighbor 169.254.1.1 remote-as 7224  # AWS ASN
  neighbor 169.254.1.1 activate
  
  # Advertise on-premises networks
  network 10.50.0.0 mask 255.255.0.0  # Datacenter CIDR
  
  # Prefer Direct Connect over VPN (higher weight)
  neighbor 169.254.1.1 weight 200

Route Propagation:
  On-premises → AWS:
    - 10.50.0.0/16 (datacenter) → Advertised via BGP
    - AWS learns route automatically
  
  AWS → On-premises:
    - 10.0.0.0/16 (VPC) → Advertised via BGP
    - Capital One router learns automatically
  
  Result: Dynamic routing (no manual route table updates)

Link Aggregation Group (LAG):
  - Combine 4 × 10 Gbps into single 40 Gbps logical link
  - Active-active: All 4 connections used simultaneously
  - Failover: Automatic (if one fails, others continue)
  - LACP protocol: IEEE 802.3ad (industry standard)

# Create LAG
aws directconnect create-lag \
    --location EqDC2 \
    --connections-bandwidth 10Gbps \
    --number-of-connections 4 \
    --lag-name "CapitalOne-LAG-40G"

Performance Metrics (2017-2019, During Migration):

Bandwidth Utilization:
    Average: 15 Gbps (38% of 40 Gbps capacity)
    Peak: 35 Gbps (88% of capacity, during data migration)
    Growth: 20%/year (as more apps moved to AWS)

Latency (us-east-1 VPC to Ashburn Datacenter):
    Direct Connect: P50 = 8ms, P95 = 10ms, P99 = 12ms
    VPN (backup): P50 = 120ms, P95 = 180ms, P99 = 300ms
    Result: Direct Connect 12× faster than VPN (10ms vs 120ms)

Failover Testing:
    Scenario: Primary Direct Connect fails (simulated fiber cut)
    Detection: BGP timeout 180 seconds (3× keepalive)
    Failover: Traffic switches to VPN automatically
    Impact: Latency increased 120ms → 180ms (noticeable but acceptable)
    Recovery: Direct Connect restored in 4 hours (fiber repair)

Monthly Data Transfer:
    2017: 1 PB/month (initial apps)
    2018: 3 PB/month (credit card migration)
    2019: 5 PB/month (core banking migration)
    2020: 0 PB/month (all in AWS, no on-premises)

Cost Over Time:
    2017: $11,760/month ($141K/year)
    2018: $11,760/month (same, data transfer within free tier)
    2019: $11,760/month + $100K data transfer = $241K/year
    2020: $0/month (Direct Connect decommissioned, all cloud)

5-year cost: $141K + $141K + $241K + $0 + $0 = $523K total

VPN Tunnel Architecture:

Capital One Router (Ashburn)
  ↓ Internet (ISP: Verizon Business)
AWS VPN Endpoint (Virtual Private Gateway)
  ↓
AWS VPC (us-east-1)

VPN Configuration:

# Create Customer Gateway (on-premises side)
aws ec2 create-customer-gateway \
    --type ipsec.1 \
    --public-ip 203.0.113.50 \    # Capital One public IP
    --bgp-asn 65001 \              # Same ASN as Direct Connect
    --tag-specifications \
        'ResourceType=customer-gateway,Tags=[{Key=Name,Value=CapitalOne-CGW}]'

# Create VPN Connection
aws ec2 create-vpn-connection \
    --type ipsec.1 \
    --customer-gateway-id cgw-abc123 \
    --vpn-gateway-id vgw-def456 \
    --options \
        StaticRoutesOnly=false,\          # Use BGP (not static)
        TunnelOptions=[{\
            TunnelInsideCidr=169.254.10.0/30,\
            PreSharedKey="32-character-random-string"\
        }]

IPsec Configuration:
Phase 1 (IKE):
    Encryption: AES-256-GCM
    Integrity: SHA-256
    DH Group: 14 (2048-bit)
    Lifetime: 28800 seconds (8 hours)

Phase 2 (IPsec):
    Encryption: AES-256-GCM
    Integrity: SHA-256
    PFS Group: 14 (perfect forward secrecy)
    Lifetime: 3600 seconds (1 hour)

BGP Configuration (Lower Priority than Direct Connect):
router bgp 65001
  neighbor 169.254.10.1 remote-as 7224  # AWS VPN endpoint
  neighbor 169.254.10.1 activate
  
  # Lower weight than Direct Connect (fallback only)
  neighbor 169.254.10.1 weight 100  # vs 200 for Direct Connect
  
  # Same networks advertised
  network 10.50.0.0 mask 255.255.0.0

Result: VPN routes installed but not preferred
    - Direct Connect active: Traffic uses Direct Connect (weight 200)
    - Direct Connect fails: Traffic fails over to VPN (weight 100)
    - Automatic: No manual intervention required

Cloud-Only Architecture (No On-Premises):

AWS VPC (Multi-Region)
├── us-east-1 (Primary)
│   ├── Availability Zones: 3 (1a, 1b, 1c)
│   ├── Applications: All banking applications
│   ├── Databases: Aurora, DynamoDB, ElastiCache
│   └── Branches: 750 branches connect via internet (SD-WAN)
│
├── us-west-2 (Disaster Recovery)
│   ├── Warm standby: All applications deployed
│   ├── Databases: Aurora Global Database (async replication)
│   └── Failover: Automated (Route 53 health checks)
│
└── eu-west-1 (Europe Operations)
    └── GDPR compliance (data residency)

Branch Connectivity (750 Branches):
Before (On-Premises):
    Branch → MPLS Network → Datacenter → Applications
    Cost: $5,000/month per branch × 750 = $3.75M/month = $45M/year
    Latency: 50-100ms (predictable)
    Bandwidth: 100 Mbps (fixed)

After (Cloud):
    Branch → Internet (SD-WAN) → AWS (CloudFront/Direct Connect)
    Cost: $500/month per branch × 750 = $375K/month = $4.5M/year
    Latency: 30-80ms (variable but acceptable)
    Bandwidth: 1 Gbps (10× faster)
    
    Savings: $45M - $4.5M = $40.5M/year on branch connectivity! 

ATM Network (15,000 ATMs):
Before:
    ATM → Dedicated Line → Datacenter → Mainframe
    Cost: $200/month per ATM × 15,000 = $3M/month = $36M/year

After:
    ATM → Internet (4G/5G backup) → AWS API Gateway → Lambda
    Cost: $50/month per ATM × 15,000 = $750K/month = $9M/year
    
    Savings: $36M - $9M = $27M/year on ATM connectivity! 

Financial Impact:

Infrastructure Costs:
    Before (8 Datacenters):
        Real estate: $50M/year (leases, power, cooling)
        Hardware: $200M/year (servers, storage, network)
        Staff: 1,000 IT ops × $100K = $100M/year
        MPLS network: $45M/year (branch connectivity)
        Total: $395M/year

    After (AWS):
        AWS compute: $150M/year (EC2, Lambda, containers)
        AWS storage: $30M/year (S3, EBS, Glacier)
        AWS network: $20M/year (data transfer, Direct Connect)
        Staff: 200 ops × $100K = $20M/year (cloud-focused, automated)
        Internet: $4.5M/year (branch connectivity)
        Total: $224.5M/year

    Savings: $395M - $224.5M = $170.5M/year 
    5-year savings: $852.5M (close to $1B!)

Innovation Speed:
    Before:
        Feature release: Quarterly (4 per year)
        New infrastructure: 6-18 months (hardware procurement)
        Testing: Manual (weeks per release)
    
    After:
        Feature release: Daily (365+ per year)
        New infrastructure: Minutes (EC2 auto-scaling)
        Testing: Automated (CI/CD pipelines)
    
    Result: 90× faster feature delivery 

Availability:
    Before (Datacenter):
        Uptime: 99.9% (8.7 hours downtime/year)
        Maintenance windows: Monthly (2 hours each)
    
    After (AWS):
        Uptime: 99.99% (52 minutes downtime/year)
        Maintenance: Zero (rolling updates, no downtime)
    
    Improvement: 10× better availability 

Security:
    Before:
        Data breaches: 1 major (2019, 106M customers affected)
        Cost: $270M+ (settlements, fines, remediation)
        Cause: Misconfigured firewall (on-premises)
    
    After:
        Data breaches: 0 (2020-2024)
        Improvements: 
            - AWS IAM (least privilege, no root access)
            - AWS GuardDuty (threat detection)
            - AWS Config (compliance monitoring)
            - Automated patching (no human error)
    
    Note: 2019 breach accelerated cloud migration
          (cloud offers better security than on-premises)

Regulatory Compliance:
    Certifications: PCI DSS Level 1, SOC 2, ISO 27001, FedRAMP
    Audits: Passed all audits 2020-2024 (AWS shared responsibility model)
    Examiners: OCC (Office of the Comptroller of the Currency) approved

Success Factors:

1. Executive Sponsorship
   - CEO: "Cloud-first or cloud-only, not hybrid forever"
   - Deadline: 2020 (enforced, not flexible)
   - Result: Organization aligned, migration prioritized

2. Incremental Migration (Not Big Bang)
   - Phase 1: Non-critical apps (learning phase)
   - Phase 2: Customer-facing apps (build confidence)
   - Phase 3: Core banking (final step, most critical)
   - Duration: 7 years (2013-2020)

3. Hybrid Connectivity Done Right
   - Direct Connect: Primary (40 Gbps, low latency)
   - VPN: Backup (automatic failover, no manual intervention)
   - Testing: Monthly failover drills (ensure VPN works)

4. Rearchitecting, Not Lift-and-Shift
   - Mainframes: Rewritten as microservices (not migrated)
   - Monoliths: Decomposed into serverless (Lambda, containers)
   - Databases: Oracle → Aurora PostgreSQL (open source)
   - Result: Better architecture, not just cloud hosting

5. Training and Culture Change
   - Engineers: 11,000 engineers trained on AWS
   - Certifications: 5,000+ AWS certifications earned
   - Culture: DevOps mindset (build it, run it, own it)

Common Mistakes to Avoid:

1. Hybrid Forever
   Problem: "We'll keep some on-premises, some cloud"
   Reality: Hybrid is transition state, not end state
   Reason: Double costs (maintain both environments)
   Capital One: Set deadline (2020), enforced migration

2. Lift and Shift Without Optimization
   Problem: Migrate as-is (VM → EC2 with no changes)
   Reality: Cloud costs more without rearchitecting
   Reason: Cloud pricing different (per-hour vs capex)
   Capital One: Rearchitected for cloud-native (serverless, containers)

3. Underestimating Network Requirements
   Problem: "We'll use VPN, it's good enough"
   Reality: VPN too slow for large data transfers
   Reason: Internet bandwidth/latency variable
   Capital One: Direct Connect essential (10 TB/day transfers)

4. Ignoring Security in Cloud
   Problem: "Cloud is less secure than on-premises"
   Reality: Cloud offers better security tools
   Reason: AWS invests billions in security
   Capital One: Zero breaches post-migration (vs 1 major before)

5. Not Planning Exit Strategy
   Problem: "Datacenters empty, but can't shut down yet"
   Reality: Idle infrastructure costs money
   Reason: Legacy dependencies discovered late
   Capital One: Mapped dependencies first (12 months planning)

Traditional Security (Perimeter-Based) - OBSOLETE:
┌─────────────────────────────────────────────────────┐
│ Corporate Network (Inside = Trusted)                 │
│ ├─ Employees: Full access to all systems           │
│ ├─ Servers: No authentication between services     │
│ └─ Data: Accessible once inside network            │
└─────────────────────────────────────────────────────┘
     ↑
  Firewall (Outside = Untrusted)
     ↓
┌─────────────────────────────────────────────────────┐
│ Internet (Blocked by default)                        │
└─────────────────────────────────────────────────────┘

Problem: Castle-and-moat model (hard shell, soft interior)
- Once inside: Full access (lateral movement easy)
- Stolen credentials: Complete compromise
- Remote work: VPN gives network access (too broad)

Zero Trust Security (Assume Breach) - MODERN:
┌─────────────────────────────────────────────────────┐
│ Identity-Based Access (Every Request Authenticated)  │
│                                                       │
│ User (Alice) → Identity Provider → Policy Engine    │
│   ↓              ↓                   ↓               │
│   ├─ MFA: Phone + Biometric        Decision         │
│   ├─ Device: Managed laptop         ├─ Allow/Deny   │
│   ├─ Location: NYC office           ├─ Conditions   │
│   └─ Risk: Low (known device)       └─ Time-based   │
│                                                       │
│ Service A → Service B                                │
│   ↓                                                   │
│   ├─ mTLS: Certificate-based auth                   │
│   ├─ JWT: Token with expiry (15 min)                │
│   └─ Scope: Read-only, specific data                │
└─────────────────────────────────────────────────────┘

Principles:
1. Never trust, always verify (even inside network)
2. Assume breach (defense in depth)
3. Verify explicitly (every request authenticated)
4. Least privilege (minimum access required)
5. Microsegmentation (isolate workloads)

Identity Providers (Multiple):

1. Employee Access (45,000 employees)
   Provider: Okta + Azure AD (hybrid)
   Authentication:
     ├─ Primary: Okta (SAML 2.0, OAuth 2.0)
     ├─ MFA: Duo Security (SMS + push + U2F)
     ├─ SSO: Single sign-on to 500+ applications
     └─ Adaptive Auth: Risk-based (location, device, behavior)

2. Developer Access (3,000 developers)
   Provider: AWS IAM + GitHub SSO
   Authentication:
     ├─ GitHub: Source of truth (teams, repos)
     ├─ AWS IAM Roles: Assume via GitHub OIDC
     ├─ Temporary Creds: 1-hour expiry (refresh needed)
     └─ Just-in-Time: Request elevated access (approve/deny)

3. Service-to-Service (10,000+ microservices)
   Provider: AWS IAM Roles for Service Accounts
   Authentication:
     ├─ No static credentials (no passwords!)
     ├─ IAM Roles: Attached to EC2, ECS, Lambda
     ├─ STS: Temporary security tokens (auto-rotated)
     └─ mTLS: Certificate-based mutual authentication

4. Customer Access (millions of clients)
   Provider: OAuth 2.0 + OpenID Connect
   Authentication:
     ├─ Marcus (consumer banking): OAuth 2.0 flows
     ├─ Goldman Sachs app: Mobile SDKs (biometric)
     ├─ API access: API keys + OAuth tokens
     └─ Rate limiting: 1,000 req/min per client

Example 1: Developer Access to Production (Least Privilege)

Bad Policy (Too Permissive):
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "*",                    # ALL actions (dangerous!)
    "Resource": "*"                   # ALL resources (overly broad!)
  }]
}

Problem: Developer can delete production databases, read customer data, etc.

Good Policy (Least Privilege):
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOnlyProductionLogs",
      "Effect": "Allow",
      "Action": [
        "logs:GetLogEvents",           # Read CloudWatch Logs
        "logs:FilterLogEvents",        # Search logs
        "logs:DescribeLogStreams"      # List log streams
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:123456789:log-group:/aws/lambda/trading-*"
      ],
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "10.50.0.0/16"             # Only from Goldman Sachs VPN
          ]
        },
        "DateGreaterThan": {
          "aws:CurrentTime": "2024-01-01T09:00:00Z"  # Business hours only
        },
        "DateLessThan": {
          "aws:CurrentTime": "2024-01-01T18:00:00Z"
        }
      }
    },
    {
      "Sid": "DeployToDevEnvironmentOnly",
      "Effect": "Allow",
      "Action": [
        "lambda:UpdateFunctionCode",   # Deploy code
        "lambda:PublishVersion"        # Create version
      ],
      "Resource": [
        "arn:aws:lambda:us-east-1:123456789:function:trading-dev-*"  # Dev only!
      ]
    },
    {
      "Sid": "DenyProductionModification",
      "Effect": "Deny",                # Explicit DENY (cannot override)
      "Action": [
        "lambda:UpdateFunctionCode",
        "rds:DeleteDBInstance",
        "s3:DeleteBucket"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Key Improvements:
1. Specific actions (not *)
2. Specific resources (ARNs, not *)
3. Conditions (IP, time, tags)
4. Explicit denies (production protected)

Example 2: Trading System Service Role (Service-to-Service)

Trading Service → Market Data Service:

# IAM Role for Trading Service (attached to ECS task)
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "execute-api:Invoke"            # Call API Gateway
    ],
    "Resource": [
      "arn:aws:execute-api:us-east-1:*/prod/GET/market-data/prices"  # Specific endpoint
    ]
  }]
}

# API Gateway Resource Policy (restricts who can call)
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::123456789:role/TradingServiceRole"  # Specific role only
    },
    "Action": "execute-api:Invoke",
    "Resource": "arn:aws:execute-api:us-east-1:*/prod/GET/market-data/prices"
  }]
}

Security Benefits:
- No API keys (compromised credentials = breach)
- Temporary credentials (auto-rotated every hour)
- Specific permissions (trading can't access settlement data)
- Audit trail (CloudTrail logs every API call with identity)

Example 3: Emergency Break-Glass Access (Production Incident)

Problem: Production down, need immediate admin access

Bad Approach: Shared "admin" account with permanent access
- Security risk: Multiple people know password
- No accountability: Can't determine who made changes
- Compliance violation: SOX requires individual accountability

Good Approach: Just-in-Time (JIT) Privileged Access
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "*",                    # Full admin (temporary only!)
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "aws:RequestedRegion": "us-east-1"
      },
      "DateGreaterThan": {
        "aws:CurrentTime": "2024-01-15T14:30:00Z"  # Incident start
      },
      "DateLessThan": {
        "aws:CurrentTime": "2024-01-15T16:30:00Z"  # 2-hour window
      }
    }
  }]
}

Process:
1. Incident declared: P1 outage (trading system down)
2. Engineer requests access: "Need RDS admin for 2 hours"
3. Manager approves: Slack approval (logged)
4. Temporary role assigned: 2-hour window
5. Access automatically revoked: After 2 hours (no manual cleanup)
6. Audit: CloudTrail logs every action with identity

Result: Emergency access + accountability + automatic expiry

Scale Metrics:

Identities:
    Employees: 45,000 (Okta managed)
    Service accounts: 10,000+ (IAM roles)
    API clients: 1,000+ (OAuth clients)
    Total: 56,000+ identities

Authentication Events:
    Daily logins: 200,000+ (employees + services)
    MFA challenges: 100,000+ per day
    API calls: 10 billion+ per day (service-to-service)
    Failed attempts: 50,000+ per day (blocked attacks)

IAM Policies:
    Custom policies: 5,000+ (specific to services)
    Managed policies: 500+ (reusable across teams)
    Policy size: Average 2 KB (complex conditions)
    Policy versions: 10,000+ (versioned for rollback)

Access Requests:
    Just-in-Time: 1,000+ per month (emergency access)
    Approval time: Average 15 minutes (manager approval)
    Auto-revocation: 100% (zero manual cleanup)
    Audit queries: 10,000+ per month (compliance reviews)

Cost:
    Okta: $8 per user/month × 45,000 = $360K/month = $4.32M/year
    Duo MFA: $3 per user/month × 45,000 = $135K/month = $1.62M/year
    AWS IAM: FREE (included with AWS)
    Staff: 20 IAM engineers × $200K = $4M/year
    Total: $9.94M/year for 45,000 users = $221/user/year

ROI:
    Without IAM: Security breaches average $4.45M each (IBM study)
    Goldman Sachs: Zero breaches 2020-2024 (attributable to Zero Trust)
    Prevented losses: $4.45M × 4 years = $17.8M
    Cost: $9.94M/year × 4 = $39.76M
    Net: -$21.96M (investment, but prevents catastrophic loss)
    
    Note: One breach could cost billions (reputation, customer loss)
          Citigroup 2020 breach: $400M+ (wire transfer error, access control failure)

1. No Long-Lived Credentials (Rotate Everything)

Bad:
    AWS Access Key: Created in 2020, never rotated
    Password: Same since hire date
    API Key: Hardcoded in source code

Good:
    AWS Access Key: Temporary (STS, 1-hour expiry)
    Password: Rotated every 90 days (enforced)
    API Key: OAuth tokens (15-minute expiry, refresh token)

Implementation:
    # Enforce 90-day password rotation
    aws iam update-account-password-policy \
        --max-password-age 90 \
        --password-reuse-prevention 5

    # Audit stale credentials
    aws iam generate-credential-report
    # Report shows: Access key age, last used date
    # Action: Deactivate keys >90 days old

2. MFA Everywhere (No Exceptions)

Coverage:
    AWS Console: MFA required (virtual MFA or U2F key)
    AWS CLI: MFA via session token
    SSH: MFA via PAM module
    VPN: MFA via Duo (push notification)
    Applications: SAML with MFA assertion

# Enforce MFA for AWS Console
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": {
      "BoolIfExists": {
        "aws:MultiFactorAuthPresent": "false"
      }
    }
  }]
}

Result: Zero access without MFA (no backdoors)

3. Separation of Duties (No Single Person with Full Access)

Principle: No one person should be able to:
    1. Write code
    2. Deploy to production
    3. Approve the deployment
    
    Requires collusion of multiple people to cause damage

Example: Production Deployment

Step 1: Developer writes code (Alice)
    - Can commit to GitHub
    - Cannot merge to main branch
    - Cannot deploy to production

Step 2: Code review (Bob)
    - Can review pull request
    - Can approve merge
    - Cannot deploy (different person required)

Step 3: Deployment (Charlie - SRE)
    - Can trigger deployment
    - Cannot modify code
    - Cannot bypass approvals

Step 4: Audit (Automated)
    - CloudTrail logs all actions
    - Alert if same person performs multiple steps
    - Security team reviews anomalies

4. Attribute-Based Access Control (ABAC)

Traditional (Role-Based):
    Problem: 1 role per team = 100 teams = 100 roles (unmanageable)

Modern (Attribute-Based):
    Solution: 1 policy using tags = scales infinitely

Example:
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:*",
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "s3:ExistingObjectTag/Team": "${aws:PrincipalTag/Team}",
        "s3:ExistingObjectTag/Environment": "${aws:PrincipalTag/Environment}"
      }
    }
  }]
}

How it works:
    User tagged: Team=Trading, Environment=Dev
    S3 object tagged: Team=Trading, Environment=Dev
    Result: Access granted (tags match)
    
    S3 object tagged: Team=Trading, Environment=Prod
    Result: Access denied (Environment mismatch)

Benefit: 1 policy for all teams (scales)

5. Regular Access Reviews (Quarterly Recertification)

Process:
    Q1: Export all IAM users + permissions
    Q2: Send to managers: "Certify your team's access"
    Q3: Managers review: "Does Alice still need RDS access?"
    Q4: Remove unused permissions: "Bob left 6 months ago" → Delete

Automation:
    # Find users with no activity in 90 days
    aws iam generate-credential-report
    # Parse CSV: last_used > 90 days ago
    # Email manager: "These users inactive, revoke access?"
    # Auto-revoke if no response in 7 days

Goldman Sachs Results:
    Inactive accounts removed: 500+ per quarter
    Permissions revoked: 2,000+ per quarter
    Over-privileged access: 10% reduction per quarter
    Compliance: 100% (SOX requires annual recertification)

Capital One Data Breach (July 2019):

What Happened:
    - 106 million customer records stolen
    - Attacker: Former AWS employee (Paige Thompson)
    - Method: SSRF (Server-Side Request Forgery) exploit
    - IAM misconfiguration: WAF role had excessive permissions

Technical Details:

Misconfiguration:
    IAM Role: WAF-Role (attached to EC2 firewall)
    Policy:
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": "*"              # PROBLEM: All S3 buckets!
    }

Attack Flow:
    1. Attacker exploited SSRF in WAF (sent request to metadata endpoint)
    2. Retrieved IAM credentials: http://169.254.169.254/latest/meta-data/iam/...
    3. Used credentials to list S3 buckets: aws s3 ls
    4. Found credit application data: capital-one-credit-apps
    5. Downloaded 700+ files: aws s3 cp s3://capital-one-credit-apps/ . --recursive
    6. Result: 106M records stolen (names, SSNs, addresses, scores)

What Should Have Been Done:

Correct Policy (Least Privilege):
{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:ListBucket"
  ],
  "Resource": [
    "arn:aws:s3:::waf-logs-only",              # Specific bucket only!
    "arn:aws:s3:::waf-logs-only/*"
  ]
}

Additional Controls:
    1. S3 Block Public Access: Enabled (prevent accidental public buckets)
    2. S3 Bucket Policies: Deny access except from specific roles
    3. VPC Endpoints: S3 access via private network (not internet)
    4. IMDSv2: Require token (prevent SSRF metadata access)
    5. GuardDuty: Alert on unusual S3 access patterns

Cost of Breach:
    Settlements: $190M+ (customers + OCC fine)
    Legal fees: $30M+
    Reputation: Immeasurable (loss of customer trust)
    Stock: -6% in week after disclosure
    Total: $270M+ direct costs

Lessons Learned:
    1. Least privilege: Only grant specific resources (never "Resource": "*")
    2. Defense in depth: Multiple layers (policy + bucket policy + VPC endpoint)
    3. Monitoring: GuardDuty would have alerted on unusual S3 access
    4. Metadata protection: IMDSv2 prevents SSRF credential theft
    5. Regular audits: Automated tools catch overly permissive policies

Goldman Sachs Response:
    - Reviewed all IAM policies: 5,000+ policies audited
    - Removed wildcards: "Resource": "*" changed to specific ARNs
    - Enabled GuardDuty: Threat detection on all accounts
    - IMDSv2 required: All EC2 instances upgraded
    - Result: Zero similar breaches (2019-2024)

Three-Tier Encryption Model:

Tier 1: Device Encryption (Hardware-Based)
┌─────────────────────────────────────────────────────────┐
│ iPhone/iPad Secure Enclave (Dedicated crypto chip)      │
│                                                           │
│ User creates passcode: "123456" (example, don't use!)   │
│   ↓                                                       │
│ Passcode + Device UID → Key Derivation Function (PBKDF2)│
│   ↓                                                       │
│ Encryption Key (256-bit AES) - NEVER leaves device      │
│   ↓                                                       │
│ All data encrypted: Photos, messages, health data, etc. │
│                                                           │
│ Key Properties:                                          │
│ - Tied to device UID (can't copy to another device)     │
│ - Requires passcode (can't decrypt without user)        │
│ - Hardware-protected (Secure Enclave, isolated)         │
│ - Auto-wipe after 10 failed attempts (security feature) │
└─────────────────────────────────────────────────────────┘

Tier 2: iCloud Encryption (Cloud Storage)
┌─────────────────────────────────────────────────────────┐
│ Standard iCloud (Apple has keys - can decrypt)          │
│                                                           │
│ User uploads photo to iCloud:                            │
│ 1. Photo encrypted on device (random key)               │
│ 2. Key wrapped with iCloud key (Apple-managed)          │
│ 3. Both uploaded to iCloud (encrypted data + wrapped key)│
│ 4. Apple can decrypt: For law enforcement, recovery     │
│                                                           │
│ Storage: AWS S3, Google Cloud, Azure (multi-cloud)      │
│ Encryption: AES-256 (at rest)                           │
│ Keys: Apple CloudHSM (Hardware Security Module)         │
└─────────────────────────────────────────────────────────┘

Tier 3: Advanced Data Protection (End-to-End Encryption)
┌─────────────────────────────────────────────────────────┐
│ iCloud E2E (Apple CANNOT decrypt - user keys only)      │
│                                                           │
│ User enables Advanced Data Protection (2022+):           │
│ 1. Photo encrypted on device (random key)               │
│ 2. Key encrypted with user's passcode (not iCloud key)  │
│ 3. Encrypted key stored locally (not in Apple's control)│
│ 4. Apple cannot decrypt: No key access                  │
│                                                           │
│ Protected data categories:                              │
│ - Photos & Videos (user's choice)                       │
│ - Notes (user's choice)                                 │
│ - Voice Memos                                           │
│ - Safari Bookmarks                                      │
│ - Wallet Passes                                         │
│                                                           │
│ Not protected (compatibility):                          │
│ - Mail (SMTP doesn't support E2E)                       │
│ - Contacts (synced with non-Apple devices)              │
│ - Calendar (shared with others)                         │
└─────────────────────────────────────────────────────────┘

iMessage E2E Encryption: Zero-knowledge architecture

Setup Phase (One-time per device):
1. iPhone generates key pair:
   - Private key: Stored in Secure Enclave (never leaves device)
   - Public key: Uploaded to Apple's IDS (Identity Service)

2. Apple IDS (Identity Directory Service):
   - Maps: phone number → public keys
   - Example: +1-555-0100 → [iPhone key, iPad key, Mac key]
   - Note: Apple doesn't store private keys (can't decrypt)

Sending Message (Alice → Bob):
┌─────────────────────────────────────────────────────────┐
│ 1. Alice types: "Hi Bob"                                │
│    ↓                                                     │
│ 2. Query Apple IDS: What are Bob's public keys?         │
│    Response: Bob has 3 devices (iPhone, iPad, Mac)      │
│    ↓                                                     │
│ 3. Encrypt message 3 times (once per device):           │
│    - Bob's iPhone key: Encrypt("Hi Bob") = Cipher1     │
│    - Bob's iPad key: Encrypt("Hi Bob") = Cipher2       │
│    - Bob's Mac key: Encrypt("Hi Bob") = Cipher3        │
│    ↓                                                     │
│ 4. Upload to Apple Push Notification Service (APNs):    │
│    - Cipher1 → iPhone                                   │
│    - Cipher2 → iPad                                     │
│    - Cipher3 → Mac                                      │
│    ↓                                                     │
│ 5. Each device decrypts with its private key            │
│    (Only Bob can read, Apple cannot decrypt)            │
└─────────────────────────────────────────────────────────┘

Group Messages (Alice → Bob + Charlie + David):
- Query IDS: Bob has 3 devices, Charlie has 2, David has 1
- Encrypt: 6 times total (once per device)
- Upload: 6 ciphertexts to APNs
- Result: All devices can read, Apple cannot

Security Properties:
End-to-end: Only sender and recipients can read
Perfect forward secrecy: Compromised key doesn't affect past messages
Zero-knowledge: Apple cannot read messages (no keys)
Multi-device: Same message readable on all user's devices
Metadata protection: Apple knows who messaged whom, but not content

Limitations:
Metadata visible: Apple knows Alice messaged Bob (timestamp, frequency)
Device compromise: If device hacked, messages readable
Backup plaintext: iCloud Backup (if enabled) contains keys (opt-in)

Key Management Infrastructure:

CloudHSM (Hardware Security Modules):
    Vendor: Thales (formerly Gemalto)
    Locations: 10+ data centers globally
    Devices: 1,000+ HSM appliances
    Cost: $30K per HSM × 1,000 = $30M initial investment
    Throughput: 10,000 operations/second per HSM
    Failover: Automatic (clustered for HA)

Key Types:

1. Master Keys (Root of Trust):
   - Location: HSM only (never exported)
   - Count: 10 master keys (one per region)
   - Rotation: Annually (migrated to new keys)
   - Access: <10 Apple employees (split knowledge)

2. Data Encryption Keys (DEKs):
   - Count: Trillions (one per file/object)
   - Generated: On-demand (per file upload)
   - Lifetime: Permanent (until file deleted)
   - Storage: Encrypted with KEKs (key encryption keys)

3. Key Encryption Keys (KEKs):
   - Count: Millions (one per user)
   - Derived: From user passcode + device UID
   - Rotation: When passcode changed
   - Storage: Secure Enclave (hardware-protected)

Key Hierarchy:
Master Key (HSM)
  ├─ KEK (user-specific)
  │   ├─ DEK (file 1)
  │   ├─ DEK (file 2)
  │   └─ DEK (file 3)
  ├─ KEK (another user)
  └─ ...

Benefit: Compromise of one DEK doesn't affect others

Key Operations Per Day:
    Encryption: 10 billion+ (files uploaded to iCloud)
    Decryption: 100 billion+ (files downloaded, 10× reads)
    Key generation: 1 billion+ (new files)
    Key rotation: 1 million+ (users change passcodes)
    Total: 111 billion+ cryptographic operations/day

Performance:
    HSM throughput: 10,000 ops/sec × 1,000 HSMs = 10M ops/sec
    Daily capacity: 10M × 86,400 = 864 billion ops/day
    Utilization: 111B ÷ 864B = 12.8% (plenty of headroom)

TLS Configuration (Apple Services):

Certificate Management:
    Issuer: Apple (self-signed root CA for internal)
    Public services: DigiCert, Let's Encrypt (third-party CA)
    Rotation: 90 days (automated, no downtime)
    Devices: 2 billion trust Apple's root CA (preinstalled)

TLS Version Enforcement:
    TLS 1.3: Required (deprecated TLS 1.0, 1.1, 1.2)
    Cipher suites: ChaCha20-Poly1305, AES-GCM only
    Forward secrecy: Mandatory (ECDHE key exchange)
    Certificate pinning: iOS apps pin Apple's certificates

# Example Nginx configuration (Apple's web servers)
ssl_protocols TLSv1.3;  # Only TLS 1.3 (strongest)
ssl_ciphers 'ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_stapling on;  # OCSP stapling (faster certificate validation)
ssl_stapling_verify on;

TLS Performance at Scale:
    Connections: 1 billion+ per day (device check-ins, App Store)
    Handshake time: 50ms average (TLS 1.3 faster than 1.2)
    Resumption: 80% (session resumption avoids full handshake)
    Overhead: <5ms per request (hardware acceleration)

Hardware Acceleration:
    Servers: AES-NI (Intel), Apple Silicon (custom crypto)
    Devices: Secure Enclave (iPhone), T2 chip (Mac)
    Throughput: 10 Gbps encrypted traffic per server (line-rate)
    CPU impact: <10% (would be 80% without acceleration)

Feature Launch: December 2022 (iOS 16.2, macOS Ventura 13.1)

Adoption Metrics (2024):
    Eligible users: 1.5B iCloud users
    Enabled: <5% (~75M users) - low adoption!
    Reason: Trade-offs not well understood

Why Low Adoption?

Drawbacks:
    1. No account recovery: Lost passcode = lost data forever
       - Apple cannot help (doesn't have keys)
       - No backdoor (by design)
       - Recovery: Only via trusted device or recovery key (physical paper)

    2. Performance impact: +10-20ms latency
       - Extra decryption step (on-device)
       - Older devices slower (iPhone 8 vs iPhone 14)

    3. Compatibility: Some features disabled
       - iCloud Web: No access (browser can't decrypt)
       - Windows iCloud: Limited support
       - Third-party apps: Can't access data

User Education Challenge:
    Survey (2023): 80% of users don't understand E2E encryption
    - Think: "My data is already encrypted" (true, but Apple has keys)
    - Don't realize: Standard iCloud = Apple can decrypt (for recovery, legal)
    - Fear: "Will I lose my photos?" (if passcode forgotten, yes!)

Apple's Response:
    - Better education: In-app warnings, explanations
    - Recovery options: Trusted device, recovery key (print and store)
    - Gradual rollout: Opt-in (not default) to avoid data loss incidents

Comparison: WhatsApp (owned by Meta):
    - E2E encryption: Default (all users, 2B+)
    - Adoption: 100% (no choice, always enabled)
    - Trade-off: Cannot recover old messages if device lost
    - User acceptance: High (privacy expected for messaging)

Difference: Storage vs messaging
    - Messages: Expected to be private (ephemeral)
    - Photos: Expected to be recoverable (precious memories)
    - Result: Users prefer recoverability over privacy for photos

Annual Encryption Infrastructure Cost:

Hardware:
    HSMs: $30M initial ÷ 10 years = $3M/year amortized
    Secure Enclave: Included in device cost (no separate charge)
    Servers: $100M (encryption-capable hardware) ÷ 5 years = $20M/year

Software:
    Development: 500 engineers × $300K = $150M/year
    Maintenance: 100 engineers × $300K = $30M/year

Operations:
    Data center: $50M/year (power, cooling, staff)
    Key rotation: $5M/year (automated, but monitoring needed)

Total: $3M + $20M + $150M + $30M + $50M + $5M = $258M/year

ROI Analysis:

Direct Benefits:
    1. Customer trust: Premium pricing (+$100 per device vs Android)
       - 200M iPhones sold/year × $100 premium = $20B/year
    
    2. Privacy reputation: Competitive advantage
       - "What happens on your iPhone, stays on your iPhone"
       - Brand value: Immeasurable (attracts privacy-conscious users)

    3. Regulatory compliance: GDPR, CCPA, HIPAA
       - Fines avoided: €20M+ per breach (GDPR maximum)
       - Apple: Zero major fines (2016-2024)

Avoided Costs (Data Breaches):
    Average breach cost: $4.45M (IBM study 2023)
    Apple scale: 1.5B users × $4.45M = Catastrophic if breached
    
    Examples of breaches (competitors):
        - T-Mobile (2021): 50M users, $350M settlement
        - Equifax (2017): 147M users, $700M settlement
        - Yahoo (2013): 3B users, $117M settlement
    
    Apple: Zero major data breaches (2010-2024)
           E2E encryption prevents breach (no data to steal)

Intangible Benefits:
    - Government pressure: Can truthfully say "We cannot decrypt"
    - Law enforcement: Avoid legal battles (FBI iPhone case 2016)
    - Public relations: Privacy leader (marketing advantage)

Total Value: $258M cost vs $20B+ brand premium = 77× ROI

Enterprise Monitoring Stack (Synthesized from all examples):

1. VPC Flow Logs (Network Traffic)
   - Enable on all VPCs (Netflix: 10 TB/day across 3 regions)
   - Send to S3 (compressed, partitioned by date)
   - Analyze with Athena (SQL queries on logs)
   - Alert on anomalies (GuardDuty integration)
   
   Cost: $0.50 per GB ingested (Netflix: $125K/month for 250 GB/day)

2. CloudTrail (API Activity)
   - Enable on all accounts (Goldman Sachs: organization-wide)
   - Capture all API calls (who, what, when, from where)
   - Detect unauthorized access (IAM policy violations)
   - Compliance: Required for SOX, PCI DSS, HIPAA
   
   Cost: $2.00 per 100,000 events (Goldman: ~$50K/month for 10B events/day)

3. GuardDuty (Threat Detection)
   - Machine learning: Detect anomalies (unusual API calls, traffic patterns)
   - Threat intelligence: Known malicious IPs, domains
   - Findings: Prioritized by severity (high, medium, low)
   - Integration: Lambda auto-response (block IP, revoke credentials)
   
   Cost: $4.80 per million events analyzed (Capital One: ~$100K/month)

4. CloudWatch (Performance Metrics)
   - Metrics: CPU, memory, network, disk (EC2, RDS, Lambda)
   - Logs: Application logs, system logs (centralized)
   - Alarms: Alert on thresholds (CPU >80%, latency >500ms)
   - Dashboards: Real-time visibility (NOC displays)
   
   Cost: $0.30 per GB ingested (varies widely, $10K-$100K/month typical)

Complete Flow Log Format (Version 2):
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

Example Record:
2 123456789012 eni-1a2b3c4d 10.0.1.5 172.217.14.206 49152 443 6 20 4000 1620000000 1620000060 ACCEPT OK

Decoded:
- version: 2 (flow log format version)
- account-id: 123456789012 (AWS account)
- interface-id: eni-1a2b3c4d (network interface)
- srcaddr: 10.0.1.5 (source IP - private EC2)
- dstaddr: 172.217.14.206 (destination IP - google.com)
- srcport: 49152 (ephemeral port)
- dstport: 443 (HTTPS)
- protocol: 6 (TCP, see below for codes)
- packets: 20 (number of packets)
- bytes: 4000 (total bytes transferred)
- start: 1620000000 (Unix timestamp start)
- end: 1620000060 (Unix timestamp end, 60 second window)
- action: ACCEPT (allowed by security group)
- log-status: OK (logged successfully)

Protocol Numbers:
1 = ICMP (ping)
6 = TCP (HTTP, HTTPS, SSH)
17 = UDP (DNS, video streaming)
58 = ICMPv6

Action Values:
ACCEPT = Traffic allowed (security group/NACL permit)
REJECT = Traffic blocked (security group/NACL deny)

-- Athena query on S3 flow logs
CREATE EXTERNAL TABLE IF NOT EXISTS vpc_flow_logs (
  version int,
  account string,
  interfaceid string,
  sourceaddress string,
  destinationaddress string,
  sourceport int,
  destinationport int,
  protocol int,
  numpackets int,
  numbytes bigint,
  starttime int,
  endtime int,
  action string,
  logstatus string
)
PARTITIONED BY (dt string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ' '
LOCATION 's3://my-vpc-flow-logs/AWSLogs/'
TBLPROPERTIES ("skip.header.line.count"="1");

-- Query: Top 10 source-destination pairs by bytes
SELECT 
    sourceaddress as source,
    destinationaddress as destination,
    SUM(numbytes) as total_bytes,
    COUNT(*) as num_connections,
    SUM(numbytes) / 1024 / 1024 / 1024 as total_gb
FROM vpc_flow_logs
WHERE dt = '2024-09-23'
  AND action = 'ACCEPT'
GROUP BY sourceaddress, destinationaddress
ORDER BY total_bytes DESC
LIMIT 10;

Results (Sample):
+----------------+-------------------+-------------+-----------------+-----------+
| source         | destination       | total_bytes | num_connections | total_gb  |
+----------------+-------------------+-------------+-----------------+-----------+
| 10.0.10.50     | 10.0.20.100      | 2.5TB       | 1,250,000       | 2,500 GB  |
| 10.0.10.51     | 54.239.28.85     | 1.8TB       | 890,000         | 1,800 GB  |
| 10.0.15.20     | 10.0.25.30       | 950GB       | 450,000         | 950 GB    |
+----------------+-------------------+-------------+-----------------+-----------+

Analysis:
- 10.0.10.50 → 10.0.20.100: 2.5 TB/day internal traffic
  - Investigation: Microservice A calling Microservice B
  - Issue: Inefficient API calls (fetching too much data per call)
  - Fix: Implement pagination, reduce payload size
  - Savings: 2.5 TB × $0.01/GB inter-AZ = $25/day = $750/month

Cost: Athena charges $5 per TB scanned
      Query above: 50 GB scanned = $0.25

-- Find rejected connections (security group blocks)
SELECT 
    sourceaddress,
    destinationaddress,
    destinationport,
    protocol,
    COUNT(*) as attempts,
    MIN(from_unixtime(starttime)) as first_attempt,
    MAX(from_unixtime(starttime)) as last_attempt
FROM vpc_flow_logs
WHERE dt >= '2024-09-20'
  AND action = 'REJECT'
GROUP BY sourceaddress, destinationaddress, destinationport, protocol
HAVING COUNT(*) > 100  -- More than 100 rejections
ORDER BY attempts DESC
LIMIT 20;

Results (Sample - SSH brute force attempt):
+-------------------+-------------------+------+----------+----------+---------------------+---------------------+
| sourceaddress     | destinationaddress| port | protocol | attempts | first_attempt       | last_attempt        |
+-------------------+-------------------+------+----------+----------+---------------------+---------------------+
| 198.51.100.42     | 10.0.1.50         | 22   | 6 (TCP)  | 15,820   | 2024-09-20 08:15:00 | 2024-09-23 14:30:00 |
+-------------------+-------------------+------+----------+----------+---------------------+---------------------+

Analysis:
- External IP 198.51.100.42 attempting SSH brute force
- 15,820 attempts over 3 days
- All REJECTED by security group (port 22 not open to 0.0.0.0/0)

Action:
1. Verify security group correct (SSH only from bastion)
2. Add NACL rule to explicitly block attacker IP
3. Alert security team (potential botnet node)
4. Consider AWS WAF if web-facing application

# Add NACL deny rule
aws ec2 create-network-acl-entry \
    --network-acl-id acl-12345 \
    --ingress \
    --rule-number 50 \
    --protocol tcp \
    --port-range From=22,To=22 \
    --cidr-block 198.51.100.42/32 \
    --rule-action deny

-- Identify expensive cross-region data transfer
-- (Assumption: Internal IPs 10.x.x.x are same region, others are cross-region or internet)
SELECT 
    sourceaddress,
    destinationaddress,
    SUM(numbytes) / 1024 / 1024 / 1024 as total_gb,
    SUM(numbytes) * 0.02 as estimated_cost_usd  -- $0.02/GB cross-region
FROM vpc_flow_logs
WHERE dt >= '2024-09-01'
  AND action = 'ACCEPT'
  AND (
    (sourceaddress LIKE '10.%' AND destinationaddress NOT LIKE '10.%')
    OR (sourceaddress NOT LIKE '10.%' AND destinationaddress LIKE '10.%')
  )
GROUP BY sourceaddress, destinationaddress
HAVING SUM(numbytes) / 1024 / 1024 / 1024 > 1000  -- More than 1 TB
ORDER BY total_gb DESC
LIMIT 10;

Results:
Source: 10.0.50.100 (us-east-1)
Destination: 52.94.76.0 (S3 endpoint, different region)
Traffic: 5,000 GB/month
Cost: 5,000 GB × $0.02 = $100/month

Fix: Use S3 VPC Gateway Endpoint in same region
     - Replicate data to us-east-1 S3 bucket
     - Update application to read from local bucket
     - Savings: $100/month = $1,200/year

VPC Flow Logs Pricing (AWS):
$0.50 per GB ingested to CloudWatch Logs
$0.03 per GB per month storage in CloudWatch
$0.01 per GB to S3 (via Kinesis Data Firehose)

Scenario: 100 GB/day flow logs

Option 1: CloudWatch Logs (Default)
    Ingestion: 100 GB/day × 30 days × $0.50/GB = $1,500/month
    Storage: 3 TB (30 days) × $0.03/GB = $90/month
    Total: $1,590/month = $19,080/year

Option 2: S3 (via Kinesis Firehose)
    Ingestion: 100 GB/day × 30 days × $0.01/GB = $30/month
    Storage: 3 TB × $0.023/GB (S3 Standard) = $69/month
    Athena queries: $5/TB × 0.1 TB/day × 30 = $15/month
    Total: $114/month = $1,368/year
    
    Savings: $19,080 - $1,368 = $17,712/year (93% reduction!)

Option 3: S3 + Sampling (50% of flows)
    Ingestion: 50 GB/day × 30 days × $0.01/GB = $15/month
    Storage: 1.5 TB × $0.023/GB = $34.50/month
    Athena: $5/TB × 0.05 TB/day × 30 = $7.50/month
    Total: $57/month = $684/year
    
    Savings: $19,080 - $684 = $18,396/year (96% reduction!)
    Trade-off: Miss 50% of flows (acceptable for cost optimization)

Recommendation: S3 with sampling for high-traffic VPCs

1. Management Events (Control Plane):
   - Creating/deleting resources (EC2, S3, RDS)
   - Modifying security settings (IAM, security groups)
   - Configuring rules (CloudWatch, Config)
   
   Example:
   {
     "eventName": "RunInstances",
     "eventSource": "ec2.amazonaws.com",
     "userIdentity": {
       "type": "IAMUser",
       "userName": "alice@company.com",
       "accountId": "123456789012"
     },
     "sourceIPAddress": "203.0.113.42",
     "requestParameters": {
       "instanceType": "t3.xlarge",
       "imageId": "ami-0c55b159cbfafe1f0",
       "keyName": "production-key"
     },
     "responseElements": {
       "instancesSet": [
         {"instanceId": "i-1234567890abcdef0"}
       ]
     },
     "eventTime": "2024-09-23T14:30:00Z"
   }

2. Data Events (Data Plane):
   - S3 object-level operations (GetObject, PutObject, DeleteObject)
   - Lambda function invocations
   - DynamoDB table operations (PutItem, GetItem)
   
   Note: Data events generate MUCH more volume
   Cost: $0.10 per 100,000 events (after free tier)
   
   Example (S3):
   {
     "eventName": "GetObject",
     "eventSource": "s3.amazonaws.com",
     "requestParameters": {
       "bucketName": "my-sensitive-data",
       "key": "financial/2024-q3.csv"
     },
     "userIdentity": {
       "userName": "bob@company.com"
     },
     "sourceIPAddress": "203.0.113.99",
     "eventTime": "2024-09-23T14:35:00Z"
   }

3. Insights Events (Anomaly Detection):
   - Unusual API activity detected by ML
   - Example: 1,000 DeleteBucket calls (vs 2 typically)
   - Cost: $0.35 per 100,000 write management events analyzed
   
   Example:
   {
     "eventName": "DeleteBucket",
     "insightType": "ApiCallRateInsight",
     "insightContext": {
       "statistics": {
         "baseline": {"average": 2},
         "insight": {"average": 1000}
       },
       "attributions": [
         {"attribute": "userIdentity", "value": "carol@company.com"}
       ]
     }
   }

# Someone deleted production RDS database
# Find WHO deleted it

aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteDBInstance \
    --start-time 2024-09-20T00:00:00Z \
    --end-time 2024-09-23T23:59:59Z

Result:
{
  "Events": [{
    "EventId": "abc123-def456-ghi789",
    "EventName": "DeleteDBInstance",
    "EventTime": "2024-09-22T03:15:42Z",
    "Username": "temp-contractor-bob",
    "Resources": [{
      "ResourceName": "production-db-primary",
      "ResourceType": "AWS::RDS::DBInstance"
    }],
    "CloudTrailEvent": {
      "userIdentity": {
        "type": "IAMUser",
        "userName": "temp-contractor-bob",
        "accountId": "123456789012",
        "principalId": "AIDAI123456EXAMPLE"
      },
      "sourceIPAddress": "198.51.100.88",
      "requestParameters": {
        "dBInstanceIdentifier": "production-db-primary",
        "skipFinalSnapshot": true
      }
    }
  }]
}

Investigation Results:
- User: temp-contractor-bob (contractor account, should not have production access)
- Time: 3:15 AM (unusual, outside business hours)
- Action: Deleted WITH skipFinalSnapshot=true (no backup!)
- IP: 198.51.100.88 (trace to contractor's home office)

Lessons Learned:
1. Contractor had too many permissions (violated least privilege)
2. No approval workflow for critical operations
3. No protection on production resources

Remediation:
1. Revoke contractor IAM credentials immediately
2. Restore database from automated backup (within recovery window)
3. Implement SCPs: Deny DeleteDBInstance for production tag
4. Add MFA requirement for destructive operations
5. Quarterly access reviews (remove unused accounts)

Service Control Policy (SCP) to prevent future incidents:
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": [
      "rds:DeleteDBInstance",
      "rds:DeleteDBCluster"
    ],
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "aws:ResourceTag/Environment": "production"
      },
      "StringNotEquals": {
        "aws:username": "admin-team-lead"
      }
    }
  }]
}

PCI DSS Requirement 10.2: Log and Monitor All Access to Cardholder Data

CloudTrail Configuration for Compliance:
1. Organization trail (all accounts, all regions)
2. Log file validation enabled (detect tampering)
3. Multi-region trail (capture all regions)
4. Management events + S3 data events (cardholder data bucket)
5. Encrypted logs (SSE-KMS)
6. Log retention: 7 years (PCI DSS requirement)
7. Real-time alerting: GuardDuty integration

Example Query: Find all access to cardholder data bucket
SELECT 
  useridentity.username,
  eventname,
  sourceipaddress,
  eventtime,
  requestparameters
FROM cloudtrail_logs
WHERE eventsource = 's3.amazonaws.com'
  AND json_extract_scalar(requestparameters, '$.bucketName') = 'cardholder-data-prod'
  AND eventname IN ('GetObject', 'PutObject', 'DeleteObject')
  AND eventtime >= '2024-09-01'
ORDER BY eventtime DESC;

Results (Sample):
- alice@company.com accessed 1,250 cardholder records (normal, customer service)
- bob@company.com accessed 50,000 records in 5 minutes (ALERT: Unusual volume!)
  
Action on Bob's Activity:
1. Automatic alert triggered (GuardDuty finding)
2. Lambda function executes:
   - Disable Bob's IAM access keys
   - Revoke active sessions
   - Send SNS notification to security team
3. Security team investigates:
   - Bob's account compromised (password leaked)
   - Attacker attempted data exfiltration
   - Stopped within 5 minutes (automated response)

Cost Breakdown (1,000-employee company):
- CloudTrail trail: $2 per 100K events
- Events/month: 50 million (50 events/employee/day × 1,000 × 30)
- Cost: 50M / 100K × $2 = $1,000/month = $12K/year
- Value: Prevented data breach (estimated $3M+ in fines/damage)
- ROI: 250× (25,000% return)

CloudTrail Pricing:
- Management events: $2.00 per 100,000 events (first trail free)
- Data events: $0.10 per 100,000 events
- Insights events: $0.35 per 100,000 write events analyzed

Scenario: Large enterprise (10,000 employees)

Without Optimization:
- Management events: 500M/month × $2/100K = $10,000/month
- S3 data events: 10B/month × $0.10/100K = $10,000/month
- Total: $20,000/month = $240K/year

With Optimization:
1. Enable data events only for sensitive buckets (not all S3)
   - Sensitive: 1B events/month × $0.10/100K = $1,000/month
   - Savings: $9,000/month

2. Use event selectors (filter by resource type)
   - Exclude read-only operations (GetObject)
   - Focus on write operations (PutObject, DeleteObject)
   - Events: 100M/month × $0.10/100K = $100/month
   - Savings: $9,900/month

3. S3 Lifecycle policy for log storage
   - Standard (30 days): $0.023/GB
   - Glacier (90 days): $0.004/GB (83% cheaper)
   - Deep Archive (7 years): $0.00099/GB (96% cheaper)
   - Savings: $5,000/month on storage

Total Optimized Cost: $10,000 + $100 + $1,000 = $11,100/month
Savings: $240K - $133K = $107K/year (45% reduction)

Finding: UnauthorizedAccess:EC2/SSHBruteForce
Severity: Medium
Description: EC2 instance i-1234567890abcdef0 is performing SSH brute force attacks

{
  "schemaVersion": "2.0",
  "accountId": "123456789012",
  "region": "us-east-1",
  "partition": "aws",
  "id": "abc123def456",
  "arn": "arn:aws:guardduty:us-east-1:123456789012:detector/xyz/finding/abc123",
  "type": "UnauthorizedAccess:EC2/SSHBruteForce",
  "resource": {
    "instanceDetails": {
      "instanceId": "i-1234567890abcdef0",
      "imageId": "ami-0c55b159cbfafe1f0",
      "tags": [
        {"key": "Name", "value": "web-server-01"},
        {"key": "Environment", "value": "production"}
      ]
    }
  },
  "service": {
    "action": {
      "networkConnectionAction": {
        "connectionDirection": "OUTBOUND",
        "remoteIpDetails": {
          "ipAddressV4": "198.51.100.42",
          "organization": {
            "asn": "12345",
            "asnOrg": "MaliciousHostingProvider"
          },
          "country": {"countryName": "Russia"}
        },
        "remotePortDetails": {"port": 22}
      }
    },
    "count": 1523,  // 1,523 SSH connection attempts!
    "eventFirstSeen": "2024-09-22T08:00:00.000Z",
    "eventLastSeen": "2024-09-23T14:30:00.000Z"
  },
  "severity": 5,  // Medium (scale 0-10)
  "title": "SSH brute force attack targeting multiple hosts",
  "description": "EC2 instance i-1234... is performing SSH brute force attacks on remote hosts"
}

Analysis:
- Your EC2 instance is ATTACKING other servers (compromised!)
- 1,523 SSH attempts to external IP in Russia
- Instance likely infected with malware/botnet

Automated Remediation (Lambda function):
1. Isolate instance:
   - Attach security group with no ingress/egress
   - Prevent lateral movement
2. Create snapshot (forensics)
3. Tag instance "QUARANTINED"
4. Send SNS alert to security team
5. Create Jira ticket for investigation

Manual Investigation:
- SSH into instance (from bastion)
- Check /var/log/auth.log for unauthorized access
- Run rootkit scanner (rkhunter, chkrootkit)
- Identify malware (find suspicious processes, cron jobs)
- Determine entry point (SSH key compromised? Vulnerable application?)

Prevention:
- Disable SSH password authentication (key-only)
- Enable AWS Systems Manager Session Manager (no SSH needed)
- Apply security patches automatically (AWS Systems Manager Patch Manager)
- Use Amazon Inspector to scan for vulnerabilities

Finding: CryptoCurrency:EC2/BitcoinTool.B!DNS
Severity: High
Description: EC2 instance is querying a domain associated with Bitcoin mining

{
  "type": "CryptoCurrency:EC2/BitcoinTool.B!DNS",
  "resource": {
    "instanceDetails": {
      "instanceId": "i-9876543210fedcba0",
      "instanceType": "c5.4xlarge",  // Compute-optimized (expensive!)
      "launchTime": "2024-09-20T10:00:00.000Z",
      "tags": [
        {"key": "Name", "value": "api-server-05"}
      ]
    }
  },
  "service": {
    "action": {
      "dnsRequestAction": {
        "domain": "mining-pool.suspicious-domain.com",
        "protocol": "UDP"
      }
    },
    "additionalInfo": {
      "threatName": "BitcoinMiner",
      "threatListName": "ProofPoint"
    }
  },
  "severity": 8,  // High
  "title": "Bitcoin mining activity detected",
  "description": "EC2 instance querying mining pool domain. Possible cryptocurrency mining."
}

Analysis:
- api-server-05 running unauthorized Bitcoin mining
- Instance type: c5.4xlarge (16 vCPUs, $0.68/hour)
- Running time: 3 days = $49 wasted
- Impact: Steals compute resources, slows legitimate workloads

Cost Impact:
- Monthly: 30 days × 24 hours × $0.68 = $489.60
- Annual: $489.60 × 12 = $5,875.20 per compromised instance
- If 10 instances compromised: $58,752/year wasted!

Automated Response:
1. Stop instance immediately (prevent further cost)
2. Create AMI snapshot (forensics)
3. Terminate instance after 24 hours (if not needed)
4. Alert finance team (unexpected EC2 costs)
5. Review IAM permissions (how was miner installed?)

Root Cause Analysis:
- Application vulnerability (unpatched WordPress plugin)
- Attacker gained web shell access
- Downloaded and compiled mining software
- Modified crontab to restart miner on boot

Prevention:
- Web application firewall (WAF) to block exploit attempts
- Immutable infrastructure (containers, not long-lived VMs)
- AWS Compute Optimizer to detect unusual CPU patterns
- Budget alerts ($500/month per instance, alert if exceeded)

Finding: Exfiltration:S3/MaliciousIPCaller.Custom
Severity: High
Description: S3 API was invoked from a malicious IP address

{
  "type": "Exfiltration:S3/MaliciousIPCaller.Custom",
  "resource": {
    "s3BucketDetails": [{
      "name": "company-sensitive-data",
      "type": "Destination",
      "tags": [
        {"key": "Classification", "value": "Confidential"}
      ],
      "defaultServerSideEncryption": {
        "encryptionType": "aws:kms",
        "kmsMasterKeyArn": "arn:aws:kms:us-east-1:123456789012:key/abc-123"
      }
    }]
  },
  "service": {
    "action": {
      "awsApiCallAction": {
        "api": "GetObject",
        "callerType": "Remote IP",
        "remoteIpDetails": {
          "ipAddressV4": "198.51.100.99",
          "organization": {
            "asn": "54321",
            "asnOrg": "TorExitNode"
          },
          "country": {"countryName": "Unknown"}
        }
      }
    },
    "count": 50000,  // 50,000 objects downloaded!
    "resourceRole": "TARGET",
    "additionalInfo": {
      "bytesOut": "10737418240",  // 10 GB
      "apiCalls": [
        {"api": "GetObject", "count": 50000}
      ]
    }
  },
  "severity": 8,
  "title": "S3 data accessed from malicious IP",
  "description": "S3 bucket company-sensitive-data was accessed from known malicious IP"
}

Analysis:
- 50,000 S3 objects downloaded (10 GB)
- Source: Tor exit node (anonymous attacker)
- IAM user credentials compromised
- Data breach in progress!

Immediate Response (Within Minutes):
1. Disable compromised IAM user access keys:
   aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive --user-name compromised-user

2. Revoke all active sessions:
   aws iam delete-user-policy --user-name compromised-user --policy-name inline-policy
   
3. Enable S3 Object Lock (prevent further exfiltration):
   aws s3api put-object-lock-configuration \
     --bucket company-sensitive-data \
     --object-lock-configuration ObjectLockEnabled=Enabled

4. Alert security team + legal team (potential breach notification required)

Post-Incident Investigation:
- How were credentials compromised?
  - Developer pushed to public GitHub repo (credential scanning found keys)
  - Attacker found within 2 hours (automated scanning)
  
- What data was accessed?
  - CloudTrail logs show 50,000 GetObject calls
  - S3 access logs show which files (PII, financial records)
  - Assess breach severity (GDPR, CCPA notification requirements)

Cost of Breach:
- Notification costs: $5 per affected customer × 100,000 = $500K
- Legal fees: $250K
- Regulatory fines: GDPR up to 4% of revenue (millions for large company)
- Reputation damage: Immeasurable
- Total: $1M+ easily

Cost of Prevention:
- GuardDuty: $4.80 per million events = $500/month = $6K/year
- ROI: Prevented $1M breach for $6K/year = 167× return

Prevention:
- AWS Secrets Manager (rotate credentials automatically)
- S3 Block Public Access (prevent accidental public buckets)
- git-secrets pre-commit hook (scan for credentials before push)
- Least privilege IAM (user only needs access to their own data)
- MFA for sensitive operations (GetObject on Confidential tag requires MFA)

# Lambda function: auto-response-guardduty-findings.py
import boto3
import json
import os

ec2 = boto3.client('ec2')
iam = boto3.client('iam')
sns = boto3.client('sns')

def lambda_handler(event, context):
    """
    Automatically respond to GuardDuty findings
    Triggered by EventBridge rule
    """
    
    # Parse GuardDuty finding
    detail = event['detail']
    finding_type = detail['type']
    severity = detail['severity']
    resource = detail['resource']
    
    # Only respond to high/critical severity
    if severity < 7:
        print(f"Severity {severity} below threshold (7), skipping")
        return
    
    print(f"Processing finding: {finding_type}, severity: {severity}")
    
    # Route to appropriate response function
    if 'EC2' in finding_type:
        response = handle_ec2_finding(detail, resource)
    elif 'IAM' in finding_type:
        response = handle_iam_finding(detail, resource)
    elif 'S3' in finding_type:
        response = handle_s3_finding(detail, resource)
    else:
        response = {"action": "none", "reason": "Unhandled finding type"}
    
    # Send notification
    send_alert(detail, response)
    
    return {
        'statusCode': 200,
        'body': json.dumps(response)
    }

def handle_ec2_finding(detail, resource):
    """Isolate compromised EC2 instance"""
    instance_id = resource['instanceDetails']['instanceId']
    
    print(f"Isolating instance: {instance_id}")
    
    # Create quarantine security group (if doesn't exist)
    try:
        quarantine_sg = ec2.describe_security_groups(
            Filters=[{'Name': 'group-name', 'Values': ['quarantine-sg']}]
        )['SecurityGroups'][0]['GroupId']
    except IndexError:
        # Create security group with no ingress/egress
        quarantine_sg = ec2.create_security_group(
            GroupName='quarantine-sg',
            Description='Quarantine security group - NO TRAFFIC ALLOWED',
            VpcId=os.environ['VPC_ID']
        )['GroupId']
        
        # Remove all egress rules (default allows all)
        ec2.revoke_security_group_egress(
            GroupId=quarantine_sg,
            IpPermissions=[{
                'IpProtocol': '-1',
                'IpRanges': [{'CidrIp': '0.0.0.0/0'}]
            }]
        )
    
    # Apply quarantine security group to instance
    ec2.modify_instance_attribute(
        InstanceId=instance_id,
        Groups=[quarantine_sg]
    )
    
    # Create snapshot for forensics
    volumes = ec2.describe_volumes(
        Filters=[{'Name': 'attachment.instance-id', 'Values': [instance_id]}]
    )['Volumes']
    
    snapshot_ids = []
    for volume in volumes:
        snapshot = ec2.create_snapshot(
            VolumeId=volume['VolumeId'],
            Description=f'Forensic snapshot - GuardDuty finding - {detail["id"]}',
            TagSpecifications=[{
                'ResourceType': 'snapshot',
                'Tags': [
                    {'Key': 'Purpose', 'Value': 'Forensics'},
                    {'Key': 'FindingId', 'Value': detail['id']},
                    {'Key': 'InstanceId', 'Value': instance_id}
                ]
            }]
        )
        snapshot_ids.append(snapshot['SnapshotId'])
    
    # Tag instance as quarantined
    ec2.create_tags(
        Resources=[instance_id],
        Tags=[
            {'Key': 'Status', 'Value': 'QUARANTINED'},
            {'Key': 'QuarantineReason', 'Value': detail['type']},
            {'Key': 'QuarantineDate', 'Value': detail['service']['eventFirstSeen']}
        ]
    )
    
    return {
        "action": "quarantine",
        "instance_id": instance_id,
        "security_group": quarantine_sg,
        "snapshots": snapshot_ids,
        "message": f"Instance {instance_id} quarantined, {len(snapshot_ids)} snapshots created"
    }

def handle_iam_finding(detail, resource):
    """Disable compromised IAM credentials"""
    principal_id = detail['resource']['accessKeyDetails']['principalId']
    user_name = detail['resource']['accessKeyDetails']['userName']
    access_key_id = detail['resource']['accessKeyDetails']['accessKeyId']
    
    print(f"Disabling access key: {access_key_id} for user: {user_name}")
    
    # Disable access key
    iam.update_access_key(
        UserName=user_name,
        AccessKeyId=access_key_id,
        Status='Inactive'
    )
    
    # Attach explicit deny policy (belt and suspenders)
    deny_policy = {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*"
        }]
    }
    
    iam.put_user_policy(
        UserName=user_name,
        PolicyName='EMERGENCY-DENY-ALL',
        PolicyDocument=json.dumps(deny_policy)
    )
    
    return {
        "action": "disable_credentials",
        "user_name": user_name,
        "access_key_id": access_key_id,
        "message": f"Disabled access key {access_key_id} and applied deny policy"
    }

def handle_s3_finding(detail, resource):
    """Restrict S3 bucket access"""
    bucket_name = resource['s3BucketDetails'][0]['name']
    
    print(f"Restricting access to bucket: {bucket_name}")
    
    # Enable MFA Delete (prevent deletion without MFA)
    # Note: Requires bucket versioning + root account to enable
    # This is a placeholder - actual implementation needs root account
    
    # Add bucket policy requiring VPC endpoint
    s3 = boto3.client('s3')
    
    bucket_policy = {
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "DenyNonVPCAccess",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                f"arn:aws:s3:::{bucket_name}",
                f"arn:aws:s3:::{bucket_name}/*"
            ],
            "Condition": {
                "StringNotEquals": {
                    "aws:sourceVpce": os.environ['VPC_ENDPOINT_ID']
                }
            }
        }]
    }
    
    s3.put_bucket_policy(
        Bucket=bucket_name,
        Policy=json.dumps(bucket_policy)
    )
    
    return {
        "action": "restrict_bucket",
        "bucket_name": bucket_name,
        "message": f"Restricted {bucket_name} to VPC endpoint access only"
    }

def send_alert(detail, response):
    """Send SNS notification to security team"""
    message = f"""
GuardDuty Finding - Automated Response

Finding Type: {detail['type']}
Severity: {detail['severity']}
Title: {detail['title']}
Description: {detail['description']}

Automated Action Taken:
{json.dumps(response, indent=2)}

View in Console:
https://console.aws.amazon.com/guardduty/home?region={detail['region']}#/findings?search=id%3D{detail['id']}

Investigation Steps:
1. Review CloudTrail logs for {detail['service']['eventFirstSeen']}
2. Check VPC Flow Logs for unusual traffic patterns
3. Verify no lateral movement to other resources
4. Update runbook with lessons learned

Time: {detail['service']['eventFirstSeen']}
    """
    
    sns.publish(
        TopicArn=os.environ['SNS_TOPIC_ARN'],
        Subject=f" GuardDuty Alert: {detail['type']} (Severity {detail['severity']})",
        Message=message
    )

# EventBridge Rule (Terraform)
"""
resource "aws_cloudwatch_event_rule" "guardduty_findings" {
  name        = "guardduty-high-severity-findings"
  description = "Capture GuardDuty findings with severity >= 7"

  event_pattern = jsonencode({
    source      = ["aws.guardduty"]
    detail-type = ["GuardDuty Finding"]
    detail = {
      severity = [7, 8, 9, 10]  # High and Critical only
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda" {
  rule      = aws_cloudwatch_event_rule.guardduty_findings.name
  target_id = "GuardDutyResponseLambda"
  arn       = aws_lambda_function.guardduty_response.arn
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.guardduty_response.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.guardduty_findings.arn
}
"""

GuardDuty Pricing (Per Account Per Region):

1. CloudTrail Events Analysis:
   $4.80 per million events after first 500K (free tier)
   
2. VPC Flow Logs Analysis:
   $1.13 per GB analyzed after first 500 GB (free tier)
   
3. DNS Logs Analysis:
   $0.40 per million requests after first 1 billion (free tier)

Example: Mid-size company (1,000 EC2 instances)

CloudTrail:
   Events: 10 million/month
   Free tier: 500K
   Billable: 9.5 million
   Cost: 9.5 × $4.80 = $45.60/month

VPC Flow Logs:
   Data: 1 TB/month
   Free tier: 500 GB
   Billable: 500 GB
   Cost: 500 × $1.13 = $565/month

DNS Logs:
   Requests: 5 billion/month
   Free tier: 1 billion
   Billable: 4 billion
   Cost: 4 × $0.40 = $1.60/month

Total: $45.60 + $565 + $1.60 = $612.20/month = $7,346/year

Value Delivered:
- Detected 15 security incidents (2024)
- Prevented: Cryptocurrency mining ($58K/year)
- Prevented: Data breach ($1M+ estimated)
- Prevented: Botnet C2 communication (reputation damage)

ROI: $1M+ prevented / $7.3K cost = 137× return

Why B is Correct (Transit Gateway):

Architecture:
Region us-east-1:
   Transit Gateway (TGW-USE1)
   ├─ VPC-01 (10.0.0.0/16)
   ├─ VPC-02 (10.1.0.0/16)
   ├─ ... (15 VPCs total)
   └─ Direct Connect Gateway → On-premises

Region eu-west-1:
   Transit Gateway (TGW-EUW1)
   ├─ VPC-20 (10.20.0.0/16)
   ├─ ... (15 VPCs total)

Region ap-southeast-1:
   Transit Gateway (TGW-APS1)
   ├─ VPC-40 (10.40.0.0/16)
   ├─ ... (20 VPCs total)

Inter-Region:
   TGW-USE1 ←→ TGW-EUW1 ←→ TGW-APS1 (peering)

Benefits:
Scalability:
   - Add new VPC: Single attachment to Transit Gateway (5 minutes)
   - No need to update 49 other VPCs (vs mesh peering)
   - Support up to 5,000 attachments per TGW

Manageability:
   - Centralized routing (TGW route tables)
   - Single place to apply firewall rules (TGW Network Firewall)
   - Centralized Flow Logs (monitor all inter-VPC traffic)

Hybrid Connectivity:
   - Direct Connect Gateway attaches to TGW (one connection, all VPCs)
   - On-prem can reach all 50 VPCs via single attachment
   - No need for 50 VPN/DX connections

Cost:
   Hourly: $0.05/hour per attachment × 50 = $2.50/hour = $1,825/month
   Data: $0.02/GB × 100 TB = $2,000/month
   Total: $3,825/month = $45,900/year

Why A is Wrong (Full Mesh Peering):
 Complexity:
   - 50 VPCs = 1,225 peering connections! (n × (n-1) / 2)
   - Add new VPC = create 49 new peering connections
   - Routing table explosion (49 routes per VPC)
   
 Management:
   - Update security groups in 1,225 places
   - No centralized monitoring
   - Difficult to troubleshoot
   
 Hybrid:
   - Need 50 Direct Connect connections (one per VPC)
   - Cost: 50 × $0.30/hour × 730 = $10,950/month (5× more expensive!)

 Scalability Limit:
   - AWS VPC peering limit: 125 per VPC
   - Can't scale beyond 125 VPCs with full mesh

Cost Comparison:
   Peering hourly: FREE (no hourly charge)
   Peering data: $0.01/GB × 100 TB = $1,000/month
   But: 50 Direct Connect attachments = $10,950/month
   Total: $11,950/month = $143,400/year
   
   Transit Gateway: $45,900/year (68% cheaper!)

Why C is Wrong (VPN):
 Performance:
   - VPN over internet: 20-100ms latency (vs <5ms TGW)
   - Throughput limited: 1.25 Gbps per VPN tunnel
   - Need 100 VPN connections (50 VPCs × 2 for redundancy)
   
 Cost:
   - $0.05/hour × 100 = $5/hour = $3,650/month
   - Data: $0.09/GB × 100 TB = $9,000/month
   - Total: $12,650/month = $151,800/year (3.3× more expensive!)
   
 Security:
   - Traverses public internet (less secure than private AWS backbone)
   - Potential for packet loss/latency spikes
   
When to use VPN: Small deployments (2-5 VPCs), temporary connections

Why D is Wrong (Hub VPC):
 Single Point of Failure:
   - Hub VPC down = all VPCs disconnected
   - No multi-AZ redundancy for hub
   
 Throughput Bottleneck:
   - All traffic routes through hub VPC instances
   - Need large NAT instances ($500+/month)
   - 100 TB/month = constant bottleneck
   
 Complexity:
   - Need to manage routing instances in hub VPC
   - Update routes when adding VPCs
   - Security group management complex
   
 Cost:
   - NAT instances: $500/month (high-throughput)
   - 49 VPC peering connections: $490/month (data)
   - Total: $990/month (but high operational burden)

Cost Comparison Summary:
Transit Gateway: $3,825/month (Best value + enterprise features)
VPC Peering + DX: $11,950/month (Too many connections)
VPN Mesh: $12,650/month (Slow + expensive)
Hub VPC: $990/month (Operational burden + SPOF)

Real-World Example - Capital One:
- Migrated 350+ VPCs to Transit Gateway (2019-2020)
- Before: Mesh peering nightmare (60,000+ peering connections!)
- After: 3 Transit Gateways (one per region)
- Result: 95% reduction in network complexity
- Saved: $2M/year in operational costs (fewer network engineers needed)

Decision Matrix:
Use Transit Gateway when:
More than 10 VPCs
Frequent VPC additions (growing architecture)
Hybrid connectivity required (Direct Connect/VPN)
Need centralized monitoring/control
Multi-region architecture

Use VPC Peering when:
2-5 VPCs (simple topology)
Static architecture (no growth expected)
Cost-sensitive (no hourly charges)
Don't need centralized control

Why B is Correct (NLB + Global Accelerator):

Architecture:
Player (Tokyo) → Global Accelerator Anycast IP → Tokyo Edge
   ↓ AWS Private Network (not internet)
Network Load Balancer (ap-northeast-1) → Game Servers (UDP)

Player (London) → Same Anycast IP → London Edge
   ↓ AWS Private Network
Network Load Balancer (eu-west-2) → Game Servers (UDP)

Components:

1. Network Load Balancer (Layer 4):
   UDP Support: Essential for real-time gaming
      - ALB doesn't support UDP (only HTTP/HTTPS)
      - Gaming uses UDP for: Low latency, no retransmission delays
   
   Ultra-Low Latency: <100µs overhead
      - ALB: 5-10ms overhead (Layer 7 processing)
      - For 50ms P95 target, every microsecond counts!
   
   Massive Throughput: 10M+ connections/sec
      - Your requirement: 100K connections/sec (well within capacity)
      - ALB: ~1M connections/sec (would need 5+ ALBs)
   
   Static IP: Elastic IP addresses
      - 2 static IPs per AZ (for redundancy)
      - Enterprise firewalls: Whitelist these IPs
      - ALB: Dynamic IPs (changes, breaks firewall rules)

2. AWS Global Accelerator:
   Anycast IP: 2 static IPs for entire global deployment
      - Player uses same IP regardless of location
      - DNS not needed (IP in game client config)
   
   Edge Network: 90+ edge locations
      - Routes traffic to nearest AWS region
      - Tokyo player → ap-northeast-1 (10ms)
      - London player → eu-west-2 (5ms)
   
   AWS Private Network: Traffic stays on AWS backbone
      - Internet path: 100-200ms (many hops, packet loss)
      - AWS network: 50ms (dedicated fiber, no congestion)
      - 50-70% latency reduction!
   
   DDoS Protection: AWS Shield Standard included
      - Absorbs network/transport layer attacks at edge
      - Protects game servers from volumetric attacks

Performance Metrics:
   Latency: 20-50ms P95 (meets <50ms requirement)
   Throughput: 100K conn/sec easily handled
   Packet Loss: <0.01% (vs 1-3% internet)
   Jitter: <5ms (smooth gameplay)

Cost (5M concurrent players, 3 regions):
   Global Accelerator: $0.025/hour = $18.25/month per accelerator
      - 3 regions × $18.25 = $54.75/month
   
   Global Accelerator DT-Premium: $0.015/GB (AWS network)
      - Assume 10 GB/player/month = 50 PB total
      - 50 PB × $0.015/GB = $750,000/month
   
   Network Load Balancer: $0.0225/hour × 6 (2 per region × 3)
      - $0.135/hour × 730 hours = $98.55/month
   
   NLCU: $0.006 per NLCU-hour
      - Connection-heavy: 100K conn/sec = 50 NLCU per NLB
      - 6 NLBs × 50 NLCU × $0.006 × 730 = $131,400/month
   
   Total: $54.75 + $750,000 + $98.55 + $131,400 = $881,553/month
   
   Note: Data transfer is the major cost (85% of total)

Why A is Wrong (ALB + Lambda@Edge):
 No UDP Support:
   - ALB only supports HTTP/HTTPS (Layer 7)
   - Gaming requires UDP (Layer 4)
   - Lambda@Edge can't handle UDP either
   
 Too Much Latency:
   - ALB: 5-10ms overhead
   - Lambda@Edge: 20-50ms cold start + 5-10ms execution
   - Total: 30-70ms JUST for routing (exceeds 50ms P95 budget!)
   
 Dynamic IPs:
   - ALB IPs change over time
   - Enterprise firewall rules break
   
When to use ALB + Lambda@Edge:
- HTTP/HTTPS applications only
- Need content-based routing (URL paths, headers)
- Example: Web application, REST API

Why C is Wrong (ALB + CloudFront):
 No UDP Support:
   - CloudFront: HTTP/HTTPS only
   - ALB: HTTP/HTTPS only
   - Can't handle UDP game traffic
   
 Wrong Use Case:
   - CloudFront designed for: Static content (images, videos, files)
   - Not for: Real-time bidirectional game state
   
When to use ALB + CloudFront:
- Serving game assets (textures, models, patches)
- Game website/portal
- NOT for real-time gameplay

Why D is Wrong (Classic Load Balancer):
 Legacy Technology:
   - Classic LB: AWS legacy (deprecated)
   - Missing features: Health check improvements, target groups
   - No UDP support either (only TCP/SSL)
   
 Performance:
   - Slower than NLB (not optimized for Layer 4)
   - No static IPs
   - Lower throughput
   
AWS Recommendation: Don't use Classic LB for new deployments
Migration path: Classic LB → NLB or ALB

Real-World Example - Epic Games (Fortnite):
Architecture:
   - Network Load Balancer for game servers (UDP)
   - 250 million players worldwide (2024)
   - Global Accelerator: 30-50% latency reduction
   - Result: <50ms P95 globally
   
Before Global Accelerator (2018):
   - Players routed via internet
   - EU players connecting to US servers: 150ms
   - Complaints about lag, unplayable
   
After Global Accelerator (2019):
   - Same EU→US connection: 80ms (47% improvement)
   - Player satisfaction: Increased 35%
   - Churn: Reduced 25% (less rage-quitting due to lag!)

Cost at Scale:
   Epic Games estimated spend: $10M+/month on Global Accelerator
   Revenue: $5.8B (2023)
   Networking: 0.2% of revenue (acceptable for global game)

Decision Matrix:
Use NLB + Global Accelerator when:
UDP/TCP Layer 4 protocol
Ultra-low latency required (<50ms)
Global audience (multi-region)
Static IPs needed (firewall whitelisting)
Massive connection rate (100K+/sec)
DDoS protection critical

Use ALB when:
HTTP/HTTPS only
Need Layer 7 features (path routing, headers)
Moderate latency acceptable (5-10ms overhead OK)

Use CloudFront when:
Static content delivery (CDN)
HTTPS (videos, images, files)
NOT real-time bidirectional traffic

Why C is Correct (Defense in Depth: SG + NACL + Flow Logs):

Architecture Layers:

Layer 1 - Network ACL (Subnet Level, Stateless):
Public Subnet NACL:
Rule# | Type    | Protocol | Port | Source/Dest      | Allow/Deny
100   | Inbound | TCP      | 443  | 0.0.0.0/0        | ALLOW
110   | Inbound | TCP      | 22   | 203.0.113.0/24   | ALLOW (bastion only)
200   | Inbound | TCP      | 1024-65535 | 0.0.0.0/0 | ALLOW (return traffic)
*     | Inbound | All      | All  | 0.0.0.0/0        | DENY (explicit)

1000  | Outbound| TCP      | 3306 | 10.0.20.0/24     | DENY (prevent web→DB)
100   | Outbound| All      | All  | 0.0.0.0/0        | ALLOW
*     | Outbound| All      | All  | 0.0.0.0/0        | DENY

Private-App Subnet NACL:
100   | Inbound | TCP      | 8080 | 10.0.1.0/24      | ALLOW (from web only)
*     | Inbound | All      | All  | 0.0.0.0/0        | DENY

Private-DB Subnet NACL:
100   | Inbound | TCP      | 3306 | 10.0.10.0/24     | ALLOW (from app only)
110   | Inbound | TCP      | 3306 | 10.0.1.0/24      | DENY (block web→DB)
*     | Inbound | All      | All  | 0.0.0.0/0        | DENY

Key: Rule 1000 outbound on public subnet + Rule 110 inbound on DB subnet
     = Double protection against web→DB direct access

Layer 2 - Security Group (Instance Level, Stateful):
SG-Web:
Inbound:
  Port 443: 0.0.0.0/0 (HTTPS from internet)
  Port 22: SG-Bastion (SSH from bastion only)
Outbound:
  Port 8080: SG-App (call app tier)
  Port 443: 0.0.0.0/0 (external APIs, OS updates)

SG-App:
Inbound:
  Port 8080: SG-Web (requests from web tier)
Outbound:
  Port 3306: SG-Database (MySQL queries)
  Port 443: 0.0.0.0/0 (external APIs if needed)

SG-Database:
Inbound:
  Port 3306: SG-App (MySQL from app tier ONLY)
Outbound:
  None (database doesn't initiate outbound, but stateful allows responses)

Layer 3 - VPC Flow Logs (Audit & Compliance):
Enable on all subnets, send to S3, analyze with Athena

-- Query to find attempts to access DB from web tier (should be blocked)
SELECT 
  sourceaddress,
  destinationaddress,
  destinationport,
  action,
  COUNT(*) as attempts
FROM vpc_flow_logs
WHERE destinationaddress BETWEEN '10.0.20.1' AND '10.0.20.254'  -- DB subnet
  AND sourceaddress BETWEEN '10.0.1.1' AND '10.0.1.254'          -- Web subnet
  AND destinationport = 3306
  AND action = 'REJECT'
GROUP BY sourceaddress, destinationaddress, destinationport, action
ORDER BY attempts DESC;

Result shows:
Source: 10.0.1.50 (compromised web server)
Destination: 10.0.20.10 (database server)
Port: 3306
Action: REJECT
Attempts: 1,523

This proves:
Attack was attempted (web server tried to access DB)
Defense worked (all attempts REJECTED by NACL rule 1000 + 110)
Audit trail exists (PCI DSS compliance requirement 10.2.7)

Benefits of Layered Approach:

1. Defense in Depth (Multiple Barriers):
   Attacker compromises web server:
      Step 1: Try to connect to database (10.0.20.10:3306)
      Step 2: Security group SG-Web doesn't allow outbound to DB
      Step 3: EVEN IF SG misconfigured, NACL rule 1000 blocks
      Step 4: EVEN IF NACL bypassed somehow, DB NACL rule 110 blocks
      Step 5: EVEN IF that fails, SG-Database only allows SG-App
   
   Result: 4 layers of protection (not just 1)

2. Stateless NACL (Can't Be Bypassed):
   Security groups: Stateful (return traffic automatic)
   NACLs: Stateless (must explicitly allow both directions)
   
   Scenario: Attacker finds SG bypass technique
      - SG bypass: Still blocked by NACL (independent layer)
      - NACL can't be bypassed from compromised instance

3. Explicit Deny (Defense Against Misconfiguration):
   Security groups: No deny rules (only allow, implicit deny)
   NACLs: Explicit deny rules (DENY takes precedence over ALLOW)
   
   Example: Junior engineer accidentally adds SG-Web → SG-Database rule
      - SG would allow connection
      - NACL explicit DENY rule 1000 still blocks
      - Misconfiguration caught in audit

4. Compliance Audit Trail:
   VPC Flow Logs show:
      - Every attempted connection (including blocked)
      - Source, destination, port, protocol
      - Action (ACCEPT or REJECT)
      - Timestamp (when attack occurred)
   
   PCI DSS 10.2.7: "All access to system components"
   VPC Flow Logs satisfy this requirement

Cost:
SG + NACL: $0 (no charge for these services)
VPC Flow Logs: $0.50 per GB ingested
   Typical: 10 GB/day = $150/month = $1,800/year
   Value: Compliance + security visibility

Why A is Wrong (Security Groups Only):
 Single Layer:
   - If SG misconfigured → direct attack surface
   - No backup defense layer
   
 No Explicit Deny:
   - SGs only have allow rules (implicit deny for everything else)
   - Can't explicitly block web→DB (only omit the allow)
   - Less clear in audit ("it's not allowed" vs "it's explicitly denied")

 Stateful Risk:
   - Return traffic automatically allowed
   - If attacker finds exploitation technique: Harder to defend

When SGs alone sufficient:
   - Development environment (not production)
   - Simple architecture (<10 instances)
   - No compliance requirements

Why B is Wrong (NACLs Only):
 Too Coarse:
   - NACLs apply to entire subnet (all instances)
   - Can't differentiate between instances in same subnet
   - Example: Subnet has web01, web02, web03
     - NACL can't allow web01→app but block web02→app
     - Security groups can (instance-level granularity)

 Management Overhead:
   - NACLs: Rule number ordering (100, 110, 120...)
   - Adding rule in middle: Renumber all subsequent rules
   - Security groups: No ordering, simpler

 Stateless Complexity:
   - Must define both inbound AND outbound rules
   - Example: Allow HTTPS
     - Inbound: Allow TCP 443 from 0.0.0.0/0
     - Outbound: Allow TCP 1024-65535 to 0.0.0.0/0 (ephemeral)
   - Security groups: Define inbound only (stateful handles return)

When NACLs alone sufficient:
   - Never! Always use SGs, optionally add NACLs for defense in depth

Why D is Wrong (WAF + SG Without NACL):
 WAF Wrong Layer:
   - WAF: Application layer (Layer 7, HTTP/HTTPS)
   - Doesn't protect: SSH brute force, DB port scanning, non-HTTP protocols
   - Example: Attacker port scans database (not HTTP)
     - WAF: Doesn't inspect (wrong layer)
     - NACL: Blocks (Layer 3/4 protection)

 Missing Network Segmentation:
   - WAF protects web application vulnerabilities (SQL injection, XSS)
   - Doesn't prevent network-level attacks (web→DB direct connection)
   - Still need NACLs for subnet-level isolation

 Cost:
   - WAF: $5/month + $1 per million requests
   - Typical: $100-500/month
   - NACL: $0 (free)
   
   Paying $100+/month for incomplete protection!

When to add WAF:
   After SGs + NACLs are configured
   For application-layer protection (OWASP Top 10)
   Budget available ($100-500/month)

Real-World Example - Capital One Breach (2019):
What Happened:
   - Misconfigured WAF allowed SSRF attack
   - Attacker accessed EC2 metadata service
   - Stole IAM credentials, accessed S3 buckets
   - 100 million customer records stolen

What Defense in Depth Could Have Prevented:
1. Security Groups:
   - Web EC2 shouldn't have S3 full access
   - Least privilege: Read-only to specific buckets
   
2. NACLs:
   - Explicit deny rules for sensitive subnets
   - Even with stolen credentials, network blocks access
   
3. VPC Flow Logs:
   - Would have shown unusual S3 API access patterns
   - Alert on 100M GetObject calls (anomaly detection)

Capital One Costs:
   - Regulatory fine: $80 million (OCC)
   - Settlement: $190 million (class action)
   - Total: $270 million breach cost
   - Compare to: $1,800/year for VPC Flow Logs (150,000× ROI!)

Best Practices Summary:

1. Always Use Security Groups (Required):
   Instance-level firewall
   Allow rules based on source security group
   Stateful (simpler management)

2. Add NACLs for High-Security (Recommended):
   Subnet-level firewall (second layer)
   Explicit deny rules (defense against misconfiguration)
   Stateless (can't be bypassed)
   Rule examples:
      - Deny all traffic between web and database subnets
      - Deny all inbound from known malicious IPs
      - Deny all outbound to non-approved ports

3. Enable VPC Flow Logs (Required for Compliance):
   Audit trail for all network traffic
   Detect attack attempts (even if blocked)
   Troubleshoot connectivity issues
   Compliance: PCI DSS, HIPAA, SOX

4. Add WAF for Web Applications (Recommended):
   After SGs and NACLs configured
   Protects against OWASP Top 10
   SQL injection, XSS, DDoS prevention

Decision Matrix:
Minimal (Dev): Security Groups only
Standard (Prod): Security Groups + VPC Flow Logs
High Security (Finance, Health): Security Groups + NACLs + Flow Logs
Public-Facing Web: Above + WAF

Why C is Correct (500 Mbps Hosted DX + MACsec):

Architecture:
On-Premises (Chicago) → Hosted DX 500 Mbps → AWS Direct Connect Location (Chicago)
   ↓ MACsec Encryption (Layer 2)
AWS us-east-1 (Private Virtual Interface) → VPC

Component Breakdown:

1. Hosted Direct Connect (500 Mbps):
   What: Shared physical connection via AWS Partner
   Speed: 50 Mbps to 10 Gbps (you choose 500 Mbps)
   Setup: 2-3 weeks (vs 4-8 weeks for Dedicated)
   
   Cost:
   Port: $0.0375/hour (500 Mbps rate)
         $0.0375 × 730 hours = $27.38/month
   
   Data Transfer Out (to on-prem): $0.02/GB
         25 TB × $0.02/GB × 1,000 = $500/month
   
   Data Transfer In (from on-prem): FREE
         25 TB × $0 = $0
   
   Partner Cross-Connect: ~$100/month (partner-specific)
   
   Subtotal: $27.38 + $500 + $100 = $627.38/month

2. MACsec Encryption (Media Access Control Security):
   What: Layer 2 encryption (faster than IPsec VPN)
   Overhead: <1ms latency (vs 5-10ms for IPsec)
   Encryption: 256-bit AES-GCM
   Compliance: FIPS 140-2 Level 2
   Cost: $0 (included in Direct Connect, no extra charge!)
   
   Key Benefit: Encrypted without VPN overhead
      - VPN over DX: 5-10ms latency added
      - MACsec over DX: <1ms latency (negligible)
      - Meets "encrypted traffic" requirement with minimal performance impact

3. Performance:
   Latency:
      - Chicago to us-east-1: ~10ms (direct fiber)
      - MACsec overhead: <1ms
      - Total: ~11ms (well under 15ms requirement )
   
   Bandwidth:
      - 500 Mbps = 1,620 TB/month theoretical
      - Your need: 50 TB/month (25 TB each way)
      - Utilization: 3% average (plenty of headroom)
   
   Reliability:
      - AWS DX SLA: 99.9% (meets requirement)
      - Partner SLA: Typically 99.9%+ (check specific partner)

Total Cost: $627.38/month = $7,529/year
Budget: $5,000/month (under budget by $4,372/month!)

Why A is Wrong (1 Gbps Dedicated DX + VPN Backup):
 Overkill Bandwidth:
   - 1 Gbps = 3,240 TB/month capacity
   - Your need: 50 TB/month
   - Utilization: 1.5% (wasting 98.5% of capacity)

 Expensive:
   Port: $0.30/hour × 730 = $219/month (1 Gbps dedicated)
   Data Out: 25 TB × $0.02/GB × 1,000 = $500/month
   Cross-Connect: $500/month (colocation facility fee)
   VPN Backup: $0.05/hour × 2 × 730 = $73/month
   VPN Data: 50 TB × $0.09/GB × 1,000 = $4,500/month (VPN path)
   Total: $5,792/month = $69,504/year
   
   Over budget by $792/month!

 Complexity:
   - Dedicated DX: Requires physical cross-connect at colocation
   - Your team: Must go to facility, connect fiber
   - Lead time: 4-8 weeks (vs 2-3 weeks hosted)
   - Operational burden: Manage physical connection

 Unnecessary Redundancy:
   - Requirement: 99.9% uptime = 43 minutes downtime/month OK
   - Single hosted DX: Meets 99.9% SLA
   - Adding VPN backup: Achieves 99.99% (overkill)
   - Extra cost for extra nine you don't need!

When to use Dedicated DX + VPN backup:
   Need 99.99%+ uptime (enterprise mission-critical)
   Sustained high bandwidth (>500 Mbps average, not peak)
   Budget allows ($5K+/month)
   In-house team to manage physical connection

Why B is Wrong (Two 1 Gbps VPNs Active/Active):
 Latency:
   - VPN over internet: 20-50ms (Chicago → us-east-1)
   - IPsec overhead: 5-10ms
   - Total: 25-60ms (exceeds 15ms requirement )
   
   ERP system with 50ms latency:
      - Users: "System is slow, unusable"
      - Productivity: Reduced 30%+ (waiting for responses)
      - Business impact: Revenue loss

 Unreliable:
   - Internet path: Best-effort (no SLA)
   - Packet loss: 1-3% typical (vs <0.01% Direct Connect)
   - Jitter: 10-50ms variability (vs <1ms DX)
   - Result: Intermittent slowness, user complaints

 Cost Not Much Cheaper:
   VPN: $0.05/hour × 2 × 730 = $73/month
   Data Out: 25 TB × $0.09/GB × 1,000 = $2,250/month
   Data In: 25 TB × $0.09/GB × 1,000 = $2,250/month
   Total: $4,573/month = $54,876/year
   
   Seems cheaper than DX, but:
      - Doesn't meet latency requirement (business requirement failed!)
      - Unreliable (user productivity reduced)
      - Hidden cost: User time wasted waiting

When to use VPN only:
   Latency not critical (>50ms acceptable)
   Burst traffic (not sustained high bandwidth)
   Temporary connection (project-based, <6 months)
   Budget constrained (<$1K/month)

Why D is Wrong (Multiple VPNs, No DX):
Same issues as option B:
 High latency (25-60ms)
 Unreliable (internet best-effort)
 Doesn't meet ERP <15ms requirement

Additional issue:
 "Multiple" doesn't solve fundamental problems:
   - 2 VPNs over same internet: Still high latency
   - 4 VPNs over same internet: Still high latency
   - Problem is internet path, not number of tunnels

Decision Matrix:

Direct Connect vs VPN Decision Tree:
1. Need low latency (<15ms)?
   YES → Direct Connect required
   NO → Go to step 2

2. Sustained bandwidth >100 Mbps?
   YES → Direct Connect recommended
   NO → Go to step 3

3. Need predictable performance?
   YES → Direct Connect recommended
   NO → VPN acceptable

4. Budget >$1,000/month?
   YES → Direct Connect feasible
   NO → VPN only option

Your Scenario:
1. Latency <15ms? YES → Direct Connect
2. Bandwidth 50 TB/month? YES → Direct Connect
3. Predictable performance? YES (ERP) → Direct Connect
4. Budget $5,000/month? YES → Direct Connect affordable

Hosted vs Dedicated Direct Connect:

Use Hosted (Your Choice ):
Bandwidth: <1 Gbps sustained
Setup: Faster (2-3 weeks vs 4-8 weeks)
Management: AWS Partner handles physical layer
Cost: Lower monthly ($27.38 vs $219 port fee)
Flexibility: Easy to upgrade (call partner, increase speed)

Use Dedicated:
Bandwidth: >1 Gbps sustained
Control: Want full control of physical connection
Multiple VIFs: Need many virtual interfaces (>50)
Resiliency: Mission-critical (pair with second dedicated)

MACsec vs IPsec VPN Over DX:

MACsec (Your Choice ):
Layer 2 encryption (data link layer)
Latency: <1ms overhead
Throughput: Near-wire speed
Cost: FREE (included in DX)
Compliance: FIPS 140-2 Level 2

IPsec VPN Over DX:
Layer 3 encryption (network layer)
 Latency: 5-10ms overhead
 Throughput: Reduced (encryption bottleneck)
 Cost: $0.05/hour VPN + $0.09/GB data
 Complexity: Manage VPN endpoints

When to use IPsec over DX:
- MACsec not supported by equipment
- Need Layer 3 encryption (rare)
- Want to use existing VPN infrastructure

Real-World Example - Healthcare Provider:
Requirement:
   - Hospital to AWS: Patient records, imaging (HIPAA)
   - Latency: <10ms (real-time access to patient data)
   - Data: 100 TB/month (medical imaging large files)
   - Compliance: Encrypted transport (HIPAA requires)

Initial Design (WRONG):
   - 2× VPN over internet
   - Cost: $9,000/month (100 TB × $0.09/GB)
   - Latency: 40-80ms (too slow, doctors complained)
   - Result: Doctors stopped using system, security risk!

Redesign (CORRECT):
   - 1 Gbps Hosted Direct Connect + MACsec
   - Cost: $2,100/month (port $54 + data $2,000 + cross-connect)
   - Latency: 8ms (meets requirement)
   - Compliance: MACsec satisfies HIPAA encryption
   - Result: Doctors happy, fast access to patient data

ROI:
   Cost savings: $9,000 - $2,100 = $6,900/month = $82,800/year
   Plus: Doctor productivity (can't quantify but significant)
   Plus: Patient safety (faster access to medical history)

Implementation Steps for Your Scenario:

1. Choose AWS Direct Connect Partner (Chicago):
   Examples: Equinix, CoreSite, Megaport, PacketFabric
   Criteria:
      - Location near your datacenter (minimize last-mile fiber cost)
      - 500 Mbps hosted connection support
      - MACsec support (check partner capabilities)
      - Pricing: Get quotes ($100-200/month typical)

2. Order Hosted Connection:
   - AWS Console: Create hosted connection request
   - Partner: Provisions within 2-3 weeks
   - Cost: $27.38/month (AWS) + partner fee

3. Configure MACsec Encryption:
   AWS Console:
      - Enable MACsec on connection
      - Generate CAK (Connectivity Association Key)
      - Configure CKN (Connectivity Association Key Name)
   
   On-Premises Router:
      - Configure MACsec (Cisco: mka pre-shared-key)
      - Apply to Direct Connect interface
   
   Test: Verify encryption active (show macsec statistics)

4. Create Private Virtual Interface (VIF):
   - VLAN: 100 (example)
   - BGP ASN: Your ASN (or use private ASN 64512-65534)
   - BGP IP: AWS provides /30 subnet (e.g., 169.254.0.0/30)
   - Advertise: Your on-prem prefixes (192.168.0.0/16)

5. Test & Monitor:
   Latency test:
      ping -c 100 10.0.1.50  # AWS VPC instance
      Result: avg 11ms (under 15ms requirement)
   
   Throughput test:
      iperf3 -c 10.0.1.50 -t 60 -P 4
      Result: 480 Mbps (96% of 500 Mbps, excellent)
   
   Monitor:
      CloudWatch metrics: ConnectionState, EgressBytes, IngressBytes
      Alert: ConnectionState != "up" for >5 minutes

Cost Breakdown Summary:

Your Solution (500 Mbps Hosted DX + MACsec):
   Monthly: $627/month
   Annual: $7,529/year
   Per TB: $12.55/TB ($627 ÷ 50 TB)

Alternative 1 (1 Gbps Dedicated DX + VPN):
   Monthly: $5,792/month (over budget!)
   Annual: $69,504/year
   Per TB: $115.84/TB

Alternative 2 (VPN Only):
   Monthly: $4,573/month (cheaper but doesn't meet requirement!)
   Annual: $54,876/year
   Per TB: $91.46/TB
   BUT: Latency 40-80ms (fails <15ms requirement)

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::company-data/*"
  }]
}

Why C is Correct (Principle of Least Privilege):

Complete Policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListAllBuckets",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    },
    {
      "Sid": "ListCompanyDataBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::company-data"
    },
    {
      "Sid": "ReadAllObjects",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::company-data/*"
    },
    {
      "Sid": "UploadToUploadsOnly",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::company-data/uploads/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    }
  ]
}

Breakdown of Each Permission:

1. s3:ListAllMyBuckets on "*":
   Required: YES
   Why: Lists all buckets in account (AWS Console S3 page)
   Scope: Account-level operation, needs "*" resource
   Security: Low risk (only reveals bucket names, not contents)
   
   Without this:
   - Console shows error "Access Denied"
   - Can't see bucket list
   - Must know exact bucket name to access

2. s3:ListBucket on "arn:aws:s3:::company-data":
   Required: YES
   Why: Lists objects within the bucket
   Resource: Bucket itself (not objects)
   Scope: company-data bucket only
   
   Without this:
   - Can't see objects in bucket
   - GetObject works if you know exact key
   - Poor user experience (blind access)
   
   Common mistake:
   Resource: "arn:aws:s3:::company-data/*"  Wrong!
   (ListBucket operates on bucket, not objects)

3. s3:GetObject on "arn:aws:s3:::company-data/*":
   Required: YES (already had this)
   Why: Read objects from bucket
   Resource: All objects in bucket
   Scope: Read-only access
   
   IAM vs Bucket Policy:
   - IAM policy: User-centric ("this user can read")
   - Bucket policy: Resource-centric ("this bucket allows")
   - Both must allow for access (implicit deny if either denies)

4. s3:PutObject on "arn:aws:s3:::company-data/uploads/*":
   Required: YES
   Why: Upload objects to uploads/ prefix only
   Resource: Objects under uploads/ prefix
   Scope: Write access limited to one prefix
   
   Condition added:
   - Requires server-side encryption
   - Prevents uploading unencrypted sensitive data
   - Security best practice
   
   Prefix restriction:
   company-data/
   ├── config/         Can't upload here
   ├── production/     Can't upload here
   └── uploads/        Can upload here
       ├── file1.txt   
       └── file2.txt   

Test the Policy:

# List all buckets (should work)
aws s3 ls
Output: 2024-09-23 10:00:00 company-data
        2024-09-23 10:00:00 other-bucket

# List objects in company-data (should work)
aws s3 ls s3://company-data/
Output: PRE config/
        PRE production/
        PRE uploads/

# Read object (should work)
aws s3 cp s3://company-data/config/app.json ./
Output: download: s3://company-data/config/app.json to ./app.json

# Upload to uploads/ (should work)
aws s3 cp file.txt s3://company-data/uploads/ --sse AES256
Output: upload: ./file.txt to s3://company-data/uploads/file.txt

# Upload to config/ (should fail - not in policy)
aws s3 cp file.txt s3://company-data/config/
Output: upload failed: An error occurred (AccessDenied)

# Delete object (should fail - not in policy)
aws s3 rm s3://company-data/uploads/file.txt
Output: delete failed: An error occurred (AccessDenied)

Least Privilege Checklist:
Can list buckets (required for console)
Can list objects in company-data (required for console)
Can read all objects in company-data (requirement met)
Can upload ONLY to uploads/ prefix (requirement met)
Cannot upload to other prefixes (security)
Cannot delete objects (security)
Cannot delete bucket (security)
Cannot modify bucket policy (security)
Cannot access other buckets (security)

Why A is Wrong (s3:* on all resources):
 Violates Least Privilege:
   - s3:* includes destructive operations:
     - DeleteObject (can delete production data!)
     - DeleteBucket (can delete entire bucket!)
     - PutBucketPolicy (can grant themselves more access!)
     - PutBucketAcl (can make bucket public!)
   
 All Resources:
   - Resource "*" means ALL buckets in account
   - Developer can access confidential-data, backup-data, etc.
   - Massive security risk

Real-world incident:
   Company: Code Spaces (2014)
   Issue: Developer credentials compromised
   Permissions: s3:* on all buckets
   Attack: Attacker deleted all S3 buckets (backups too!)
   Result: Company went out of business
   
   Lesson: Least privilege could have limited damage

Why B is Wrong (Missing ListAllMyBuckets):
 Console Won't Work:
   - AWS Console S3 page requires ListAllMyBuckets
   - Without it: "Access Denied" error on console
   - CLI works (if you know exact bucket name)
   
 Poor User Experience:
   Developer must type: aws s3 ls s3://company-data/
   Can't discover bucket name from console
   
When B is acceptable:
   - Programmatic access only (no console needed)
   - Bucket name hardcoded in application
   - Single-purpose service account

Why D is Wrong (Administrator Access):
 Massive Security Risk:
   Administrator: arn:aws:iam::aws:policy/AdministratorAccess
   Permissions: *:* on *:* (EVERYTHING!)
   
   Can do:
   - Delete production databases
   - Terminate EC2 instances
   - Modify IAM policies (escalate privileges)
   - Access all S3 buckets (including confidential)
   - Cost: Spin up expensive resources ($100K+ bill)
   
 Violates Separation of Duties:
   Developer should develop, not administer
   Admins should administer, not develop
   
 Compliance Violations:
   PCI DSS 7.1: "Limit access to system components"
   SOX: Separation of duties required
   HIPAA: Minimum necessary standard
   
   Audit finding: "Developer has admin access"
   Result: Failed compliance audit, potential fines

Real-World Example - Capital One Breach (2019):
Root cause: SSRF vulnerability in WAF
Developer IAM role: Too permissive (list + read on ALL S3 buckets)
Should have had: Access only to specific bucket
Result: 100 million customer records stolen
Cost: $270 million in fines and settlements

Prevention with Least Privilege:
{
  "Effect": "Allow",
  "Action": ["s3:ListBucket", "s3:GetObject"],
  "Resource": [
    "arn:aws:s3:::developer-sandbox",
    "arn:aws:s3:::developer-sandbox/*"
  ]
}

Result: Even with SSRF, attacker limited to sandbox bucket
Customer data in other buckets: Protected 

IAM Policy Best Practices:

1. Start with Nothing:
   - New user/role: Zero permissions
   - Add only what's needed
   - Don't use AWS managed policies for specific needs

2. Use Conditions:
   - Restrict by IP: aws:SourceIp
   - Require MFA: aws:MultiFactorAuthPresent
   - Limit time: aws:CurrentTime
   - Restrict encryption: s3:x-amz-server-side-encryption
   
   Example:
   "Condition": {
     "IpAddress": {"aws:SourceIp": "203.0.113.0/24"},
     "Bool": {"aws:MultiFactorAuthPresent": "true"}
   }

3. Use Permission Boundaries:
   - Maximum permissions (ceiling)
   - User can't escalate beyond boundary
   - Use for delegated admin scenarios
   
   Example: Developer can create IAM roles, but only with specific permissions

4. Regular Audits:
   - IAM Access Analyzer: Identifies unused permissions
   - AWS Access Advisor: Shows last accessed time
   - Remove unused permissions quarterly
   
   Command: aws iam generate-service-last-accessed-details

5. Use SCPs for Organization-Wide Controls:
   Service Control Policy (AWS Organizations):
   {
     "Effect": "Deny",
     "Action": "s3:DeleteBucket",
     "Resource": "*"
   }
   
   Result: No one can delete buckets (including admins)
   Use for: Prevent accidental deletions, compliance

Troubleshooting IAM Policies:

Tool: IAM Policy Simulator
URL: https://policysim.aws.amazon.com

Test:
1. Select user/role
2. Select service (S3)
3. Select action (PutObject)
4. Select resource (arn:aws:s3:::company-data/uploads/test.txt)
5. Click "Simulate"

Result shows:
Allowed: Policy grants permission
Denied: Shows which policy denies
? Implicit Deny: No policy allows

Common IAM Issues:

Issue 1: "Access Denied" on ListBucket
Cause: Resource should be bucket, not objects
Wrong: "Resource": "arn:aws:s3:::bucket/*"
Right: "Resource": "arn:aws:s3:::bucket"

Issue 2: "Access Denied" on GetObject
Cause: Resource should be objects, not bucket
Wrong: "Resource": "arn:aws:s3:::bucket"
Right: "Resource": "arn:aws:s3:::bucket/*"

Issue 3: Works in CLI, fails in Console
Cause: Console requires additional permissions
Missing: s3:ListAllMyBuckets, s3:GetBucketLocation
Fix: Add these to policy

Issue 4: Cross-Account Access Denied
Cause: Need both IAM policy AND bucket policy
IAM: Allow user to access bucket
Bucket: Allow external account to access
Both must allow!

Why B is Correct (CloudHSM for PCI DSS Level 3):

Architecture:
Application Servers (us-east-1)
   ↓ PKCS#11, JCE, CNG API
CloudHSM Cluster
   ├─ HSM-1 (us-east-1a): Primary
   ├─ HSM-2 (us-east-1b): Replica (automatic sync)
   └─ HSM-3 (us-east-1c): Replica (automatic sync)
   
Encryption Flow:
1. App encrypts credit card data
2. Calls CloudHSM API for data key
3. CloudHSM generates key inside HSM (never leaves device)
4. Returns encrypted data key (wrapped with master key)
5. App stores: Encrypted data + Encrypted data key

CloudHSM Key Features for PCI DSS:

1. FIPS 140-2 Level 3 Compliance:
   Required: PCI DSS 3.2.1 Requirement 3.6.1
   CloudHSM: Level 3 certified
   KMS: Level 2 certified (not sufficient for PCI DSS!)
   
   Level 3 requirements:
   - Physical tamper-evident security
   - Tamper-responsive (zeroizes keys if tampered)
   - Identity-based authentication
   - Dedicated hardware (not multi-tenant)

2. Customer-Controlled Keys:
   You own the keys (not AWS)
   Keys never leave HSM in plaintext
   AWS cannot access your keys
   Key material generated inside HSM
   
   vs KMS:
   - KMS: AWS manages hardware, you manage keys
   - CloudHSM: You manage both hardware and keys
   - PCI DSS: Requires customer control

3. Automatic Key Synchronization:
   - Keys replicated across all HSMs in cluster
   - Sync: Real-time (within milliseconds)
   - Encryption: Replicated data encrypted in transit
   
   High Availability:
   - HSM-1 fails → HSM-2 takes over (no downtime)
   - Cluster remains operational with 1 HSM
   - Best practice: 3+ HSMs for production

4. Audit Logging:
   CloudHSM logs to CloudWatch Logs:
   - Every encryption/decryption operation
   - Key creation, deletion, rotation
   - User authentication attempts
   - Timestamp, user, operation, key ID
   
   PCI DSS 10.2: "Implement automated audit trails"
   CloudHSM satisfies this requirement

Cost Analysis (3 HSMs, us-east-1):

Monthly Costs:
   CloudHSM: $1.60/hour per HSM
   3 HSMs: 3 × $1.60 × 730 hours = $3,504/month
   
   ENI (Elastic Network Interface): $0.01/hour
   3 ENIs: 3 × $0.01 × 730 = $21.90/month
   
   Data transfer: Negligible (encryption happens in-VPC)
   
   Total: $3,504 + $21.90 = $3,525.90/month = $42,311/year
   
   Budget: $5,000/month
   Under budget: $1,474/month ($17,688/year) 

Additional Costs (Optional):
   Backup HSM (4th): $1,168/month (disaster recovery)
   Additional region: $3,525/month (multi-region)

Performance:
   Throughput: 10,000 operations/second per HSM
   Cluster: 30,000 ops/sec total (3 HSMs)
   Latency: <5ms typical (in-region)
   
   Workload example (payment processor):
   - 1,000 transactions/second
   - Each: 2 crypto operations (encrypt card, decrypt for processing)
   - Total: 2,000 ops/sec
   - Capacity: 30,000 ops/sec (15× headroom) 

Key Rotation (90-day requirement):

Automatic Rotation Not Supported:
   CloudHSM: Manual key rotation required
   Process:
   1. Generate new master key (newKey)
   2. Re-encrypt all data keys with newKey
   3. Update application to use newKey
   4. Delete oldKey after grace period
   
   Frequency: Every 90 days (PCI DSS requirement)
   Downtime: Zero (blue-green deployment)
   
   Automation:
   Lambda function (scheduled CloudWatch Event):
   - Trigger: Every 90 days
   - Action: Rotate keys automatically
   - Notify: Security team via SNS

Why A is Wrong (KMS Customer Managed CMK):
 FIPS 140-2 Level 2 Only:
   - KMS: Level 2 certified
   - PCI DSS 3.6.1: Requires Level 3
   - Result: Non-compliant 
   
   Level 2 vs Level 3:
   Level 2: Software-based encryption (multi-tenant)
   Level 3: Hardware-based encryption (dedicated device)
   
   PCI DSS rationale: Payment data requires highest security

 AWS-Managed Hardware:
   - KMS: AWS manages physical HSMs
   - You: Manage keys, not hardware
   - PCI DSS: Some interpretations require customer control of hardware
   - Auditor may reject KMS for strict PCI DSS compliance

 Multi-Tenant:
   - KMS: Shared hardware across AWS customers
   - CloudHSM: Dedicated hardware for you only
   - PCI DSS concern: Data commingling

When KMS is sufficient:
   FIPS 140-2 Level 2 acceptable
   AWS-managed hardware acceptable
   Not PCI DSS (or lenient interpretation)
   Cost-sensitive (<$500/month budget)
   Automatic key rotation important

KMS Cost Comparison:
   CMK: $1/month per key
   API calls: $0.03 per 10,000 requests
   
   Example (same workload):
   CMKs: 10 keys × $1 = $10/month
   API calls: 2,000 ops/sec × 2.6M sec/month = 5.2B calls
             5.2B ÷ 10K × $0.03 = $15,600/month
   Total: $15,610/month
   
   CloudHSM: $3,526/month (77% cheaper!)
   
   Why KMS more expensive: Per-API-call pricing at high volume
   When KMS cheaper: Low volume (<1M calls/month)

Why C is Wrong (Client-Side with On-Prem Keys):
 Operational Burden:
   - Manage on-premises HSM hardware
   - Physical security requirements
   - 24/7 monitoring and support
   - Key distribution to cloud applications
   
 Latency:
   - Every encryption: Call on-premises from AWS
   - Latency: 50-100ms+ (internet or Direct Connect)
   - CloudHSM: <5ms (same VPC)
   
 Single Point of Failure:
   - On-prem datacenter down → AWS apps can't encrypt
   - Need redundant on-prem HSMs + connectivity
   
 Complexity:
   - VPN or Direct Connect required ($500+/month)
   - Key synchronization between on-prem and cloud
   - Disaster recovery planning

When on-prem keys acceptable:
   - Regulatory requirement (data residency, not encryption)
   - Existing on-prem HSM investment
   - Hybrid cloud transition phase

Why D is Wrong (KMS Automatic Rotation + AWS Managed):
 AWS Managed Keys: You don't control keys
   - AWS manages key lifecycle
   - AWS could theoretically access (they don't, but auditor concern)
   - PCI DSS: Customer must control encryption keys
   
 Limited Rotation Control:
   - KMS automatic rotation: Annually (365 days)
   - Your requirement: 90 days
   - Can't change rotation frequency with AWS managed keys
   
 Still FIPS Level 2:
   Same issue as option A (not Level 3)

KMS Customer vs AWS Managed Keys:
Customer Managed:
   You control rotation schedule
   You can disable/delete keys
   You manage key policies
   Supports cross-account access
   Cost: $1/month per key

AWS Managed:
   AWS controls rotation (annual only)
   Can't delete (only disable)
   Limited policy control
   Cost: FREE
   
   Use for: Non-sensitive data, cost optimization

CloudHSM Setup Guide:

1. Create CloudHSM Cluster:
aws cloudhsmv2 create-cluster \
  --hsm-type hsm1.medium \
  --subnet-ids subnet-abc123 subnet-def456 subnet-ghi789

Output: Cluster ID (cluster-abc123def456)

2. Create HSMs (one per AZ):
aws cloudhsmv2 create-hsm \
  --cluster-id cluster-abc123def456 \
  --availability-zone us-east-1a

aws cloudhsmv2 create-hsm \
  --cluster-id cluster-abc123def456 \
  --availability-zone us-east-1b

aws cloudhsmv2 create-hsm \
  --cluster-id cluster-abc123def456\
  --availability-zone us-east-1c

Status: Wait 10-15 minutes (HSM provisioning)

3. Initialize Cluster:
# Download cluster CSR
aws cloudhsmv2 describe-clusters \
  --filters clusterIds=cluster-abc123def456 \
  --output text \
  --query 'Clusters[0].Certificates.ClusterCsr' > cluster.csr

# Sign CSR (create self-signed cert for dev, use CA for prod)
openssl genrsa -out customerCA.key 2048
openssl req -new -x509 -days 3652 -key customerCA.key -out customerCA.crt
openssl x509 -req -days 3652 -in cluster.csr -CA customerCA.crt -CAkey customerCA.key -CAcreateserial -out cluster.crt

# Initialize cluster with signed cert
aws cloudhsmv2 initialize-cluster \
  --cluster-id cluster-abc123def456 \
  --signed-cert file://cluster.crt \
  --trust-anchor file://customerCA.crt

4. Activate Cluster:
# Install CloudHSM client on EC2 instance
sudo yum install -y aws-cloudhsmv2-client

# Configure cluster
sudo /opt/cloudhsm/bin/configure -a <HSM_IP_ADDRESS>

# Create initial admin user
/opt/cloudhsm/bin/cloudhsm_mgmt_util /opt/cloudhsm/etc/cloudhsm_mgmt_util.cfg

cloudhsm-cli> loginHSM PRECO admin password
cloudhsm-cli> changePswd PRECO admin <NEW_PASSWORD>
cloudhsm-cli> createUser CU app-user <PASSWORD>

5. Use in Application (Python example):
import boto3
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend

# Initialize CloudHSM client
hsm = boto3.client('cloudhsmv2')

def encrypt_credit_card(card_number, key_handle):
    """Encrypt credit card using CloudHSM"""
    # Call CloudHSM to encrypt
    # Key never leaves HSM!
    response = hsm.encrypt(
        KeyHandle=key_handle,
        Plaintext=card_number.encode(),
        Algorithm='AES_GCM'
    )
    return response['CiphertextBlob']

def decrypt_credit_card(ciphertext, key_handle):
    """Decrypt credit card using CloudHSM"""
    response = hsm.decrypt(
        KeyHandle=key_handle,
        CiphertextBlob=ciphertext,
        Algorithm='AES_GCM'
    )
    return response['Plaintext'].decode()

Real-World Example - Payment Processor:
Company: Stripe (estimated, based on public information)
Requirement: PCI DSS Level 1 (>6M transactions/year)
Solution: CloudHSM clusters in multiple regions

Architecture:
   us-east-1: 5 HSMs (primary region)
   eu-west-1: 5 HSMs (EU processing)
   ap-southeast-1: 3 HSMs (APAC processing)
   
   Total: 13 HSMs = $20,800/month = $249,600/year

Scale:
   Transactions: 1 billion+/year
   Operations: 10,000/second peak
   Availability: 99.99%+ (4 nines)

Cost per transaction: $0.00025 (0.025 cents)
   Acceptable for payment processing industry

Alternative (if used KMS):
   API calls: 10,000/sec × 86,400 sec/day = 864M calls/day
   Cost: 864M ÷ 10K × $0.03 = $2,592/day = $77,760/month
   
   CloudHSM: $20,800/month (73% cheaper at scale!)

PCI DSS Compliance Checklist:

Requirement 3.5: Protect keys used for encryption
CloudHSM: Keys never leave HSM in plaintext

Requirement 3.6: Fully document and implement key management
CloudHSM: Key lifecycle (generation, rotation, deletion) documented
Rotation: Every 90 days (automated via Lambda)

Requirement 3.6.1: Generation of strong cryptographic keys
CloudHSM: FIPS 140-2 Level 3 (meets requirement)
KMS: FIPS 140-2 Level 2 (does not meet for strict interpretation)

Requirement 10.2: Implement automated audit trails
CloudHSM: All operations logged to CloudWatch Logs

Requirement 10.3: Record audit trail entries
User ID, Type of event, Date/time, Success/failure, Origination

CloudHSM Limitations:

1. No Automatic Key Rotation:
   Must implement custom rotation logic
   Recommended: Lambda function + CloudWatch Events

2. Manual Backups:
   Automatic: Daily cluster state
   Manual: Export keys for disaster recovery
   Important: Store backups encrypted in S3

3. Regional:
   Not multi-region by default
   Must set up separate clusters per region
   Key synchronization: Manual (security requirement)

4. Learning Curve:
   Complex setup vs KMS (click and go)
   Requires HSM expertise
   Training: 2-4 weeks for team

Best Practices:

1. Deploy 3+ HSMs (high availability)
2. Multi-AZ (survive AZ failure)
3. Multi-region for critical workloads
4. Rotate keys every 90 days (PCI DSS)
5. Backup keys to S3 (encrypted, versioned)
6. Monitor CloudWatch metrics (availability, latency)
7. Regular security audits (quarterly)
8. Least privilege IAM (who can access HSM)

Decision Matrix:

Use CloudHSM when:
PCI DSS Level 1 compliance (strict interpretation)
FIPS 140-2 Level 3 required
Customer must control hardware
High throughput (>10K ops/sec sustained)
Budget allows ($3K+/month)

Use KMS when:
FIPS 140-2 Level 2 sufficient
AWS-managed hardware acceptable
Lower throughput (<10K ops/sec)
Budget constrained (<$500/month)
Want simplicity (no HSM expertise needed)

/search?q='; DROP TABLE users; --
/search?q=1' UNION SELECT * FROM credit_cards--
/search?q=admin' OR '1'='1

Why C is Correct (Comprehensive WAF Protection):

Complete WAF Configuration:

# 1. Create Web ACL
aws wafv2 create-web-acl \
  --name ecommerce-sql-protection \
  --scope REGIONAL \
  --region us-east-1 \
  --default-action Allow={} \
  --rules file://waf-rules.json \
  --visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=EcommerceSQLiWAF

# 2. WAF Rules Configuration (waf-rules.json)
[
  {
    "Name": "AWSManagedRulesSQLiRuleSet",
    "Priority": 1,
    "Statement": {
      "ManagedRuleGroupStatement": {
        "VendorName": "AWS",
        "Name": "AWSManagedRulesSQLiRuleSet",
        "ExcludedRules": []
      }
    },
    "OverrideAction": {"None": {}},
    "VisibilityConfig": {
      "SampledRequestsEnabled": true,
      "CloudWatchMetricsEnabled": true,
      "MetricName": "SQLiRuleSet"
    }
  },
  {
    "Name": "RateLimitRule",
    "Priority": 2,
    "Statement": {
      "RateBasedStatement": {
        "Limit": 2000,
        "AggregateKeyType": "IP",
        "ScopeDownStatement": {
          "ByteMatchStatement": {
            "SearchString": "/search",
            "FieldToMatch": {"UriPath": {}},
            "TextTransformations": [{"Priority": 0, "Type": "LOWERCASE"}],
            "PositionalConstraint": "CONTAINS"
          }
        }
      }
    },
    "Action": {"Block": {}},
    "VisibilityConfig": {
      "SampledRequestsEnabled": true,
      "CloudWatchMetricsEnabled": true,
      "MetricName": "SearchRateLimit"
    }
  },
  {
    "Name": "KnownBadInputs",
    "Priority": 3,
    "Statement": {
      "OrStatement": {
        "Statements": [
          {
            "ByteMatchStatement": {
              "SearchString": "union select",
              "FieldToMatch": {"QueryString": {}},
              "TextTransformations": [
                {"Priority": 0, "Type": "LOWERCASE"},
                {"Priority": 1, "Type": "URL_DECODE"}
              ],
              "PositionalConstraint": "CONTAINS"
            }
          },
          {
            "ByteMatchStatement": {
              "SearchString": "drop table",
              "FieldToMatch": {"QueryString": {}},
              "TextTransformations": [
                {"Priority": 0, "Type": "LOWERCASE"},
                {"Priority": 1, "Type": "URL_DECODE"}
              ],
              "PositionalConstraint": "CONTAINS"
            }
          }
        ]
      }
    },
    "Action": {"Block": {}},
    "VisibilityConfig": {
      "SampledRequestsEnabled": true,
      "CloudWatchMetricsEnabled": true,
      "MetricName": "KnownBadInputs"
    }
  }
]

# 3. Enable Logging to S3
aws wafv2 put-logging-configuration \
  --logging-configuration ResourceArn=arn:aws:wafv2:us-east-1:123456789012:regional/webacl/ecommerce-sql-protection/abc123,LogDestinationConfigs=arn:aws:s3:::waf-logs-ecommerce,RedactedFields=[{UriPath={}}],ManagedByFirewallManager=false

# 4. Associate with ALB
aws wafv2 associate-web-acl \
  --web-acl-arn arn:aws:wafv2:us-east-1:123456789012:regional/webacl/ecommerce-sql-protection/abc123 \
  --resource-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/ecommerce-alb/abc123

AWS Managed SQLi Rule Group Features:

1. SQL Injection Detection:
   Patterns detected:
   - UNION SELECT statements
   - DROP/DELETE/UPDATE commands
   - OR 1=1 conditions
   - Comment sequences (-- /* */)
   - Boolean logic attacks
   - Time-based blind SQLi (SLEEP, WAITFOR)
   
   Transformations applied before matching:
   - URL_DECODE: %27 → '
   - HTML_ENTITY_DECODE: ' → '
   - LOWERCASE: Admin → admin
   - COMPRESS_WHITE_SPACE: Multiple spaces → single space
   - SQL_HEX_DECODE: 0x41646D696E → Admin

2. False Positive Handling:
   Legitimate search: "O'Reilly books"
   - Contains single quote (potential SQLi indicator)
   - But: No SQL keywords (SELECT, UNION, DROP)
   - Result: ALLOWED 
   
   Legitimate search: "1+1=2"
   - Contains equals and math
   - But: No SQL commands
   - Result: ALLOWED 
   
   Attack: "' OR '1'='1"
   - Contains SQL logic (OR statement)
   - Pattern matches boolean injection
   - Result: BLOCKED 
   
   Attack: "admin'--"
   - Contains comment sequence (SQL terminator)
   - Pattern matches authentication bypass
   - Result: BLOCKED 

3. Rate Limiting (2000 requests per 5 minutes per IP):
   Why needed:
   - Automated SQLi scanners try thousands of payloads
   - Rate limiting blocks brute-force attempts
   - Legitimate users: <100 searches per 5 min
   - Attackers: 1000+ attempts per minute
   
   Configuration:
   - Scope: /search endpoint only (not homepage)
   - Key: Source IP address
   - Limit: 2000 req/5min = 400 req/min = 6.6 req/sec per IP
   - Action: Block for 10 minutes after threshold
   
   Example:
   Attacker (IP 203.0.113.50):
   - Time 00:00: Request 1 (allowed)
   - Time 00:01: Request 100 (allowed)
   - Time 00:03: Request 1000 (allowed)
   - Time 00:04: Request 2001 (BLOCKED - exceeded limit)
   - Time 00:14: Block expires, can retry

4. Logging to S3:
   Log format (JSON):
   {
     "timestamp": 1695477600000,
     "formatVersion": 1,
     "webaclId": "arn:aws:wafv2:...:webacl/ecommerce-sql-protection",
     "terminatingRuleId": "AWSManagedRulesSQLiRuleSet",
     "terminatingRuleType": "MANAGED_RULE_GROUP",
     "action": "BLOCK",
     "httpSourceName": "ALB",
     "httpSourceId": "123456789012-app/ecommerce-alb/abc123",
     "ruleGroupList": [{
       "ruleGroupId": "AWSManagedRulesSQLiRuleSet",
       "terminatingRule": {
         "ruleId": "SQLi_QUERYARGUMENTS",
         "action": "BLOCK"
       },
       "nonTerminatingMatchingRules": [],
       "excludedRules": null
     }],
     "httpRequest": {
       "clientIp": "203.0.113.50",
       "country": "US",
       "uri": "/search",
       "args": "q='+UNION+SELECT+*+FROM+users--",
       "httpMethod": "GET",
       "headers": [
         {"name": "Host", "value": "shop.example.com"},
         {"name": "User-Agent", "value": "sqlmap/1.5"}
       ]
     }
   }
   
   S3 bucket structure:
   s3://waf-logs-ecommerce/
   ├── AWSLogs/
   │   └── 123456789012/
   │       └── WAFLogs/
   │           └── us-east-1/
   │               └── ecommerce-sql-protection/
   │                   ├── 2024/09/23/10/
   │                   │   ├── 10-00.json.gz
   │                   │   ├── 10-05.json.gz
   │                   │   └── 10-10.json.gz
   
   Athena query to analyze attacks:
   CREATE EXTERNAL TABLE waf_logs (
     timestamp bigint,
     action string,
     httprequest struct<
       clientip:string,
       country:string,
       uri:string,
       args:string,
       httpmethod:string
     >,
     terminatingruleid string
   )
   ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
   LOCATION 's3://waf-logs-ecommerce/AWSLogs/123456789012/WAFLogs/';
   
   -- Find top attacking IPs
   SELECT httprequest.clientip, COUNT(*) as attack_count
   FROM waf_logs
   WHERE action = 'BLOCK'
     AND terminatingruleid LIKE '%SQLi%'
     AND timestamp > (unix_timestamp() - 86400) * 1000
   GROUP BY httprequest.clientip
   ORDER BY attack_count DESC
   LIMIT 10;
   
   Output:
   clientip         attack_count
   203.0.113.50     15,234
   198.51.100.22    8,901
   192.0.2.100      5,432

Cost Analysis (10,000 req/sec):

WAF Pricing (us-east-1):
   Web ACL: $5.00/month
   Rules: $1.00/month per rule × 3 rules = $3.00/month
   Requests: $0.60 per 1 million requests
   
   Requests/month:
   10,000 req/sec × 86,400 sec/day × 30 days = 25.92 billion requests
   25,920 million × $0.60 = $15,552/month
   
   Total: $5 + $3 + $15,552 = $15,560/month = $186,720/year

S3 Logging Cost:
   Log size: ~1 KB per request (blocked only, ~5% of traffic)
   Blocked: 25.92B × 0.05 = 1.296B requests
   Storage: 1.296B × 1 KB = 1.296 TB
   S3 cost: 1,296 GB × $0.023/GB = $29.81/month
   
   Lifecycle policy (delete after 90 days):
   Average storage: 1,296 GB × 3 months / 12 = 324 GB average
   Cost: 324 GB × $0.023 = $7.45/month

Total Monthly Cost: $15,560 + $7.45 = $15,567.45/month

Cost Optimization:
   - Sample logging: Log 10% of requests (saves 90% on S3)
   - New cost: $0.75/month for S3
   - Total: $15,560.75/month

Why A is Wrong (Default SQLi Rules Without Rate Limiting):
 No Rate Limit Protection:
   Attack scenario:
   1. Attacker uses SQLmap tool
   2. Tool tries 10,000 different SQLi payloads
   3. Even if all blocked, consumes resources:
      - WAF evaluation: 10,000 evaluations
      - ALB processing: 10,000 requests forwarded to WAF
      - Cost: 10,000 × $0.60/million = $0.006 (minimal)
      - But: ALB bandwidth, CloudWatch metrics add up
   
   With rate limiting:
   - First 2000 requests: Evaluated
   - Request 2001+: Blocked at WAF (no SQLi evaluation needed)
   - Saves: ALB bandwidth, backend protection

 No Logging:
   Without logs, you can't:
   - Identify attacking IPs (can't block at network level)
   - Understand attack patterns (improve rules)
   - Compliance reporting (PCI DSS 10.2 requires logging)
   - Incident response (who attacked, when, from where)

When default rules sufficient:
   Low-traffic site (<1000 req/sec)
   Non-critical application
   Limited security requirements
   Cost-constrained (<$100/month)

Why B is Wrong (Blocking All Special Characters):
 Massive False Positives:

   Legitimate searches blocked:
   - "O'Reilly books" (contains ')
   - "Rock & Roll" (contains &)
   - "Cost <$50" (contains < and $)
   - "Jack's Diner" (contains ')
   - "C++ Programming" (contains +)
   - "AT&T service" (contains &)
   
   Customer impact:
   - 30-50% of searches blocked (estimated)
   - Frustration: "Why can't I search for books?"
   - Lost sales: Can't find products
   - Support tickets: "Your site is broken"

 Doesn't Block Encoded Attacks:
   Attacker can URL-encode payloads:
   ' OR '1'='1     →     %27%20OR%20%271%27%3D%271
   
   Your rule: Blocks literal '
   Encoded attack: No literal ' (blocked? NO!)
   
   WAF must decode THEN check (TextTransformations)
   Your simple rule: Doesn't decode

 Bypassable:
   SQL injection without quotes:
   - ?q=admin OR 1=1 (no quotes needed!)
   - ?q=1 UNION SELECT CHAR(97,100,109,105,110) (uses CHAR function)
   - ?q=admin OR TRUE (boolean without quotes)

Real-world example - UK Police Website (2013):
   Problem: Blocked apostrophes to prevent SQLi
   Result: Residents named O'Brien, D'Angelo couldn't report crimes
   Outcome: Had to whitelist certain names (complex)
   Better solution: Parameterized queries (application fix) + WAF

Why D is Wrong (Lambda@Edge for Every Request):
 Performance Penalty:
   Lambda@Edge execution time: 10-50ms per request
   Direct to origin: 5ms
   Overhead: 2-10× slower
   
   At 10,000 req/sec:
   - Total Lambda invocations: 864 million/day
   - User experience: Noticeably slower (10-50ms added latency)

 Cost Explosion:
   Lambda@Edge pricing:
   - $0.60 per 1 million requests
   - $0.00005001 per GB-second (memory)
   
   Assuming 128 MB, 20ms average:
   Requests: 25.92B/month × $0.60/million = $15,552/month
   Compute: 25.92B × 0.020 sec × 0.125 GB × $0.00005001 = $3,240/month
   
   Total: $18,792/month
   
   vs WAF: $15,560/month
   
   Delta: $3,232/month more expensive (21% costlier)

 Complexity:
   Lambda@Edge code (must write and maintain):
   - SQL injection regex patterns (100+ patterns)
   - URL decoding logic
   - False positive handling
   - Testing across all edge locations
   - Deployment: 15-30 minutes globally
   
   WAF Managed Rules:
   - AWS maintains patterns (updated automatically)
   - No code to write
   - Deployment: Instant

 Regional Limitations:
   Lambda@Edge: Available in limited regions only
   WAF: Available in all regions

When Lambda@Edge appropriate:
   Complex custom logic (e.g., A/B testing)
   Request/response modification beyond WAF
   Low traffic (<100 req/sec)
   Accept latency overhead

Advanced WAF Techniques:

1. Geo-Blocking (if attacks from specific countries):
aws wafv2 create-rule \
  --name BlockCountries \
  --statement '{"GeoMatchStatement":{"CountryCodes":["CN","RU"]}}' \
  --action Block={}

Use case: 99% of attacks from CN/RU, zero customers there
Saves: WAF request costs (blocks before SQLi evaluation)

2. IP Reputation Lists (AWS Managed):
{
  "Name": "AWSManagedRulesAmazonIpReputationList",
  "Priority": 0,
  "Statement": {
    "ManagedRuleGroupStatement": {
      "VendorName": "AWS",
      "Name": "AWSManagedRulesAmazonIpReputationList"
    }
  },
  "OverrideAction": {"None": {}}
}

Blocks: Known malicious IPs (botnets, scanners)
Updated: Automatically by AWS

3. Custom Response (friendly error page):
"Action": {
  "Block": {
    "CustomResponse": {
      "ResponseCode": 403,
      "CustomResponseBodyKey": "blocked-response"
    }
  }
}

Custom response body:
{
  "blocked-response": {
    "ContentType": "TEXT_HTML",
    "Content": "<html><body><h1>Request Blocked</h1><p>Your request was blocked by our security system. If you believe this is an error, contact support@example.com with reference ID: {REQUEST_ID}</p></body></html>"
  }
}

User experience: Clear message instead of generic 403

4. WAF Captcha (AWS WAF CAPTCHA):
For ambiguous requests (might be bot, might be human):
"Action": {
  "Captcha": {
    "CustomRequestHandling": {
      "InsertHeaders": [{
        "Name": "X-WAF-Captcha",
        "Value": "Required"
      }]
    }
  }
}

Use case: Rate limit exceeded, but not definite attack
User: Solves CAPTCHA to continue
Bot: Can't solve, blocked

Testing WAF Rules:

1. Count Mode (before blocking):
"Action": {"Count": {}}

Result: Logs matches but doesn't block
Use: Test rules before production (measure false positives)

# Run for 7 days in Count mode
# Analyze CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/WAFV2 \
  --metric-name CountedRequests \
  --dimensions Name=Rule,Value=SQLiRuleSet Name=WebACL,Value=ecommerce-sql-protection \
  --start-time 2024-09-16T00:00:00Z \
  --end-time 2024-09-23T00:00:00Z \
  --period 86400 \
  --statistics Sum

If 0 false positives after 7 days → Switch to Block mode

2. Test Requests:
# Simulate attack (should be blocked)
curl -X GET "https://shop.example.com/search?q='+UNION+SELECT+*+FROM+users--" \
  -H "User-Agent: sqlmap/1.5"

Expected response: 403 Forbidden

# Legitimate search (should be allowed)
curl -X GET "https://shop.example.com/search?q=O%27Reilly+books" \
  -H "User-Agent: Mozilla/5.0"

Expected response: 200 OK

3. Sampled Requests (WAF Console):
Shows:
- Last 3 hours of requests
- Which rule matched
- Full request details
- Action taken (Block, Allow, Count)

Monitoring & Alerts:

CloudWatch Alarm for high block rate:
aws cloudwatch put-metric-alarm \
  --alarm-name WAF-High-Block-Rate \
  --alarm-description "Alert when WAF blocks >1000 requests in 5 minutes" \
  --metric-name BlockedRequests \
  --namespace AWS/WAFV2 \
  --statistic Sum \
  --period 300 \
  --threshold 1000 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=Rule,Value=All Name=WebACL,Value=ecommerce-sql-protection \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:security-alerts

When alarm triggers:
1. SNS notification to security team
2. Check WAF logs for attack details
3. Consider additional countermeasures:
   - Block attacking IPs at Network Firewall
   - Enable Shield Advanced for DDoS protection
   - Rate limit more aggressively

Real-World Example - Large E-commerce Site:
Company: Shopify (estimated, public information)
Scale: 1M+ merchants, 10K+ req/sec per large merchant
WAF: Enabled on all ALBs

Configuration:
- Core managed rules: SQLi, XSS, known bad inputs
- Rate limiting: 5000 req/5min per IP per merchant
- Geo-blocking: None (global customers)
- Logging: 1% sample (cost optimization)
- Cost: ~$20,000/month for WAF ($240K/year)

Metrics:
- Requests/month: 25 billion+
- Blocked: 5% (1.25 billion malicious requests)
- False positives: <0.01% (2.5 million legitimate blocked)
- False positive rate: 0.01% acceptable (1 in 10,000)

Incident response:
- Attack detected: CloudWatch alarm triggers
- Investigation: Athena query on WAF logs
- Response: IP-based blocking, rule tuning
- Resolution time: <15 minutes

ROI:
- SQL injection breach cost: $4M+ (IBM 2023 Cost of Breach)
- WAF cost: $240K/year
- Breaches prevented: Estimated 50+/year
- ROI: $200M+ saved / $240K spent = 833× return

Best Practices:

1. Start with managed rules (don't reinvent the wheel)
2. Enable logging (at least 1% sample for cost)
3. Use Count mode first (test rules before blocking)
4. Monitor false positives (review sampled requests weekly)
5. Rate limiting per endpoint (not global)
6. Custom error pages (better user experience)
7. Regular rule reviews (quarterly)
8. Document exceptions (why certain rules excluded)
9. Integrate with SIEM (Splunk, Sumo Logic)
10. Test DR: What if WAF fails? (should fail open, not block all traffic)

Why C is Correct (Global Accelerator + Multi-Region + Shield Advanced):

Complete Architecture:

┌─────────────────────────────────────────────────────────────────┐
│                     AWS Global Accelerator                       │
│  Anycast IPs: 3.125.23.144, 75.2.76.100 (2 static IPs)         │
│  DDoS Protection: AWS Shield Advanced (automatic mitigation)     │
└─────────────────────────────────────────────────────────────────┘
                           ↓
         ┌─────────────────┼─────────────────┐
         ↓                 ↓                  ↓
   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
   │  us-east-1  │   │  eu-west-1  │   │ ap-south-1  │
   │  (Primary)  │   │  (Europe)   │   │   (Asia)    │
   ├─────────────┤   ├─────────────┤   ├─────────────┤
   │ NLB (60%)   │   │ NLB (30%)   │   │  NLB (10%)  │
   │ 50 EC2 inst │   │ 30 EC2 inst │   │  15 EC2 inst│
   │ RDS Primary │   │ RDS Replica │   │ RDS Replica │
   │ ElastiCache │   │ ElastiCache │   │ ElastiCache │
   └─────────────┘   └─────────────┘   └─────────────┘
          ↓                 ↓                  ↓
   ┌─────────────────────────────────────────────────┐
   │         Aurora Global Database                   │
   │  Primary: us-east-1  |  Replicas: eu-west-1,    │
   │  RPO: <1 sec  |  RTO: <1 minute                 │
   └─────────────────────────────────────────────────┘

Traffic Flow (Normal Operation):
1. User (New York) → DNS: app.example.com
2. DNS returns: Global Accelerator Anycast IP
3. User → Nearest AWS Edge Location (NYC)
4. Global Accelerator routes to: us-east-1 (60% traffic)
5. NLB distributes to: 50 EC2 instances
6. Response path: EC2 → NLB → Global Accelerator → User

Traffic Flow (us-east-1 Under Attack):
1. DDoS attack: 1 Tbps targeting us-east-1
2. Shield Advanced detects: Volumetric attack (layer 3/4)
3. Automatic mitigation: Enabled within 30 seconds
4. Global Accelerator health checks: us-east-1 unhealthy (high latency)
5. Automatic failover: Traffic redirected to eu-west-1 + ap-south-1
6. New distribution: eu-west-1 (70%), ap-south-1 (30%)
7. User experience: Slight latency increase (+50ms), but service available

Terraform Configuration:

# 1. Global Accelerator
resource "aws_globalaccelerator_accelerator" "main" {
  name            = "saas-platform-accelerator"
  ip_address_type = "IPV4"
  enabled         = true
  
  attributes {
    flow_logs_enabled   = true
    flow_logs_s3_bucket = aws_s3_bucket.flow_logs.id
    flow_logs_s3_prefix = "global-accelerator/"
  }
}

# 2. Listener (Port 443 HTTPS)
resource "aws_globalaccelerator_listener" "https" {
  accelerator_arn = aws_globalaccelerator_accelerator.main.id
  protocol        = "TCP"
  port_ranges {
    from_port = 443
    to_port   = 443
  }
}

# 3. Endpoint Groups (Multi-Region)
resource "aws_globalaccelerator_endpoint_group" "us_east_1" {
  listener_arn = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "us-east-1"
  traffic_dial_percentage = 60
  
  health_check_interval_seconds = 10
  health_check_path             = "/health"
  health_check_port             = 443
  health_check_protocol         = "HTTPS"
  threshold_count               = 2
  
  endpoint_configuration {
    endpoint_id = aws_lb.us_east_1_nlb.arn
    weight      = 100
    client_ip_preservation_enabled = true
  }
}

resource "aws_globalaccelerator_endpoint_group" "eu_west_1" {
  listener_arn = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "eu-west-1"
  traffic_dial_percentage = 30
  
  health_check_interval_seconds = 10
  health_check_path             = "/health"
  health_check_port             = 443
  health_check_protocol         = "HTTPS"
  threshold_count               = 2
  
  endpoint_configuration {
    endpoint_id = aws_lb.eu_west_1_nlb.arn
    weight      = 100
    client_ip_preservation_enabled = true
  }
}

resource "aws_globalaccelerator_endpoint_group" "ap_south_1" {
  listener_arn = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "ap-south-1"
  traffic_dial_percentage = 10
  
  health_check_interval_seconds = 10
  health_check_path             = "/health"
  health_check_port             = 443
  health_check_protocol         = "HTTPS"
  threshold_count               = 2
  
  endpoint_configuration {
    endpoint_id = aws_lb.ap_south_1_nlb.arn
    weight      = 100
    client_ip_preservation_enabled = true
  }
}

# 4. Shield Advanced Protection
resource "aws_shield_protection" "accelerator" {
  name         = "saas-platform-shield"
  resource_arn = aws_globalaccelerator_accelerator.main.id
}

# 5. WAF Web ACL (DDoS Layer 7 protection)
resource "aws_wafv2_web_acl" "ddos_protection" {
  name  = "ddos-layer7-protection"
  scope = "CLOUDFRONT"
  
  default_action {
    allow {}
  }
  
  rule {
    name     = "RateLimitRule"
    priority = 1
    
    statement {
      rate_based_statement {
        limit              = 10000
        aggregate_key_type = "IP"
      }
    }
    
    action {
      block {}
    }
    
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "RateLimitRule"
      sampled_requests_enabled   = true
    }
  }
  
  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "DDosProtectionWAF"
    sampled_requests_enabled   = true
  }
}

Global Accelerator Key Features:

1. Anycast IP Addresses:
   Traditional routing (Unicast):
   User (NYC) → DNS lookup → Closest server IP (us-east-1)
   Problem: DDoS attack on that IP → entire IP unreachable
   
   Anycast routing:
   User (NYC) → Global Accelerator IP (3.125.23.144)
   AWS routes to: Nearest edge location (20+ locations)
   Edge location routes to: Healthy endpoint (us-east-1 or failover)
   
   DDoS benefit:
   - Attack on us-east-1: Only affects that region's routing
   - Anycast IP: Still reachable from other edge locations
   - Traffic automatically flows to healthy regions

2. Automatic Health Checks:
   Configuration:
   - Interval: 10 seconds (vs Route 53: 30 seconds)
   - Threshold: 2 consecutive failures (20 seconds total)
   - Health check: HTTPS GET /health
   - Expected: 200 OK response
   
   Failure scenarios:
   Scenario 1: All EC2 instances unhealthy
   - NLB returns: 503 Service Unavailable
   - Global Accelerator marks: us-east-1 unhealthy
   - Action: Stop sending traffic to us-east-1
   
   Scenario 2: Network isolation (DDoS attack)
   - Health check: Times out (no response)
   - After 20 seconds: Endpoint marked unhealthy
   - Action: Failover to eu-west-1 + ap-south-1
   
   Scenario 3: High latency (>500ms)
   - Health check: Slow response (600ms)
   - Depends on: health_check_timeout_seconds setting
   - If timeout=5 sec: Still considered healthy
   - If timeout=0.5 sec: Marked unhealthy

3. Traffic Dials (Weighted Traffic Distribution):
   Normal operation:
   - us-east-1: 60% (closest to 60% of users)
   - eu-west-1: 30% (closest to 30% of users)
   - ap-south-1: 10% (closest to 10% of users)
   
   During us-east-1 failure:
   - us-east-1: 0% (unhealthy, dial=0 automatically)
   - eu-west-1: 75% (30% / (30%+10%) × 100)
   - ap-south-1: 25% (10% / (30%+10%) × 100)
   
   Gradual traffic shift (Blue-Green deployment):
   Phase 1: us-east-1=100%, eu-west-1=0%, ap-south-1=0%
   Phase 2: us-east-1=90%, eu-west-1=10%, ap-south-1=0%
   Phase 3: us-east-1=60%, eu-west-1=30%, ap-south-1=10%
   
   Use case: Test new region before full traffic

4. Client IP Preservation:
   Without preservation:
   Application sees: Global Accelerator IP (internal AWS)
   Problem: Can't identify user IP (no geolocation, no IP-based auth)
   
   With preservation (client_ip_preservation_enabled=true):
   Application sees: Original user IP (203.0.113.50)
   Benefit: Geolocation works, IP-based rate limiting works
   
   How it works: X-Forwarded-For header + preserve source IP at NLB

Shield Advanced Features:

1. Always-On DDoS Detection:
   Layer 3 (Network): SYN flood, UDP reflection
   Layer 4 (Transport): Connection exhaustion
   Layer 7 (Application): HTTP flood, Slowloris
   
   Detection methods:
   - Baseline traffic: ML model learns normal traffic patterns
   - Anomaly detection: Identifies deviations (10× normal traffic)
   - Signature-based: Known attack patterns (Mirai botnet)

2. Automatic Mitigation:
   When attack detected:
   1. Shield Advanced engages: Within 30 seconds
   2. Mitigation techniques:
      - Traffic scrubbing: Filter malicious packets
      - Rate limiting: Limit requests per IP
      - Geo-blocking: Block countries not in whitelist
      - Challenge-response: CAPTCHA for suspicious clients
   3. Legitimate traffic: Continues flowing (minimal impact)
   
   Mitigation SLA:
   - Time to mitigate: <30 seconds (Shield Advanced)
   - vs Shield Standard: Minutes to hours (manual)

3. DDoS Response Team (DRT):
   Included with Shield Advanced:
   - 24/7 support: Call AWS DRT during attack
   - Expert assistance: DDoS mitigation engineers
   - Proactive engagement: DRT monitors, suggests mitigations
   - WAF rule updates: DRT can update WAF rules on your behalf
   
   Cost: Included (no additional charge beyond Shield Advanced)

4. Cost Protection:
   Shield Advanced includes:
   - DDoS-related cost spikes: Waived by AWS
   - Example: Attack causes autoscaling to 500 instances
   - Normal cost: $50,000 extra for 6-hour attack
   - With Shield Advanced: $0 (costs waived)
   
   Eligibility:
   - Must be Shield Advanced subscriber ($3,000/month)
   - Attack must be verified DDoS (not legitimate traffic spike)
   - Submit cost protection claim within 30 days

Cost Analysis (Multi-Region + Shield Advanced):

Global Accelerator:
   Fixed fee: $0.025/hour = $18/month
   Data transfer:
     Inbound (internet to AWS): Free
     Outbound (AWS to internet): $0.015/GB (premium tier)
   
   Traffic: 50M users, average 10 MB/user/month = 500 TB/month
   Outbound: 500 TB × 1024 GB × $0.015 = $7,680/month
   
   Total Global Accelerator: $18 + $7,680 = $7,698/month

Shield Advanced:
   Subscription: $3,000/month (covers up to 100 resources)
   Resources protected:
     1× Global Accelerator
     3× NLBs (us-east-1, eu-west-1, ap-south-1)
     3× CloudFront distributions (for static assets)
     Total: 7 resources (well under 100 limit)
   
   DDoS Response Team: Included
   Cost protection: Included
   
   Total Shield Advanced: $3,000/month

WAF (Layer 7 protection):
   Web ACL: $5/month
   Rules: 1 rule × $1 = $1/month
   Requests: $0.60 per 1M requests
   
   Requests/month: 50M users × 100 requests/month = 5B requests
   WAF cost: 5,000 million × $0.60 = $3,000/month
   
   Total WAF: $3,006/month

Multi-Region Infrastructure (existing costs, not DDoS-specific):
   EC2 (95 instances total): ~$20,000/month
   RDS Aurora Global: ~$5,000/month
   ElastiCache: ~$2,000/month
   Data transfer (inter-region): ~$1,000/month
   
   Total Infrastructure: ~$28,000/month (baseline)

DDoS Protection Cost Summary:
   Global Accelerator: $7,698/month
   Shield Advanced: $3,000/month
   WAF: $3,006/month
   
   Total DDoS Protection: $13,704/month
   
   Budget: $50,000/month
   Remaining: $36,296/month (73% under budget) 

ROI:
   Outage cost: $2M per 6-hour attack
   Shield Advanced prevents: Estimated 5 attacks/year
   Savings: $10M/year
   Protection cost: $13,704 × 12 = $164,448/year
   ROI: $10M / $164K = 60.9× return

Why A is Wrong (CloudFront + Shield Standard + Single Region):
 Single Region is Single Point of Failure:
   Attack targets: us-east-1
   Impact: All traffic to that region affected
   Failover: None (no other regions)
   Result: Complete outage 
   
 Shield Standard Limited Protection:
   Protection: Layer 3/4 only (SYN flood, UDP reflection)
   NOT protected: Layer 7 HTTP floods
   Mitigation time: Minutes to hours (vs seconds with Advanced)
   DRT access: No (must mitigate yourself)
   
 No Cost Protection:
   Attack causes: Auto Scaling to 500 instances
   Cost: $50,000 for 6-hour attack
   Shield Standard: You pay full cost
   Shield Advanced: AWS waives cost

When single region acceptable:
   Users in one geography only (e.g., US-only service)
   Can tolerate regional outage (non-critical app)
   Cost-sensitive (<$5,000/month budget)

Why B is Wrong (Route 53 + Active-Passive + Shield Advanced):
 Active-Passive Wastes Resources:
   Active-passive:
   - Primary: us-east-1 (handles 100% traffic)
   - Passive: eu-west-1, ap-south-1 (idle, waiting for failover)
   
   Resource utilization:
   - us-east-1: 100%
   - eu-west-1: 0% (running but not serving traffic)
   - ap-south-1: 0% (running but not serving traffic)
   
   Cost: Pay for idle resources in eu-west-1 + ap-south-1
   
   Active-active (Option C):
   - us-east-1: 60% traffic
   - eu-west-1: 30% traffic
   - ap-south-1: 10% traffic
   - Resource utilization: 100% in all regions
   - Cost: Same infrastructure, but fully utilized

 Higher Latency for Non-US Users:
   European user:
   - Route 53 routes to: us-east-1 (active region)
   - Latency: 100-150ms (transatlantic)
   
   With active-active:
   - Global Accelerator routes to: eu-west-1 (closest region)
   - Latency: 10-20ms (within Europe)
   
   95th percentile latency:
   - Active-passive: 150ms (fails <200ms requirement marginally)
   - Active-active: 30ms 

 Slower Failover:
   Route 53 health checks:
   - Interval: 30 seconds (vs Global Accelerator: 10 seconds)
   - Threshold: 3 failures = 90 seconds
   - DNS TTL: 60 seconds (wait for clients to refresh)
   - Total failover time: 150 seconds (2.5 minutes)
   
   Global Accelerator:
   - Health check: 10 seconds interval
   - Threshold: 2 failures = 20 seconds
   - No DNS (Anycast routing, instant)
   - Total failover time: 20-30 seconds 

When active-passive acceptable:
   DR only (not performance requirement)
   Can tolerate 2-3 minute failover
   Cost-sensitive (can use smaller passive instances)

Why D is Wrong (Multiple CloudFront + Manual Failover):
 Manual Failover is Too Slow:
   Process:
   1. Detect outage (monitoring alerts): 2-5 minutes
   2. On-call engineer responds: 5-15 minutes (depends on time of day)
   3. Update DNS to point to backup CloudFront: 2 minutes
   4. DNS propagation (TTL): 5-60 minutes
   5. Total downtime: 15-80 minutes 
   
   Requirement: <30 seconds automated failover
   Manual failover: 15-80 minutes (30-160× slower)
   
 Human Error Risk:
   3 AM on Sunday: DDoS attack detected
   Tired engineer: Updates wrong DNS record
   Result: Complete outage extended
   
   Automated failover: No human error

 CloudFront Not Designed for Multi-Origin Failover:
   CloudFront origin groups:
   - Primary origin: us-east-1 ALB
   - Secondary origin: eu-west-1 ALB (failover)
   - Limitation: Failover triggers on 5xx errors, not DDoS
   
   DDoS scenario:
   - us-east-1 under attack: Slow responses (200 OK but slow)
   - CloudFront: Considers it healthy (no 5xx errors)
   - Result: No failover, users experience slowness

When CloudFront sufficient:
   Static content only (S3 origins with cross-region replication)
   Can tolerate manual intervention
   Low-risk application (not revenue-critical)

Operational Procedures:

1. DDoS Attack Response (Automated):
   Time 00:00: Attack begins (1 Tbps volumetric)
   Time 00:00:10: Shield Advanced detects anomaly
   Time 00:00:30: Mitigation engaged automatically
   Time 00:00:20: Global Accelerator health checks fail
   Time 00:00:30: Traffic rerouted to healthy regions
   Time 00:00:45: CloudWatch alarm triggers (SNS notification)
   Time 00:01:00: Engineer notified (low priority, already mitigated)
   
   No manual intervention needed 

2. DDoS Attack Response (Manual Escalation):
   Time 00:05: Attack continues, mitigation partially effective
   Time 00:05: Engineer calls AWS DRT (DDoS Response Team)
   Time 00:10: DRT analyzes traffic patterns
   Time 00:15: DRT updates WAF rules (block attack signatures)
   Time 00:20: Attack fully mitigated
   Time 01:00: Incident review, update runbooks

3. Failback (After Attack Ends):
   Automatic:
   Time 06:00: Attack ends
   Time 06:00:30: us-east-1 health checks pass (2 consecutive)
   Time 06:01:00: Global Accelerator gradually restores traffic
     - Minute 1: us-east-1 receives 10% traffic
     - Minute 5: us-east-1 receives 30% traffic
     - Minute 10: us-east-1 receives 60% traffic (normal)
   
   Gradual failback prevents: Thundering herd problem

Monitoring & Dashboards:

CloudWatch Dashboard (DDoS Monitoring):

Metrics:
1. Global Accelerator:
   - ProcessedBytesIn: Traffic volume (detect volumetric attacks)
   - NewFlowCount: New connections per second (detect SYN flood)
   
2. Shield Advanced:
   - DDoSDetected: Binary (0=no attack, 1=attack detected)
   - DDoSAttackBitsPerSecond: Attack volume in bps
   - DDoSAttackPacketsPerSecond: Attack volume in pps
   
3. NLB:
   - ActiveFlowCount: Concurrent connections (detect connection exhaustion)
   - HealthyHostCount: Number of healthy targets
   - UnHealthyHostCount: Number of unhealthy targets
   
4. WAF:
   - BlockedRequests: Requests blocked by WAF rules
   - AllowedRequests: Requests allowed
   - CountedRequests: Requests in count mode (testing)

Alarms:
1. High Attack Volume:
   DDoSAttackBitsPerSecond > 10 Gbps for 1 minute
   Action: SNS notification (low priority, auto-mitigated)
   
2. Region Unhealthy:
   HealthyHostCount = 0 for 1 minute
   Action: SNS notification + PagerDuty (high priority)
   
3. Failover Occurred:
   Custom metric: us-east-1 traffic dial = 0
   Action: SNS notification (informational)

Real-World Example - Gaming Platform:
Company: Riot Games (estimated, public information)
Scale: 100M+ players worldwide
DDoS history: Frequent attacks (competitive gaming attracts attackers)

Architecture:
   Global Accelerator: 5 regions (NA, EU, Asia, SA, OCE)
   Shield Advanced: Enabled on all resources
   WAF: Custom rules for gaming-specific attacks
   Cost: Estimated $100K+/month for DDoS protection

Attack incident (2020):
   Attack: 1.2 Tbps DDoS targeting League of Legends
   Duration: 8 hours
   Shield Advanced: Mitigated automatically
   Player impact: Minimal (< 50ms latency increase)
   Downtime: Zero
   
   Without Shield Advanced:
   - Estimated downtime: 4-8 hours
   - Player churn: 5-10% (players leave after bad experience)
   - Revenue loss: $10M+ (skins, battle passes)

Comparison Table:

Feature                   | Option A    | Option B       | Option C (Correct) | Option D
Single point of failure   | Yes (us-1)  | No (multi-reg) | No (multi-region)  | Yes (manual)
Automatic failover        | No          | Yes (90 sec)   | Yes (30 sec)      | No (manual)
DDoS protection level     | Basic (L3)  | Advanced       | Advanced          | Basic
Multi-region active       | No          | No (passive)   | Yes               | No
Cost protection           | No          | Yes            | Yes               | No
DRT access                | No          | Yes            | Yes               | No
Latency (95th %)          | 100ms       | 150ms          | 30ms              | 100ms
Monthly cost              | $5K         | $20K           | $13.7K            | $5K
Meets requirements        |          | (latency)   |                 | 

Why B is Correct (PrivateLink + Security Groups + IAM + VPC Endpoints):

Zero Trust Architecture:

┌──────────────────────────────────────────────────────────────────┐
│                      Identity Provider                            │
│  AWS SSO / Okta / Azure AD (SAML 2.0 federation)                │
│  MFA Required for all users                                      │
└──────────────────────────────────────────────────────────────────┘
                           ↓
┌──────────────────────────────────────────────────────────────────┐
│                    IAM Identity Center                            │
│  Permission Sets: Least privilege access                         │
│  Session duration: 1 hour (re-authenticate)                      │
└──────────────────────────────────────────────────────────────────┘
                           ↓
        ┌──────────────────┴──────────────────┐
        ↓                                      ↓
┌──────────────────┐                  ┌──────────────────┐
│  Application VPC  │                  │   Services VPC   │
│                   │                  │                  │
│ ┌──────────────┐ │                  │ ┌──────────────┐ │
│ │   Lambda     │ │  AWS PrivateLink │ │   RDS        │ │
│ │ (App Logic)  │ ├──────────────────┤ │ (Database)   │ │
│ └──────────────┘ │  Endpoint        │ └──────────────┘ │
│                   │  Service         │                  │
│ IAM Role:         │ ←──────────────→ │ Security Group:  │
│ - S3 read only    │  No internet     │ - Port 5432      │
│ - KMS decrypt     │  No VPC peering  │ - Source: Lambda │
│                   │                  │   SG only        │
└──────────────────┘                  └──────────────────┘
        ↓                                      ↓
┌──────────────────┐                  ┌──────────────────┐
│ VPC Endpoint (S3)│                  │  CloudTrail      │
│ - No NAT Gateway │                  │  Every API call  │
│ - No IGW         │                  │  logged to S3    │
└──────────────────┘                  └──────────────────┘

Zero Trust Principles Implementation:

1. Never Trust, Always Verify:
   Traditional network: "Inside network = trusted"
   Zero Trust: "All requests authenticated, even internal"
   
   Example - Lambda calling RDS:
   Traditional:
   - Lambda in VPC → RDS in same VPC
   - Assumption: Network location = trust
   - Security: Security Group allows "all from VPC"
   
   Zero Trust:
   - Lambda in VPC → RDS in separate VPC
   - No VPC peering (forces authentication)
   - Lambda → PrivateLink endpoint → RDS
   - IAM role: Verified at RDS (IAM database authentication)
   - Security Group: Explicit allow (Lambda SG only, not "all VPC")
   
   Code (Lambda connecting to RDS with IAM auth):
   import boto3
   import pymysql
   
   def lambda_handler(event, context):
       # Get IAM auth token (verifies Lambda's IAM role)
       rds = boto3.client('rds')
       token = rds.generate_db_auth_token(
           DBHostname='db-cluster.cluster-abc.us-east-1.rds.amazonaws.com',
           Port=5432,
           DBUsername='lambda_user',
           Region='us-east-1'
       )
       
       # Connect using token (not static password)
       conn = pymysql.connect(
           host='db-cluster.cluster-abc.us-east-1.rds.amazonaws.com',
           user='lambda_user',
           password=token,  # IAM token, expires in 15 minutes
           database='financial_db',
           ssl={'ssl': True}  # Enforce TLS
       )
       
       # Every connection authenticated via IAM
       # CloudTrail logs: Who, what, when
   
   Benefits:
   - No static passwords (rotated automatically)
   - IAM policies control who can connect
   - CloudTrail logs every connection attempt
   - Token expires in 15 minutes (limit blast radius)

2. Micro-Segmentation:
   Traditional: Flat network, "allow all" within VPC
   Zero Trust: Every workload isolated, explicit allow only
   
   Security Group Strategy:
   
   # Application tier
   sg-app (Lambda functions):
     Outbound:
       - Port 5432 to sg-db (PostgreSQL) 
       - Port 443 to VPC endpoint (S3) 
       - DENY all other outbound 
   
   # Database tier
   sg-db (RDS):
     Inbound:
       - Port 5432 from sg-app ONLY 
       - DENY all other inbound 
     Outbound:
       - DENY all (database doesn't initiate connections) 
   
   # No implicit trust:
   - Web tier CANNOT access database (no rule)
   - Database CANNOT access S3 (no rule)
   - Lambda CANNOT access internet (no NAT Gateway)
   
   Terraform example:
   resource "aws_security_group" "app" {
     name   = "app-tier-sg"
     vpc_id = aws_vpc.app.id
     
     egress {
       description = "PostgreSQL to database tier"
       from_port   = 5432
       to_port     = 5432
       protocol    = "tcp"
       security_groups = [aws_security_group.db.id]
     }
     
     egress {
       description = "HTTPS to S3 via VPC endpoint"
       from_port   = 443
       to_port     = 443
       protocol    = "tcp"
       prefix_list_ids = [aws_vpc_endpoint.s3.prefix_list_id]
     }
     
     # NO default "allow all" egress rule
   }
   
   resource "aws_security_group" "db" {
     name   = "db-tier-sg"
     vpc_id = aws_vpc.db.id
     
     ingress {
       description = "PostgreSQL from app tier only"
       from_port   = 5432
       to_port     = 5432
       protocol    = "tcp"
       security_groups = [aws_security_group.app.id]
     }
     
     # NO egress rules (database doesn't initiate connections)
     # NO default "allow all" egress
   }
   
   Result:
   - Lateral movement: Impossible (no implicit allow)
   - Compromised Lambda: Can only access database + S3 (limited blast radius)
   - Compromised database: Cannot access anything (no outbound)

3. Least Privilege IAM:
   Traditional: Broad permissions ("s3:*")
   Zero Trust: Minimum required permissions, time-limited
   
   Example - Lambda IAM role:
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "ReadSpecificS3Bucket",
         "Effect": "Allow",
         "Action": [
           "s3:GetObject",
           "s3:ListBucket"
         ],
         "Resource": [
           "arn:aws:s3:::financial-data",
           "arn:aws:s3:::financial-data/transactions/*"
         ],
         "Condition": {
           "StringEquals": {
             "s3:x-amz-server-side-encryption": "aws:kms"
           }
         }
       },
       {
         "Sid": "DecryptSpecificKMSKey",
         "Effect": "Allow",
         "Action": ["kms:Decrypt"],
         "Resource": "arn:aws:kms:us-east-1:123456789012:key/abc-123",
         "Condition": {
           "StringEquals": {
             "kms:ViaService": "s3.us-east-1.amazonaws.com"
           }
         }
       },
       {
         "Sid": "ConnectToDatabase",
         "Effect": "Allow",
         "Action": ["rds-db:connect"],
         "Resource": "arn:aws:rds-db:us-east-1:123456789012:dbuser:cluster-abc/lambda_user"
       }
     ]
   }
   
   Principle: Specific actions, specific resources, conditions
   NOT: s3:*, kms:*, rds:* (too broad)

4. AWS PrivateLink (Service-to-Service Without Internet):
   Traditional: VPC peering or internet
   Zero Trust: PrivateLink (private, scalable, no route propagation)
   
   Use case: Application VPC → RDS in Services VPC
   
   Without PrivateLink (wrong approach):
   Option 1: VPC peering
   - Creates route table entries (app VPC can reach ALL of services VPC)
   - Violates micro-segmentation (too much trust)
   
   Option 2: Internet
   - Must traverse internet gateway
   - Even with TLS, violates "zero trust" (external routing)
   
   With PrivateLink:
   1. Create endpoint service (Services VPC):
      - Expose NLB fronting RDS Proxy
      - Principal: Application VPC account (IAM condition)
   
   2. Create interface endpoint (Application VPC):
      - Connects to endpoint service
      - Private IP in Application VPC (10.1.0.50)
      - Lambda connects to: 10.1.0.50:5432
   
   3. Traffic flow:
      Lambda → Interface endpoint → PrivateLink → NLB → RDS Proxy → RDS
      
      All traffic: Private AWS network (never internet)
      No route tables: Endpoint is just an IP (no CIDR propagation)
      IAM-gated: Endpoint service checks Lambda's IAM role
   
   Terraform configuration:
   # Services VPC (where RDS lives)
   resource "aws_vpc_endpoint_service" "rds" {
     acceptance_required        = false
     network_load_balancer_arns = [aws_lb.rds_proxy.arn]
     allowed_principals         = [
       "arn:aws:iam::123456789012:root"  # Application account
     ]
   }
   
   # Application VPC (where Lambda lives)
   resource "aws_vpc_endpoint" "rds" {
     vpc_id              = aws_vpc.app.id
     service_name        = aws_vpc_endpoint_service.rds.service_name
     vpc_endpoint_type   = "Interface"
     subnet_ids          = [aws_subnet.app_private.id]
     security_group_ids  = [aws_security_group.app.id]
     private_dns_enabled = true
   }

5. VPC Endpoints (AWS Services Without Internet):
   Traditional: Lambda → NAT Gateway → Internet Gateway → S3 (public API)
   Zero Trust: Lambda → VPC Endpoint → S3 (private connection)
   
   Gateway Endpoints (S3, DynamoDB):
   # Create gateway endpoint
   resource "aws_vpc_endpoint" "s3" {
     vpc_id       = aws_vpc.app.id
     service_name = "com.amazonaws.us-east-1.s3"
     route_table_ids = [aws_route_table.private.id]
   }
   
   # Endpoint policy (limit to specific bucket)
   resource "aws_vpc_endpoint_policy" "s3" {
     vpc_endpoint_id = aws_vpc_endpoint.s3.id
     policy = jsonencode({
       Statement = [{
         Effect = "Allow"
         Principal = "*"
         Action = ["s3:GetObject", "s3:PutObject"]
         Resource = "arn:aws:s3:::financial-data/*"
       }]
     })
   }
   
   Benefits:
   - No NAT Gateway ($0.045/hour = $32.40/month saved)
   - No Internet Gateway (attack surface reduced)
   - Traffic stays on AWS backbone (never internet)
   - Endpoint policy: Defense in depth (even if IAM misconfigured)
   
   Interface Endpoints (most AWS services):
   Services: Lambda, SNS, SQS, Secrets Manager, Systems Manager
   
   Example - Secrets Manager endpoint:
   resource "aws_vpc_endpoint" "secrets" {
     vpc_id              = aws_vpc.app.id
     service_name        = "com.amazonaws.us-east-1.secretsmanager"
     vpc_endpoint_type   = "Interface"
     subnet_ids          = [aws_subnet.app_private.id]
     security_group_ids  = [aws_security_group.endpoints.id]
     private_dns_enabled = true
   }
   
   Lambda code (no change needed):
   import boto3
   secrets = boto3.client('secretsmanager')
   
   # SDK automatically uses VPC endpoint (via private DNS)
   secret = secrets.get_secret_value(SecretId='db-password')

6. CloudTrail (Audit All Access):
   Zero Trust requirement: "Trust but verify" → verify via audit logs
   
   CloudTrail configuration:
   resource "aws_cloudtrail" "main" {
     name                          = "zero-trust-audit"
     s3_bucket_name                = aws_s3_bucket.cloudtrail.id
     include_global_service_events = true
     is_multi_region_trail         = true
     enable_log_file_validation    = true  # Cryptographically signed
     
     event_selector {
       read_write_type           = "All"
       include_management_events = true
       
       data_resource {
         type   = "AWS::S3::Object"
         values = ["arn:aws:s3:::financial-data/"]
       }
       
       data_resource {
         type   = "AWS::Lambda::Function"
         values = ["arn:aws:lambda:*:*:function/*"]
       }
     }
     
     insight_selector {
       insight_type = "ApiCallRateInsight"  # Detect unusual API activity
     }
   }
   
   Example log entry (Lambda accessing S3):
   {
     "eventTime": "2024-09-23T10:30:00Z",
     "eventName": "GetObject",
     "userIdentity": {
       "type": "AssumedRole",
       "principalId": "AIDAI...EXAMPLE:lambda-execution",
       "arn": "arn:aws:sts::123456789012:assumed-role/LambdaExecutionRole/lambda-fn",
       "sessionContext": {
         "sessionIssuer": {
           "type": "Role",
           "arn": "arn:aws:iam::123456789012:role/LambdaExecutionRole"
         }
       }
     },
     "eventSource": "s3.amazonaws.com",
     "requestParameters": {
       "bucketName": "financial-data",
       "key": "transactions/2024-09-23.csv"
     },
     "responseElements": null,
     "sourceIPAddress": "10.1.0.25",  # Lambda's private IP
     "vpcEndpointId": "vpce-abc123"    # Accessed via VPC endpoint (zero trust )
   }
   
   Athena query (detect access from outside VPC):
   SELECT eventtime, useridentity.arn, requestparameters.bucketName, sourceipaddress
   FROM cloudtrail_logs
   WHERE eventsource = 's3.amazonaws.com'
     AND eventname = 'GetObject'
     AND vpcendpointid IS NULL  # NOT via VPC endpoint = potential breach
     AND year = '2024' AND month = '09' AND day = '23'
   
   If result = any rows → Investigation needed (should be zero in zero trust)

Cost Analysis (Zero Trust Architecture):

VPC Endpoints:
   Gateway endpoints (S3, DynamoDB): FREE
   Interface endpoints: $0.01/hour per AZ + data transfer
   
   Example (3 interface endpoints × 2 AZs):
     Secrets Manager: $0.01/hour × 2 AZ × 730 hours = $14.60/month
     SNS: $0.01/hour × 2 AZ × 730 hours = $14.60/month
     SQS: $0.01/hour × 2 AZ × 730 hours = $14.60/month
     Data transfer: $0.01/GB (intra-AZ)
   Total: ~$50/month
   
   Savings from removing NAT Gateway:
     NAT Gateway: $0.045/hour × 2 AZ × 730 = $65.70/month
     Data processing: $0.045/GB (outbound)
   Total savings: $70+/month
   
   Net cost: -$20/month (save money with endpoints!)

AWS PrivateLink:
   Endpoint service: FREE (on consumer side)
   Interface endpoint: $0.01/hour × 2 AZ = $14.60/month
   Data transfer: $0.01/GB
   Total: ~$15-30/month (depends on traffic)

IAM:
   IAM roles, policies, users: FREE
   IAM Identity Center: FREE (up to AWS SSO limits)
   Total: $0

CloudTrail:
   Management events: FREE (first copy)
   Data events: $0.10 per 100,000 events
   
   Example (1M Lambda invocations, 1M S3 API calls):
     Lambda events: 1M × $0.10 / 100K = $1.00
     S3 events: 1M × $0.10 / 100K = $1.00
     Storage: 10 GB × $0.023 = $0.23
   Total: ~$2.23/month

Total Zero Trust Cost: ~$50/month (vs $70 NAT Gateway = net savings)

Why A is Wrong (Traditional VPC with Public/Private Subnets):
 Implicit Trust Within VPC:
   Traditional security model:
   - Public subnet: "untrusted" (internet-facing)
   - Private subnet: "trusted" (internal only)
   - Assumption: Private = safe
   
   Problem: Lateral movement
   - Attacker compromises one EC2 in private subnet
   - Can access ALL resources in private subnet (implicit trust)
   - No micro-segmentation
   
 NAT Gateway = Internet Access:
   Private subnet → NAT Gateway → Internet
   - Traffic leaves AWS network (even if to AWS service)
   - Violates "private connectivity" principle
   - Attack vector: Exfiltrate data to external IP
   
 No Identity-Based Access:
   Security Groups: Network-based (IP addresses, ports)
   Zero Trust: Identity-based (IAM roles, users)
   
   Example:
   Traditional: Allow port 5432 from 10.0.1.0/24 (any IP in subnet)
   Zero Trust: Allow port 5432 from sg-app (only authorized workloads) + IAM auth

When traditional VPC acceptable:
   Non-regulated industry (not finance, healthcare)
   No compliance requirements (PCI DSS, HIPAA)
   Cost-sensitive (<$1000/month budget)

Why C is Wrong (VPN + Active Directory + NACLs):
 VPN Provides Network Access (Too Much Trust):
   VPN model:
   - User connects via VPN
   - Gets access to entire VPC CIDR (e.g., 10.0.0.0/16)
   - Can access ANY resource in VPC
   
   Zero Trust model:
   - User authenticates to identity provider
   - Gets access to SPECIFIC resource (e.g., one S3 bucket)
   - Cannot access other resources
   
 Network ACLs Are Stateless and Coarse:
   NACLs:
   - Subnet-level (all resources in subnet treated equally)
   - Stateless (must configure inbound AND outbound)
   - IP-based (not identity-based)
   
   vs Security Groups (Zero Trust):
   - Instance-level (each resource independent)
   - Stateful (automatic return traffic)
   - Can reference other SGs (not just IPs)
   
 Active Directory = Centralized Trust:
   AD model: User in domain → trusted for all domain resources
   Zero Trust: User authenticated per resource, per request
   
   Example:
   AD: Alice logs in → can access all file shares (domain member)
   Zero Trust: Alice requests file A → checks IAM policy → allow/deny
              Alice requests file B → checks IAM policy → allow/deny

When VPN + AD acceptable:
   Existing AD investment (sunk cost)
   Hybrid migration phase (temporary, 6-12 months)
   Legacy applications that require domain join

Why D is Wrong (Transit Gateway + Firewall Manager + GuardDuty):
 Transit Gateway Enables Network-Based Trust:
   Transit Gateway:
   - Connects multiple VPCs
   - Route propagation (VPC-A can reach VPC-B's CIDR)
   - Network-level connectivity
   
   Zero Trust principle:
   - No network-level trust
   - Each connection authenticated and authorized
   - PrivateLink (not routing) for service-to-service
   
 Firewall Manager = Network Firewall (Layer 3/4):
   Firewall Manager: Centrally manage Security Groups, WAF
   Still network-based: IP addresses, ports, protocols
   
   Zero Trust: Identity-based
   - Who (IAM role), not where (IP address)
   - What (specific resource), not how (port)
   
 GuardDuty is Detective, Not Preventive:
   GuardDuty: Detects threats (after the fact)
   Zero Trust: Prevents unauthorized access (before it happens)
   
   GuardDuty finding: "EC2 instance communicating with known malicious IP"
   Zero Trust: EC2 cannot communicate with external IPs (no NAT Gateway)
   
   Both are useful, but GuardDuty doesn't implement Zero Trust

When Transit Gateway + Firewall Manager acceptable:
   Large multi-VPC environment (hundreds of VPCs)
   Centralized management requirement
   Complementary to Zero Trust (not replacement)

Real-World Example - Goldman Sachs (Public Information):
Company: Goldman Sachs
Regulation: Highly regulated (SEC, FINRA)
Security model: Zero Trust

Implementation (estimated based on public AWS case study):
   - All services communicate via PrivateLink (no VPC peering)
   - Every API call authenticated via IAM
   - MFA required for all human access
   - Service-to-service: Mutual TLS + IAM roles
   - Audit: CloudTrail + centralized SIEM (Splunk)
   
Architecture:
   - 500+ VPCs (separated by line of business)
   - No internet gateways (except bastion VPC)
   - All AWS service access via VPC endpoints
   - PrivateLink for internal services (trading platform → risk engine)

Benefit:
   - Compliance: Passed SOC 2, ISO 27001 audits
   - Security: Zero breaches in AWS environment (as of public record)
   - Lateral movement: Impossible (micro-segmentation)
   
Cost:
   - Estimated $500K+/year for VPC endpoints, PrivateLink
   - ROI: Prevents $100M+ breach (Equifax-level breach = $1.4B cost)
   - Compliance: Required for operating in financial services

Zero Trust Migration Path (18-Month Hybrid):

Phase 1 (Months 1-6): Greenfield Zero Trust in AWS
   1. Deploy new apps in AWS with Zero Trust from day one
   2. VPC endpoints for all AWS services
   3. IAM roles (no long-term credentials)
   4. CloudTrail logging to centralized bucket
   5. On-prem apps: Continue with existing security

Phase 2 (Months 7-12): Hybrid Connectivity
   1. AWS Site-to-Site VPN: On-prem → AWS Transit Gateway
   2. But: Add VPN firewall rules (explicit allow only)
   3. On-prem apps accessing AWS: Via PrivateLink endpoint service
   4. AWS apps accessing on-prem: Via VPN (no VPC peering)
   5. Gradual migration: 10 apps/month

Phase 3 (Months 13-18): Full Zero Trust
   1. Last on-prem apps migrated to AWS
   2. Decommission VPN (no longer needed)
   3. All services communicate via PrivateLink
   4. On-prem datacenter: Decommissioned or used for DR
   5. Zero Trust 100% complete

Best Practices:

1. Start with identity: IAM roles for everything (no users)
2. Eliminate implicit trust: Remove default "allow" egress in SGs
3. Use PrivateLink: Service-to-service without VPC peering
4. VPC endpoints: All AWS services (no NAT Gateway)
5. Audit everything: CloudTrail data events + management events
6. Principle of least privilege: Specific IAM policies (not s3:*)
7. Short-lived credentials: IAM temporary tokens (15 min - 1 hour)
8. Monitor and alert: CloudWatch + GuardDuty + Security Hub
9. Regular reviews: Quarterly IAM policy audits (remove unused)
10. Automated remediation: Lambda responds to GuardDuty findings

Monitoring Dashboard (Zero Trust):

CloudWatch dashboard metrics:
1. VPC endpoint usage: Bytes transferred (should be high)
2. NAT Gateway usage: Bytes transferred (should be zero)
3. IAM AssumeRole calls: Count (all service access should use roles)
4. CloudTrail events without vpc_endpoint_id: Count (should be zero)
5. Security Group changes: Count (alert on changes, investigate)

Alarms:
1. NAT Gateway data transfer > 0 GB:
   Action: SNS alert (investigate why traffic to internet)
   
2. S3 access without VPC endpoint:
   Query: vpcEndpointId IS NULL in CloudTrail
   Action: PagerDuty critical (potential data exfiltration)
   
3. IAM long-term credentials created:
   Action: Auto-delete + alert security team (violates Zero Trust)

Why B is Correct (IAM Roles with Cross-Account Assume Role):

Complete Cross-Account Architecture:

┌───────────────────────────────────────────────────────────────┐
│  Account A (Parent): 111111111111                             │
│                                                                │
│  ┌─────────────┐        ┌──────────────────┐                │
│  │  Developer  │        │  CI/CD Pipeline  │                │
│  │  (Alice)    │        │  (CodePipeline)  │                │
│  └──────┬──────┘        └────────┬─────────┘                │
│         │ Assumes                 │ Assumes                  │
│         │ (via STS)               │ (via STS)                │
│         ↓                         ↓                          │
│  ┌─────────────────┐      ┌──────────────────┐             │
│  │ IAM Role:       │      │ IAM Role:         │             │
│  │ DevReadOnlyRole │      │ CICDDeployRole    │             │
│  └─────────────────┘      └──────────────────┘             │
└───────────────────────────────────────────────────────────────┘
           ↓                         ↓
     (Cross-account)           (Cross-account)
           ↓                         ↓
┌───────────────────────────────────────────────────────────────┐
│  Account B (Acquired): 222222222222                           │
│                                                                │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ IAM Role: CrossAccountReadOnlyRole                       │ │
│  │ Trust policy: Allow 111111111111 (Account A)            │ │
│  │ External ID: "acquisition-project-2024" (prevents CSRF) │ │
│  │ Permissions: CloudWatch Logs read-only                   │ │
│  └─────────────────────────────────────────────────────────┘ │
│                               ↓                               │
│  ┌─────────────────────────────────────────────────────────┐ │
│  │ IAM Role: CrossAccountDeployRole                         │ │
│  │ Trust policy: Allow 111111111111 (Account A)            │ │
│  │ External ID: "acquisition-project-2024"                  │ │
│  │ Permissions: EC2, ECS, Lambda deploy permissions         │ │
│  └─────────────────────────────────────────────────────────┘ │
│                                                                │
│  ┌──────────────────────────────────────────────────────────┐ │
│  │ CloudTrail: Logs all AssumeRole calls                    │ │
│  │ - Who (Account A user/role)                              │ │
│  │ - When (timestamp)                                        │ │
│  │ - What (actions taken after assume)                      │ │
│  └──────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘

Step-by-Step Implementation:

Step 1: Create Role in Account B (Target Account):

# In Account B (222222222222)
resource "aws_iam_role" "cross_account_readonly" {
  name = "CrossAccountReadOnlyRole"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:aws:iam::111111111111:root"  # Account A
      }
      Action = "sts:AssumeRole"
      Condition = {
        StringEquals = {
          "sts:ExternalId" = "acquisition-project-2024"
        }
      }
    }]
  })
}

# Attach permissions (read-only logs)
resource "aws_iam_role_policy" "readonly_permissions" {
  role = aws_iam_role.cross_account_readonly.id
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid = "ReadCloudWatchLogs"
        Effect = "Allow"
        Action = [
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "logs:FilterLogEvents"
        ]
        Resource = "arn:aws:logs:*:222222222222:log-group:/aws/lambda/*"
      },
      {
        Sid = "ReadS3Logs"
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:ListBucket"]
        Resource = [
          "arn:aws:s3:::account-b-logs",
          "arn:aws:s3:::account-b-logs/*"
        ]
      }
    ]
  })
}

Trust Policy Breakdown:
1. Principal: "arn:aws:iam::111111111111:root"
   - Meaning: Account A (entire account, not specific user)
   - Account A administrator decides WHO in Account A can assume
   
2. ExternalId: "acquisition-project-2024"
   - Purpose: Prevents "confused deputy" attack
   - Scenario: Third-party (e.g., monitoring SaaS) shouldn't be able to assume
   - Solution: Only requests with correct external ID succeed
   
   Confused Deputy Example (without External ID):
   1. Company gives role ARN to Monitoring SaaS
   2. SaaS can assume role (has Account A credentials)
   3. SaaS can access ANY Account B that trusts Account A
   4. Problem: SaaS accesses competitors' accounts!
   
   With External ID:
   1. Company gives: Role ARN + External ID to Monitoring SaaS
   2. SaaS assumes with External ID
   3. Works only for Company's Account B (unique External ID)
   4. Competitors use different External IDs → SaaS can't access

Step 2: Grant Permission in Account A (Source Account):

# In Account A (111111111111)
resource "aws_iam_policy" "assume_accountb_readonly" {
  name = "AssumeAccountBReadOnly"
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = "sts:AssumeRole"
      Resource = "arn:aws:iam::222222222222:role/CrossAccountReadOnlyRole"
      Condition = {
        StringEquals = {
          "sts:ExternalId" = "acquisition-project-2024"
        }
      }
    }]
  })
}

# Attach to developers group
resource "aws_iam_group_policy_attachment" "developers" {
  group      = "Developers"
  policy_arn = aws_iam_policy.assume_accountb_readonly.arn
}

Result:
- Developers in Account A can assume CrossAccountReadOnlyRole in Account B
- Must use external ID (prevents accidental assumptions)
- Account B controls what permissions granted (read-only logs only)

Step 3: Developer Assumes Role (AWS CLI):

# Developer Alice in Account A
$ aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/CrossAccountReadOnlyRole \
  --role-session-name alice-troubleshooting \
  --external-id acquisition-project-2024

Output:
{
  "Credentials": {
    "AccessKeyId": "ASIA...",
    "SecretAccessKey": "...",
    "SessionToken": "...",
    "Expiration": "2024-09-23T11:30:00Z"  # 1 hour from now
  },
  "AssumedRoleUser": {
    "AssumedRoleId": "AROA...:alice-troubleshooting",
    "Arn": "arn:aws:sts::222222222222:assumed-role/CrossAccountReadOnlyRole/alice-troubleshooting"
  }
}

# Set environment variables (temporary credentials)
$ export AWS_ACCESS_KEY_ID="ASIA..."
$ export AWS_SECRET_ACCESS_KEY="..."
$ export AWS_SESSION_TOKEN="..."

# Now can access Account B resources
$ aws logs tail /aws/lambda/payment-processor --follow
2024-09-23 10:45:23 INFO Processing payment for order #12345
2024-09-23 10:45:24 INFO Payment successful, amount: $99.99

# Credentials expire in 1 hour (must re-assume)

Step 4: CI/CD Pipeline Assumes Role (Automated):

# In Account A CI/CD pipeline (CodeBuild buildspec.yml)
version: 0.2
phases:
  pre_build:
    commands:
      # Assume deploy role in Account B
      - |
        CREDS=$(aws sts assume-role \
          --role-arn arn:aws:iam::222222222222:role/CrossAccountDeployRole \
          --role-session-name codebuild-deploy \
          --external-id acquisition-project-2024 \
          --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
          --output text)
      - export AWS_ACCESS_KEY_ID=$(echo $CREDS | cut -d' ' -f1)
      - export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | cut -d' ' -f2)
      - export AWS_SESSION_TOKEN=$(echo $CREDS | cut -d' ' -f3)
  
  build:
    commands:
      # Deploy to Account B
      - aws lambda update-function-code \
          --function-name payment-processor \
          --zip-file fileb://function.zip
      
      - aws ecs update-service \
          --cluster production \
          --service api-service \
          --force-new-deployment

Result:
- CI/CD in Account A can deploy to Account B
- No long-term credentials (temporary STS tokens only)
- External ID prevents unauthorized deployments
- CloudTrail logs who deployed what

CloudTrail Audit (Account B Logs):

AssumeRole Event:
{
  "eventTime": "2024-09-23T10:30:00Z",
  "eventName": "AssumeRole",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "AIDA...EXAMPLE",
    "arn": "arn:aws:iam::111111111111:user/alice",
    "accountId": "111111111111"  # Account A
  },
  "requestParameters": {
    "roleArn": "arn:aws:iam::222222222222:role/CrossAccountReadOnlyRole",
    "roleSessionName": "alice-troubleshooting",
    "externalId": "acquisition-project-2024"
  },
  "responseElements": {
    "credentials": {
      "expiration": "Sep 23, 2024, 11:30:00 AM"
    }
  },
  "sourceIPAddress": "203.0.113.50",
  "userAgent": "aws-cli/2.13.0"
}

Actions Performed After AssumeRole:
{
  "eventTime": "2024-09-23T10:31:00Z",
  "eventName": "GetLogEvents",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROA...:alice-troubleshooting",
    "arn": "arn:aws:sts::222222222222:assumed-role/CrossAccountReadOnlyRole/alice-troubleshooting",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROA...",
        "arn": "arn:aws:iam::222222222222:role/CrossAccountReadOnlyRole",
        "accountId": "222222222222"
      },
      "attributes": {
        "creationDate": "2024-09-23T10:30:00Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "requestParameters": {
    "logGroupName": "/aws/lambda/payment-processor"
  },
  "sourceIPAddress": "203.0.113.50"
}

Athena Query (Who Accessed What from Account A):
SELECT useridentity.arn as user,
       eventname,
       requestparameters.logGroupName as accessed_resource,
       sourceipaddress,
       eventtime
FROM cloudtrail_logs
WHERE useridentity.type = 'AssumedRole'
  AND useridentity.arn LIKE '%Cross AccountReadOnlyRole%'
  AND year = '2024' AND month = '09' AND day = '23'
ORDER BY eventtime DESC

Output:
user                                         | eventname     | accessed_resource            | sourceipaddress | eventtime
arn:aws:sts::222222222222:assumed-role/...  | GetLogEvents  | /aws/lambda/payment-processor| 203.0.113.50    | 2024-09-23T10:31:00Z

Cost Analysis:

IAM Roles: FREE
STS AssumeRole calls: FREE
CloudTrail: Management events FREE (first copy)
Total cost: $0

vs Alternatives:
- Third-party identity provider (Okta): $5-15/user/month = $3,250-9,750/month for 650 users
- VPN between accounts: $0.05/hour per tunnel = $36/month (but less secure)

Benefits:
- No additional cost over baseline AWS usage
- Native AWS service (no external dependencies)
- Automatic credential rotation (1-hour expiration)

Advanced Features:

1. Session Duration Control:
# Shorter sessions for high-risk operations
$ aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/CrossAccountAdminRole \
  --role-session-name emergency-access \
  --duration-seconds 900  # 15 minutes (min: 900, max: 43200)

Use case: Break-glass access (emergency admin, expires quickly)

2. MFA Required for Assume Role:
# In Account B trust policy
{
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::111111111111:root"},
  "Action": "sts:AssumeRole",
  "Condition": {
    "Bool": {"aws:MultiFactorAuthPresent": "true"}
  }
}

# Developer must use MFA device
$ aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/CrossAccountAdminRole \
  --role-session-name alice-admin \
  --serial-number arn:aws:iam::111111111111:mfa/alice \
  --token-code 123456  # MFA token

Result: Even if credentials compromised, attacker can't assume without MFA

3. Permission Boundaries (Limit Maximum Permissions):
# In Account A, limit what developers can do EVEN IF they assume powerful role
resource "aws_iam_policy" "developer_boundary" {
  name = "DeveloperPermissionBoundary"
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "logs:*",
        "cloudwatch:*",
        "s3:GetObject",
        "s3:ListBucket"
      ]
      Resource = "*"
    },{
      Effect = "Deny"
      Action = [
        "iam:*",
        "organizations:*",
        "account:*"
      ]
      Resource = "*"
    }]
  })
}

# Attach boundary to developer user
resource "aws_iam_user_policy_attachments_exclusive" "alice" {
  user_name = "alice"
  policy_arns = [aws_iam_policy.developer_boundary.arn]
  
  # This is the permission boundary (ceiling)
  # Even if Alice assumes role with admin access,
  # she can only do what's in this boundary
}

Why A is Wrong (Shared IAM User Credentials):
 Massive Security Risk:
   Scenario:
   - Create IAM user in Account B: "shared-account-a"
   - Share credentials: Access key + secret key to 500 Account A developers
   - All developers use same credentials
   
   Problems:
   1. No Individual Attribution:
      CloudTrail log: "User: shared-account-a accessed S3"
      Question: Which of 500 developers was it?
      Answer: Unknown (can't audit)
   
   2. Credential Leakage:
      - 500 developers with credentials = 500 leak opportunities
      - Shared Slack channel, email, GitHub repo (accidental commit)
      - Once leaked: Can't determine who leaked
   
   3. No Expiration:
      - Long-term credentials (valid until manually rotated)
      - Developer leaves company → credentials still valid
      - Must manually track and rotate
   
   4. Overly Broad Permissions:
      - Shared user needs permissions for all 500 developers' use cases
      - Result: Over-privileged (violates least privilege)
   
   5. Credential Rotation Nightmare:
      - Rotate credentials → must update 500 developers
      - Deployment scripts, CI/CD pipelines, local configs
      - Hours of downtime during rotation
   
   Real-world incident - Uber breach (2016):
   - Developers shared AWS credentials in private GitHub repo
   - Repo leaked (insider threat)
   - Attacker accessed 57 million customer records
   - Cost: $148 million settlement + CEO fired

Why C is Wrong (VPC Peering + Security Groups):
 VPC Peering is Network Connectivity, Not Access Control:
   VPC Peering:
   - Enables network routing (Account A VPC ↔ Account B VPC)
   - Traffic flows like single network
   - Security Groups can reference peer VPC SGs
   
   Problem: Doesn't solve access management
   - Developer in Account A wants CloudWatch Logs in Account B
   - VPC peering: Enables EC2 in A to reach EC2 in B (network layer)
   - CloudWatch Logs: Accessed via API (not network), needs IAM permissions
   - Result: VPC peering irrelevant for API access
   
 When VPC Peering is Appropriate:
   Use case: Application in Account A needs database in Account B
   - Peering: Enable network connectivity (VPC-A → VPC-B)
   - Security Group: database-sg allows app-sg (specific instance)
   - IAM role: Still needed (application needs sts:AssumeRole to get DB creds)
   
   VPC peering + IAM roles: Complementary, not替代

Why D is Wrong (AWS Organizations SCPs Only):
 SCPs are Guardrails, Not Access Grants:
   Service Control Policy (SCP):
   - Maximum permissions (ceiling)
   - Applied at OU or account level
   - Doesn't GRANT permissions, only RESTRICTS
   
   Example SCP (prevent S3 bucket deletion):
   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Deny",
       "Action": "s3:DeleteBucket",
       "Resource": "*"
     }]
   }
   
   Result: Even account administrator can't delete S3 buckets
   But: Doesn't grant anyone permission to READ buckets
   
 Still Need IAM Roles for Access:
   To access Account B from Account A:
   1. SCP: Ensures Account A can't do forbidden actions (guardrail)
   2. IAM role: Grants specific permissions (access)
   
   Both needed: SCPs (prevent), IAM roles (permit)

When Organizations + SCPs Appropriate:
   Prevent account-level actions (e.g., leaving organization)
   Enforce policies (e.g., require encryption)
   Complement IAM roles (not replace)

12-Month Integration Timeline:

Month 1-3 (Initial Access):
   - Create cross-account read-only roles (troubleshooting)
   - Enable CloudTrail logging in both accounts
   - Train developers on assume-role workflow
   - Document external ID (shared secret)

Month 4-6 (CI/CD Integration):
   - Create cross-account deploy roles
   - Migrate CI/CD pipelines to assume roles
   - Implement least privilege (specific Lambda functions only)
   - Quarterly IAM policy review

Month 7-9 (Shared Services):
   - Centralize monitoring in Account A (Grafana, CloudWatch dashboards)
   - Account B exports metrics to Account A (assume role)
   - Centralize logging (Account B → CloudWatch Logs → Kinesis → Account A S3)
   - Cost allocation tags (separate billing maintained)

Month 10-12 (Prepare for Merge):
   - Document all cross-account dependencies
   - Plan account consolidation (Account B → Account A OUs)
   - Migrate resources to Account A (CloudFormation StackSets)
   - Final cutover (Month 12): Account B becomes empty, decommissioned

Post-Month 12 (Consolidated):
   - Single account (Account A) with OUs for organizational units
   - No more cross-account roles (same-account IAM)
   - Simplified billing (single payer account)
   - Reduced operational overhead

Monitoring & Compliance:

CloudWatch Dashboard (Cross-Account Access):
Metrics:
1. AssumeRole Success/Failure Count:
   - Success: Normal operation
   - Failures: Investigate (wrong external ID? Permission denied?)
   
2. AssumeRole by User:
   - Top 10 users assuming roles
   - Detect anomalies (user suddenly assuming 100× more)
   
3. Session Duration:
   - Average: 30-60 minutes typical
   - >1 hour: Long-lived sessions (potential credential theft)

Athena Query (Compliance Report):
-- All cross-account access in last 30 days
SELECT 
  DATE_TRUNC('day', from_iso8601_timestamp(eventtime)) as date,
  useridentity.arn as user,
  requestparameters.roleArn as assumed_role,
  COUNT(*) as assume_count,
  SUM(CASE WHEN errorcode IS NULL THEN 0 ELSE 1 END) as failed_attempts
FROM cloudtrail_logs
WHERE eventname = 'AssumeRole'
  AND requestparameters.roleArn LIKE '%222222222222%'
  AND timestamp > current_timestamp - INTERVAL '30' DAY
GROUP BY 1, 2, 3
ORDER BY date DESC, assume_count DESC

Output (CSV for auditors):
date       | user                               | assumed_role                                        | assume_count | failed_attempts
2024-09-23 | arn:aws:iam::111111111111:user/alice | arn:aws:iam::222222222222:role/CrossAccountReadOnly | 45           | 0
2024-09-23 | arn:aws:iam::111111111111:user/bob   | arn:aws:iam::222222222222:role/CrossAccountDeploy   | 12           | 2

Best Practices:

1. Use external IDs (prevent confused deputy)
2. Require MFA for sensitive roles (admin, deploy)
3. Short session durations (1 hour max, 15 min for admin)
4. Permission boundaries (limit maximum permissions)
5. Regular audits (quarterly CloudTrail review)
6. Least privilege (specific resources, not "*")
7. Separate roles by function (read-only, deploy, admin)
8. Document roles (README with purpose, who uses, permissions)
9. Automate credential expiration (no long-term keys)
10. Monitor and alert (unexpected AssumeRole calls)

psycopg2.OperationalError: could not connect to server: Connection timed out

Why B is Correct (Network ACL Ephemeral Ports):

Connection Flow (TCP 3-Way Handshake):

1. EC2 Initiates Connection:
   Source: 10.0.1.50:54321 (ephemeral port, randomly chosen 1024-65535)
   Destination: 10.0.2.100:5432 (RDS PostgreSQL)
   
   Packet flow:
   EC2 (10.0.1.50) → SG sg-app (outbound check) → NACL subnet-1 (outbound check) 
   → NACL subnet-2 (inbound check) → SG sg-db (inbound check) → RDS (10.0.2.100)
   
    sg-app outbound: Allows 0.0.0.0/0 (all destinations) 
    NACL subnet-1 outbound: Default allows all 
    NACL subnet-2 inbound: Default allows all 
    sg-db inbound: Allows port 5432 from sg-app 
   
   Result: Packet reaches RDS 

2. RDS Responds (SYN-ACK):
   Source: 10.0.2.100:5432 (RDS database port)
   Destination: 10.0.1.50:54321 (EC2 ephemeral port)
   
   Packet flow:
   RDS (10.0.2.100) → SG sg-db (outbound check) → NACL subnet-2 (outbound check)
   → NACL subnet-1 (inbound check) → SG sg-app (inbound check) → EC2 (10.0.1.50)
   
    sg-db outbound: STATEFUL - automatically allows return traffic 
    NACL subnet-2 outbound: Default allows all 
    NACL subnet-1 inbound: Default allows all, BUT...
      If custom NACL was created and only allows port 5432 inbound:
      BLOCKS port 54321 (ephemeral port) 
   
   Result: Packet dropped at NACL 

Problem: Network ACLs are STATELESS (unlike Security Groups)
- Outbound rule: Allows request (10.0.1.50:54321 → 10.0.2.100:5432)
- Inbound rule: Must explicitly allow response (10.0.2.100:5432 → 10.0.1.50:54321)
- If NACL only allows port 5432 inbound, blocks ephemeral port 54321
- Result: SYN packet reaches RDS, but SYN-ACK response blocked

Example Custom NACL (WRONG - causes this issue):
resource "aws_network_acl" "public" {
  vpc_id     = aws_vpc.main.id
  subnet_ids = [aws_subnet.public.id]
  
  # Inbound rules
  ingress {
    rule_no    = 100
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 22   # SSH
    to_port    = 22
  }
  
  ingress {
    rule_no    = 110
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 443  # HTTPS
    to_port    = 443
  }
  
  ingress {
    rule_no    = 120
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "10.0.2.0/24"  # Private subnet
    from_port  = 5432           # PostgreSQL
    to_port    = 5432
  }
  # MISSING: Ephemeral ports 1024-65535 for return traffic!
  
  # Outbound rules
  egress {
    rule_no    = 100
    protocol   = "-1"  # All protocols
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }
}

Fixed NACL (CORRECT - allows ephemeral ports):
resource "aws_network_acl" "public" {
  vpc_id     = aws_vpc.main.id
  subnet_ids = [aws_subnet.public.id]
  
  # Inbound rules
  ingress {
    rule_no    = 100
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 22
    to_port    = 22
  }
  
  ingress {
    rule_no    = 110
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 443
    to_port    = 443
  }
  
  ingress {
    rule_no    = 120
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "10.0.2.0/24"
    from_port  = 5432
    to_port    = 5432
  }
  
  #  CRITICAL: Allow ephemeral ports for return traffic
  ingress {
    rule_no    = 130
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "10.0.2.0/24"  # RDS subnet
    from_port  = 1024            # Ephemeral port range start
    to_port    = 65535           # Ephemeral port range end
  }
  
  # Outbound rules
  egress {
    rule_no    = 100
    protocol   = "-1"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }
}

Ephemeral Port Ranges (Operating System Specific):
Linux (Amazon Linux 2): 32768-60999
Windows Server: 49152-65535
NAT Gateway: 1024-65535
Best Practice: Allow 1024-65535 (covers all OS)

Verification Commands:

# Check NACL rules
$ aws ec2 describe-network-acls \
  --filters "Name=vpc-id,Values=vpc-abc123" \
  --query 'NetworkAcls[*].[NetworkAclId,Associations[0].SubnetId,Entries]'

# Test connectivity (from EC2 to RDS)
$ nc -zv db.abc123.us-east-1.rds.amazonaws.com 5432
Connection to db.abc123.us-east-1.rds.amazonaws.com port 5432 [tcp/postgresql] succeeded!

# If fails, check return path
$ sudo tcpdump -i eth0 'tcp port 5432' -n
Listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:30:00.123456 IP 10.0.1.50.54321 > 10.0.2.100.5432: Flags [S], seq 1234567890
#  Outbound SYN packet sent
#  No SYN-ACK response = return traffic blocked

# Check Security Group (should show return traffic allowed)
$ aws ec2 describe-security-groups --group-ids sg-abc123
{
  "SecurityGroups": [{
    "IpPermissionsEgress": [{
      "IpProtocol": "-1",
      "IpRanges": [{"CidrIp": "0.0.0.0/0"}]
    }]
  }]
}
# Security Groups are STATEFUL (automatically allow return traffic)

Why A is Wrong (Security Group Stateless):
 Security Groups are STATEFUL, not stateless:
   
   Stateful firewall (Security Groups):
   - Tracks connection state
   - Outbound rule allows request → automatically allows response
   - No need to explicitly allow return traffic
   
   Example:
   EC2 (10.0.1.50:54321) → RDS (10.0.2.100:5432)
   sg-app outbound: Allows 0.0.0.0/0 (covers RDS)
   
   Response:
   RDS (10.0.2.100:5432) → EC2 (10.0.1.50:54321)
   sg-app inbound: NO RULE for port 54321
   But: ALLOWED anyway (tracked as part of same connection) 
   
   Security Groups automatically allow:
   - Return traffic for outbound connections
   - Response traffic for inbound connections
   
   sg-db configuration (RDS):
   Inbound: Port 5432 from sg-app 
   Outbound: Not needed (automatically allowed as return traffic) 
   
   Result: Security Groups correctly configured, not the issue

Stateful vs Stateless Comparison:

Feature              | Security Groups | Network ACLs
---------------------|-----------------|---------------
Stateful?            | Yes             | No
Return traffic       | Automatic       | Must explicitly allow
Scope                | Instance-level  | Subnet-level
Rule evaluation      | All rules       | Rules in order (lowest number first)
Default              | Deny all        | Allow all (default NACL)
Use case             | Primary defense | Additional subnet-level control

Why C is Wrong (Missing Route):
 Route table has local route for VPC CIDR:
   
   Route table (both subnets):
   Destination      Target
   10.0.0.0/16      local      #  Covers both 10.0.1.0/24 and 10.0.2.0/24
   0.0.0.0/0        igw-abc123 # Internet gateway (public subnet only)
   
   How "local" route works:
   - Automatically created when VPC created
   - Covers all subnets within VPC CIDR
   - Cannot be deleted
   - Enables intra-VPC communication
   
   EC2 (10.0.1.50) → RDS (10.0.2.100):
   1. EC2 checks route table: "Where is 10.0.2.100?"
   2. Route table: "10.0.2.0 matches 10.0.0.0/16 → use local route"
   3. Packet sent via VPC internal routing 
   
   No explicit route to 10.0.2.0/24 needed (covered by 10.0.0.0/16 local)

When you WOULD need explicit routes:
   Scenario: VPN or Transit Gateway to on-premises
   
   Route table:
   Destination      Target
   10.0.0.0/16      local           # VPC CIDR
   192.168.0.0/16   vgw-abc123      # On-prem via VPN
   172.16.0.0/12    tgw-abc123      # Other VPCs via Transit Gateway
   
   Without these: Can't reach on-prem or other VPCs

Why D is Wrong (Need IGW for RDS):
 Private subnet communication doesn't need internet gateway:
   
   RDS is PRIVATE resource:
   - Lives in private subnet (10.0.2.0/24)
   - No public IP address
   - Accessed via private IP (10.0.2.100)
   
   EC2 → RDS communication:
   - Uses VPC internal routing (local route)
   - Never leaves VPC
   - No internet gateway involved
   
   When internet gateway IS needed:
   Scenario 1: EC2 in public subnet needs to download packages from internet
   EC2 (10.0.1.50) → IGW → Internet (93.184.216.34 - example.com)
   
   Scenario 2: EC2 in private subnet needs to download packages
   EC2 (10.0.2.50) → NAT Gateway (10.0.1.100) → IGW → Internet
   
   Scenario 3: EC2 needs to access RDS
   EC2 (10.0.1.50) → RDS (10.0.2.100)
   No IGW involved! (intra-VPC communication) 

Troubleshooting Methodology (Systematic Approach):

Step 1: Check Security Groups (Stateful):
# EC2 Security Group (source)
$ aws ec2 describe-security-groups --group-ids sg-app \
  --query 'SecurityGroups[0].IpPermissionsEgress'

Expected: Allow port 5432 outbound (or 0.0.0.0/0)
If missing: Add rule

# RDS Security Group (destination)
$ aws ec2 describe-security-groups --group-ids sg-db \
  --query 'SecurityGroups[0].IpPermissions'

Expected: Allow port 5432 inbound from sg-app (or EC2's IP)
If missing: Add rule

Step 2: Check Network ACLs (Stateless):
# Public subnet NACL (where EC2 lives)
$ aws ec2 describe-network-acls \
  --filters "Name=association.subnet-id,Values=subnet-public" \
  --query 'NetworkAcls[0].Entries' | jq .

Expected inbound rules:
- Allow ephemeral ports (1024-65535) from private subnet CIDR
- Allow any required management ports (22, 443)

Expected outbound rules:
- Allow port 5432 to private subnet CIDR

# Private subnet NACL (where RDS lives)
$ aws ec2 describe-network-acls \
  --filters "Name=association.subnet-id,Values=subnet-private" \
  --query 'NetworkAcls[0].Entries' | jq .

Expected inbound rules:
- Allow port 5432 from public subnet CIDR

Expected outbound rules:
- Allow ephemeral ports (1024-65535) to public subnet CIDR

Step 3: Check Route Tables:
# Public subnet route table
$ aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-public" \
  --query 'RouteTables[0].Routes'

Expected:
- 10.0.0.0/16 → local (VPC CIDR)
- 0.0.0.0/0 → igw-xxx (internet access, optional)

# Private subnet route table
$ aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-private" \
  --query 'RouteTables[0].Routes'

Expected:
- 10.0.0.0/16 → local (VPC CIDR)
- 0.0.0.0/0 → nat-xxx (internet via NAT, optional)

Step 4: Test Connectivity:
# From EC2 instance
$ telnet db.abc123.us-east-1.rds.amazonaws.com 5432

Success: "Connected to db.abc123.us-east-1.rds.amazonaws.com"
Failure: "Connection timed out" → Network issue (NACL, route table, or firewall)
Failure: "Connection refused" → RDS not listening on port (RDS stopped, wrong port)

# Packet capture (advanced)
$ sudo tcpdump -i any 'host 10.0.2.100 and port 5432' -nn -vv
10:30:00.000 IP 10.0.1.50.54321 > 10.0.2.100.5432: Flags [S], seq 123
10:30:00.050 IP 10.0.2.100.5432 > 10.0.1.50.54321: Flags [S.], seq 456, ack 124

If you see SYN but no SYN-ACK: Outbound path works, return path blocked (NACL!)
If you see SYN-ACK: Both paths work, application issue (not network)

Step 5: Check VPC Flow Logs:
# Enable flow logs (if not already)
$ aws ec2 create-flow-logs \
  --resource-type VPC \
  --resource-ids vpc-abc123 \
  --traffic-type ALL \
  --log-destination-type cloud-watch-logs \
  --log-group-name /aws/vpc/flowlogs

# Query logs (Athena after 5-10 minutes)
SELECT srcaddr, dstaddr, srcport, dstport, action
FROM vpc_flow_logs
WHERE srcaddr = '10.0.1.50'
  AND dstaddr = '10.0.2.100'
  AND dstport = 5432
LIMIT 10

Expected:
srcaddr   | dstaddr   | srcport | dstport | action
10.0.1.50 | 10.0.2.100| 54321   | 5432    | ACCEPT  (outbound from EC2)
10.0.2.100| 10.0.1.50 | 5432    | 54321   | ACCEPT  (return from RDS)

If first ACCEPT but second REJECT: Return traffic blocked by NACL! 

Real-World Example - E-Commerce Outage:
Company: Medium-sized e-commerce platform
Issue: Application could not connect to database after network changes
Impact: 2-hour complete outage, $200K revenue loss

Root cause analysis:
1. Security engineer created custom NACL for "tighter security"
2. NACL allowed only port 3306 (MySQL) inbound
3. Forgot to allow ephemeral ports 1024-65535
4. Outbound connections worked (SYN packet reached RDS)
5. Return traffic blocked (SYN-ACK couldn't reach EC2)
6. Application saw "Connection timed out" errors

Resolution:
1. Identified via VPC Flow Logs (REJECT on return path)
2. Added NACL inbound rule: 1024-65535 from private subnet
3. Connectivity restored within 10 minutes
4. Total outage: 2 hours (including investigation)

Prevention:
1. Always use default NACL (allows all) unless specific requirement
2. If custom NACL needed: Allow ephemeral ports 1024-65535
3. Test connectivity in staging environment first
4. Enable VPC Flow Logs for faster troubleshooting
5. Document NACL changes (why custom NACL needed)

Best Practices:

1. Security Groups for Primary Defense:
   - Instance-level control
   - Stateful (easier to configure)
   - Most use cases covered by Security Groups alone

2. Network ACLs for Additional Layer:
   - Use default NACL (allows all) for most subnets
   - Custom NACL only for high-security requirements
   - Always allow ephemeral ports (1024-65535)

3. VPC Flow Logs for Troubleshooting:
   - Enable for all VPCs
   - Store in S3 (cheaper than CloudWatch Logs)
   - Athena queries for fast investigation

4. Testing:
   - Test connectivity after any network change
   - Use telnet or nc for quick port checks
   - tcpdump for packet-level troubleshooting

5. Documentation:
   - Document why custom NACLs created
   - Standard NACL templates (with ephemeral ports)
   - Runbook for network troubleshooting

Common NACL Mistakes:

Mistake 1: Forgetting ephemeral ports (this question!)
Solution: Always add 1024-65535 inbound rule

Mistake 2: Wrong rule order
NACL evaluates rules in order (lowest number first)
Rule 100: ALLOW 10.0.0.0/16
Rule 200: DENY 10.0.1.0/24
Result: 10.0.1.0/24 is ALLOWED (rule 100 matches first)

Mistake 3: Not allowing ICMP
Cannot ping without ICMP echo reply allowed
Add: Protocol ICMP, Type 0 (echo reply), Code 0

Mistake 4: Explicit deny for internet
Rule 100: ALLOW 0.0.0.0/0
Rule 200: DENY 0.0.0.0/0
Result: All allowed (rule 100 matches first, rule 200 never evaluated)

Mistake 5: Different NAC Ls for request/response path
Request: Public subnet NACL (allows outbound port 5432)
Response: Private subnet NACL (doesn't allow outbound ephemeral ports)
Result: Response can't leave private subnet

Why B is Correct (ACM Wildcard + Regional + DNS Validation):

Certificate Architecture:

┌─────────────────────────────────────────────────────────────────┐
│                     CloudFront (Global)                          │
│  Certificate: *.example.com (ACM us-east-1)                     │
│  Domains: www.example.com, api.example.com, {customer}.example.com│
│  Cost: FREE (ACM on CloudFront)                                 │
└─────────────────────────────────────────────────────────────────┘
              ↓                            ↓
   ┌──────────────────┐         ┌──────────────────┐
   │  ALB us-east-1   │         │  ALB eu-west-1   │
   │  Certificate:    │         │  Certificate:    │
   │  *.example.com   │         │  *.example.com   │
   │  (ACM regional)  │         │  (ACM regional)  │
   │  Cost: FREE      │         │  Cost: FREE      │
   └──────────────────┘         └──────────────────┘

Partner Domain (www.partner-site.com):
   ┌─────────────────────────────────────────────────────┐
   │ ACM Certificate: www.partner-site.com               │
   │ Validation: DNS (CNAME in partner's Route 53)      │
   │ Partner adds: _acm-validation.partner-site.com     │
   │ Value: _xxx.acm-validations.aws.                   │
   │ Cost: FREE                                          │
   └─────────────────────────────────────────────────────┘

Terraform Configuration:

# 1. Wildcard Certificate (us-east-1 for CloudFront)
resource "aws_acm_certificate" "wildcard_cloudfront" {
  provider          = aws.us-east-1  # CloudFront requires us-east-1
  domain_name       = "*.example.com"
  validation_method = "DNS"
  
  subject_alternative_names = [
    "example.com",      # Apex domain
    "www.example.com"   # Explicit www (covered by wildcard, but explicit better)
  ]
  
  lifecycle {
    create_before_destroy = true
  }
}

# 2. DNS Validation Records (Route 53)
resource "aws_route53_record" "wildcard_validation" {
  for_each = {
    for dvo in aws_acm_certificate.wildcard_cloudfront.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }
  
  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.main.zone_id
}

# 3. Certificate Validation (wait for DNS propagation)
resource "aws_acm_certificate_validation" "wildcard" {
  certificate_arn         = aws_acm_certificate.wildcard_cloudfront.arn
  validation_record_fqdns = [for record in aws_route53_record.wildcard_validation : record.fqdn]
  
  timeouts {
    create = "45m"  # DNS propagation can take time
  }
}

# 4. CloudFront Distribution (uses wildcard cert)
resource "aws_cloudfront_distribution" "main" {
  enabled = true
  aliases = ["www.example.com", "api.example.com", "*.example.com"]
  
  viewer_certificate {
    acm_certificate_arn      = aws_acm_certificate_validation.wildcard.certificate_arn
    ssl_support_method       = "sni-only"  # FREE (vs dedicated IP $600/month)
    minimum_protocol_version = "TLSv1.2_2021"
  }
  
  # Origins, behaviors, etc.
}

# 5. Regional Certificate (us-east-1 for ALB)
resource "aws_acm_certificate" "wildcard_us_east_1" {
  provider          = aws.us-east-1
  domain_name       = "*.example.com"
  validation_method = "DNS"
  
  subject_alternative_names = ["example.com"]
  
  lifecycle {
    create_before_destroy = true
  }
}

# 6. Regional Certificate (eu-west-1 for ALB)
resource "aws_acm_certificate" "wildcard_eu_west_1" {
  provider          = aws.eu-west-1
  domain_name       = "*.example.com"
  validation_method = "DNS"
  
  subject_alternative_names = ["example.com"]
  
  lifecycle {
    create_before_destroy = true
  }
}

# 7. ALB (uses regional cert)
resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.main.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS-1-2-2017-01"
  certificate_arn   = aws_acm_certificate_validation.wildcard_us_east_1.certificate_arn
  
  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

# 8. Partner Domain Certificate
resource "aws_acm_certificate" "partner" {
  domain_name       = "www.partner-site.com"
  validation_method = "DNS"
  
  # Partner must add DNS record in their Route 53/DNS provider
}

Wildcard Certificate Coverage:

*.example.com covers:
www.example.com
api.example.com
customer1.example.com
customer2.example.com
customer1000.example.com
Any single-level subdomain

*.example.com does NOT cover:
example.com (apex domain - add as SAN)
sub.api.example.com (multi-level subdomain)
partner-site.com (different domain entirely)

For multi-level subdomains (sub.api.example.com):
Option 1: Second wildcard *.api.example.com
Option 2: Explicit SAN: sub.api.example.com

DNS Validation Process:

1. Request Certificate:
   AWS ACM provides CNAME record:
   Name: _abc123.example.com
   Value: _xyz789.acm-validations.aws.

2. Add to Route 53:
   resource "aws_route53_record" "validation" {
     name    = "_abc123.example.com"
     type    = "CNAME"
     records = ["_xyz789.acm-validations.aws."]
     ttl     = 60
     zone_id = aws_route53_zone.main.zone_id
   }

3. ACM Validates (checks DNS):
   Query: _abc123.example.com
   Expected: _xyz789.acm-validations.aws.
   Result: MATCH → Certificate issued 

4. Automatic Renewal:
   - ACM checks DNS record still exists
   - If present: Certificate auto-renewed (no action needed)
   - If removed: Renewal fails (must re-validate)
   
   Timeline:
   - Certificate issued: Day 0
   - Expiry: Day 395 (13 months)
   - Renewal starts: Day 335 (60 days before expiry)
   - Renewal completes: Day 345 (if DNS record present)
   - No downtime: New cert deployed automatically

Partner Domain Validation:

Challenge: Partner owns www.partner-site.com
You need: TLS certificate for partner's domain

Process:
1. Request certificate in ACM: www.partner-site.com
2. ACM provides validation CNAME: _123.partner-site.com → _456.acm-validations.aws.
3. Send to partner: "Please add this DNS record"
4. Partner adds record in their DNS provider
5. ACM validates: Certificate issued 
6. Renewal: Partner must keep DNS record (don't delete!)

Alternative - Email Validation (NOT recommended):
- ACM sends email to: admin@partner-site.com, webmaster@partner-site.com
- Partner clicks link to approve
- Problem: Renewal requires partner click link again (manual)
- DNS validation: Automatic renewal (partner never touches again)

Cost Analysis:

ACM Certificates:
   Public certificates: FREE
   Private certificates: $400/month per CA (not needed here)
   
   Your architecture:
   - Wildcard *.example.com (CloudFront): $0
   - Wildcard *.example.com (ALB us-east-1): $0
   - Wildcard *.example.com (ALB eu-west-1): $0
   - www.partner-site.com: $0
   
   Total: $0/month 

CloudFront SSL:
   SNI (Server Name Indication): FREE
   Dedicated IP: $600/month
   
   Your choice: SNI (covers 99% of clients, IE on Windows XP not supported)
   Total: $0/month 

Data transfer (not cert-related, but relevant):
   CloudFront to internet: $0.085/GB
   ALB to internet: $0.09/GB
   
   If 10 TB/month:
   CloudFront: 10,000 GB × $0.085 = $850/month
   ALB only: 10,000 GB × $0.09 = $900/month
   
   Savings: $50/month (plus performance benefits)

Total Certificate Cost: $0/month (well under $1,000 budget) 

Why A is Wrong (Third-Party Certificate + Manual Renewal):
 Manual Renewal Process:
   Third-party CA (e.g., DigiCert, Sectigo):
   - Purchase wildcard cert: $300-500/year
   - Expires: 397 days (13 months, CA/Browser Forum limit)
   - Renewal:
     Day 365: Receive email "Certificate expiring in 30 days"
     Day 370: Engineer purchases renewal ($300-500)
     Day 372: Download new certificate
     Day 373: Import to ACM (aws acm import-certificate)
     Day 374: Update CloudFront/ALB to use new cert ARN
     Day 375: Test in production
   
   Risk:
   - Forget to renew → Certificate expires → HTTPS down → Revenue loss
   - Real-world: Many outages caused by expired certificates
   
   Examples:
   - Microsoft Teams (2020): Certificate expired, 4-hour outage
   - LinkedIn (2021): Certificate expired, 2-hour outage
   - Spotify (2016): Certificate expired, intermittent errors

 Higher Cost:
   Third-party wildcard: $300-500/year
   ACM wildcard: $0/year
   
   Savings: $300-500/year with ACM

 Import to ACM Limitations:
   - Must import to EACH region (us-east-1, eu-west-1)
   - Must update before expiry (no auto-renewal)
   - ACM doesn't manage private key (you do)

When third-party certificate appropriate:
   Non-AWS services (on-premises servers, other clouds)
   Client compatibility requirements (ancient clients not supporting SNI)
   Specific CA requirement (e.g., government mandates specific CA)

Why C is Wrong (Let's Encrypt on EC2):
 Operational Burden:
   Let's Encrypt setup (per EC2 instance):
   1. Install certbot: sudo yum install certbot
   2. Request cert: sudo certbot certonly --webroot -d www.example.com
   3. Configure cron: sudo crontab -e
      0 0 * * * /usr/bin/certbot renew --quiet
   4. Reload web server after renewal: sudo systemctl reload nginx
   
   Scaling to 50 EC2 instances:
   - 50× setup (can automate with user data)
   - 50× renewal monitoring (What if one fails?)
   - 50× certificate files (/etc/letsencrypt/live/)
   - Complexity: High vs ACM (centralized)

 Load Balancer Challenge:
   ALB/NLB handles TLS termination (not EC2):
   - Let's Encrypt on EC2 → Can't use with ALB
   - Must terminate TLS on EC2 (end-to-end encryption)
   - ALB → EC2 on port 443 (not 80)
   - Performance: Slower (EC2 handles encryption, not ALB)
   
   With ACM on ALB:
   - ALB terminates TLS (high-performance hardware offload)
   - ALB → EC2 on port 80 (plain HTTP internal)
   - EC2 resources freed (no encryption overhead)

 Certificate Distribution:
   Auto Scaling group:
   - New EC2 launched → Must install certbot + request cert
   - Launch time: +2-3 minutes (certbot setup)
   - vs ACM: ALB already has cert, new EC2 ready instantly
   
 Renewal Failures:
   Let's Encrypt rate limits:
   - 50 certificates per domain per week
   - If Auto Scaling launches 60 instances → 10 fail (rate limit)
   - Result: 10 instances without valid certificates

When Let's Encrypt appropriate:
   Non-AWS environment (on-premises, other clouds)
   Single server (not auto-scaling)
   Full control requirement (private key on instance)

Why D is Wrong (Single Certificate with 1,000+ SANs):
 Certificate Size Limit:
   ACM limit: 10 SANs per certificate
   - domain.com (primary domain)
   - +9 SANs = 10 total
   
   Your need: 1,000+ customer subdomains
   - customer1.example.com
   - customer2.example.com
   - ...
   - customer1000.example.com
   
   Required: 1,000 SANs (exceeds limit by 100×!)
   
   Even if allowed:
   - Certificate size: ~500 KB (vs typical 2 KB)
   - TLS handshake: Slower (large cert transfer)
   - Management: Adding customer = reissue cert (for all 1,000)

 Wildcard Solution:
   Instead of 1,000 SANs: Use *.example.com wildcard
   - Covers customer1.example.com, customer2.example.com, ...
   - Certificate size: 2 KB (same as single domain)
   - Adding customer: No certificate change needed 

When multiple SANs appropriate:
   Few specific domains (2-10): www.example.com, api.example.com, example.com
   Many similar subdomains: Use wildcard instead

Advanced Certificate Scenarios:

1. Multi-Level Subdomains:
   Requirement: api.v2.example.com
   Solution:
   - Wildcard *.example.com: Doesn't cover (only single-level)
   - Wildcard *.v2.example.com: Covers api.v2.example.com 
   - Or explicit SAN: api.v2.example.com
   
   Terraform:
   resource "aws_acm_certificate" "multi_wildcard" {
     domain_name = "*.example.com"
     subject_alternative_names = [
       "*.v2.example.com",
       "*.api.example.com"
     ]
     validation_method = "DNS"
   }

2. Apex Domain + Wildcard:
   Requirement: example.com AND *.example.com
   Solution: Add apex as SAN
   
   resource "aws_acm_certificate" "wildcard_with_apex" {
     domain_name = "*.example.com"
     subject_alternative_names = ["example.com"]  # Apex domain
     validation_method = "DNS"
   }
   
   Covers:
   example.com
   www.example.com
   api.example.com
   customer123.example.com

3. Multiple Domains (Completely Different):
   Requirement: example.com, partner.com, another-domain.com
   Solution: Separate certificates (can't wildcard across different domains)
   
   resource "aws_acm_certificate" "example" {
     domain_name = "*.example.com"
   }
   
   resource "aws_acm_certificate" "partner" {
     domain_name = "*.partner.com"
   }
   
   CloudFront: Can attach multiple certificates to single distribution
   ALB: One certificate per listener (use SNI for multiple domains)

Certificate Monitoring:

CloudWatch Alarm (Certificate Expiration):
# ACM automatically renews, but alarm for safety
resource "aws_cloudwatch_metric_alarm" "cert_expiry" {
  alarm_name          = "acm-certificate-expiring"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "1"
  metric_name         = "DaysToExpiry"
  namespace           = "AWS/CertificateManager"
  period              = "86400"  # Daily
  statistic           = "Minimum"
  threshold           = "30"  # Alert if <30 days to expiry
  alarm_description   = "ACM certificate expiring soon"
  alarm_actions       = [aws_sns_topic.alerts.arn]
  
  dimensions = {
    CertificateArn = aws_acm_certificate.wildcard.arn
  }
}

Renewal Check (Lambda):
import boto3
from datetime import datetime, timedelta

def lambda_handler(event, context):
    acm = boto3.client('acm')
    
    # List all certificates
    certs = acm.list_certificates(CertificateStatuses=['ISSUED'])
    
    for cert_summary in certs['CertificateSummaryList']:
        cert = acm.describe_certificate(CertificateArn=cert_summary['CertificateArn'])
        not_after = cert['Certificate']['NotAfter']
        days_remaining = (not_after - datetime.now(not_after.tzinfo)).days
        
        if days_remaining < 30:
            print(f"WARNING: {cert_summary['DomainName']} expires in {days_remaining} days")
            # Send SNS notification
        else:
            print(f"OK: {cert_summary['DomainName']} expires in {days_remaining} days")

Real-World Example - Netflix:
Company: Netflix (estimated based on public information)
Domains: netflix.com, *.netflix.com, netflix.ca, netflix.co.uk (200+ country TLDs)

Certificate strategy:
- Primary: *.netflix.com (covers most subdomains)
- Country-specific: *.netflix.ca, *.netflix.co.uk (separate certs)
- Total certificates: ~250 (one per country TLD)
- Management: ACM for AWS-hosted, Let's Encrypt for CDN edge (Open Connect)

Automation:
- Terraform/IaC: All certificates defined in code
- Auto-renewal: ACM handles (zero manual intervention)
- Monitoring: Datadog tracks certificate expiration
- Incidents: Zero certificate-related outages (public record)

Scale:
- TLS handshakes: 100M+/day
- Certificate distribution: Edge servers worldwide (thousands)
- Cost: $0 for certificates (ACM + Let's Encrypt both free)

Best Practices:

1. Use ACM for All AWS Services:
   - CloudFront, ALB, API Gateway: ACM native support
   - Cost: FREE
   - Renewal: Automatic

2. DNS Validation Over Email:
   - DNS: Add CNAME once, renewal automatic forever
   - Email: Must click link for every renewal (manual)

3. Wildcard for Subdomains:
   - If >3 subdomains: Use wildcard (*.example.com)
   - If <3 specific domains: Use SANs (www.example.com, api.example.com)

4. Regional Certificates:
   - CloudFront: Cert in us-east-1 only
   - ALB: Cert in same region as ALB
   - Don't share certificates across regions (must create per region)

5. Monitor Expiration:
   - Even with auto-renewal, monitor as backup
   - CloudWatch alarm <30 days to expiry
   - SNS notification to operations team

6. Terraform/IaC:
   - Define certificates in code (version control)
   - Auto-create validation records
   - Deploy to multiple regions consistently

7. Certificate Rotation:
   - ACM rotates automatically (~60 days before expiry)
   - No action needed (transparent to services)
   - Old certificate deprecated gracefully (no downtime)

Why B is Correct (AS_PATH Prepending):

BGP Path Selection Process:

AWS receives two BGP advertisements for your network (192.168.0.0/16):

Path 1 (Direct Connect):
AS_PATH: 65000
Next-hop: Direct Connect virtual interface
MED: 0
Local Preference: Not applicable (received from external peer)

Path 2 (VPN):
AS_PATH: 65000 65000 65000  # Prepended (appears 3 hops away)
Next-hop: VPN tunnel
MED: 0
Local Preference: Not applicable

BGP Best Path Algorithm (in order):
1. Highest Weight (Cisco-specific, not applicable here)
2. Highest Local Preference (N/A - external BGP)
3. Locally originated routes (neither is local to AWS)
4. Shortest AS_PATH  THIS DECIDES!
   - Direct Connect AS_PATH length: 1 (65000)
   - VPN AS_PATH length: 3 (65000 65000 65000)
   - Winner: Direct Connect (shorter path) 
5. Lowest origin type (IGP < EGP < Incomplete)
6. Lowest MED (Multi-Exit Discriminator)
7. eBGP over iBGP
8. Lowest IGP metric to next-hop
9. Lowest router ID

Result: AWS prefers Direct Connect due to shorter AS_PATH

Failover Scenario:
Normal: Direct Connect UP → AS_PATH length 1 → Traffic uses Direct Connect
Failure: Direct Connect DOWN → Only VPN path available → Traffic uses VPN
Recovery: Direct Connect UP → AS_PATH length 1 again → Traffic returns to Direct Connect

Configuration Example (Your Router):

# Direct Connect BGP Configuration (Cisco IOS)
router bgp 65000
 bgp log-neighbor-changes
 neighbor 169.254.0.1 remote-as 7224  # AWS Direct Connect peer
 !
 address-family ipv4
  network 192.168.0.0 mask 255.255.0.0
  neighbor 169.254.0.1 activate
  neighbor 169.254.0.1 soft-reconfiguration inbound
  ! No AS_PATH prepending (advertise shortest path)
 exit-address-family

# VPN BGP Configuration
router bgp 65000
 neighbor 169.254.1.1 remote-as 7224  # AWS VPN tunnel 1 peer
 neighbor 169.254.1.5 remote-as 7224  # AWS VPN tunnel 2 peer
 !
 address-family ipv4
  network 192.168.0.0 mask 255.255.0.0
  ! Prepend AS 3 times (make path longer, less preferred)
  neighbor 169.254.1.1 route-map PREPEND-AS out
  neighbor 169.254.1.5 route-map PREPEND-AS out
  neighbor 169.254.1.1 activate
  neighbor 169.254.1.5 activate
 exit-address-family

# Route map for AS_PATH prepending
route-map PREPEND-AS permit 10
 set as-path prepend 65000 65000 65000
 ! Prepends your AS number 3 times
 ! Resulting AS_PATH: 65000 65000 65000 (length 3)

Verification Commands:

# On your router - Check BGP status
Router# show ip bgp summary
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
169.254.0.1     4  7224    5000    4995        12    0    0 7d23h           50  # Direct Connect
169.254.1.1     4  7224    5000    4995        12    0    0 7d23h           50  # VPN tunnel 1
169.254.1.5     4  7224    5000    4995        12    0    0 7d23h           50  # VPN tunnel 2

# Check advertised routes
Router# show ip bgp neighbors 169.254.0.1 advertised-routes
Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.0.0/16  0.0.0.0                  0         32768 i
# AS_PATH to AWS: (empty, since we're AS 65000 advertising to AWS)
# AWS sees: AS_PATH 65000 (length 1)

Router# show ip bgp neighbors 169.254.1.1 advertised-routes
Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.0.0/16  0.0.0.0                  0         32768 65000 65000 65000 i
# AS_PATH to AWS: 65000 65000 65000 (prepended)
# AWS sees: AS_PATH 65000 65000 65000 (length 3)

# On AWS side (via AWS Console or CLI)
$ aws ec2 describe-vpn-connections --query 'VpnConnections[0].Routes'
[
  {
    "DestinationCidrBlock": "192.168.0.0/16",
    "Source": "propagated",
    "State": "available"
  }
]

# Check Transit Gateway route table (if using TGW)
$ aws ec2 describe-transit-gateway-route-tables --query 'TransitGatewayRouteTables[0].Routes'
[
  {
    "DestinationCidrBlock": "192.168.0.0/16",
    "TransitGatewayAttachments": [
      {
        "ResourceId": "dxcon-fg5example",  # Direct Connect
        "ResourceType": "direct-connect-gateway",
        "State": "available"
      }
    ],
    "Type": "propagated",
    "State": "active"
  }
]
# Direct Connect route is active (preferred due to shorter AS_PATH)

Traffic Flow Analysis:

Normal Operation (Direct Connect UP):
   AWS VPC (10.0.0.0/16)
         ↓
   Transit Gateway
         ↓
   [Route table: 192.168.0.0/16 via Direct Connect]  ← Chosen (AS_PATH length 1)
   [Route table: 192.168.0.0/16 via VPN]             ← Not used (AS_PATH length 3)
         ↓
   Direct Connect Gateway
         ↓
   Direct Connect (10 Gbps)
         ↓
   Your datacenter (192.168.0.0/16)
   
   Latency: 5ms
   Bandwidth: 10 Gbps
   Cost: $0.02/GB outbound

Failover Scenario (Direct Connect DOWN):
   AWS VPC (10.0.0.0/16)
         ↓
   Transit Gateway
         ↓
   [Route table: 192.168.0.0/16 via Direct Connect]  ← UNAVAILABLE (path down)
   [Route table: 192.168.0.0/16 via VPN]             ← Activated (only available path)
         ↓
   Virtual Private Gateway
         ↓
   Site-to-Site VPN (2 × IPsec tunnels)
         ↓
   Your datacenter (192.168.0.0/16)
   
   Latency: 50ms (10× slower)
   Bandwidth: 1.25 Gbps (8× slower)
   Cost: $0.05/hour per tunnel
   
   Failover time: 30-180 seconds (BGP convergence)
   - BGP hold timer: 90 seconds (default)
   - Route withdrawal propagation: 10-30 seconds
   - New route advertisement: 10-30 seconds
   - Total: ~2-3 minutes worst case

Cost Comparison:

Normal operation (1000 GB/month outbound):
   Direct Connect: $1,620/month + (1,000 GB × $0.02) = $1,640/month
   VPN: $73/month (idle, no data transfer)
   Total: $1,713/month

During failover (assume 1% of time = 7.2 hours/month):
   Direct Connect: $1,620/month (port fee still charged)
   VPN data transfer: 1,000 GB × 1% × $0.09/GB = $0.90
   Total: $1,620.90/month
   
   Difference: Negligible (VPN backup adds minimal cost)

Why A is Wrong (Same AS_PATH Length):
 No Path Preference:
   
   Configuration:
   - Direct Connect: AS_PATH 65000 (length 1)
   - VPN: AS_PATH 65000 (length 1, no prepending)
   
   BGP decision:
   Step 4: AS_PATH length → TIE (both length 1)
   Step 5: Origin type → TIE (both IGP)
   Step 6: MED → TIE (both 0)
   Step 7: eBGP vs iBGP → TIE (both eBGP)
   Step 8: IGP metric → Depends on AWS internal routing
   Step 9: Router ID → AWS chooses arbitrary path
   
   Result: AWS might choose VPN instead of Direct Connect!
   - Or worse: Load balance across both (ECMP)
   - Asymmetric routing: Outbound via Direct Connect, inbound via VPN
   - Packet loss: TCP connections broken by asymmetric paths

 Asymmetric Routing Example:
   Outbound (AWS → Your datacenter):
   - AWS chooses Direct Connect (lower latency)
   - Packet: Source 10.0.1.50, Destination 192.168.0.100
   - Path: AWS VPC → Direct Connect → Datacenter
   
   Inbound (Your datacenter → AWS):
   - Your router chooses VPN (both paths equal, chooses based on local preference)
   - Packet: Source 192.168.0.100, Destination 10.0.1.50
   - Path: Datacenter → VPN → AWS VPC
   
   Problem: Different paths for each direction
   - Stateful firewalls: May drop packets (unexpected return path)
   - NAT devices: Connection tracking fails
   - Performance: VPN is slower, affects return traffic

Why C is Wrong (Local Preference):
 Local Preference Only Works for iBGP (Internal BGP):
   
   BGP attribute: Local Preference
   - Scope: Local to your AS (doesn't cross AS boundaries)
   - Used for: Choosing exit point from your AS
   - Propagated: Only within iBGP (same AS)
   - NOT propagated: Across eBGP (different AS)
   
   Your scenario:
   - Your AS: 65000
   - AWS AS: 7224
   - Connection: eBGP (External BGP, different AS)
   - Local Preference: Not sent to AWS 
   
   What happens:
   1. You set Local Preference 200 on Direct Connect
   2. Your router prefers Direct Connect for outbound traffic 
   3. You advertise routes to AWS via BGP
   4. Local Preference: NOT included in BGP advertisement to AWS
   5. AWS: Doesn't see Local Preference, ignores it
   
   Result: Solves outbound (your DC → AWS), doesn't solve inbound (AWS → your DC)

When Local Preference is appropriate:
   Scenario: Multiple routers in your AS, choosing which router sends to AWS
   
   Your network:
   Router A (New York) → Direct Connect to AWS us-east-1
   Router B (London) → Direct Connect to AWS eu-west-1
   
   Both routers: iBGP peers (same AS 65000)
   
   Configuration (Router A):
   router bgp 65000
    neighbor 10.0.0.2 remote-as 65000  # Router B, iBGP
    !
    address-family ipv4
     ! Set Local Preference 200 for routes learned from AWS us-east-1
     neighbor 169.254.0.1 route-map SET-LOCAL-PREF in
   
   route-map SET-LOCAL-PREF permit 10
    set local-preference 200
   
   Result: All routers in AS 65000 prefer paths via Router A (higher Local Pref)

Why D is Wrong (ECMP - Equal-Cost Multi-Path):
 ECMP Causes Packet Reordering and Connection Issues:
   
   ECMP configuration:
   - Advertise same AS_PATH length on both paths (no prepending)
   - AWS enables ECMP (load balances across both paths)
   - Traffic split: 50% Direct Connect, 50% VPN
   
   Problems:
   1. Performance Mismatch:
      - Direct Connect: 5ms latency, 10 Gbps
      - VPN: 50ms latency, 1.25 Gbps
      - Result: Half of packets take 10× longer!
   
   2. TCP Reordering:
      Flow: AWS VPC (10.0.1.50) → Your server (192.168.0.100)
      - Packet 1: Via Direct Connect (arrives in 5ms)
      - Packet 2: Via VPN (arrives in 50ms)
      - Packet 3: Via Direct Connect (arrives in 10ms)
      - Your server: Receives out of order (1, 3, 2)
      - TCP: Duplicate ACKs, triggers fast retransmit
      - Performance: 50% throughput reduction
   
   3. Bandwidth Imbalance:
      - ECMP: Splits traffic 50/50 by number of flows
      - But: VPN max 1.25 Gbps, Direct Connect max 10 Gbps
      - If VPN saturated: 50% of flows experience packet loss
   
   4. Cost Increase:
      - VPN: $0.09/GB (4.5× more expensive than Direct Connect $0.02/GB)
      - 50% of traffic via VPN: Unnecessary cost increase

When ECMP is appropriate:
   Scenario: Multiple Direct Connect connections of same speed
   
   Your network:
   Direct Connect 1: 10 Gbps to us-east-1a
   Direct Connect 2: 10 Gbps to us-east-1b
   
   Both: Same latency, same bandwidth, same cost
   ECMP: Doubles throughput to 20 Gbps 
   
   Configuration:
   - Advertise same AS_PATH on both Direct Connects
   - AWS load balances across both
   - No performance mismatch (identical paths)

Advanced BGP Tuning:

1. BFD (Bidirectional Forwarding Detection):
   Purpose: Faster failover detection
   
   Problem: BGP hold timer default 90 seconds
   - Direct Connect fails at 10:00:00
   - BGP detects at 10:01:30 (90 seconds later)
   - Total downtime: 90 seconds
   
   Solution: Enable BFD (detects failures in <1 second)
   
   Configuration:
   interface TenGigabitEthernet0/0/0
    bfd interval 300 min_rx 300 multiplier 3
    ! Check every 300ms, 3 missed = failure (900ms)
   
   router bgp 65000
    neighbor 169.254.0.1 fall-over bfd
   
   Result: Failover in <1 second (vs 90 seconds)

2. BGP Graceful Restart:
   Purpose: Avoid route flapping during planned maintenance
   
   Scenario: AWS Direct Connect maintenance (15 minutes)
   Without graceful restart:
   - BGP session down → All routes withdrawn
   - Traffic fails over to VPN
   - 15 minutes later: Direct Connect up → Routes re-advertised
   - Traffic returns to Direct Connect
   - Flapping: Causes brief outages during transitions
   
   With graceful restart:
   - BGP session down → Routes marked "stale" (not withdrawn)
   - Traffic continues using "stale" routes (Direct Connect)
   - If Direct Connect actually down: Timer expires, switch to VPN
   - If maintenance: Direct Connect up before timer, no failover
   
   Configuration:
   router bgp 65000
    bgp graceful-restart restart-time 120
    neighbor 169.254.0.1 remote-as 7224
    neighbor 169.254.0.1 activate
    neighbor 169.254.0.1 send-community
    neighbor 169.254.0.1 capability graceful-restart

3. Community Tags (Advanced Routing Control):
   Purpose: Tag routes for AWS to handle specially
   
   AWS communities:
   - 7224:9100 (Local Preference 100)
   - 7224:9200 (Local Preference 200)
   - 7224:9300 (Local Preference 300)
   
   Use case: Prefer Direct Connect even more
   
   Configuration:
   route-map SET-COMMUNITY permit 10
    set community 7224:9300  # Highest Local Pref in AWS
   
   router bgp 65000
    neighbor 169.254.0.1 route-map SET-COMMUNITY out
   
   Result: AWS VPC strongly prefers Direct Connect path

Real-World Example - Enterprise Hybrid Cloud:
Company: Large financial services firm
Workload: Trading platform (latency-sensitive)
Architecture:
   - Primary: AWS Direct Connect (100 Gbps)
   - Backup: 2× Site-to-Site VPN (1.25 Gbps each)

BGP configuration:
   Direct Connect: AS_PATH 65001 (length 1)
   VPN 1: AS_PATH 65001 65001 65001 (prepended 3×, length 3)
   VPN 2: AS_PATH 65001 65001 65001 (prepended 3×, length 3)

Performance:
   Normal: 99.9% traffic via Direct Connect, <5ms latency
   Failover: VPN activates within 30 seconds (BFD enabled)
   Recovery: Automatic return to Direct Connect
   
   Failover events: 2× per year (planned maintenance)
   Unplanned outages: 0 in last 3 years

Cost:
   Direct Connect: $16,200/month (100 Gbps port)
   VPN: $146/month (2 tunnels × 2 connections)
   Data transfer: $40K/month (2 PB/month × $0.02/GB)
   Total: $56,346/month
   
   VPN cost during failover: Negligible (<$100/month)

Monitoring:

CloudWatch Metrics:
1. Direct Connect:
   - ConnectionState: UP/DOWN
   - ConnectionBpsEgress: Outbound throughput
   - ConnectionPpsEgress: Packets per second
   
2. VPN:
   - TunnelState: UP/DOWN
   - TunnelDataIn: Bytes received (should be near-zero normally)
   - TunnelDataOut: Bytes sent
   
3. Alarms:
   - Direct Connect DOWN + VPN traffic >1 GB/min → SNS alert
   - Both paths DOWN → PagerDuty critical

BGP Monitoring:
# Check BGP neighbor status
$ watch -n 5 'show ip bgp summary'

# Alert if BGP peer down
if [ $(show ip bgp summary | grep -c "169.254.0.1.*Active") -gt 0 ]; then
  echo "CRITICAL: Direct Connect BGP peer down" | mail -s "BGP Alert" ops@company.com
fi

Best Practices:

1. Always prepend AS_PATH for backup paths (3× typical)
2. Enable BFD for fast failover (<1 second)
3. Test failover quarterly (planned maintenance window)
4. Monitor BGP metrics (peer status, route advertisements)
5. Document BGP configuration (route maps, communities)
6. Use identical BGP timers on all peers (hold timer, keepalive)
7. Configure graceful restart for planned maintenance
8. Audit route advertisements (ensure prepending works)
9. Set up alerting for asymmetric routing (netflow analysis)
10. Have rollback plan (remove prepending if issues)

# Attacker tries to copy data to their bucket
aws s3 cp s3://company-medical-records/patient-records.csv s3://attacker-bucket/stolen/

# Error:
# An error occurred (AccessDenied) when calling the CopyObject operation:
# VPC endpoint policy does not allow access to this bucket

import boto3
import json

iam = boto3.client('iam')
sns = boto3.client('sns')

def lambda_handler(event, context):
    finding = event['detail']
    
    # Extract user identity
    user_arn = finding['resource']['accessKeyDetails']['userArn']
    access_key_id = finding['resource']['accessKeyDetails']['accessKeyId']
    
    # Severity
    severity = finding['severity']
    
    if severity >= 7:  # HIGH severity
        # IMMEDIATE ACTION: Disable access key
        response = iam.update_access_key(
            UserName=user_arn.split('/')[-1],
            AccessKeyId=access_key_id,
            Status='Inactive'
        )
        
        action_taken = f"DISABLED access key {access_key_id}"
        
        # Notify security team
        sns.publish(
            TopicArn='arn:aws:sns:us-east-1:123456789012:security-incidents',
            Subject=f'CRITICAL: Data Exfiltration Detected - {user_arn}',
            Message=json.dumps({
                'Action': action_taken,
                'User': user_arn,
                'AccessKey': access_key_id,
                'Finding': finding['type'],
                'Description': finding['description'],
                'Time': finding['createdAt'],
                'Region': finding['region']
            }, indent=2)
        )
        
        return {
            'statusCode': 200,
            'body': f'Remediation: {action_taken}'
        }

index=cloudtrail eventName=GetObject bucket=company-medical-records
| stats sum(bytesTransferredOut) as total_bytes by userIdentity.principalId
| where total_bytes > 1073741824  # 1 GB threshold
| table userIdentity.principalId, total_bytes, _time
| sort -total_bytes