Configuration Management using AWS SSM for EC2/EKS

Overview

This document outlines our environment configuration management strategy using AWS Systems Manager Parameter Store. The system is designed to manage Docker Compose environment variables across different deployment environments (DEV, BETA, PROD).

Infrastructure Components

• AWS Systems Manager Parameter Store: Central configuration storage
• EC2 instances: Application hosting environment
• EKS: Configs and Secrets management
• IAM Roles: Access control for configurations
• Shell Scripts: Configuration deployment utilities

Flow Diagram

%%{init: {"theme": "redux", "look":"neo"} }%%



flowchart TD
 subgraph AWS["AWS Services"]
    direction LR
        SSM["AWS SSM Parameter Store"]

        IAM["IAM Roles & Policies"]

  end

 subgraph DEV["Development Environment (EC2)"]

        D_Params["Dev Parameters"]

        D_Script["generate"]

        D_Env[".env"]

        D_Docker["Docker Compose"]

        D_App["Application"]

  end

 subgraph BETA["Beta Environment (EC2)"]

        B_Params["Beta Parameters"]

        B_Script["generate"]

        B_Env[".env"]

        B_Docker["Docker Compose"]

        B_App["Application"]

  end

 subgraph EKS_Components["EKS Components"]

        IRSA["IAM Role"]

        SecretStore["External Secrets Operator"]

        ConfigMap["ConfigMaps"]

        K8S_Secret["Kubernetes Secrets"]

  end

 subgraph EKS_Deploy["Deployment"]

        Pod1["Pod 1"]

        Pod2["Pod 2"]

        Pod3["Pod 3"]

  end

 subgraph EKS["Production Environment (EKS)"]

    direction TB

        EKS_Components

        EKS_Deploy

  end

    SSM --> D_Params & B_Params & SecretStore

    IRSA --> SecretStore

    D_Params --> D_Script

    D_Script --> D_Env

    D_Env --> D_Docker

    D_Docker --> D_App

    B_Params --> B_Script

    B_Script --> B_Env

    B_Env --> B_Docker

    B_Docker --> B_App

    SecretStore --> K8S_Secret & ConfigMap

    K8S_Secret --> Pod1 & Pod2 & Pod3

    ConfigMap --> Pod1 & Pod2 & Pod3

     SSM:::aws

     IAM:::aws

     D_Script:::script

     D_Docker:::docker

     D_App:::app

     B_Script:::script

     B_Docker:::docker

     B_App:::app

     SecretStore:::k8s

     ConfigMap:::k8s

     K8S_Secret:::k8s

     Pod1:::k8s

     Pod2:::k8s

     Pod3:::k8s

    classDef aws fill:#FF9900,stroke:#232F3E,stroke-width:2px,color:white

    classDef env fill:#E8F6F3,stroke:#148F77,stroke-width:2px

    classDef role fill:#D4EFDF,stroke:#27AE60,stroke-width:2px

    classDef script fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px

    classDef docker fill:#D6EAF8,stroke:#2E86C1,stroke-width:2px

    classDef app fill:#FCF3CF,stroke:#D4AC0D,stroke-width:2px

    classDef eks fill:#F5EEF8,stroke:#8E44AD,stroke-width:2px

    classDef k8s fill:#EBF5FB,stroke:#3498DB,stroke-width:2px

    style EKS fill:#BBDEFB

    style BETA fill:#E1BEE7

    style DEV fill:#C8E6C9

Loading

Parameter Store Structure

Parameters are organized hierarchically in SSM Parameter Store using the following convention:

/[application]/[environment]/[parameter_name]

Example:
/myapp/dev/DB_HOST
/myapp/dev/DB_PORT
/myapp/beta/DB_HOST
/myapp/prod/DB_HOST

Environment Parameters

Common parameters across environments:

DATABASE_CONFIG
├── DB_HOST
├── DB_PORT
├── DB_NAME
├── DB_USER
└── DB_PASSWORD (SecureString)

APP_CONFIG
├── API_KEY (SecureString)
├── LOG_LEVEL
├── APP_PORT
└── DEBUG_MODE

Parameter Versioning

• All parameters maintain version history in SSM
• Latest version is automatically used
• Version labels:
  • latest: Most recent version
  • stable: Last known working version
  • previous: Prior version for rollback

Version Management Guidelines

1. Always include change description when updating parameters
2. Test parameter changes in DEV before promoting to higher environments
3. Maintain at least 5 previous versions for rollback capability
4. Document version changes in change log

IAM Roles & Permissions

EC2 Instance Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:GetParametersByPath"
            ],
            "Resource": [
                "arn:aws:ssm:region:account:parameter/myapp/dev/*"
            ]
        }
    ]
}

Environment-Specific Access

• DEV environment: Full read/write access to /myapp/dev/*
• BETA environment: Read-only access to /myapp/beta/*
• PROD environment: Read-only access to /myapp/prod/*

Deployment Configuration

Pre-deployment Script

Location: /opt/myapp/scripts/generate-env.sh

Purpose: Generates environment-specific .env file from SSM parameters

Usage: ./generate-env.sh <environment>
Example: ./generate-env.sh dev

Deployment Flow

1. CI/CD pipeline triggers deployment
2. Generate-env script pulls parameters from SSM
3. .env file is created in application directory
4. Docker Compose uses .env file for environment variables
5. Application containers are started

Deployment Verification

1. Parameter retrieval validation
2. Environment file syntax check
3. Container environment variable verification
4. Application health check

Security Guidelines

Parameter Security

1. Use SecureString for sensitive values
2. Encrypt parameters using AWS KMS
3. Regular rotation of sensitive parameters
4. Audit parameter access through CloudTrail

Access Control

1. Principle of least privilege
2. Environment isolation
3. Regular access review
4. MFA for parameter modifications

Monitoring and Alerts

CloudWatch Alarms

• Parameter changes in PROD
• Failed parameter access attempts
• Configuration retrieval failures

Logging

• Parameter version changes
• Access patterns
• Configuration updates
• Deployment events

Disaster Recovery

Backup Procedures

1. Daily parameter snapshot
2. Cross-region replication for PROD
3. Version history maintenance
4. Regular recovery testing

Recovery Steps

1. Identify last known good configuration
2. Restore parameters from backup/previous version
3. Verify parameter integrity
4. Update application configuration
5. Validate application functionality

Change Management

Parameter Updates

1. Create change request
2. Test in lower environment
3. Peer review required
4. Schedule update window
5. Apply changes
6. Verify application stability

Emergency Changes

1. Emergency change approval process
2. Immediate rollback plan
3. Post-change review
4. Documentation update

Best Practices

Parameter Management

1. Use descriptive parameter names
2. Include parameter descriptions
3. Maintain parameter documentation
4. Regular parameter review
5. Clean up unused parameters

Development Guidelines

1. Never hardcode environment values
2. Use parameter hierarchies effectively
3. Implement proper error handling
4. Regular testing of configuration changes

Troubleshooting Guide

Common Issues

1. Parameter access denied

   • Verify IAM roles and permissions
   • Check parameter path
   • Validate AWS credentials

2. Missing parameters

   • Verify parameter existence
   • Check environment name
   • Validate parameter path

3. Invalid configuration

   • Verify parameter format
   • Check environment file syntax
   • Validate parameter values