Configuration Merge Pattern: Secure Config Management Made Simple

The Security Problem

🚨 Never Commit Secrets to Version Control

In production applications, you face a critical security challenge: how to manage configuration without exposing sensitive data.

The Dilemma:

• ✅ Non-sensitive config (hosts, ports, timeouts) → Safe to version control

• ❌ Sensitive config (passwords, API keys, certificates) → Must NEVER be in version control

What Goes Wrong:

# ❌ DANGEROUS: All config in one file
config = {
    "database": {
        "host": "prod-db.company.com",      # Safe to share
        "port": 5432,                       # Safe to share
        "username": "app_user",             # Safe to share
        "password": "super_secret_123!"     # 🚨 LEAKED TO GIT!
    }
}

Real-world consequences:

• 🔓 Credentials exposed in git history (permanent damage)

• 🤖 Bots scraping GitHub for secrets (immediate exploitation)

• 👥 Team members accidentally access production secrets

• 🔄 Copy-paste errors spreading secrets across environments

---

Why Existing Solutions Fall Short

The Problems with Manual Config Assembly

Let’s see what happens when developers try to solve this manually:

# Setup for demonstration
import json
from rich import print as rprint

def jprint(data: dict):
    """Pretty print JSON data"""
    rprint(json.dumps(data, indent=2))

# ❌ Attempt 1: Simple dict.update()
base_config = {
    "database": {"host": "prod-db.com", "port": 5432, "pool_size": 20},
    "cache": {"host": "redis.com", "ttl": 3600}
}

secrets = {
    "database": {"password": "secret123"},  # Only has password
    "cache": {"password": "redis-secret"}   # Only has password
}

# This OVERWRITES the entire nested dict!
broken_config = base_config.copy()
broken_config.update(secrets)

print("❌ Broken result with dict.update():")
jprint(broken_config)

❌ Broken result with dict.update():

{
  "database": {
    "password": "secret123"
  },
  "cache": {
    "password": "redis-secret"
  }
}

# ❌ Attempt 2: Manual nested merging
def manual_merge(base, secrets):
    result = base.copy()
    for key, value in secrets.items():
        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
            # Manual nested merge - but what about deeper nesting? Lists?
            result[key].update(value)
        else:
            result[key] = value
    return result

manual_result = manual_merge(base_config, secrets)
print("⚠️ Manual merge - works but limited:")
jprint(manual_result)

⚠️ Manual merge - works but limited:

{
  "database": {
    "host": "prod-db.com",
    "port": 5432,
    "pool_size": 20,
    "password": "secret123"
  },
  "cache": {
    "host": "redis.com",
    "ttl": 3600,
    "password": "redis-secret"
  }
}

Problems with manual approaches:

• 🔄 Doesn’t handle deep nesting (3+ levels)

• 📋 Can’t merge lists (loses relationships)

• 🐛 No type validation (silent data corruption)

• 🚀 Not reusable (reimplemented everywhere)

• ❌ No error reporting (fails silently)

---

The Smart Merge Solution

Introducing deep_merge

The deep_merge function provides intelligent, structure-aware merging that solves all the problems above:

from configcraft.api import deep_merge

# ✅ The same data as before
base_config = {
    "database": {"host": "prod-db.com", "port": 5432, "pool_size": 20},
    "cache": {"host": "redis.com", "ttl": 3600}
}

secrets = {
    "database": {"password": "secret123"},
    "cache": {"password": "redis-secret"}
}

# ✅ Smart merging
smart_result = deep_merge(base_config, secrets)
print("✅ Smart merge result:")
jprint(smart_result)

✅ Smart merge result:

{
  "database": {
    "host": "prod-db.com",
    "port": 5432,
    "pool_size": 20,
    "password": "secret123"
  },
  "cache": {
    "host": "redis.com",
    "ttl": 3600,
    "password": "redis-secret"
  }
}

Why This Works Better

🧠 Intelligence: Understands data structure and preserves relationships

🛡️ Safety: Immutable operations (never modifies originals)

🔍 Validation: Type-checked merging with clear error messages

📈 Scalability: Handles arbitrary nesting depth and complexity

---

Core Merge Behaviors

Understanding how deep_merge works will help you predict and control the merge behavior:

1. Adding New Keys (No Conflicts)

When keys exist in only one dictionary, they’re simply added:

config_data = {
    "dev": {"host": "dev-server.com"}
}
secrets_data = {
    "prod": {"host": "prod-server.com"}  # Completely new environment
}

result = deep_merge(config_data, secrets_data)
print("New keys are simply added:")
jprint(result)

New keys are simply added:

{
  "dev": {
    "host": "dev-server.com"
  },
  "prod": {
    "host": "prod-server.com"
  }
}

2. Recursive Dictionary Merging

When both inputs have the same key with dictionary values, they merge recursively:

config_data = {
    "database": {
        "host": "db.company.com",
        "port": 5432,
        "connection": {
            "timeout": 30,
            "retry_attempts": 3
        }
    }
}

secrets_data = {
    "database": {
        "username": "app_user",
        "password": "secret_password",
        "connection": {
            "ssl_cert": "/path/to/cert.pem"
        }
    }
}

result = deep_merge(config_data, secrets_data)
print("Recursive dictionary merging:")
jprint(result)

Recursive dictionary merging:

{
  "database": {
    "host": "db.company.com",
    "port": 5432,
    "connection": {
      "timeout": 30,
      "retry_attempts": 3,
      "ssl_cert": "/path/to/cert.pem"
    },
    "password": "secret_password",
    "username": "app_user"
  }
}

3. Positional List Merging

Critical Behavior: Lists are merged by position to maintain relationships:

# User configuration with roles
user_config = {
    "users": [
        {"username": "alice", "role": "admin", "department": "engineering"},
        {"username": "bob", "role": "user", "department": "sales"},
        {"username": "charlie", "role": "moderator", "department": "support"}
    ]
}

# Corresponding passwords (same order!)
password_config = {
    "users": [
        {"password": "alice_secure_123"},      # For alice
        {"password": "bob_password_456"},      # For bob
        {"password": "charlie_secret_789"}     # For charlie
    ]
}

result = deep_merge(user_config, password_config)
print("Positional list merging maintains relationships:")
jprint(result)

Positional list merging maintains relationships:

{
  "users": [
    {
      "username": "alice",
      "role": "admin",
      "department": "engineering",
      "password": "alice_secure_123"
    },
    {
      "username": "bob",
      "role": "user",
      "department": "sales",
      "password": "bob_password_456"
    },
    {
      "username": "charlie",
      "role": "moderator",
      "department": "support",
      "password": "charlie_secret_789"
    }
  ]
}

Why positional merging matters:

• 🎯 Maintains relationships: user[0] password matches user[0] username

• 🔒 Security critical: Wrong password assignments = security breach

• 📊 Data integrity: Preserves logical connections between data elements

---

Step-by-Step Guide

Basic Configuration + Secrets Pattern

The most common use case: separating configuration from secrets.

Step 1: Create your base configuration

# config.json - Safe to commit to version control
base_config = {
    "app_name": "MyApplication",
    "environments": {
        "dev": {
            "database": {
                "host": "dev-db.company.com",
                "port": 5432,
                "name": "myapp_dev",
                "pool_size": 5
            },
            "redis": {
                "host": "dev-redis.company.com",
                "port": 6379,
                "db": 0
            },
            "features": {
                "debug_mode": True,
                "rate_limiting": False
            }
        },
        "prod": {
            "database": {
                "host": "prod-db.company.com",
                "port": 5432,
                "name": "myapp_prod",
                "pool_size": 20
            },
            "redis": {
                "host": "prod-redis.company.com",
                "port": 6379,
                "db": 1
            },
            "features": {
                "debug_mode": False,
                "rate_limiting": True
            }
        }
    }
}

print("📄 Base configuration (safe to commit):")
jprint(base_config)

📄 Base configuration (safe to commit):

{
  "app_name": "MyApplication",
  "environments": {
    "dev": {
      "database": {
        "host": "dev-db.company.com",
        "port": 5432,
        "name": "myapp_dev",
        "pool_size": 5
      },
      "redis": {
        "host": "dev-redis.company.com",
        "port": 6379,
        "db": 0
      },
      "features": {
        "debug_mode": true,
        "rate_limiting": false
      }
    },
    "prod": {
      "database": {
        "host": "prod-db.company.com",
        "port": 5432,
        "name": "myapp_prod",
        "pool_size": 20
      },
      "redis": {
        "host": "prod-redis.company.com",
        "port": 6379,
        "db": 1
      },
      "features": {
        "debug_mode": false,
        "rate_limiting": true
      }
    }
  }
}

Step 2: Create your secrets configuration

# secrets.json - NEVER commit to version control
# Load from environment variables, secret management system, etc.
secrets_config = {
    "environments": {
        "dev": {
            "database": {
                "username": "dev_user",
                "password": "dev_secret_123"
            },
            "redis": {
                "password": "dev_redis_pwd"
            },
            "api_keys": {
                "stripe": "sk_test_dev_key_123",
                "sendgrid": "SG.dev.api.key"
            }
        },
        "prod": {
            "database": {
                "username": "prod_user",
                "password": "super_secure_prod_password_456"
            },
            "redis": {
                "password": "prod_redis_secure_789"
            },
            "api_keys": {
                "stripe": "sk_live_prod_key_789",
                "sendgrid": "SG.prod.live.key"
            }
        }
    }
}

print("🔒 Secrets configuration (NEVER commit):")
jprint(secrets_config)

🔒 Secrets configuration (NEVER commit):

{
  "environments": {
    "dev": {
      "database": {
        "username": "dev_user",
        "password": "dev_secret_123"
      },
      "redis": {
        "password": "dev_redis_pwd"
      },
      "api_keys": {
        "stripe": "sk_test_dev_key_123",
        "sendgrid": "SG.dev.api.key"
      }
    },
    "prod": {
      "database": {
        "username": "prod_user",
        "password": "super_secure_prod_password_456"
      },
      "redis": {
        "password": "prod_redis_secure_789"
      },
      "api_keys": {
        "stripe": "sk_live_prod_key_789",
        "sendgrid": "SG.prod.live.key"
      }
    }
  }
}

Step 3: Merge them safely

# Merge configurations
final_config = deep_merge(base_config, secrets_config)

print("✅ Final merged configuration:")
jprint(final_config)

✅ Final merged configuration:

{
  "app_name": "MyApplication",
  "environments": {
    "dev": {
      "database": {
        "host": "dev-db.company.com",
        "port": 5432,
        "name": "myapp_dev",
        "pool_size": 5,
        "password": "dev_secret_123",
        "username": "dev_user"
      },
      "redis": {
        "host": "dev-redis.company.com",
        "port": 6379,
        "db": 0,
        "password": "dev_redis_pwd"
      },
      "features": {
        "debug_mode": true,
        "rate_limiting": false
      },
      "api_keys": {
        "stripe": "sk_test_dev_key_123",
        "sendgrid": "SG.dev.api.key"
      }
    },
    "prod": {
      "database": {
        "host": "prod-db.company.com",
        "port": 5432,
        "name": "myapp_prod",
        "pool_size": 20,
        "password": "super_secure_prod_password_456",
        "username": "prod_user"
      },
      "redis": {
        "host": "prod-redis.company.com",
        "port": 6379,
        "db": 1,
        "password": "prod_redis_secure_789"
      },
      "features": {
        "debug_mode": false,
        "rate_limiting": true
      },
      "api_keys": {
        "stripe": "sk_live_prod_key_789",
        "sendgrid": "SG.prod.live.key"
      }
    }
  }
}

Step 4: Use environment-specific config

# Extract environment-specific configuration
env = "dev"  # or "prod" in production
app_config = final_config["environments"][env]

print(f"🎯 Configuration for {env} environment:")
jprint(app_config)

# Now you can safely use this config in your application
# database_url = f"postgresql://{app_config['database']['username']}:{app_config['database']['password']}@{app_config['database']['host']}:{app_config['database']['port']}/{app_config['database']['name']}"

🎯 Configuration for dev environment:

{
  "database": {
    "host": "dev-db.company.com",
    "port": 5432,
    "name": "myapp_dev",
    "pool_size": 5,
    "password": "dev_secret_123",
    "username": "dev_user"
  },
  "redis": {
    "host": "dev-redis.company.com",
    "port": 6379,
    "db": 0,
    "password": "dev_redis_pwd"
  },
  "features": {
    "debug_mode": true,
    "rate_limiting": false
  },
  "api_keys": {
    "stripe": "sk_test_dev_key_123",
    "sendgrid": "SG.dev.api.key"
  }
}

---

Advanced Use Cases

Complex List Merging with Multiple Services

Real-world applications often have lists of services, databases, or API endpoints:

# Base service configuration
service_config = {
    "microservices": [
        {
            "name": "user-service",
            "port": 8001,
            "replicas": 3,
            "health_check": "/health"
        },
        {
            "name": "order-service",
            "port": 8002,
            "replicas": 2,
            "health_check": "/status"
        },
        {
            "name": "payment-service",
            "port": 8003,
            "replicas": 5,
            "health_check": "/ping"
        }
    ]
}

# Service secrets (API keys, database passwords, etc.)
service_secrets = {
    "microservices": [
        {
            "api_key": "user_service_key_123",
            "db_password": "user_db_secret"
        },
        {
            "api_key": "order_service_key_456",
            "db_password": "order_db_secret"
        },
        {
            "api_key": "payment_service_key_789",
            "db_password": "payment_db_secret"
        }
    ]
}

merged_services = deep_merge(service_config, service_secrets)
print("🔧 Merged service configuration:")
jprint(merged_services)

🔧 Merged service configuration:

{
  "microservices": [
    {
      "name": "user-service",
      "port": 8001,
      "replicas": 3,
      "health_check": "/health",
      "db_password": "user_db_secret",
      "api_key": "user_service_key_123"
    },
    {
      "name": "order-service",
      "port": 8002,
      "replicas": 2,
      "health_check": "/status",
      "db_password": "order_db_secret",
      "api_key": "order_service_key_456"
    },
    {
      "name": "payment-service",
      "port": 8003,
      "replicas": 5,
      "health_check": "/ping",
      "db_password": "payment_db_secret",
      "api_key": "payment_service_key_789"
    }
  ]
}

Multi-Environment Database Configuration

# Database topology configuration
db_topology = {
    "environments": {
        "dev": {
            "databases": [
                {"role": "primary", "host": "dev-db-1.internal", "port": 5432},
                {"role": "replica", "host": "dev-db-2.internal", "port": 5432}
            ]
        },
        "prod": {
            "databases": [
                {"role": "primary", "host": "prod-db-1.internal", "port": 5432},
                {"role": "replica", "host": "prod-db-2.internal", "port": 5432},
                {"role": "replica", "host": "prod-db-3.internal", "port": 5432}
            ]
        }
    }
}

# Database credentials (different for each database)
db_credentials = {
    "environments": {
        "dev": {
            "databases": [
                {"username": "dev_primary_user", "password": "dev_primary_secret"},
                {"username": "dev_replica_user", "password": "dev_replica_secret"}
            ]
        },
        "prod": {
            "databases": [
                {"username": "prod_primary_user", "password": "prod_primary_secret"},
                {"username": "prod_replica_user", "password": "prod_replica_secret_1"},
                {"username": "prod_replica_user", "password": "prod_replica_secret_2"}
            ]
        }
    }
}

merged_db_config = deep_merge(db_topology, db_credentials)
print("🗄️ Complete database configuration:")
jprint(merged_db_config)

🗄️ Complete database configuration:

{
  "environments": {
    "dev": {
      "databases": [
        {
          "role": "primary",
          "host": "dev-db-1.internal",
          "port": 5432,
          "password": "dev_primary_secret",
          "username": "dev_primary_user"
        },
        {
          "role": "replica",
          "host": "dev-db-2.internal",
          "port": 5432,
          "password": "dev_replica_secret",
          "username": "dev_replica_user"
        }
      ]
    },
    "prod": {
      "databases": [
        {
          "role": "primary",
          "host": "prod-db-1.internal",
          "port": 5432,
          "password": "prod_primary_secret",
          "username": "prod_primary_user"
        },
        {
          "role": "replica",
          "host": "prod-db-2.internal",
          "port": 5432,
          "password": "prod_replica_secret_1",
          "username": "prod_replica_user"
        },
        {
          "role": "replica",
          "host": "prod-db-3.internal",
          "port": 5432,
          "password": "prod_replica_secret_2",
          "username": "prod_replica_user"
        }
      ]
    }
  }
}

---

Error Handling & Troubleshooting

Understanding when and why deep_merge fails helps you design better configuration structures:

1. List Length Mismatches

Problem: Lists must have the same length to maintain positional relationships.

# ❌ This will fail - different number of items
try:
    config_with_mismatch = {
        "users": [
            {"username": "alice"},
            {"username": "bob"},
            {"username": "charlie"}  # 3 users
        ]
    }

    secrets_with_mismatch = {
        "users": [
            {"password": "alice_pwd"},
            {"password": "bob_pwd"}  # Only 2 passwords!
        ]
    }

    deep_merge(config_with_mismatch, secrets_with_mismatch)

except ValueError as e:
    print(f"❌ Error: {e}")

❌ Error: list length mismatch: path = '.users'

Solution: Ensure lists have matching lengths:

# ✅ Fixed - same number of items
config_fixed = {
    "users": [
        {"username": "alice"},
        {"username": "bob"},
        {"username": "charlie"}
    ]
}

secrets_fixed = {
    "users": [
        {"password": "alice_pwd"},
        {"password": "bob_pwd"},
        {"password": "charlie_pwd"}  # Now we have 3 passwords
    ]
}

result = deep_merge(config_fixed, secrets_fixed)
print("✅ Fixed - all users have passwords:")
jprint(result)

✅ Fixed - all users have passwords:

{
  "users": [
    {
      "username": "alice",
      "password": "alice_pwd"
    },
    {
      "username": "bob",
      "password": "bob_pwd"
    },
    {
      "username": "charlie",
      "password": "charlie_pwd"
    }
  ]
}

2. Type Incompatibility

Problem: Can’t merge different data types (string with dict, etc.).

# ❌ This will fail - trying to merge incompatible types
try:
    incompatible_config = {
        "database": "simple_connection_string"  # String
    }

    incompatible_secrets = {
        "database": {"password": "secret"}      # Dict
    }

    deep_merge(incompatible_config, incompatible_secrets)

except TypeError as e:
    print(f"❌ Error: {e}")

❌ Error: type of value at '.database' in data1 and data2 has to be both dict or list of dict to merge! they are <class 'str'> and <class 'dict'>.

Solution: Ensure compatible data structures:

# ✅ Fixed - both are dictionaries
compatible_config = {
    "database": {"connection_string": "postgresql://host:port/db"}  # Dict
}

compatible_secrets = {
    "database": {"password": "secret"}  # Dict
}

result = deep_merge(compatible_config, compatible_secrets)
print("✅ Fixed - compatible types:")
jprint(result)

✅ Fixed - compatible types:

{
  "database": {
    "connection_string": "postgresql://host:port/db",
    "password": "secret"
  }
}

3. Non-Dict Items in Lists

Problem: List items must be dictionaries to merge properly.

# ❌ This will fail - list contains non-dict items
try:
    config_with_scalars = {
        "ports": [8001, 8002, 8003]  # Numbers, not dicts
    }

    secrets_with_scalars = {
        "ports": [9001, 9002, 9003]  # Numbers, not dicts
    }

    deep_merge(config_with_scalars, secrets_with_scalars)

except TypeError as e:
    print(f"❌ Error: {e}")

❌ Error: items in '.ports' are not dict, so you cannot merge them!

Solution: Use dictionaries in lists when merging is needed:

# ✅ Fixed - use dicts in lists
config_with_dicts = {
    "services": [
        {"name": "api", "port": 8001},
        {"name": "worker", "port": 8002},
        {"name": "scheduler", "port": 8003}
    ]
}

secrets_with_dicts = {
    "services": [
        {"api_key": "api_secret"},
        {"api_key": "worker_secret"},
        {"api_key": "scheduler_secret"}
    ]
}

result = deep_merge(config_with_dicts, secrets_with_dicts)
print("✅ Fixed - dictionaries in lists:")
jprint(result)

✅ Fixed - dictionaries in lists:

{
  "services": [
    {
      "name": "api",
      "port": 8001,
      "api_key": "api_secret"
    },
    {
      "name": "worker",
      "port": 8002,
      "api_key": "worker_secret"
    },
    {
      "name": "scheduler",
      "port": 8003,
      "api_key": "scheduler_secret"
    }
  ]
}

---

Best Practices

1. Configuration File Organization

Recommended structure:

project/
├── config/
│   ├── base.json              # ✅ Safe to commit
│   ├── environments/
│   │   ├── dev.json           # ✅ Safe to commit
│   │   ├── staging.json       # ✅ Safe to commit
│   │   └── prod.json          # ✅ Safe to commit
│   └── secrets/               # ❌ Never commit this folder!
│       ├── dev-secrets.json   # ❌ Add to .gitignore
│       ├── staging-secrets.json
│       └── prod-secrets.json
└── .gitignore                 # Must include config/secrets/

2. Validation Before Merging

def validate_config_structure(base_config, secrets_config):
    """Validate configs have compatible structure before merging"""

    def validate_list_lengths(base, secrets, path=""):
        for key in base.keys() & secrets.keys():
            current_path = f"{path}.{key}" if path else key
            base_val, secret_val = base[key], secrets[key]

            if isinstance(base_val, list) and isinstance(secret_val, list):
                if len(base_val) != len(secret_val):
                    raise ValueError(
                        f"List length mismatch at {current_path}: "
                        f"base has {len(base_val)} items, secrets has {len(secret_val)} items"
                    )

            elif isinstance(base_val, dict) and isinstance(secret_val, dict):
                validate_list_lengths(base_val, secret_val, current_path)

    validate_list_lengths(base_config, secrets_config)
    print("✅ Configuration structure validation passed")

# Example usage
base = {"users": [{"username": "alice"}, {"username": "bob"}]}
secrets = {"users": [{"password": "pwd1"}, {"password": "pwd2"}]}

validate_config_structure(base, secrets)
result = deep_merge(base, secrets)

✅ Configuration structure validation passed

3. Environment-Specific Loading

import os
import json

def load_configuration(environment: str):
    """Load and merge configuration for specific environment"""

    # Load base configuration
    with open(f"config/environments/{environment}.json") as f:
        base_config = json.load(f)

    # Load secrets (from secure location, not git)
    secrets_path = f"config/secrets/{environment}-secrets.json"
    if os.path.exists(secrets_path):
        with open(secrets_path) as f:
            secrets_config = json.load(f)
    else:
        print(f"⚠️ No secrets file found at {secrets_path}")
        secrets_config = {}

    # Merge and return
    return deep_merge(base_config, secrets_config)

# Usage
# config = load_configuration("prod")

4. CI/CD Integration

# Example: Inject secrets at deployment time
def prepare_deployment_config(base_config_path: str, environment: str):
    """Prepare config for deployment by injecting secrets from environment variables"""

    with open(base_config_path) as f:
        base_config = json.load(f)

    # Build secrets from environment variables
    secrets = {
        "database": {
            "username": os.environ["DB_USERNAME"],
            "password": os.environ["DB_PASSWORD"]
        },
        "api_keys": {
            "stripe": os.environ["STRIPE_API_KEY"],
            "sendgrid": os.environ["SENDGRID_API_KEY"]
        }
    }

    return deep_merge(base_config, secrets)

# In your deployment script:
# deployment_config = prepare_deployment_config("config/prod.json", "prod")

---

Summary

The deep_merge pattern provides a secure, scalable solution for configuration management:

🎯 Key Benefits

• 🔒 Security: Keeps secrets out of version control

• 🧠 Intelligence: Structure-aware merging preserves relationships

• 🛡️ Safety: Immutable operations prevent accidental data corruption

• 📈 Scalability: Handles complex, deeply nested configurations

• 🔍 Validation: Clear error messages for troubleshooting

🚀 When to Use This Pattern

• ✅ Multi-environment applications (dev/staging/prod)

• ✅ Microservice configurations with shared structure

• ✅ CI/CD pipelines that inject secrets at deployment

• ✅ Applications requiring compliance with security standards

• ✅ Teams that need to separate config ownership

💡 Next Steps

1. Identify configuration vs secrets in your application

2. Separate them into different files/sources

3. Structure your configs with compatible schemas

4. Merge them safely with deep_merge

5. Automate the process in your deployment pipeline

---

Remember: Configuration management is a security practice, not just a convenience. Use deep_merge to build robust, secure applications that scale with your team and infrastructure needs.