Hierarchical Configuration Inheritance Pattern: A Complete Guide¶
The Problem: Configuration Duplication¶
Why Traditional Configuration Management Is Painful¶
Imagine you’re building a microservice that needs to run in multiple environments. Without inheritance patterns, your configuration might look like this:
[1]:
# ❌ Traditional approach - lots of duplication
traditional_config = {
"dev": {
"database": {
"host": "dev-db.example.com",
"port": 5432,
"pool_size": 10,
"timeout": 30,
"ssl_mode": "prefer",
"retry_attempts": 3
},
"redis": {
"host": "dev-redis.example.com",
"port": 6379,
"timeout": 10,
"pool_size": 20
},
"logging": {
"level": "DEBUG",
"format": "detailed"
}
},
"staging": {
"database": {
"host": "staging-db.example.com",
"port": 5432, # 🔄 DUPLICATE
"pool_size": 10, # 🔄 DUPLICATE
"timeout": 30, # 🔄 DUPLICATE
"ssl_mode": "prefer", # 🔄 DUPLICATE
"retry_attempts": 3 # 🔄 DUPLICATE
},
"redis": {
"host": "staging-redis.example.com",
"port": 6379, # 🔄 DUPLICATE
"timeout": 10, # 🔄 DUPLICATE
"pool_size": 20 # 🔄 DUPLICATE
},
"logging": {
"level": "INFO",
"format": "detailed" # 🔄 DUPLICATE
}
},
"prod": {
"database": {
"host": "prod-db.example.com",
"port": 5432, # 🔄 DUPLICATE
"pool_size": 50, # Different value, but pattern repeats
"timeout": 30, # 🔄 DUPLICATE
"ssl_mode": "require", # Different value
"retry_attempts": 3 # 🔄 DUPLICATE
},
"redis": {
"host": "prod-redis.example.com",
"port": 6379, # 🔄 DUPLICATE
"timeout": 10, # 🔄 DUPLICATE
"pool_size": 50 # Different value
},
"logging": {
"level": "ERROR",
"format": "detailed" # 🔄 DUPLICATE
}
}
}
The Pain Points¶
🔄 Massive Duplication: 80% of configuration values are repeated across environments
🐛 Error Prone: Change a default port? You must remember to update it in 3+ places
📈 Scales Poorly: Adding a new environment means copying and modifying everything
🔍 Hard to Understand: What values are defaults vs environment-specific overrides?
🚀 Maintenance Nightmare: Updating shared settings requires touching multiple sections
The Solution: Hierarchical Inheritance¶
Core Concepts¶
Setup¶
[3]:
import json
from rich import print as rprint
from configcraft.api import SHARED, apply_inheritance, inherit_value
def jprint(data: dict):
"""Pretty print JSON data"""
rprint(json.dumps(data, indent=2))
2. JSON Path Patterns¶
JSON paths specify where shared values should be applied:
Pattern |
Meaning |
Example |
|---|---|---|
|
All top-level keys |
|
|
Specific environment |
|
|
Nested paths |
|
|
Multiple wildcards |
|
3. Non-Destructive Inheritance¶
Key Principle: Shared values are only applied when the target key doesn’t already exist.
[5]:
config = {
"_shared": {"*.memory": 2},
"dev": {}, # ✅ Will get memory: 2
"prod": {"memory": 8} # ✅ Keeps existing memory: 8
}
Basic Usage¶
Example 1: Simple Environment Defaults¶
[6]:
# Define configuration with shared defaults
config_data = {
"_shared": {
"*.memory": 2, # Default memory for all environments
"*.cpu": 1 # Default CPU for all environments
},
"dev": {}, # Empty - will inherit all defaults
"staging": {
"memory": 4 # Override memory, inherit CPU
},
"prod": {
"memory": 8, # Override memory
"cpu": 4 # Override CPU
}
}
# Apply inheritance
apply_inheritance(config_data)
jprint(config_data)
{ "dev": { "memory": 2, "cpu": 1 }, "staging": { "memory": 4, "cpu": 1 }, "prod": { "memory": 8, "cpu": 4 } }
Example 2: Nested Configuration Inheritance¶
[7]:
config_data = {
"_shared": {
"*.database.port": 5432,
"*.database.pool_size": 10,
"*.cache.ttl": 3600
},
"dev": {
"database": {
"host": "localhost"
# port and pool_size will be inherited
},
"cache": {
"host": "localhost"
# ttl will be inherited
}
},
"prod": {
"database": {
"host": "prod-db.com",
"pool_size": 50 # Override default
# port will be inherited
},
"cache": {
"host": "prod-cache.com",
"ttl": 7200 # Override default
}
}
}
apply_inheritance(config_data)
jprint(config_data)
{ "dev": { "database": { "host": "localhost", "port": 5432, "pool_size": 10 }, "cache": { "host": "localhost", "ttl": 3600 } }, "prod": { "database": { "host": "prod-db.com", "pool_size": 50, "port": 5432 }, "cache": { "host": "prod-cache.com", "ttl": 7200 } } }
Advanced Patterns¶
Path Execution Order: Exception-Then-Default Pattern¶
🚨 Critical Behavior: Within a single _shared section, paths are processed from top to bottom. If multiple paths affect the same node, the earlier path wins due to setdefault behavior.
This enables powerful exception-then-default patterns:
[8]:
# Example: CPU allocation with exceptions
config_data = {
"_shared": {
# ⚠️ ORDER MATTERS! Exception MUST come first
"*.servers.high_memory.cpu": 8, # Exception: high_memory gets 8 CPU
"*.servers.*.cpu": 2 # Default: all other servers get 2 CPU
},
"dev": {
"servers": {
"web": {}, # Gets cpu=2 (default rule)
"high_memory": {}, # Gets cpu=8 (exception rule)
"worker": {} # Gets cpu=2 (default rule)
}
},
"prod": {
"servers": {
"web": {}, # Gets cpu=2 (default rule)
"high_memory": {}, # Gets cpu=8 (exception rule)
"database": {"cpu": 16} # Keeps cpu=16 (existing value)
}
}
}
apply_inheritance(config_data)
jprint(config_data)
{ "dev": { "servers": { "web": { "cpu": 2 }, "high_memory": { "cpu": 8 }, "worker": { "cpu": 2 } } }, "prod": { "servers": { "web": { "cpu": 2 }, "high_memory": { "cpu": 8 }, "database": { "cpu": 16 } } } }
❌ Wrong Order Example:
[9]:
# This WON'T work as expected - wrong order!
config_data = {
"_shared": {
"*.servers.*.cpu": 2, # 🚫 Default comes first
"*.servers.high_memory.cpu": 8 # 🚫 Exception comes second - TOO LATE!
},
"dev": {
"servers": {
"high_memory": {} # Gets cpu=2 (not 8!) because default ran first
}
}
}
✅ Design Logic:
The exception-then-default pattern leverages Python’s dict.setdefault() behavior:
First path sets
high_memory.cpu = 8(exception)Second path tries to set
high_memory.cpu = 2, but key already exists → ignoredResult: Exception takes precedence, defaults fill gaps
Working with Lists of Objects¶
The inheritance pattern works seamlessly with lists of dictionaries:
[11]:
config_data = {
"_shared": {
"*.databases.port": 5432, # Apply to ALL database objects
"*.databases.timeout": 30 # Apply to ALL database objects
},
"dev": {
"databases": [
{"host": "dev-primary.com", "type": "primary"},
{"host": "dev-replica.com", "type": "replica"}
# Both will inherit port and timeout
]
},
"prod": {
"databases": [
{"host": "prod-primary.com", "type": "primary"},
{"host": "prod-replica.com", "type": "replica", "port": 5433}
# First inherits port, second keeps override
]
}
}
apply_inheritance(config_data)
jprint(config_data)
{ "dev": { "databases": [ { "host": "dev-primary.com", "type": "primary", "port": 5432, "timeout": 30 }, { "host": "dev-replica.com", "type": "replica", "port": 5432, "timeout": 30 } ] }, "prod": { "databases": [ { "host": "prod-primary.com", "type": "primary", "port": 5432, "timeout": 30 }, { "host": "prod-replica.com", "type": "replica", "port": 5433, "timeout": 30 } ] } }
Specific Environment Targeting¶
Sometimes you want to set defaults for specific environments only:
[12]:
config_data = {
"_shared": {
"dev.*.memory": 4, # Only dev environments get 4GB
"prod.*.memory": 16, # Only prod environments get 16GB
"*.log_level": "INFO" # All environments get INFO logging
},
"dev": {
"web": {}, # Will get memory: 4, log_level: "INFO"
"worker": {} # Will get memory: 4, log_level: "INFO"
},
"staging": {
"web": {}, # Will get log_level: "INFO" only
"worker": {"memory": 8} # Custom memory, inherits log_level
},
"prod": {
"web": {}, # Will get memory: 16, log_level: "INFO"
"worker": {"memory": 32} # Custom memory, inherits log_level
}
}
apply_inheritance(config_data)
jprint(config_data)
{ "dev": { "web": { "memory": 4 }, "worker": { "memory": 4 }, "log_level": "INFO" }, "staging": { "web": {}, "worker": { "memory": 8 }, "log_level": "INFO" }, "prod": { "web": { "memory": 16 }, "worker": { "memory": 32 }, "log_level": "INFO" } }
Understanding the API¶
The hierarchical configuration pattern provides two main functions with different purposes:
inherit_value() - Low-Level Inheritance¶
This is the core building block that applies a single shared value to its target location(s):
[14]:
from configcraft.api import inherit_value
# Example: Set a default value only where it doesn't exist
data = {
"dev": {"host": "dev.com"},
"prod": {"host": "prod.com", "port": 8080} # Already has port
}
# Apply default port to all environments
inherit_value(path="*.port", value=3000, data=data)
jprint(data)
{ "dev": { "host": "dev.com", "port": 3000 }, "prod": { "host": "prod.com", "port": 8080 } }
When to use ``inherit_value()``:
✅ Building custom inheritance logic
✅ Applying a single default value
✅ Fine-grained control over inheritance behavior
❌ Processing complete
_sharedconfigurations (useapply_inheritanceinstead)
apply_inheritance() - High-Level Configuration Processing¶
This is the main entry point that processes entire configuration structures with _shared sections:
[15]:
from configcraft.api import apply_inheritance
# Example: Process a complete configuration
config = {
"_shared": {
"*.port": 3000,
"*.timeout": 30
},
"dev": {"host": "dev.com"},
"prod": {"host": "prod.com", "port": 8080}
}
apply_inheritance(config)
jprint(config)
{ "dev": { "host": "dev.com", "port": 3000, "timeout": 30 }, "prod": { "host": "prod.com", "port": 8080, "timeout": 30 } }
When to use ``apply_inheritance()``:
✅ Processing complete configuration files
✅ Standard use case with
_sharedsections✅ Production configuration management
✅ Most common use case - start here!
Real-World Examples¶
Example 1: Microservice Configuration¶
[16]:
# Real-world microservice configuration
microservice_config = {
"_shared": {
# Database defaults
"*.database.pool_size": 10,
"*.database.timeout": 30,
"*.database.retry_attempts": 3,
# Redis defaults
"*.redis.timeout": 5,
"*.redis.pool_size": 20,
# Logging defaults
"*.logging.format": "json",
"*.logging.level": "INFO",
# HTTP defaults
"*.http.timeout": 10,
"*.http.retry_attempts": 3
},
"local": {
"database": {
"host": "localhost",
"port": 5432,
"name": "myapp_dev"
},
"redis": {
"host": "localhost",
"port": 6379
},
"logging": {
"level": "DEBUG" # Override for local development
},
"http": {
"base_url": "http://localhost:8000"
}
},
"staging": {
"database": {
"host": "staging-db.company.com",
"port": 5432,
"name": "myapp_staging",
"pool_size": 20 # Override for staging load
},
"redis": {
"host": "staging-redis.company.com",
"port": 6379
},
"logging": {},
"http": {
"base_url": "https://staging-api.company.com"
}
},
"production": {
"database": {
"host": "prod-db.company.com",
"port": 5432,
"name": "myapp_prod",
"pool_size": 50, # Production needs more connections
"timeout": 60 # Production can wait longer
},
"redis": {
"host": "prod-redis.company.com",
"port": 6379,
"pool_size": 100 # Production needs larger pool
},
"logging": {
"level": "ERROR" # Production only logs errors
},
"http": {
"base_url": "https://api.company.com",
"timeout": 30 # Production can wait longer
}
}
}
apply_inheritance(microservice_config)
After processing, each environment gets:
✅ All the appropriate defaults from
_shared✅ Environment-specific host/URL configurations
✅ Performance tuning overrides where needed
✅ No duplication of common settings
[17]:
jprint(microservice_config)
{ "local": { "database": { "host": "localhost", "port": 5432, "name": "myapp_dev", "pool_size": 10, "timeout": 30, "retry_attempts": 3 }, "redis": { "host": "localhost", "port": 6379, "timeout": 5, "pool_size": 20 }, "logging": { "level": "DEBUG", "format": "json" }, "http": { "base_url": "http://localhost:8000", "timeout": 10, "retry_attempts": 3 } }, "staging": { "database": { "host": "staging-db.company.com", "port": 5432, "name": "myapp_staging", "pool_size": 20, "timeout": 30, "retry_attempts": 3 }, "redis": { "host": "staging-redis.company.com", "port": 6379, "timeout": 5, "pool_size": 20 }, "logging": { "format": "json", "level": "INFO" }, "http": { "base_url": "https://staging-api.company.com", "timeout": 10, "retry_attempts": 3 } }, "production": { "database": { "host": "prod-db.company.com", "port": 5432, "name": "myapp_prod", "pool_size": 50, "timeout": 60, "retry_attempts": 3 }, "redis": { "host": "prod-redis.company.com", "port": 6379, "pool_size": 100, "timeout": 5 }, "logging": { "level": "ERROR", "format": "json" }, "http": { "base_url": "https://api.company.com", "timeout": 30, "retry_attempts": 3 } } }
Example 2: Multi-Tenant SaaS Configuration¶
[18]:
# SaaS application with multiple tenants
saas_config = {
"_shared": {
# Default resource limits
"*.tenants.*.cpu_limit": 1,
"*.tenants.*.memory_limit": 2,
"*.tenants.*.storage_limit": 10,
# Default feature flags
"*.tenants.*.features.analytics": True,
"*.tenants.*.features.api_access": True,
"*.tenants.*.features.custom_domain": False,
# Default billing
"*.tenants.*.billing.plan": "basic",
"*.tenants.*.billing.trial_days": 14
},
"dev": {
"tenants": {
"test_tenant": {
"name": "Test Company",
# Gets all defaults
"features": {},
"billing": {}
}
}
},
"prod": {
"tenants": {
"startup_co": {
"name": "Startup Co",
"billing": {"plan": "startup"}, # Override plan
# Other defaults inherited
"features": {},
"billing": {}
},
"enterprise_corp": {
"name": "Enterprise Corp",
"cpu_limit": 8, # Enterprise gets more resources
"memory_limit": 16,
"storage_limit": 1000,
"features": {
"custom_domain": True, # Enterprise feature
"sso": True # Additional enterprise feature
},
"billing": {
"plan": "enterprise",
"trial_days": 30 # Longer trial
}
}
}
}
}
apply_inheritance(saas_config)
This pattern allows you to:
🎯 Set sensible defaults for all tenants
🚀 Quickly onboard new tenants with minimal configuration
💰 Easily implement tiered pricing with resource overrides
🔧 Maintain consistent feature flags across environments
[19]:
jprint(saas_config)
{ "dev": { "tenants": { "test_tenant": { "name": "Test Company", "features": { "analytics": true, "api_access": true, "custom_domain": false }, "billing": { "plan": "basic", "trial_days": 14 }, "cpu_limit": 1, "memory_limit": 2, "storage_limit": 10 } } }, "prod": { "tenants": { "startup_co": { "name": "Startup Co", "billing": { "plan": "basic", "trial_days": 14 }, "features": { "analytics": true, "api_access": true, "custom_domain": false }, "cpu_limit": 1, "memory_limit": 2, "storage_limit": 10 }, "enterprise_corp": { "name": "Enterprise Corp", "cpu_limit": 8, "memory_limit": 16, "storage_limit": 1000, "features": { "custom_domain": true, "sso": true, "analytics": true, "api_access": true }, "billing": { "plan": "enterprise", "trial_days": 30 } } } } }
Best Practices¶
1. Design Patterns¶
✅ DO: Start with Broad Defaults, Then Specialize¶
[20]:
# Good: Broad defaults with specific overrides
config = {
"_shared": {
"*.memory": 2, # Broad default
"prod.*.memory": 8 # Environment-specific override
},
"dev": {"api": {}, "worker": {}},
"prod": {"api": {}, "worker": {"memory": 16}} # Service-specific override
}
2. Path Pattern Guidelines¶
Use Wildcards Strategically¶
[23]:
# ✅ Good patterns
"*.timeout" # All environments
"*.database.port" # All database configs
"prod.*.memory" # All prod services
"*.services.*.cpu" # All services in all environments
# ❌ Avoid these patterns
"*.*.*.*" # Too generic
"very.specific.deep.path.field" # Too specific
;
[23]:
''
Establish Naming Conventions¶
[24]:
# ✅ Consistent naming helps pattern matching
config = {
"_shared": {
"*.database_primary.port": 5432,
"*.database_replica.port": 5433,
"*.cache_redis.port": 6379
}
}
3. Configuration Organization¶
Group Related Settings¶
[25]:
# ✅ Well-organized configuration
config = {
"_shared": {
# Database cluster
"*.database.port": 5432,
"*.database.pool_size": 10,
"*.database.timeout": 30,
# Caching layer
"*.cache.ttl": 3600,
"*.cache.max_size": 1000,
# Monitoring
"*.monitoring.enabled": True,
"*.monitoring.interval": 60
}
}
Document Your Patterns¶
[26]:
config = {
"_shared": {
# Resource defaults - production overrides these
"*.memory": 2, # GB
"*.cpu": 1, # cores
# Network timeouts - keep aggressive for responsiveness
"*.timeout": 30, # seconds
"*.retry_attempts": 3, # count
# Feature flags - enable by default, disable selectively
"*.features.metrics": True,
"*.features.tracing": True
}
}
4. Test Your Configuration Processing¶
[27]:
def test_config_inheritance():
config = {
"_shared": {"*.port": 3000},
"dev": {"host": "localhost"},
"prod": {"host": "prod.com", "port": 8080}
}
apply_inheritance(config)
# Validate inheritance worked
assert config["dev"]["port"] == 3000 # Inherited
assert config["prod"]["port"] == 8080 # Preserved override
assert "_shared" not in config # Cleaned up
test_config_inheritance()
5. Error Prevention¶
Validate Paths Before Processing¶
[28]:
def validate_shared_paths(shared_config):
"""Validate _shared path patterns"""
for path in shared_config.keys():
if path.endswith("*"):
raise ValueError(f"Path cannot end with '*': {path}")
if ".." in path:
raise ValueError(f"Path cannot contain '..': {path}")
Handle Missing Intermediate Keys¶
[29]:
# ❌ This will raise KeyError if 'database' doesn't exist
config = {
"_shared": {"*.database.port": 5432},
"dev": {} # No 'database' key
}
# ✅ Better: Ensure intermediate structures exist
config = {
"_shared": {"*.database.port": 5432},
"dev": {"database": {}} # Provide empty database config
}
Summary¶
The Hierarchical Configuration Inheritance Pattern solves the fundamental problem of configuration duplication in multi-environment applications. By using _shared sections and JSON path patterns, you can:
🎯 Key Benefits¶
Eliminate Duplication: Define common settings once
Reduce Errors: Single source of truth for defaults
Scale Easily: Add new environments with minimal config
Override Flexibly: Keep environment-specific customizations
Maintain Simply: Change defaults in one place
🚀 When to Use This Pattern¶
✅ Multi-environment deployments (dev/staging/prod)
✅ Microservice configurations with shared defaults
✅ Multi-tenant applications with tiered features
✅ Configuration templates with customization points
✅ Any scenario with repetitive configuration data
🛠️ Getting Started¶
Identify duplicated configuration values
Extract them to a
_sharedsectionUse
*.fieldpatterns for broad defaultsUse
env.fieldpatterns for specific overridesCall
apply_inheritance()to process your config
The pattern transforms configuration management from a maintenance burden into a powerful tool for organizing and scaling your application configurations.