1233 lines
30 KiB
Markdown
1233 lines
30 KiB
Markdown
# KMS Deployment Guide
|
|
|
|
## Table of Contents
|
|
1. [Deployment Overview](#deployment-overview)
|
|
2. [Container Architecture](#container-architecture)
|
|
3. [Environment Configuration](#environment-configuration)
|
|
4. [Docker Compose Setup](#docker-compose-setup)
|
|
5. [Production Deployment](#production-deployment)
|
|
6. [Monitoring and Health Checks](#monitoring-and-health-checks)
|
|
7. [Security Configuration](#security-configuration)
|
|
8. [Backup and Recovery](#backup-and-recovery)
|
|
9. [Troubleshooting](#troubleshooting)
|
|
|
|
---
|
|
|
|
## Deployment Overview
|
|
|
|
The KMS is designed as a containerized application using Docker and Docker Compose for orchestration. The system consists of multiple services working together to provide secure API key management capabilities.
|
|
|
|
### Service Architecture
|
|
- **kms-nginx**: Reverse proxy and load balancer
|
|
- **kms-api**: Go backend API service
|
|
- **kms-frontend**: React TypeScript frontend
|
|
- **kms-postgres**: PostgreSQL database
|
|
- **kms-redis**: Redis cache (optional)
|
|
|
|
### Deployment Modes
|
|
- **Development**: Docker Compose with hot reload
|
|
- **Testing**: Isolated container environment
|
|
- **Staging**: Production-like with monitoring
|
|
- **Production**: High availability with load balancing
|
|
|
|
---
|
|
|
|
## Container Architecture
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph "External Network"
|
|
Internet[Internet]
|
|
CDN[Content Delivery Network<br/>Static Assets]
|
|
DNS[DNS<br/>Load Balancing]
|
|
end
|
|
|
|
subgraph "Load Balancer Tier"
|
|
LB1[Load Balancer 1<br/>HAProxy/Nginx]
|
|
LB2[Load Balancer 2<br/>HAProxy/Nginx]
|
|
VIP[Virtual IP<br/>Failover]
|
|
end
|
|
|
|
subgraph "Container Orchestration - Docker Compose"
|
|
subgraph "Reverse Proxy"
|
|
Nginx1[Nginx Container<br/>kms-nginx<br/>Port 8081:80]
|
|
end
|
|
|
|
subgraph "API Tier"
|
|
API1[API Service Container<br/>kms-api-service<br/>Port 8080:8080]
|
|
Metrics1[Metrics Endpoint<br/>Port 9090:9090<br/>Prometheus]
|
|
end
|
|
|
|
subgraph "Frontend Tier"
|
|
Frontend1[React SPA Container<br/>kms-frontend<br/>Port 3000:80]
|
|
StaticAssets[Static Assets<br/>Nginx served]
|
|
end
|
|
|
|
subgraph "Network"
|
|
InternalNet[kms-network<br/>Bridge Network<br/>Internal Communication]
|
|
end
|
|
end
|
|
|
|
subgraph "Data Tier"
|
|
subgraph "Primary Database"
|
|
PostgreSQL1[PostgreSQL 15<br/>kms-postgres<br/>Port 5432:5432]
|
|
DBData[Persistent Volume<br/>postgres_data]
|
|
end
|
|
|
|
subgraph "Cache Layer"
|
|
Redis1[Redis Cache<br/>Optional<br/>Session Store]
|
|
end
|
|
|
|
subgraph "Migration System"
|
|
Migrations[Database Migrations<br/>Auto-run on startup<br/>Volume mounted]
|
|
end
|
|
end
|
|
|
|
subgraph "Monitoring & Observability"
|
|
subgraph "Metrics Collection"
|
|
Prometheus[Prometheus<br/>Metrics Scraping]
|
|
Grafana[Grafana<br/>Dashboards]
|
|
end
|
|
|
|
subgraph "Logging"
|
|
LogAggregator[Log Aggregation<br/>ELK Stack / Loki]
|
|
StructuredLogs[Structured Logging<br/>JSON format]
|
|
end
|
|
|
|
subgraph "Health Monitoring"
|
|
HealthCheck[Health Checks<br/>/health endpoint]
|
|
Alerting[Alerting<br/>PagerDuty/Slack]
|
|
end
|
|
end
|
|
|
|
subgraph "Security & Compliance"
|
|
subgraph "Certificate Management"
|
|
TLS[TLS Certificates<br/>Let's Encrypt/Manual]
|
|
CertRotation[Certificate Rotation<br/>Automated]
|
|
end
|
|
|
|
subgraph "Secrets Management"
|
|
EnvVars[Environment Variables<br/>Container secrets]
|
|
VaultIntegration[Vault Integration<br/>Secret rotation]
|
|
end
|
|
|
|
subgraph "Backup & Recovery"
|
|
DBBackup[Database Backups<br/>Automated daily]
|
|
DisasterRecovery[Disaster Recovery<br/>Multi-region]
|
|
end
|
|
end
|
|
|
|
subgraph "Development Environment"
|
|
LocalDev[Local Development<br/>docker-compose.yml]
|
|
TestEnv[Test Environment<br/>Isolated containers]
|
|
CICDPipeline[CI/CD Pipeline<br/>GitHub Actions]
|
|
end
|
|
|
|
%% External Connections
|
|
Internet --> DNS
|
|
DNS --> VIP
|
|
CDN --> Frontend1
|
|
|
|
%% Load Balancer Configuration
|
|
VIP --> LB1
|
|
VIP --> LB2
|
|
LB1 --> Nginx1
|
|
LB2 --> Nginx1
|
|
|
|
%% Container Communication
|
|
Nginx1 --> API1
|
|
Nginx1 --> Frontend1
|
|
API1 --> PostgreSQL1
|
|
API1 --> Redis1
|
|
API1 --> Metrics1
|
|
|
|
%% Data Flow
|
|
PostgreSQL1 --> DBData
|
|
API1 --> Migrations
|
|
Migrations --> PostgreSQL1
|
|
|
|
%% Network Isolation
|
|
InternalNet --> API1
|
|
InternalNet --> Frontend1
|
|
InternalNet --> PostgreSQL1
|
|
InternalNet --> Redis1
|
|
InternalNet --> Nginx1
|
|
|
|
%% Monitoring Connections
|
|
API1 --> Prometheus
|
|
Prometheus --> Grafana
|
|
API1 --> LogAggregator
|
|
LogAggregator --> StructuredLogs
|
|
HealthCheck --> API1
|
|
HealthCheck --> PostgreSQL1
|
|
HealthCheck --> Alerting
|
|
|
|
%% Security Connections
|
|
TLS --> Nginx1
|
|
CertRotation --> TLS
|
|
EnvVars --> API1
|
|
VaultIntegration --> EnvVars
|
|
DBBackup --> PostgreSQL1
|
|
DisasterRecovery --> DBBackup
|
|
|
|
%% Development Flow
|
|
CICDPipeline --> LocalDev
|
|
LocalDev --> TestEnv
|
|
TestEnv --> API1
|
|
|
|
%% Styling
|
|
classDef external fill:#ffecb3,stroke:#f57c00,stroke-width:2px
|
|
classDef loadbalancer fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px
|
|
classDef container fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
|
|
classDef data fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
|
|
classDef monitoring fill:#fff3e0,stroke:#ef6c00,stroke-width:2px
|
|
classDef security fill:#ffcdd2,stroke:#d32f2f,stroke-width:2px
|
|
classDef development fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
|
|
|
|
class Internet,CDN,DNS external
|
|
class LB1,LB2,VIP loadbalancer
|
|
class Nginx1,API1,Frontend1,Metrics1,InternalNet,StaticAssets container
|
|
class PostgreSQL1,Redis1,DBData,Migrations data
|
|
class Prometheus,Grafana,LogAggregator,StructuredLogs,HealthCheck,Alerting monitoring
|
|
class TLS,CertRotation,EnvVars,VaultIntegration,DBBackup,DisasterRecovery security
|
|
class LocalDev,TestEnv,CICDPipeline development
|
|
```
|
|
|
|
---
|
|
|
|
## Environment Configuration
|
|
|
|
### Environment Variables
|
|
|
|
#### **Database Configuration**
|
|
```bash
|
|
# PostgreSQL Database
|
|
DB_HOST=postgres
|
|
DB_PORT=5432
|
|
DB_NAME=kms
|
|
DB_USER=postgres
|
|
DB_PASSWORD=secure_password_here
|
|
DB_MAX_OPEN_CONNECTIONS=25
|
|
DB_MAX_IDLE_CONNECTIONS=5
|
|
DB_CONNECTION_MAX_LIFETIME=300s
|
|
DB_CONNECTION_MAX_IDLE_TIME=60s
|
|
```
|
|
|
|
#### **Server Configuration**
|
|
```bash
|
|
# API Server
|
|
SERVER_HOST=0.0.0.0
|
|
SERVER_PORT=8080
|
|
SERVER_READ_TIMEOUT=30s
|
|
SERVER_WRITE_TIMEOUT=30s
|
|
SERVER_IDLE_TIMEOUT=120s
|
|
|
|
# Environment
|
|
ENVIRONMENT=production
|
|
LOG_LEVEL=info
|
|
DEBUG_MODE=false
|
|
```
|
|
|
|
#### **Authentication Configuration**
|
|
```bash
|
|
# Authentication Provider
|
|
AUTH_PROVIDER=header
|
|
AUTH_HEADER_USER_EMAIL=X-User-Email
|
|
AUTH_SIGNING_KEY=your_hmac_signing_key_here
|
|
|
|
# JWT Configuration
|
|
JWT_SECRET=your_jwt_secret_here
|
|
JWT_ISSUER=kms-api-service
|
|
JWT_EXPIRY=1h
|
|
|
|
# OAuth2 Configuration (optional)
|
|
OAUTH2_CLIENT_ID=your_oauth2_client_id
|
|
OAUTH2_CLIENT_SECRET=your_oauth2_client_secret
|
|
OAUTH2_REDIRECT_URL=https://your-domain.com/api/oauth2/callback
|
|
OAUTH2_PROVIDER_URL=https://oauth-provider.com
|
|
|
|
# SAML Configuration (optional)
|
|
SAML_IDP_URL=https://saml-provider.com/sso
|
|
SAML_CERT_PATH=/etc/ssl/certs/saml.crt
|
|
SAML_KEY_PATH=/etc/ssl/private/saml.key
|
|
```
|
|
|
|
#### **Security Configuration**
|
|
```bash
|
|
# Rate Limiting
|
|
RATE_LIMIT_ENABLED=true
|
|
RATE_LIMIT_RPS=100
|
|
RATE_LIMIT_BURST=200
|
|
AUTH_RATE_LIMIT_RPS=5
|
|
AUTH_RATE_LIMIT_BURST=10
|
|
|
|
# Security Settings
|
|
CSRF_TOKEN_MAX_AGE=1h
|
|
MAX_AUTH_FAILURES=5
|
|
IP_BLOCK_DURATION=1h
|
|
AUTH_FAILURE_WINDOW=15m
|
|
REQUEST_MAX_AGE=5m
|
|
|
|
# CORS Settings
|
|
CORS_ALLOWED_ORIGINS=https://your-frontend-domain.com,https://admin.your-domain.com
|
|
CORS_ALLOWED_METHODS=GET,POST,PUT,DELETE,OPTIONS
|
|
CORS_ALLOWED_HEADERS=Origin,Content-Type,Accept,Authorization,X-User-Email,X-CSRF-Token
|
|
```
|
|
|
|
#### **Monitoring Configuration**
|
|
```bash
|
|
# Metrics
|
|
METRICS_ENABLED=true
|
|
METRICS_PORT=9090
|
|
METRICS_PATH=/metrics
|
|
|
|
# Health Checks
|
|
HEALTH_CHECK_ENABLED=true
|
|
HEALTH_CHECK_INTERVAL=30s
|
|
```
|
|
|
|
### Configuration Management
|
|
|
|
#### **Environment Files**
|
|
```bash
|
|
# Development environment
|
|
.env.development
|
|
|
|
# Testing environment
|
|
.env.test
|
|
|
|
# Staging environment
|
|
.env.staging
|
|
|
|
# Production environment
|
|
.env.production
|
|
```
|
|
|
|
#### **Docker Environment**
|
|
```yaml
|
|
# docker-compose.override.yml for local development
|
|
version: '3.8'
|
|
services:
|
|
kms-api:
|
|
environment:
|
|
- LOG_LEVEL=debug
|
|
- DEBUG_MODE=true
|
|
volumes:
|
|
- ./:/app
|
|
command: air -c .air.toml # Hot reload
|
|
```
|
|
|
|
---
|
|
|
|
## Docker Compose Setup
|
|
|
|
### Development Configuration
|
|
|
|
#### **docker-compose.yml**
|
|
```yaml
|
|
version: '3.8'
|
|
|
|
services:
|
|
# PostgreSQL Database
|
|
kms-postgres:
|
|
image: postgres:15-alpine
|
|
container_name: kms-postgres
|
|
environment:
|
|
POSTGRES_DB: ${DB_NAME:-kms}
|
|
POSTGRES_USER: ${DB_USER:-postgres}
|
|
POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
|
|
POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --lc-collate=C --lc-ctype=C"
|
|
volumes:
|
|
- postgres_data:/var/lib/postgresql/data
|
|
- ./migrations:/docker-entrypoint-initdb.d:ro
|
|
ports:
|
|
- "5432:5432"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres}"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 5
|
|
start_period: 30s
|
|
networks:
|
|
- kms-network
|
|
restart: unless-stopped
|
|
|
|
# Redis Cache (Optional)
|
|
kms-redis:
|
|
image: redis:7-alpine
|
|
container_name: kms-redis
|
|
command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD:-redis_password}
|
|
volumes:
|
|
- redis_data:/data
|
|
ports:
|
|
- "6379:6379"
|
|
healthcheck:
|
|
test: ["CMD", "redis-cli", "ping"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
networks:
|
|
- kms-network
|
|
restart: unless-stopped
|
|
|
|
# API Service
|
|
kms-api:
|
|
build:
|
|
context: .
|
|
dockerfile: Dockerfile
|
|
target: production
|
|
container_name: kms-api
|
|
environment:
|
|
- DB_HOST=kms-postgres
|
|
- DB_PORT=5432
|
|
- DB_NAME=${DB_NAME:-kms}
|
|
- DB_USER=${DB_USER:-postgres}
|
|
- DB_PASSWORD=${DB_PASSWORD:-postgres}
|
|
- REDIS_HOST=kms-redis
|
|
- REDIS_PORT=6379
|
|
- REDIS_PASSWORD=${REDIS_PASSWORD:-redis_password}
|
|
- SERVER_PORT=8080
|
|
- JWT_SECRET=${JWT_SECRET}
|
|
- AUTH_SIGNING_KEY=${AUTH_SIGNING_KEY}
|
|
- LOG_LEVEL=${LOG_LEVEL:-info}
|
|
- RATE_LIMIT_ENABLED=true
|
|
ports:
|
|
- "8080:8080"
|
|
- "9090:9090" # Metrics port
|
|
depends_on:
|
|
kms-postgres:
|
|
condition: service_healthy
|
|
kms-redis:
|
|
condition: service_healthy
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 60s
|
|
networks:
|
|
- kms-network
|
|
restart: unless-stopped
|
|
volumes:
|
|
- ./logs:/app/logs
|
|
|
|
# Frontend Service
|
|
kms-frontend:
|
|
build:
|
|
context: ./kms-frontend
|
|
dockerfile: Dockerfile
|
|
target: production
|
|
container_name: kms-frontend
|
|
environment:
|
|
- REACT_APP_API_BASE_URL=http://localhost:8081/api
|
|
ports:
|
|
- "3000:80"
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:80"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
networks:
|
|
- kms-network
|
|
restart: unless-stopped
|
|
|
|
# Nginx Reverse Proxy
|
|
kms-nginx:
|
|
image: nginx:alpine
|
|
container_name: kms-nginx
|
|
ports:
|
|
- "8081:80"
|
|
- "8443:443"
|
|
volumes:
|
|
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
|
- ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
|
|
- ./ssl:/etc/ssl/certs:ro
|
|
depends_on:
|
|
- kms-api
|
|
- kms-frontend
|
|
healthcheck:
|
|
test: ["CMD", "nginx", "-t"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
networks:
|
|
- kms-network
|
|
restart: unless-stopped
|
|
|
|
networks:
|
|
kms-network:
|
|
driver: bridge
|
|
ipam:
|
|
config:
|
|
- subnet: 172.20.0.0/16
|
|
|
|
volumes:
|
|
postgres_data:
|
|
driver: local
|
|
redis_data:
|
|
driver: local
|
|
```
|
|
|
|
#### **Dockerfile (Multi-stage)**
|
|
```dockerfile
|
|
# Build stage
|
|
FROM golang:1.21-alpine AS builder
|
|
|
|
WORKDIR /app
|
|
|
|
# Install dependencies
|
|
RUN apk add --no-cache git ca-certificates tzdata
|
|
|
|
# Copy go mod files
|
|
COPY go.mod go.sum ./
|
|
RUN go mod download
|
|
|
|
# Copy source code
|
|
COPY . .
|
|
|
|
# Build the binary
|
|
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main ./cmd/server
|
|
|
|
# Production stage
|
|
FROM alpine:latest AS production
|
|
|
|
RUN apk --no-cache add ca-certificates curl
|
|
|
|
WORKDIR /root/
|
|
|
|
# Copy the binary from builder stage
|
|
COPY --from=builder /app/main .
|
|
COPY --from=builder /app/migrations ./migrations
|
|
|
|
# Create non-root user
|
|
RUN addgroup -g 1001 appgroup && \
|
|
adduser -D -u 1001 -G appgroup appuser
|
|
|
|
USER appuser
|
|
|
|
EXPOSE 8080 9090
|
|
|
|
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
|
CMD curl -f http://localhost:8080/health || exit 1
|
|
|
|
CMD ["./main"]
|
|
```
|
|
|
|
### Production Configuration
|
|
|
|
#### **docker-compose.prod.yml**
|
|
```yaml
|
|
version: '3.8'
|
|
|
|
services:
|
|
kms-postgres:
|
|
image: postgres:15-alpine
|
|
environment:
|
|
POSTGRES_DB: ${DB_NAME}
|
|
POSTGRES_USER: ${DB_USER}
|
|
POSTGRES_PASSWORD: ${DB_PASSWORD}
|
|
POSTGRES_MAX_CONNECTIONS: 100
|
|
volumes:
|
|
- postgres_data:/var/lib/postgresql/data
|
|
- ./postgres/postgresql.conf:/etc/postgresql/postgresql.conf
|
|
- ./postgres/pg_hba.conf:/etc/postgresql/pg_hba.conf
|
|
command: postgres -c config_file=/etc/postgresql/postgresql.conf
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 1G
|
|
cpus: '0.5'
|
|
reservations:
|
|
memory: 512M
|
|
cpus: '0.25'
|
|
restart: always
|
|
logging:
|
|
driver: "json-file"
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
kms-api:
|
|
image: kms-api:latest
|
|
environment:
|
|
- DB_HOST=kms-postgres
|
|
- DB_NAME=${DB_NAME}
|
|
- DB_USER=${DB_USER}
|
|
- DB_PASSWORD=${DB_PASSWORD}
|
|
- JWT_SECRET=${JWT_SECRET}
|
|
- AUTH_SIGNING_KEY=${AUTH_SIGNING_KEY}
|
|
- LOG_LEVEL=info
|
|
- ENVIRONMENT=production
|
|
deploy:
|
|
replicas: 3
|
|
resources:
|
|
limits:
|
|
memory: 512M
|
|
cpus: '0.5'
|
|
reservations:
|
|
memory: 256M
|
|
cpus: '0.25'
|
|
restart_policy:
|
|
condition: on-failure
|
|
delay: 5s
|
|
max_attempts: 3
|
|
window: 120s
|
|
healthcheck:
|
|
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
logging:
|
|
driver: "json-file"
|
|
options:
|
|
max-size: "50m"
|
|
max-file: "5"
|
|
|
|
kms-nginx:
|
|
image: nginx:alpine
|
|
ports:
|
|
- "80:80"
|
|
- "443:443"
|
|
volumes:
|
|
- ./nginx/nginx.prod.conf:/etc/nginx/nginx.conf:ro
|
|
- ./ssl:/etc/ssl/certs:ro
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 256M
|
|
cpus: '0.25'
|
|
restart: always
|
|
```
|
|
|
|
---
|
|
|
|
## Production Deployment
|
|
|
|
### Load Balancer Configuration
|
|
|
|
#### **HAProxy Configuration**
|
|
```bash
|
|
# /etc/haproxy/haproxy.cfg
|
|
global
|
|
daemon
|
|
maxconn 4096
|
|
log stdout local0
|
|
|
|
defaults
|
|
mode http
|
|
timeout connect 5000ms
|
|
timeout client 50000ms
|
|
timeout server 50000ms
|
|
option httplog
|
|
|
|
frontend kms_frontend
|
|
bind *:80
|
|
bind *:443 ssl crt /etc/ssl/certs/kms.pem
|
|
redirect scheme https if !{ ssl_fc }
|
|
|
|
# Rate limiting
|
|
stick-table type ip size 100k expire 30s store http_req_rate(10s)
|
|
http-request track-sc0 src
|
|
http-request reject if { sc_http_req_rate(0) gt 20 }
|
|
|
|
default_backend kms_api_servers
|
|
|
|
backend kms_api_servers
|
|
balance roundrobin
|
|
option httpchk GET /health
|
|
|
|
server api1 kms-api-1:8080 check
|
|
server api2 kms-api-2:8080 check
|
|
server api3 kms-api-3:8080 check
|
|
```
|
|
|
|
#### **Nginx Load Balancer**
|
|
```nginx
|
|
# /etc/nginx/nginx.conf
|
|
upstream kms_api_backend {
|
|
least_conn;
|
|
server kms-api-1:8080 weight=3 max_fails=3 fail_timeout=30s;
|
|
server kms-api-2:8080 weight=3 max_fails=3 fail_timeout=30s;
|
|
server kms-api-3:8080 weight=3 max_fails=3 fail_timeout=30s;
|
|
}
|
|
|
|
upstream kms_frontend_backend {
|
|
server kms-frontend-1:80;
|
|
server kms-frontend-2:80;
|
|
}
|
|
|
|
server {
|
|
listen 80;
|
|
server_name kms.yourdomain.com;
|
|
return 301 https://$server_name$request_uri;
|
|
}
|
|
|
|
server {
|
|
listen 443 ssl http2;
|
|
server_name kms.yourdomain.com;
|
|
|
|
# SSL Configuration
|
|
ssl_certificate /etc/ssl/certs/kms.crt;
|
|
ssl_certificate_key /etc/ssl/private/kms.key;
|
|
ssl_protocols TLSv1.2 TLSv1.3;
|
|
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384;
|
|
ssl_prefer_server_ciphers off;
|
|
|
|
# Security Headers
|
|
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
|
|
add_header X-Frame-Options DENY;
|
|
add_header X-Content-Type-Options nosniff;
|
|
add_header X-XSS-Protection "1; mode=block";
|
|
add_header Referrer-Policy "strict-origin-when-cross-origin";
|
|
|
|
# Rate Limiting
|
|
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
|
|
limit_req_zone $binary_remote_addr zone=auth:10m rate=5r/s;
|
|
|
|
# API Routes
|
|
location /api/ {
|
|
limit_req zone=api burst=20 nodelay;
|
|
proxy_pass http://kms_api_backend;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
}
|
|
|
|
# Authentication Routes (stricter rate limiting)
|
|
location /api/login {
|
|
limit_req zone=auth burst=10 nodelay;
|
|
proxy_pass http://kms_api_backend;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
}
|
|
|
|
# Frontend Routes
|
|
location / {
|
|
proxy_pass http://kms_frontend_backend;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
|
|
# SPA routing support
|
|
try_files $uri $uri/ /index.html;
|
|
}
|
|
|
|
# Health Check
|
|
location /health {
|
|
access_log off;
|
|
proxy_pass http://kms_api_backend;
|
|
}
|
|
|
|
# Metrics (restricted access)
|
|
location /metrics {
|
|
allow 10.0.0.0/8;
|
|
allow 172.16.0.0/12;
|
|
allow 192.168.0.0/16;
|
|
deny all;
|
|
proxy_pass http://kms_api_backend:9090;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Database High Availability
|
|
|
|
#### **PostgreSQL Primary-Replica Setup**
|
|
```yaml
|
|
# docker-compose.ha.yml
|
|
version: '3.8'
|
|
|
|
services:
|
|
postgres-primary:
|
|
image: postgres:15-alpine
|
|
environment:
|
|
POSTGRES_DB: kms
|
|
POSTGRES_USER: postgres
|
|
POSTGRES_PASSWORD: ${DB_PASSWORD}
|
|
POSTGRES_REPLICATION_USER: replicator
|
|
POSTGRES_REPLICATION_PASSWORD: ${REPLICATION_PASSWORD}
|
|
volumes:
|
|
- postgres_primary_data:/var/lib/postgresql/data
|
|
- ./postgres/primary.conf:/etc/postgresql/postgresql.conf
|
|
- ./postgres/setup-replication.sh:/docker-entrypoint-initdb.d/setup-replication.sh
|
|
command: postgres -c config_file=/etc/postgresql/postgresql.conf
|
|
ports:
|
|
- "5432:5432"
|
|
|
|
postgres-replica:
|
|
image: postgres:15-alpine
|
|
environment:
|
|
POSTGRES_USER: postgres
|
|
POSTGRES_PASSWORD: ${DB_PASSWORD}
|
|
POSTGRES_PRIMARY_HOST: postgres-primary
|
|
POSTGRES_REPLICATION_USER: replicator
|
|
POSTGRES_REPLICATION_PASSWORD: ${REPLICATION_PASSWORD}
|
|
volumes:
|
|
- postgres_replica_data:/var/lib/postgresql/data
|
|
- ./postgres/replica.conf:/etc/postgresql/postgresql.conf
|
|
- ./postgres/setup-replica.sh:/docker-entrypoint-initdb.d/setup-replica.sh
|
|
command: postgres -c config_file=/etc/postgresql/postgresql.conf
|
|
ports:
|
|
- "5433:5432"
|
|
depends_on:
|
|
- postgres-primary
|
|
|
|
volumes:
|
|
postgres_primary_data:
|
|
postgres_replica_data:
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring and Health Checks
|
|
|
|
### Health Check Configuration
|
|
|
|
#### **Application Health Checks**
|
|
```go
|
|
// File: internal/handlers/health.go
|
|
type HealthResponse struct {
|
|
Status string `json:"status"`
|
|
Timestamp time.Time `json:"timestamp"`
|
|
Services map[string]string `json:"services"`
|
|
Version string `json:"version"`
|
|
}
|
|
|
|
func (h *HealthHandler) GetHealth(c *gin.Context) {
|
|
response := &HealthResponse{
|
|
Status: "healthy",
|
|
Timestamp: time.Now(),
|
|
Services: make(map[string]string),
|
|
Version: h.version,
|
|
}
|
|
|
|
// Check database connectivity
|
|
if err := h.db.Ping(); err != nil {
|
|
response.Status = "unhealthy"
|
|
response.Services["database"] = "unhealthy"
|
|
} else {
|
|
response.Services["database"] = "healthy"
|
|
}
|
|
|
|
// Check Redis connectivity
|
|
if err := h.redis.Ping(); err != nil {
|
|
response.Services["cache"] = "unhealthy"
|
|
} else {
|
|
response.Services["cache"] = "healthy"
|
|
}
|
|
|
|
statusCode := http.StatusOK
|
|
if response.Status == "unhealthy" {
|
|
statusCode = http.StatusServiceUnavailable
|
|
}
|
|
|
|
c.JSON(statusCode, response)
|
|
}
|
|
```
|
|
|
|
#### **Docker Health Checks**
|
|
```dockerfile
|
|
# API Service Health Check
|
|
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
|
CMD curl -f http://localhost:8080/health || exit 1
|
|
|
|
# Database Health Check
|
|
HEALTHCHECK --interval=30s --timeout=10s --retries=5 \
|
|
CMD pg_isready -U postgres || exit 1
|
|
|
|
# Frontend Health Check
|
|
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
|
|
CMD curl -f http://localhost:80 || exit 1
|
|
```
|
|
|
|
### Prometheus Metrics
|
|
|
|
#### **Metrics Configuration**
|
|
```yaml
|
|
# prometheus.yml
|
|
global:
|
|
scrape_interval: 15s
|
|
evaluation_interval: 15s
|
|
|
|
scrape_configs:
|
|
- job_name: 'kms-api'
|
|
static_configs:
|
|
- targets: ['kms-api:9090']
|
|
metrics_path: /metrics
|
|
scrape_interval: 15s
|
|
|
|
- job_name: 'nginx'
|
|
static_configs:
|
|
- targets: ['kms-nginx:9113']
|
|
|
|
- job_name: 'postgres'
|
|
static_configs:
|
|
- targets: ['postgres-exporter:9187']
|
|
|
|
- job_name: 'redis'
|
|
static_configs:
|
|
- targets: ['redis-exporter:9121']
|
|
```
|
|
|
|
#### **Grafana Dashboards**
|
|
```json
|
|
{
|
|
"dashboard": {
|
|
"title": "KMS System Metrics",
|
|
"panels": [
|
|
{
|
|
"title": "Request Rate",
|
|
"type": "graph",
|
|
"targets": [
|
|
{
|
|
"expr": "rate(http_requests_total[5m])",
|
|
"legendFormat": "{{method}} {{status}}"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"title": "Response Time",
|
|
"type": "graph",
|
|
"targets": [
|
|
{
|
|
"expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
|
|
"legendFormat": "95th percentile"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"title": "Database Connections",
|
|
"type": "graph",
|
|
"targets": [
|
|
{
|
|
"expr": "postgres_stat_database_numbackends",
|
|
"legendFormat": "Active connections"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"title": "Authentication Success Rate",
|
|
"type": "stat",
|
|
"targets": [
|
|
{
|
|
"expr": "rate(auth_attempts_success_total[5m]) / rate(auth_attempts_total[5m])",
|
|
"legendFormat": "Success rate"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Security Configuration
|
|
|
|
### SSL/TLS Configuration
|
|
|
|
#### **Certificate Management**
|
|
```bash
|
|
# Let's Encrypt certificate generation
|
|
docker run --rm -it \
|
|
-v /etc/letsencrypt:/etc/letsencrypt \
|
|
-v /var/lib/letsencrypt:/var/lib/letsencrypt \
|
|
certbot/certbot certonly \
|
|
--standalone \
|
|
--email admin@yourdomain.com \
|
|
--agree-tos \
|
|
--no-eff-email \
|
|
-d kms.yourdomain.com
|
|
|
|
# Certificate renewal cron job
|
|
0 2 * * * docker run --rm -it \
|
|
-v /etc/letsencrypt:/etc/letsencrypt \
|
|
-v /var/lib/letsencrypt:/var/lib/letsencrypt \
|
|
certbot/certbot renew --quiet
|
|
```
|
|
|
|
#### **SSL Configuration**
|
|
```nginx
|
|
# Strong SSL configuration
|
|
ssl_protocols TLSv1.2 TLSv1.3;
|
|
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305;
|
|
ssl_prefer_server_ciphers off;
|
|
ssl_session_cache shared:SSL:10m;
|
|
ssl_session_timeout 1d;
|
|
ssl_session_tickets off;
|
|
ssl_stapling on;
|
|
ssl_stapling_verify on;
|
|
|
|
# HSTS
|
|
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
|
|
```
|
|
|
|
### Secrets Management
|
|
|
|
#### **Docker Secrets**
|
|
```yaml
|
|
version: '3.8'
|
|
|
|
secrets:
|
|
db_password:
|
|
file: ./secrets/db_password.txt
|
|
jwt_secret:
|
|
file: ./secrets/jwt_secret.txt
|
|
auth_signing_key:
|
|
file: ./secrets/auth_signing_key.txt
|
|
|
|
services:
|
|
kms-api:
|
|
image: kms-api:latest
|
|
secrets:
|
|
- db_password
|
|
- jwt_secret
|
|
- auth_signing_key
|
|
environment:
|
|
- DB_PASSWORD_FILE=/run/secrets/db_password
|
|
- JWT_SECRET_FILE=/run/secrets/jwt_secret
|
|
- AUTH_SIGNING_KEY_FILE=/run/secrets/auth_signing_key
|
|
```
|
|
|
|
#### **HashiCorp Vault Integration**
|
|
```go
|
|
// Vault secret retrieval
|
|
func getSecretFromVault(path string) (string, error) {
|
|
client, err := vault.NewClient(&vault.Config{
|
|
Address: os.Getenv("VAULT_ADDR"),
|
|
})
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
|
|
client.SetToken(os.Getenv("VAULT_TOKEN"))
|
|
|
|
secret, err := client.Logical().Read(path)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
|
|
if secret == nil || secret.Data == nil {
|
|
return "", fmt.Errorf("secret not found")
|
|
}
|
|
|
|
value, ok := secret.Data["value"].(string)
|
|
if !ok {
|
|
return "", fmt.Errorf("invalid secret format")
|
|
}
|
|
|
|
return value, nil
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Backup and Recovery
|
|
|
|
### Database Backup Strategy
|
|
|
|
#### **Automated Backup Script**
|
|
```bash
|
|
#!/bin/bash
|
|
# backup.sh
|
|
|
|
set -e
|
|
|
|
# Configuration
|
|
BACKUP_DIR="/backups"
|
|
POSTGRES_CONTAINER="kms-postgres"
|
|
S3_BUCKET="kms-backups"
|
|
RETENTION_DAYS=30
|
|
|
|
# Create backup directory
|
|
mkdir -p $BACKUP_DIR
|
|
|
|
# Generate backup filename with timestamp
|
|
BACKUP_NAME="kms_backup_$(date +%Y%m%d_%H%M%S).sql"
|
|
BACKUP_PATH="$BACKUP_DIR/$BACKUP_NAME"
|
|
|
|
# Create database dump
|
|
echo "Creating database backup..."
|
|
docker exec $POSTGRES_CONTAINER pg_dump -U postgres kms > $BACKUP_PATH
|
|
|
|
# Compress backup
|
|
gzip $BACKUP_PATH
|
|
|
|
# Upload to S3
|
|
echo "Uploading backup to S3..."
|
|
aws s3 cp "$BACKUP_PATH.gz" "s3://$S3_BUCKET/$(date +%Y/%m/%d)/$BACKUP_NAME.gz"
|
|
|
|
# Clean up local files older than retention period
|
|
find $BACKUP_DIR -name "*.gz" -mtime +$RETENTION_DAYS -delete
|
|
|
|
# Clean up S3 files older than retention period
|
|
aws s3 ls "s3://$S3_BUCKET/" --recursive | while read -r line; do
|
|
createDate=$(echo $line | awk '{print $1" "$2}')
|
|
createDate=$(date -d "$createDate" +%s)
|
|
olderThan=$(date -d "$RETENTION_DAYS days ago" +%s)
|
|
|
|
if [[ $createDate -lt $olderThan ]]; then
|
|
fileName=$(echo $line | awk '{$1=$2=$3=""; print $0}' | sed 's/^[ \t]*//')
|
|
aws s3 rm "s3://$S3_BUCKET/$fileName"
|
|
fi
|
|
done
|
|
|
|
echo "Backup completed successfully"
|
|
```
|
|
|
|
#### **Backup Cron Job**
|
|
```bash
|
|
# Daily backup at 2 AM
|
|
0 2 * * * /opt/kms/scripts/backup.sh >> /var/log/kms-backup.log 2>&1
|
|
|
|
# Weekly full backup at 1 AM on Sundays
|
|
0 1 * * 0 /opt/kms/scripts/full-backup.sh >> /var/log/kms-backup.log 2>&1
|
|
```
|
|
|
|
### Disaster Recovery
|
|
|
|
#### **Recovery Procedure**
|
|
```bash
|
|
#!/bin/bash
|
|
# restore.sh
|
|
|
|
set -e
|
|
|
|
BACKUP_FILE=$1
|
|
POSTGRES_CONTAINER="kms-postgres"
|
|
|
|
if [ -z "$BACKUP_FILE" ]; then
|
|
echo "Usage: $0 <backup_file>"
|
|
exit 1
|
|
fi
|
|
|
|
# Download backup from S3 if needed
|
|
if [[ $BACKUP_FILE == s3://* ]]; then
|
|
LOCAL_FILE="/tmp/$(basename $BACKUP_FILE)"
|
|
aws s3 cp "$BACKUP_FILE" "$LOCAL_FILE"
|
|
BACKUP_FILE=$LOCAL_FILE
|
|
fi
|
|
|
|
# Extract if compressed
|
|
if [[ $BACKUP_FILE == *.gz ]]; then
|
|
gunzip -c "$BACKUP_FILE" > "/tmp/backup.sql"
|
|
BACKUP_FILE="/tmp/backup.sql"
|
|
fi
|
|
|
|
# Stop API services
|
|
echo "Stopping API services..."
|
|
docker-compose stop kms-api kms-frontend
|
|
|
|
# Drop and recreate database
|
|
echo "Recreating database..."
|
|
docker exec $POSTGRES_CONTAINER psql -U postgres -c "DROP DATABASE IF EXISTS kms;"
|
|
docker exec $POSTGRES_CONTAINER psql -U postgres -c "CREATE DATABASE kms;"
|
|
|
|
# Restore database
|
|
echo "Restoring database..."
|
|
docker exec -i $POSTGRES_CONTAINER psql -U postgres kms < "$BACKUP_FILE"
|
|
|
|
# Start services
|
|
echo "Starting services..."
|
|
docker-compose up -d
|
|
|
|
# Verify restoration
|
|
echo "Verifying restoration..."
|
|
sleep 30
|
|
curl -f http://localhost:8081/health || {
|
|
echo "Health check failed after restoration"
|
|
exit 1
|
|
}
|
|
|
|
echo "Database restoration completed successfully"
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### **Database Connection Issues**
|
|
```bash
|
|
# Check database container logs
|
|
docker logs kms-postgres
|
|
|
|
# Test database connectivity
|
|
docker exec kms-postgres pg_isready -U postgres
|
|
|
|
# Check connection pool status
|
|
curl http://localhost:8080/health | jq '.services.database'
|
|
```
|
|
|
|
#### **Authentication Problems**
|
|
```bash
|
|
# Check JWT secret configuration
|
|
docker exec kms-api env | grep JWT_SECRET
|
|
|
|
# Verify HMAC key configuration
|
|
docker exec kms-api env | grep AUTH_SIGNING_KEY
|
|
|
|
# Test authentication endpoint
|
|
curl -X POST http://localhost:8081/api/login \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"app_id": "test-app", "user_id": "test@example.com"}'
|
|
```
|
|
|
|
#### **Performance Issues**
|
|
```bash
|
|
# Check API response times
|
|
curl -w "@curl-format.txt" -s -o /dev/null http://localhost:8081/api/health
|
|
|
|
# Monitor database connections
|
|
docker exec kms-postgres psql -U postgres -c "SELECT count(*) FROM pg_stat_activity;"
|
|
|
|
# Check memory usage
|
|
docker stats kms-api
|
|
```
|
|
|
|
#### **SSL/TLS Issues**
|
|
```bash
|
|
# Test SSL certificate
|
|
openssl s_client -connect kms.yourdomain.com:443 -servername kms.yourdomain.com
|
|
|
|
# Check certificate expiration
|
|
curl -vI https://kms.yourdomain.com 2>&1 | grep -i expire
|
|
|
|
# Verify certificate chain
|
|
curl -I https://kms.yourdomain.com
|
|
```
|
|
|
|
### Debugging Commands
|
|
|
|
#### **Container Debugging**
|
|
```bash
|
|
# View container logs
|
|
docker logs -f kms-api
|
|
|
|
# Execute shell in container
|
|
docker exec -it kms-api /bin/sh
|
|
|
|
# Inspect container configuration
|
|
docker inspect kms-api
|
|
|
|
# Check resource usage
|
|
docker stats
|
|
```
|
|
|
|
#### **Network Debugging**
|
|
```bash
|
|
# Test inter-container connectivity
|
|
docker exec kms-api ping kms-postgres
|
|
|
|
# Check port binding
|
|
netstat -tlnp | grep :8080
|
|
|
|
# Inspect Docker network
|
|
docker network inspect kms_kms-network
|
|
```
|
|
|
|
This deployment guide provides comprehensive instructions for deploying, configuring, and maintaining the KMS in various environments, from development to production-scale deployments. |