19: Security, Secrets Management & Hardening¶

This section explains how to secure your entire LocalCloudLab environment. Security is not a single component — it is a multi-layer discipline involving:

• Secrets management
• Network policies
• RBAC (Role-Based Access Control)
• API hardening (rate limits, input validation)
• Pod and container security
• Node-level security
• Service-to-service encryption
• Key rotation
• Protecting PostgreSQL, Redis, RabbitMQ
• Protecting your Kubernetes cluster & Envoy Gateway
• Securing CI/CD (GitHub Actions)

This is one of the most critical sections in the entire project.

19.1 Security Overview for LocalCloudLab¶

Security in Kubernetes must follow defense in depth:

Layer 1 — Code security
Layer 2 — API security
Layer 3 — TLS & encryption
Layer 4 — Secrets management
Layer 5 — Network policies
Layer 6 — Pod security
Layer 7 — Cluster RBAC
Layer 8 — Node security
Layer 9 — Backup security
Layer 10 — CI/CD security

If any layer is weak → the system is vulnerable.

19.2 Secrets Management in LocalCloudLab¶

Kubernetes Secrets are base64-encoded (NOT encrypted). This means:

✗ NEVER commit real secrets into Git
✗ Base64 ≠ security
✓ Use Encryption-at-Rest
✓ Use Sealed Secrets or External Secrets (future)

19.2.1 Create Kubernetes Secrets Safely¶

Example:

kubectl create secret generic search-api-secret       --from-literal=postgres-password="XXXX"       --from-literal=jwt-key="YYYY"       -n search

This generates:

/var/lib/rancher/k3s/server/db/state.db

19.2.2 Encrypt Secrets at Rest (k3s)¶

Check if encryption is enabled:

cat /var/lib/rancher/k3s/server/cred/encryption-config.json

If not present, enable:

Create /var/lib/rancher/k3s/server/encryption-config.yaml:

apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: <64-character-base64-secret> - identity: {}
Restart k3s:

systemctl restart k3s

Now secrets are encrypted on disk.

19.3 Sealed Secrets (Optional Future)¶

You may want to keep encrypted secrets in Git safely.

Tool: Bitnami Sealed Secrets.

Workflow:

kubeseal --format yaml < secret.yaml > secret-sealed.yaml

You commit secret-sealed.yaml, but only your cluster can decrypt it.

This enables:

✓ GitOps
✓ Versioning
✓ Zero fear of leaks

Recommend for future expansion.

19.4 External Secrets Operator (Advanced Future Option)¶

If you want secrets pulled automatically from:

• AWS Secrets Manager
• GCP Secret Manager
• HashiCorp Vault

This allows:

• External rotation
• Stronger security
• Centralized secrets management

LocalCloudLab doesn’t require it yet, but it is compatible.

19.5 Network Policies (Critical for Isolation)¶

By default: All pods can talk to all pods.

This is insecure.

NetworkPolicies restrict communication.

Example: Search API can only talk to Redis + PostgreSQL:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: search-api-policy
  namespace: search
spec:
  podSelector:
    matchLabels:
      app: search-api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: envoy-gateway
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: db
    - namespaceSelector:
        matchLabels:
          name: caching

This prevents:

✓ Pod-to-pod attacks
✓ Lateral movement
✓ Misconfigured services talking everywhere

19.6 RBAC — Role-Based Access Control¶

Use kubectl auth can-i to check permissions:

kubectl auth can-i get pods -n search

Principles:

• Use least privilege
• No root on production namespaces
• Robots (CI/CD) must have limited permissions

Example of restricting GitHub Actions:

Create a ServiceAccount:

kubectl create sa cicd-deployer -n kube-system

Apply permissions:

kind: ClusterRole
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["patch", "get", "list"]

Bind role:

kind: ClusterRoleBinding
roleRef:
  kind: ClusterRole
  name: cicd-deployer
subjects:
- kind: ServiceAccount
  name: cicd-deployer
  namespace: kube-system

Use this SA’s token in GitHub Actions instead of full admin.

19.7 Pod Security Standards (PSS)¶

Apply:

• Disallow privileged containers
• Prevent host network access
• Drop root privileges

Example:

securityContext:
  runAsUser: 1000
  runAsGroup: 1000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true

This prevents:

✓ container breakout
✓ privilege escalation
✓ malicious file writes

19.8 TLS Everywhere¶

Already implemented:

• cert-manager
• Let’s Encrypt
• TLS termination at Envoy

Additional steps:

• Internal mTLS (future upgrade)
• Service-to-service encrypted calls

Optional: Istio or Linkerd for internal mTLS.

19.9 API Security Best Practices¶

Every endpoint must:

1. Validate inputs
2. Limit body size
3. Enforce auth where required
4. Rate limit sensitive endpoints

Input Validation¶

Use FluentValidation in Application Layer:

RuleFor(x => x.Email).EmailAddress();
RuleFor(x => x.Date).GreaterThan(DateTime.UtcNow);

Body Size Limit¶

Program.cs:

app.Use((context, next) =>
{
    context.Features.Get<IHttpMaxRequestBodySizeFeature>().MaxRequestBodySize = 10_000_000;
    return next();
});

Rate Limit (Envoy)¶

As configured in Section 15.

19.10 Protecting PostgreSQL¶

Checklist:

✓ Only accessible inside cluster
✓ Strong passwords
✓ No outside exposure via LoadBalancer
✓ Enforce TLS (future option)
✓ Limit max connections
✓ Frequent backups
✓ WAL archiving

Block external access:

NetworkPolicy
and
No Service type LoadBalancer

19.11 Protecting Redis¶

Checklist:

✓ NO external exposure
✓ Require auth
✓ Disable dangerous commands (e.g., FLUSHALL)
✓ Enable persistence
✓ Enable snapshotting

Disable dangerous commands:

rename-command FLUSHALL ""
rename-command FLUSHDB  ""

19.12 Protecting RabbitMQ¶

Checklist:

✓ Use strong credentials
✓ Use virtual hosts
✓ Limit user permissions
✓ Disable guest user
✓ Restrict management UI
✓ Use durable queues
✓ Use mirrored queues (optional)

19.13 Securing GitHub Actions (Critical)¶

GitHub Actions is a security boundary. If compromised → attackers deploy code into your cluster.

Checklist:

✓ Use fine-grained PAT
✓ Limit PAT to only packages:write
✓ Store kubeconfig securely
✓ Use ServiceAccount token instead of kubeconfig (recommended future)
✓ Restrict workflow triggers
✓ Use branch protection rules
✓ Require PR approvals

Never store:

✗ SSH keys in repo
✗ Secrets in YAML
✗ Plain kubeconfig

Rotate secrets every 90 days.

19.14 Summary of Section 19¶

You now have:

✔ Secure secrets management
✔ Encrypted secrets at rest
✔ Network policies
✔ RBAC & least privilege
✔ Pod security contexts
✔ API hardening
✔ PostgreSQL/Redis/RabbitMQ hardening
✔ CI/CD security
✔ Full defense-in-depth strategy

This section transforms LocalCloudLab from “functional” into secure.

Next section begins automatically:

Section 20 — LocalCloudLab Deployment Lifecycle & Operations