10: SSL Certificates, cert-manager & Envoy TLS¶
10.1 Introduction¶
Every production‑grade system—whether large or small—must run entirely over HTTPS.
In LocalCloudLab, TLS is required for:
• Your public APIs:
- https://search.hershkowitz.co.il
- https://checkin.hershkowitz.co.il
- any additional microservices
• The Envoy Gateway listener
• Backend communication that requires encryption
• Compliance with security and browser standards
• Protecting sensitive tokens, cookies, and personal data
• Preventing man‑in‑the‑middle (MITM) attacks
• Enabling HTTP/2 support for improved performance
The correct Kubernetes-native solution for automated TLS certificate management is cert‑manager.
cert-manager provides:
✔ Automatic Let’s Encrypt certificates
✔ Renewal before expiry
✔ ACME HTTP‑01 challenge
✔ DNS‑01 challenge (optional advanced)
✔ Integration with Envoy Gateway
This section will take you from a clean cluster with no HTTPS to a fully automated TLS infrastructure.
We will cover:
• Installing cert-manager
• Creating ClusterIssuers
• Generating certificates for your domains
• Configuring Envoy Gateway to terminate TLS
• Validating ACME challenges
• Troubleshooting failed certificates
10.2 Installing cert-manager (Non‑Deprecated Helm Chart)¶
cert-manager is deployed into its own namespace and requires CRDs (Custom Resource Definitions).
10.2.1 Add Jetstack Helm repository¶
helm repo add jetstack https://charts.jetstack.io
helm repo update
10.2.2 Install CRDs (required)¶
cert-manager CRDs must be installed before deploying the chart:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.crds.yaml
10.2.3 Create namespace¶
kubectl create namespace cert-manager
10.2.4 Install cert-manager via Helm¶
Use a modern stable version, such as v1.15.x:
helm install cert-manager jetstack/cert-manager -n cert-manager --version v1.15.0 --set installCRDs=false
Explanation:
• installCRDs=false – CRDs already installed
• v1.15.0 – choose a recent stable release
10.2.5 Validate installation¶
Check Pods:
kubectl get pods -n cert-manager
Expected:
cert-manager-xxxxx Running
cert-manager-cainjector-xxxxx Running
cert-manager-webhook-xxxxx Running
If ANY of the webhook containers is not running, cert-manager cannot issue certificates.
10.3 Creating a ClusterIssuer (Let’s Encrypt Production)¶
A ClusterIssuer is a global certificate authority configuration.
For LocalCloudLab, we use:
ACME → Let’s Encrypt → HTTP‑01 challenge
The HTTP‑01 challenge requires:
✔ Your domain A record pointing to the cluster
✔ Port 80 open on Envoy Gateway
✔ Cert-manager able to route ACME challenge paths
10.3.1 Production ClusterIssuer¶
Create:
k8s/cert-manager/cluster-issuer-prod.yaml
Contents:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: your-email@domain.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: envoy-gateway
Apply:
kubectl apply -f k8s/cert-manager/cluster-issuer-prod.yaml
Replace email with a real address—Let’s Encrypt uses it for expiry notices.
10.3.2 Staging ClusterIssuer (Recommended for Testing)¶
Let’s Encrypt rate-limits certificate issuance. During development, always use staging.
Create:
k8s/cert-manager/cluster-issuer-staging.yaml
Contents:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: your-email@domain.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: envoy-gateway
Apply:
kubectl apply -f k8s/cert-manager/cluster-issuer-staging.yaml
Now you have:
✔ letsencrypt-prod
✔ letsencrypt-staging
Ready for issuing certificates.
10.4 How Envoy Gateway Uses Certificates¶
Envoy Gateway automatically loads TLS certificates stored in Kubernetes Secrets.
Flow:
1. You define a Certificate resource
2. cert-manager requests a certificate from Let’s Encrypt
3. Let’s Encrypt verifies ACME challenge
4. cert-manager creates a TLS secret
5. Envoy Gateway loads the secret
6. HTTPS works automatically
Envoy integrates with cert-manager via certificateRefs in HTTPRoute.
Example snippet:
tls:
mode: Terminate
certificateRefs:
- name: search-hershkowitz-tls
This refers to:
secret/search-hershkowitz-tls
Automatically created by cert-manager.
(End of Part 1 — Part 2 will cover Certificate resources, HTTPRoute TLS, Let’s Encrypt validation, testing HTTPS, and troubleshooting.)
10.5 Creating Certificate Resources¶
With cert-manager and ClusterIssuers configured, the next step is to define Certificate resources for your domains.
For LocalCloudLab, you want TLS for:
• search.hershkowitz.co.il
• checkin.hershkowitz.co.il
These certificates will:
• Be requested from Let’s Encrypt
• Be renewed automatically
• Be stored as Kubernetes Secrets
• Be consumed by Envoy Gateway
10.5.1 Single Certificate for Multiple Hosts¶
You can create one certificate that covers both domains.
Create file:
k8s/cert-manager/search-checkin-cert.yaml
Contents:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: search-checkin-tls
namespace: envoy-gateway-system
spec:
secretName: search-checkin-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- "search.hershkowitz.co.il"
- "checkin.hershkowitz.co.il"
Apply:
kubectl apply -f k8s/cert-manager/search-checkin-cert.yaml
cert-manager will:
• Create an Order and Challenge
• Solve ACME HTTP-01 using Envoy Gateway
• On success → create Secret `search-checkin-tls` in `envoy-gateway-system` namespace
10.5.2 Verifying Certificate Status¶
Check:
kubectl get certificate -n envoy-gateway-system
You should see:
NAME READY SECRET
search-checkin-tls True search-checkin-tls
More details:
kubectl describe certificate -n envoy-gateway-system search-checkin-tls
Look for:
Conditions:
Type: Ready
Status: True
10.6 Adding TLS to Envoy HTTPRoutes¶
Now that you have a valid TLS secret, you must tell Envoy Gateway to use it.
There are two ways:
• TLS at Gateway level (one cert shared by many routes)
• TLS at Route level (per-host configuration)
In LocalCloudLab we keep the Gateway simple and configure TLS via HTTPRoutes.
10.6.1 Update main Gateway (if not already using TLS)¶
Your Gateway (simplified):
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: main-gateway
namespace: envoy-gateway-system
spec:
gatewayClassName: envoy-gateway
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
- name: https
protocol: HTTPS
port: 443
allowedRoutes:
namespaces:
from: All
Apply (if not already done):
kubectl apply -f k8s/gateway/gateway.yaml
10.6.2 Enable TLS in Search API HTTPRoute¶
Update k8s/gateway/search-api-route.yaml:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: search-api-route
namespace: search
spec:
parentRefs:
- name: main-gateway
namespace: envoy-gateway-system
hostnames:
- "search.hershkowitz.co.il"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: search-api-svc
port: 80
tls:
mode: Terminate
certificateRefs:
- name: search-checkin-tls
kind: Secret
Key points:
• tls.mode=Terminate → Envoy terminates HTTPS
• certificateRefs → uses Secret created by cert-manager
• Secret must be in same namespace as the Gateway (envoy-gateway-system)
10.6.3 Enable TLS in Checkin API HTTPRoute¶
Update k8s/gateway/checkin-api-route.yaml:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: checkin-api-route
namespace: checkin
spec:
parentRefs:
- name: main-gateway
namespace: envoy-gateway-system
hostnames:
- "checkin.hershkowitz.co.il"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: checkin-api-svc
port: 80
tls:
mode: Terminate
certificateRefs:
- name: search-checkin-tls
kind: Secret
Apply both routes:
kubectl apply -f k8s/gateway/search-api-route.yaml
kubectl apply -f k8s/gateway/checkin-api-route.yaml
10.7 Testing HTTPS & Certificate Health¶
Once certificates are ready and HTTPRoutes are updated, test:
• Browser access over HTTPS
• Certificate validity
• TLS handshake success
10.7.1 Browser tests¶
From your PC:
https://search.hershkowitz.co.il
https://checkin.hershkowitz.co.il
Check:
• No "Not secure" warning
• Certificate issued by "R3" / Let's Encrypt
• Correct hostnames in certificate
10.7.2 curl tests¶
From your machine:
curl -v https://search.hershkowitz.co.il/health
curl -v https://checkin.hershkowitz.co.il/health
You should see:
* TLS handshake success
* HTTP/2 or HTTP/1.1
* Response body:
ok from Search
ok from Checkin
10.7.3 Check Certificate resources and Orders¶
If things fail, inspect Certificates:
kubectl describe certificate -n envoy-gateway-system search-checkin-tls
And related Orders & Challenges:
kubectl get orders.acme.cert-manager.io -A
kubectl get challenges.acme.cert-manager.io -A
Then:
kubectl describe challenge -n <namespace> <name>
10.7.4 Common ACME HTTP-01 problems¶
• DNS not pointing to Envoy external IP
• Port 80 blocked by firewall
• Incorrect Envoy HTTP routing for /.well-known/acme-challenge/*
• ClusterIssuer misconfigured (wrong ingress class, wrong email, etc.)
10.8 Automatic Renewal¶
cert-manager tracks certificate expiry and renews automatically.
By default:
• It attempts renewal when ~2/3 of certificate lifetime has passed
• Let’s Encrypt certificates are valid for 90 days
• Renewal is attempted ~30 days before expiry
10.8.1 Checking renewal status¶
List all Certificates:
kubectl get certificate --all-namespaces
Inspect a specific one:
kubectl describe certificate -n envoy-gateway-system search-checkin-tls
You will see fields like:
Not After: 2025-08-01T10:23:45Z
Renewal Time: 2025-07-02T10:23:45Z
10.8.2 Forcing a manual renewal (for debugging)¶
You can delete the secret to force re-issuance:
kubectl delete secret search-checkin-tls -n envoy-gateway-system
cert-manager will notice the missing secret and try to re-issue the certificate.
Be careful: for a brief window, HTTPS may break.
10.9 Summary of Section 10¶
In this section you:
✔ Installed cert-manager with a non-deprecated Helm chart
✔ Configured Let’s Encrypt production and staging ClusterIssuers
✔ Created Certificate resources for search.hershkowitz.co.il and checkin.hershkowitz.co.il
✔ Integrated Envoy Gateway with cert-manager via TLS in HTTPRoutes
✔ Successfully enabled HTTPS for your APIs
✔ Learned how to debug ACME challenges and certificate failures
✔ Understood how automatic renewal works and how to verify it
Your LocalCloudLab cluster now supports fully automated, production-grade TLS.
Next section (Section 11) will focus on:
• GitOps-friendly directory structure for all YAML files
• Recommended repository layout for LocalCloudLab
• Branching strategy, environments, and CI/CD pipeline structure
(End of Section 10 — Complete)