05: Envoy Gateway¶

5.1 Introduction¶

Envoy Gateway is the modern, standardized, Kubernetes-native gateway built on top of the Envoy Proxy. Unlike traditional ingress controllers, Envoy Gateway uses the Gateway API specification—a more powerful, expressive, extensible API created by the Kubernetes networking SIG.

In LocalCloudLab, Envoy Gateway is responsible for:

• Handling all inbound HTTP/HTTPS traffic into the cluster
• Terminating TLS for your public domains
• Routing requests to Search API and Checkin API
• Enforcing routing rules and security policies
• Integrating with cert-manager for automatic SSL certificates
• Providing a clear separation between application logic and traffic management

Envoy Gateway transforms your cluster into a production-grade platform with:

• Host-based routing
• Path-based routing
• TLS configuration
• Request/response transformations
• Extensible policies
• High-performance data plane (Envoy Proxy)

This section is one of the most important in LocalCloudLab because it is the entry point for all external clients accessing your APIs—including real users, test tools, browser requests, and mobile apps.

You will learn how to install Envoy Gateway using supported Helm charts, configure Gateway API resources, issue certificates using cert-manager, and validate the final routing setup.

5.2 Envoy Gateway vs Envoy Proxy vs Ingress Controller¶

Before installation, it is important to differentiate the three major concepts you will encounter:

5.2.1 Envoy Proxy¶

Envoy Proxy is a high-performance L7 proxy used widely in service meshes (e.g., Istio) and gateways.

It is a: • High-performance data plane • Proxy engine capable of advanced traffic operations • Foundation for many modern cloud-native networking tools

But by itself, Envoy Proxy does NOT include: • A Kubernetes-native API • Reconciliation controllers • CRDs for gateway resources • Automatic configuration management

Envoy Proxy alone requires manual config files—too low-level for Kubernetes.

5.2.2 Envoy Gateway¶

Envoy Gateway is: • A Kubernetes controller • A translation layer between Gateway API resources and Envoy configuration • A manager that deploys Envoy data-plane pods • A controller that reconciles Gateway, HTTPRoute, ReferenceGrant, etc.

In short: • Gateway API config → Envoy Gateway → Envoy Proxy dataplane

Envoy Gateway automates: • Deployment of Envoy proxies • Certificate wiring • Configuration management • Traffic routing • Error handling

This is perfect for LocalCloudLab.

5.2.3 Ingress Controller (NGINX, Traefik, etc.)¶

Traditional Ingress Controllers use the older Ingress API.

Limitations of the Ingress API: • Weak expressive power • No separation between listeners and routes • Not very extensible • Harder to model complex routing • Vendor-specific annotations everywhere

Gateway API solves these issues and is the future of Kubernetes ingress.

Envoy Gateway fully embraces Gateway API.

5.3 Why Envoy Gateway for LocalCloudLab?¶

There are several reasons why Envoy Gateway is chosen over alternatives:

✔ Modern, future-proof API (Gateway API)¶

Kubernetes Ingress is effectively being replaced by Gateway API.

Envoy Gateway is the reference implementation for Gateway API.

✔ Very high performance¶

Envoy is battle-tested at: • Lyft • Google • Apple • Cloudflare • Amazon AppMesh

It is one of the fastest proxies in the world.

✔ Easy integration with cert-manager¶

Envoy Gateway + cert-manager provide: • Automatic HTTPS • Automatic renewal • DNS-based or HTTP-based validation

✔ Clear separation of concerns¶

Gateway = listener configuration HTTPRoutes = application routing rules

✔ Perfect match for microservices¶

You will manage routes separately for: • Search API • Checkin API • Additional APIs in the future

✔ Production-grade capabilities¶

Envoy supports: • mTLS • JWT authentication • Rate limiting • Load balancing strategies • Timeout & retry policies • Response rewriting • Request header manipulations

LocalCloudLab doesn’t require all these features now, but the architecture supports them.

5.4 Helm Installation (Non-Deprecated Chart)¶

Next, we install Envoy Gateway using the official Helm chart.

We use: • Latest stable version • Recommended repository • Best-practice installation pattern

5.4.1 Add the Envoy Gateway Helm repo¶

helm repo add envoy-gateway https://gateway.envoyproxy.io/helm
helm repo update

5.4.2 Create namespace¶

kubectl create namespace envoy-gateway-system

5.4.3 Install Envoy Gateway¶

helm install eg envoy-gateway/envoy-gateway -n envoy-gateway-system

This installation deploys: • envoy-gateway controller • envoy proxy daemonset (or deployment depending on chart version) • Gateway API CRDs (if not already installed)

Check resources:

kubectl get pods -n envoy-gateway-system

You should see: • envoy-gateway controller • envoy-proxy pods

5.4.4 Verify CRDs were installed¶

kubectl get crds | grep gateway

You should see: • gateways.gateway.networking.k8s.io • httproutes.gateway.networking.k8s.io • referencegrants.gateway.networking.k8s.io • backendpolicies.gateway.envoyproxy.io • tlsroutes.gateway.networking.k8s.io • more depending on version

If CRDs are missing, the installation must be repeated.

(End of Part 1 — Part 2 will continue with Gateway architecture and creating the main Gateway resource.)

5.5 Envoy Gateway Architecture¶

To use Envoy Gateway effectively, it is essential to understand how its architecture is structured. Envoy Gateway consists of two major layers:

1. The **Control Plane** (Envoy Gateway controller)
2. The **Data Plane** (Envoy Proxy instances)

These work together to translate Gateway API resources into real routing behavior.

5.5.1 Control Plane: Envoy Gateway Controller¶

The controller is the brain of the system.

It performs the following responsibilities:

• Watches Kubernetes resources:
    - Gateway
    - HTTPRoute
    - ReferenceGrant
    - Service
    - Secrets (TLS certs)

• Validates configurations
• Creates Envoy Proxy bootstrap configurations
• Ensures Envoy pods reflect the desired routing state
• Updates configuration when new routes, certs, or domains appear

You can view the controller logs:

kubectl logs -n envoy-gateway-system deploy/envoy-gateway

5.5.2 Data Plane: Envoy Proxy¶

Envoy Proxy handles real network traffic.

Envoy performs:

• TLS termination
• Routing decisions
• Load balancing across backends
• HTTP/2 and gRPC support
• Retry and timeout policies
• Header manipulation
• Error handling
• Metrics emission

Envoy is deployed either as:

• a DaemonSet (one per node), or
• a Deployment (one or more pods)

Check Envoy pods:

kubectl get pods -n envoy-gateway-system -l app.kubernetes.io/name=envoy

5.5.3 Gateway API as the Configuration Layer¶

Gateway API CRDs give you a clean and modern configuration model.

Core resources include:

• Gateway
• HTTPRoute
• TLSRoute
• TCPRoute
• ReferenceGrant

Envoy Gateway specifically uses:

• Gateway
• HTTPRoute
• ReferenceGrant

This model separates:

Listener configuration (Gateway)
from
Routing logic (HTTPRoute)

Which is significantly cleaner than the old Ingress model.

5.5.4 Reconciliation Flow (Important)¶

The sequence for routing traffic is:

1. You apply a Gateway manifest
2. Envoy Gateway controller receives it
3. You apply HTTPRoutes referencing the Gateway
4. Controller validates routes and services
5. Controller generates Envoy configuration
6. Envoy proxies reload their dynamic config
7. Traffic immediately reflects the new rules

This architecture makes routing highly dynamic and robust.

5.6 Creating the Main Gateway Resource¶

The Gateway resource defines:

• Which ports Envoy listens on
• Which protocols (HTTP, HTTPS)
• Whether TLS termination is enabled
• Which routes may attach to it

This is the “front door” of your cluster.

5.6.1 Create gateway.yaml¶

You will store this in your Git repo, under:

k8s/gateway/gateway.yaml

Content:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-cert
            kind: Secret
      allowedRoutes:
        namespaces:
          from: All

Explanation:

• Listener 1: HTTP on port 80
• Listener 2: HTTPS on port 443 with TLS termination
• Envoy Gateway will read TLS certs from a Secret named tls-cert
• All routes in any namespace may attach to this Gateway

Apply it:

kubectl apply -f k8s/gateway/gateway.yaml

5.6.2 MetalLB will assign an External IP¶

Because the Gateway spawns a Service of type LoadBalancer, MetalLB assigns an IP from the pool:

kubectl get svc -n envoy-gateway-system

Expected:

NAME            TYPE           EXTERNAL-IP        PORT(S)
envoy-gateway   LoadBalancer   172.18.255.200     80, 443

This will be your public entry point.

5.7 Attaching Routes to the Gateway¶

Defining the Gateway is only part 1.

Now we attach HTTPRoutes for:

• Search API
• Checkin API

Each route maps:

• Hostnames (e.g., search.hershkowitz.co.il)
• Paths (e.g., /health, /search)
• Backend services (Service objects)

We cover full route examples in the next section.

5.8 Creating HTTPRoutes for Search and Checkin APIs¶

Now that the Gateway is created and exposed via MetalLB, we attach routing rules to it. This is done through HTTPRoute resources.

Each HTTPRoute defines:

• Which hostnames it matches
• Which paths it matches
• Which Kubernetes Services receive the traffic
• Optional filters (headers, redirects, rewrites)

In LocalCloudLab, we create two routes:

1. Search API route
2. Checkin API route

5.8.1 Prerequisites¶

Before creating HTTPRoutes, ensure that:

• The Search API and Checkin API Deployments are running
• Each has a stable Kubernetes Service (ClusterIP)
• DNS records point to the Envoy external IP

Examples: search.hershkowitz.co.il → 172.18.255.200 checkin.hershkowitz.co.il → 172.18.255.200

5.8.2 Search API HTTPRoute¶

Create file:

k8s/gateway/search-api-route.yaml

Contents:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: search-api-route
  namespace: search
spec:
  parentRefs:
    - name: main-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "search.hershkowitz.co.il"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: search-api-svc
          port: 80

Explanation:

• All requests to search.hershkowitz.co.il go to search-api-svc:80
• PathPrefix "/" means everything is routed
• No rewrites needed unless APIs require them

Apply:

kubectl apply -f k8s/gateway/search-api-route.yaml

5.8.3 Checkin API HTTPRoute¶

Create:

k8s/gateway/checkin-api-route.yaml

Contents:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkin-api-route
  namespace: checkin
spec:
  parentRefs:
    - name: main-gateway
      namespace: envoy-gateway-system
  hostnames:
    - "checkin.hershkowitz.co.il"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: checkin-api-svc
          port: 80

Apply:

kubectl apply -f k8s/gateway/checkin-api-route.yaml

5.8.4 Verifying Routes¶

Run:

kubectl get httproutes --all-namespaces

You should see:

search    search-api-route     Accepted
checkin   checkin-api-route    Accepted

Use describe for debugging:

kubectl describe httproute -n search search-api-route

5.9 Integrating cert-manager for TLS¶

To enable HTTPS termination at Envoy Gateway, you need certificates.

cert-manager automates:

• Certificate creation
• Renewal
• DNS validation
• Secret management

Envoy Gateway reads TLS certs from Kubernetes Secrets referenced in the Gateway resource.

5.9.1 Install cert-manager (non-deprecated Helm chart)¶

Add repo:

helm repo add jetstack https://charts.jetstack.io
helm repo update

Install:

helm install cert-manager jetstack/cert-manager       --namespace cert-manager       --create-namespace       --set crds.enabled=true

Verify:

kubectl get pods -n cert-manager

5.9.2 Create ClusterIssuer for Let's Encrypt (Production)¶

Create:

k8s/cert-manager/cluster-issuer.yaml

Contents:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: your-email@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: acme-private-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRef:
              name: main-gateway
              namespace: envoy-gateway-system

Apply:

kubectl apply -f k8s/cert-manager/cluster-issuer.yaml

5.9.3 Create Certificate resource for the Gateway¶

Create:

k8s/cert-manager/tls-cert.yaml

Contents:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-cert
  namespace: envoy-gateway-system
spec:
  secretName: tls-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - "search.hershkowitz.co.il"
    - "checkin.hershkowitz.co.il"

Apply:

kubectl apply -f k8s/cert-manager/tls-cert.yaml

Verify:

kubectl describe certificate -n envoy-gateway-system tls-cert

5.9.4 Envoy Gateway picks up TLS automatically¶

Envoy Gateway listens for Secrets referenced in Gateway spec.

Your main-gateway already contains:

  tls:
    mode: Terminate
    certificateRefs:
      - name: tls-cert
        kind: Secret

As soon as cert-manager populates the secret, Envoy Gateway reloads configuration.

5.10 Validating Envoy Gateway Routing¶

Now it’s time to validate that everything works end-to-end.

5.10.1 Validate Gateway status¶

kubectl get gateway -n envoy-gateway-system
kubectl describe gateway -n envoy-gateway-system main-gateway

5.10.2 Validate Envoy service external IP¶

kubectl get svc -n envoy-gateway-system

Expected:

envoy-gateway   LoadBalancer   172.18.255.200

5.10.3 Validate HTTP routing¶

From your PC:

curl http://search.hershkowitz.co.il/health
curl http://checkin.hershkowitz.co.il/health

Expected responses:

ok from Search
ok from Checkin

5.10.4 Validate HTTPS routing¶

curl -I https://search.hershkowitz.co.il
curl -I https://checkin.hershkowitz.co.il

Expected:

HTTP/2 200
server: envoy

5.10.5 Validate certificate issuance¶

kubectl get certificate -n envoy-gateway-system

Expected:

tls-cert   True   Ready

5.10.6 Inspect Envoy logs (optional)¶

kubectl logs -n envoy-gateway-system -l app=envoy -f

5.11 Troubleshooting Envoy Gateway¶

5.11.1 Route shows “NotAccepted”¶

Check:

kubectl describe httproute -n search search-api-route

Common issues:

• Wrong parentRef name
• Wrong namespace
• Service not found
• TLS misconfiguration

5.11.2 HTTPS returns 404¶

Likely cause:

Certificate dnsNames do not include the hostname.

5.11.3 Gateway has no External IP¶

MetalLB misconfiguration:

• IPAddressPool incorrect
• L2Advertisement missing
• MetalLB pods not running

5.11.4 Cert-manager challenges fail¶

Check:

kubectl describe challenges.acme.cert-manager.io

5.12 Summary of Section 5¶

In this section you learned how to:

• Install Envoy Gateway via Helm
• Understand the Envoy architecture (control plane + data plane)
• Create the main Gateway with HTTP and HTTPS listeners
• Attach HTTPRoutes for Search API and Checkin API
• Issue HTTPS certificates using cert-manager + Let’s Encrypt
• Validate routing and TLS termination
• Troubleshoot common issues

Envoy Gateway is the core entry point of LocalCloudLab. From here, all external traffic goes through a modern, cloud-native gateway with full observability and routing control.

Next section (Section 6) will cover: Prometheus, Grafana, Jaeger, Loki, and log/trace correlation.

(End of Section 05 — Complete)