Debugging Kubernetes Gateway API HTTPRoute Issues

Case Study: Fixing vault.1-z.me Routing

This document explains the debugging process we followed to fix a failing Kubernetes Gateway API route.\ It focuses on HTTPRoute, Gateway listeners, TLS secrets, and ReferenceGrant behavior.

---

1. The HTTPRoute was attached to the wrong Gateway listener

The route originally contained:

parentRefs:
  - name: public-gw
    namespace: default
    sectionName: https-root

But the Gateway listener https-root only allowed traffic for:

hostname: hershkowitz.co.il

Meanwhile, the HTTPRoute requested:

vault.1-z.me

Because the hostname did not match the listener hostname, the Gateway rejected the route.

Kubernetes reported this in the route status:

Reason: NoMatchingListenerHostname

Meaning:

The hostname in the route does not match the hostname allowed by that listener.

---

2. The correct listener already existed

The Gateway configuration already contained a listener designed specifically for Vault:

- name: https-vault
  hostname: vault.1-z.me

This listener perfectly matched the route's hostname.

However, the HTTPRoute was pointing to the wrong listener.

We changed the route from:

sectionName: https-root

to:

sectionName: https-vault

Now the route attaches to the correct Gateway listener.

---

3. The Vault listener itself was broken

Even after identifying the correct listener, the Gateway reported another problem:

RefNotPermitted
Certificate ref to secret vault/vault-tls not permitted by any ReferenceGrant

This occurred because:

• The Gateway is in namespace default
• The TLS secret vault-tls is in namespace vault

By default, Kubernetes blocks cross‑namespace secret references for security reasons.

Therefore the Gateway could not use the TLS certificate.

---

4. Fixing cross‑namespace TLS permissions

We resolved this by creating a ReferenceGrant resource.

This object explicitly allows a resource in one namespace to reference another namespace.

The ReferenceGrant was created in the vault namespace:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-public-gw-to-use-vault-tls
  namespace: vault
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: default
  to:
    - group: ""
      kind: Secret
      name: vault-tls

This allowed:

Gateway (default namespace)
        ↓
Secret vault-tls (vault namespace)

After applying this object, the TLS listener became valid.

---

5. Final working architecture

After the fixes, the request flow looks like this:

User
  │
  │ https://vault.1-z.me
  ▼
Envoy Gateway (public-gw)
  │
  │ Listener: https-vault
  │ TLS termination using vault-tls
  ▼
HTTPRoute: vault-route
  │
  ▼
Service: vault:8200
  │
  ▼
Vault Pod

The routing chain now works correctly:

DNS → Gateway → Listener → HTTPRoute → Service → Pod

---

6. Debugging pattern for Gateway API

When troubleshooting Gateway routing problems, follow this order.

Step 1 --- Inspect the HTTPRoute

kubectl describe httproute

Look for reasons like:

• NoMatchingParent
• NoMatchingListenerHostname
• BackendNotFound

These indicate how the route failed to attach.

---

Step 2 --- Inspect the Gateway

kubectl describe gateway

Look for problems such as:

• Invalid
• RefNotPermitted
• Programmed: False

These indicate listener configuration issues.

---

Step 3 --- Verify the backend service

kubectl get svc -n <namespace>

Confirm that:

• the Service exists
• the correct port is exposed
• the HTTPRoute backendRefs match the Service

---

7. Why this Gateway architecture is good

Your Gateway configuration uses a clean multi‑listener design.

You have one central Gateway:

public-gw

And multiple listeners such as:

https-docs
https-keycloak
https-cms
https-vault
https-hoppscotch
https-hlg-bz

This architecture is good because:

• Each hostname has its own listener
• TLS configuration is explicit
• Routing rules are easy to understand
• Debugging is simpler

It effectively creates a central edge router for the cluster.

---

8. Suggested improvement

Instead of storing TLS secrets in many namespaces, consider storing them centrally in a shared namespace such as:

infra

Example layout:

infra
 ├─ hershkowitz-co-il-tls
 ├─ hlg-tls
 ├─ vault-tls
 └─ hoppscotch-tls

Then the Gateway can reference the secrets directly without requiring ReferenceGrant resources.

This simplifies operations and reduces configuration complexity.

---

Summary

The routing problem was caused by two separate issues.

1. The HTTPRoute pointed to the wrong listener (https-root instead of https-vault).
2. The Gateway listener could not access the TLS secret because cross‑namespace references were blocked.

We fixed it by:

• Creating a ReferenceGrant allowing the Gateway to use the Vault TLS secret.
• Updating the HTTPRoute to use the correct listener (https-vault).

Once these changes were applied, the full routing chain worked correctly:

DNS → Gateway → Listener → HTTPRoute → Service → Pod