Published on

How a request reaches your pod in Kubernetes — DNS to Ingress to Service to Pod

Authors
  • avatar
    Name
    Krzysztof Kozłowski
    Twitter

I'll be honest. For a long time I could deploy to Kubernetes without really knowing how a request reached my container.

I wrote the Deployment. I wrote the Service. I wrote the Ingress. Traffic flowed. It just worked.

Until it didn't.

One day an app started returning 503, and I had no mental map of where the request even went. I was poking at random pieces, hoping one of them was the broken one. I burned an afternoon on it.

The fix, in the end, was small. But the slow part wasn't the fix — it was that I didn't know the path. I couldn't walk it in my head.

So here's the full path, hop by hop, from a user's browser to the container inside your pod. Once you can trace this, "why is my app returning 503" stops being a guessing game.

The whole chain in one frame

A request takes six hops. Two of them are outside the cluster. Four are inside.

Read it left to right:

DNS → Load balancer → Ingress / Gateway API → Service → Pod → Container

That arrow chain is the whole post. Everything below is just "what does this hop actually do".

DNS — find the front door

The browser starts with a name, not an address. api.example.com means nothing to the network until it's resolved to an IP.

So the browser asks DNS, and DNS answers with an IP.

Here's the part I had wrong for a while: that IP doesn't belong to any pod. It belongs to the load balancer sitting in front of the cluster. Pods get new IPs every time they restart — you'd never put a pod IP in DNS. The stable, public address is the load balancer.

Load balancer — the public IP

The cloud load balancer (Azure, AWS, GCP) is what actually listens on that public IP. The request lands here first.

Its job at this point is simple: take the traffic from the internet and forward it into the cluster — usually straight to the Ingress controller.

This is the boundary. Everything before this hop is the public internet. Everything after it is inside your cluster.

Ingress / Gateway API — the traffic router

This is where routing decisions happen.

The Ingress (or its newer successor, the Gateway API) looks at the request's host and path and decides which Service should handle it. api.example.com goes to one Service, app.example.com to another, /admin maybe to a third.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: edge
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-api # routes to this Service
                port:
                  number: 80

Two things usually live here too: TLS termination (the HTTPS connection ends at the Ingress, plain HTTP continues inside), and the fact that one entry point fans out to many backends.

One Ingress. Many Services behind it. That's the normal shape.

Service — the load-balancing layer

The Ingress forwards to a Service. And the Service is the hop people misunderstand most.

A Service is a stable internal address — a ClusterIP. It runs nothing itself. It doesn't hold your code. What it does is keep a list of healthy pods and spread traffic across them.

apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  selector:
    app: my-api # which pods belong to this Service
  ports:
    - port: 80
      targetPort: 8080

So the load balancing across your pods doesn't happen at the Ingress. It happens here, at the Service. The Ingress just picks which Service. The Service picks which pod.

I'll come back to why that distinction matters.

Pod — and who's even eligible

The Service picks one pod from its list and forwards the request.

But not every running pod is in that list. Only pods that pass their readiness probe are. A readiness probe is the pod telling Kubernetes "I'm ready to take traffic".

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

If that probe fails, the pod is quietly pulled from the Service's list of endpoints. It keeps running. It just stops getting traffic. This is by design — you don't want requests hitting a pod that's still starting up or has lost its database connection.

Container — finally

Inside the pod, the request hits your container, on the port your app actually listens on (targetPort from the Service).

That's the last hop. Browser to DNS to load balancer to Ingress to Service to pod to container. Six steps, and your code runs.

Two things that click once you see this

The Service is the load-balancing layer, not the Ingress. The Ingress routes by host and path — it decides which Service. The Service spreads traffic across pods — it decides which pod. People mix these up constantly and then get confused about where traffic is actually being balanced.

Readiness probes decide who's in the Service's list. A failing readiness probe removes a pod from the path silently. No error, no crash. The pod is just... not getting requests anymore. If you don't know this hop exists, a half-empty endpoint list is invisible to you.

The payoff — debugging a 503 by walking the chain

This is the part that made the whole picture worth learning.

When something returns 503, you don't guess anymore. You walk the chain and find the hop that breaks. Each hop has a command:

# DNS — does the name resolve, and to the load balancer IP?
nslookup api.example.com

# Ingress — is there a rule for this host/path?
kubectl describe ingress edge

# Service — does it exist and have the right selector?
kubectl get svc my-api

# Endpoints — THE big one. Are there any pods in the list?
kubectl get endpoints my-api

# Pods — are they running AND ready (the second number in READY)?
kubectl get pods -l app=my-api

That kubectl get endpoints line is the one that ends most 503 hunts. If it shows <none>, the Service has no healthy pods to send traffic to — and now you know exactly where to look next: the readiness probe, or the Service selector not matching any pod labels.

Before I knew the chain, a 503 was "something is broken somewhere". After, it's "the chain breaks at hop four, the endpoint list is empty, the readiness probe is failing". Same bug. A fraction of the time.

Pitfalls I hit

Thinking the Ingress load-balances across pods. It doesn't. It routes to a Service, and the Service spreads traffic. I spent time looking at Ingress config for a balancing problem that lived in the Service.

Forgetting the endpoint list can be empty. A Service with a selector that matches nothing, or with every pod failing readiness, looks fine in kubectl get svc — but kubectl get endpoints shows the truth. Always check endpoints, not just the Service.

Readiness probe too aggressive. A probe with a tight timeout under load can flap pods in and out of the list. Traffic gets weird and you don't see a crash anywhere. The pods are healthy; the probe is just too strict.

Assuming DNS points at a pod. It points at the load balancer. Pod IPs are ephemeral and never belong in DNS.

Mixing up port and targetPort. The Service listens on port. Your container listens on targetPort. Swap them and traffic reaches the Service and dies one hop short of your app.

Key takeaways

  • A request takes six hops: DNS → Load balancer → Ingress → Service → Pod → Container.
  • The DNS name and public IP belong to the load balancer, never to a pod.
  • The Ingress routes by host and path. The Service load-balances across pods. Different jobs.
  • Only pods passing their readiness probe are in the Service's endpoint list — a failing probe removes a pod silently.
  • To debug a 503, walk the chain and find the broken hop. kubectl get endpoints ends most hunts.

What's next

If you came here from the Service types question — ClusterIP vs NodePort vs LoadBalancer — this is the picture those types live inside. I wrote that one up separately: Kubernetes Service types — ClusterIP vs NodePort vs LoadBalancer.

If I'd had this chain in my head the first time I saw a 503, I'd have saved myself an afternoon. That's the whole point of drawing it once — the next outage, you just walk the path.

I work through Kubernetes one confusing corner at a time on LinkedIn — that's the best place to push back on anything here.