Nginx Ingress to Cilium: Migration Retrospective

TL;DR — Replacing Ingress-NGINX turned into a full networking-stack swap (CNI + kube-proxy + ingress). The migration itself went to plan; the interesting part came after. Over the next ten days, services that "used to just work" broke one by one — and almost every root cause was the same shape: something kube-proxy or a shared IP had been handling implicitly now required an explicit declaration.

Summary

We switched a production Kubernetes cluster's CNI from flannel to Cilium. The end-of-life of the Ingress-NGINX Controller was the direct trigger, but in practice this was a simultaneous change across the CNI, kube-proxy, and ingress layers. Over a weekend we reinstalled the cluster and backed up and restored users, Projects, ImageHub, and Workloads in full.

The migration itself finished as planned. What was interesting came next. Services that worked perfectly on migration day revealed problems one at a time over the following ten days. Domains, timeouts, external VIPs, monitoring — all things that "used to work," and all with root causes in the same place.

This post covers why we chose Cilium, what actually changed, and the five issues that surfaced after the switch.

Why Cilium

It started as a notice, not a choice. Maintenance of the community-managed Ingress-NGINX Controller ended — meaning no more security patches or bug fixes. Ingress is the point every bit of external traffic passes through, so "let's wait and see" isn't an option for that component.

As we evaluated alternatives, our reasoning came together like this:

• Kubernetes officially recommends the Gateway API as the successor to Ingress. If we're switching anyway, moving to the next generation of the ingress resource makes sense.

• Among Gateway API implementations, the mature options are essentially Cilium and Istio.

• Choosing Cilium means not just swapping the ingress controller but also gaining the performance benefits of an eBPF-based CNI.

• Cilium adds no sidecars, so it's resource-efficient. In an environment that's mostly GPU nodes, per-node sidecar cost is not negligible.

So a review that began as "replace the ingress controller" concluded as "replace the entire networking stack." That expansion is exactly what made this work hard.

What actually changes

As we prepared the customer advisory, the scope turned out to be wider than expected.

Two things mattered most here.

One, kube-proxy goes away. Because Cilium handles service load balancing directly with eBPF, kube-proxy becomes an unnecessary component and we removed it. This single decision would later create two of the issues.

Two, the external exposure path splits in two. Previously a single ingress was shared by both the platform and user workloads. After the switch, we separated an Ingress for the platform's main services and a Gateway for user workloads. This kept workload traffic from affecting platform access — but it required three VIPs. And this domain rearrangement planted the seed of the SSO issue.

As a prerequisite, the cluster had to be reinstalled: swapping the CNI isn't the kind of thing you hot-swap on a running cluster without downtime.

The migration — one weekend

We coordinated backup approach, upgrade order, and an hour-by-hour plan with the customer and SI a month ahead. Before the actual work, we validated the following on a separate cluster first:

• Storage backup/restore testing (at real volume scale)

• Writing and verifying backup & restore scripts for DB, Project, ImageHub, and Workload individually

• The new-version migration and a functional check

Work began Saturday morning with file intake, then proceeded over two days: cluster reinstall → data restore → availability testing. Availability testing ran a scenario of removing the network interface and rebooting the OS; nothing failed.

Backup/restore was problem-free except for taking longer than expected. The time we'd spent on pre-validation was the clearest example of reducing risk on the actual day. Because we didn't have to worry about whether restore would work, we could focus on the other problems that came up that day.

The trouble was that those "other problems" kept arriving after the work was done.

Five issues that surfaced

1) TLS handshake failure — move a domain, move its SNI

This was the issue that consumed the most time and energy.

This customer's setup had users connecting to a.example.com, with a commercial SSO gateway (reverse-proxy style) in front handling TLS termination and authentication. DNS was configured so that a.example.com and b.example.com pointed at the same IP.

In this migration we decided to use b.example.com as a Workspace-dedicated domain, so we separated that domain's IP. Right after install, connecting through SSO failed.

Finding the cause took a day. Connecting directly from the SSO host with curl and openssl worked fine. But going through the SSO application produced a TLS error. With the network healthy but only auth failing, we even suspected a TLS-version issue at one point.

The actual cause was simple. The SSO gateway was sending b.example.com in the SNI field when connecting to the backend. Before the switch the two domains shared an IP, so there was no problem — and so no one was conscious of that setting. Once the domains were separated, the SNI and the actual ingress host diverged, and the TLS handshake broke right there.

Setting the SNI to a.example.com fixed it immediately. Applying that fix required an emergency call in the small hours of Sunday, and the harder part was that almost no one on the customer side knew the history of that setting precisely.

Lesson: If you classify an ingress migration as a "networking change," you'll miss issues like this. Changes that affect domains, certificates, or SNI should be treated as identity/authentication changes, with their own checklist and rollback scenario. And the change history of such settings must live in documentation, not one person's memory, so a successor can respond.

2) Cutting off at exactly five minutes — there are three timeouts

A week after the switch, we got a report: "A service that used to be fine now cuts off at exactly the five-minute mark with a 504 Gateway Timeout." The target was an LLM serving workload, and the user-side application timeouts were all set to 20 minutes or more.

In the NGINX days, this was a two-line annotation:

proxy-read-timeout: 3600
proxy-send-timeout: 3600

In Cilium, getting the same result means raising all three of these:

yaml

envoy:
  idleTimeoutDurationSeconds: 3600        # ① backend L7 proxy route idle (default 60)
  streamIdleTimeoutDurationSeconds: 3600  # ② backend L7 proxy stream idle (default 300)
extraConfig:
  proxy-stream-idle-timeout-seconds: "3600"  # ③ gateway stream idle (default 300)

Why three? Because workload HTTP traffic passes through the gateway Envoy and then makes one more hop through the backend node's L7 visibility proxy. Raise only the gateway (③) and it still cuts at the backend proxy's default. On top of that, ③ isn't honored via envoy.streamIdleTimeoutDurationSeconds — it must go into extraConfig (a known Cilium bug).

The value actually applied at install time was this:

yaml

# incorrect config
envoy:
  idleTimeoutDurationSeconds: 3600
  streamIdleTimeoutDurationSeconds: 3600
  extraConfig:                            # ← nested under envoy
    proxy-stream-idle-timeout-seconds: "3600"

extraConfig was nested one level too deep. Because of a two-space indent, one of the three values wasn't applied, and so it cut off at exactly 300 seconds (five minutes). The config we handed over was correct and the understanding was correct — but the step to verify the applied result was missing.

The fix was editing the cilium-config ConfigMap and restarting the cilium operator. Workloads aren't restarted and existing connections aren't dropped. Because it's a cluster-wide setting, there was no need to reconfigure per workload.

Incidentally, the question the customer asked alongside was actually more important: "With the method you gave us for the old NGINX environment, we can't check the current values." Not just how to change settings but how to read them had entirely changed — and that wasn't in the handover doc.

Lesson: When you replace a proxy, migrate not just the config values but the operational procedures for reading and changing config. And "config was handed over" and "config was applied" are different events. Post-apply measurement (how many seconds it actually cuts at) has to be a verification step.

3) NodePort fails only from outside — Cilium needs a declaration

This is the most important issue in this post. The cause is clear, and it shows the essence of a Cilium migration.

This cluster had an internal integration system communicating directly with a platform-internal service via NodePort. A few days after the switch, we got a report that the integration wasn't working. The customer's guess was "the target service must be down."

The symptom was odd:

A tcpdump showed this:

<external system> > 10.x.x.10.31441: Flags [S]     ← SYN reaches the node (path is fine)
10.x.x.10.31441 > <external system>: Flags [R.]    ← RST in 20µs

An RST in 20µs is a refusal sent directly by the kernel — a "there's no listener on that port" response. Yet the Service, the Endpoint, and the Pod were all healthy. A contradiction: the service is fine but the kernel says the port doesn't exist.

Opening Cilium's load-balancing table gave the answer:

# cilium bpf lb list | grep 31441
10.x.x.11:31441/TCP → <pod IP>:<port>   ← only the node IP is registered
→ no 10.x.x.10:31441 entry              ← the decisive evidence

10.x.x.10 was an HA VIP raised separately by keepalived. The node's primary IP is 10.x.x.11. In other words, this VIP wasn't registered as a NodePort frontend in Cilium's ledger. Connecting to the node IP (10.x.x.11) did work.

Two questions remain here.

Why did it work from inside? Cilium translates internal traffic to the pod IP at the connect() syscall stage (socket-LB). The packet carrying the VIP is never even created, so it works regardless of registration. Only external traffic goes through exact IP:PORT matching at the NIC — and with the VIP unregistered, matching fails, it passes to the kernel, and gets an RST.

Why did it work under kube-proxy? This is the crux. The iptables rules kube-proxy creates match NodePort with --dst-type LOCAL, which implicitly includes all of the node's local IPs — and the VIP keepalived raised on the interface is automatically included too. Cilium, by contrast, only accepts declared IPs as frontends.

To summarize:

The keepalived VIP was an IP created outside Kubernetes and had never been declared to Cilium. This wasn't a failure — it was a configuration gap. And this gap couldn't even exist under kube-proxy, because it was being covered implicitly.

The fix was to declare that IP explicitly on the Service:

kubectl -n <namespace> patch svc <service> --type merge -p '{
  "spec": {
    "externalIPs": ["10.x.x.10"],
    "ports": [
      {"name": "svc",     "port": <service port>, "targetPort": <target port>, "nodePort": 31441, "protocol": "TCP"},
      {"name": "svc-ext", "port": 31441,          "targetPort": <target port>, "protocol": "TCP"}
    ]
  }
}'

externalIPs is the key — this declaration is what makes Cilium register the VIP as a NodePort frontend. Opening port 31441 separately keeps the existing address and port so the integration system's config doesn't have to change, and since --type merge replaces the whole ports array, the existing port definitions have to be included too.

Lesson: To expose a NodePort externally under Cilium, traffic arriving on any address other than the node IP needs a separate declaration. Removing kube-proxy means everything kube-proxy handled implicitly must now be declared explicitly. If there's NodePort traffic arriving on IPs created outside the cluster (keepalived VIPs, secondary IPs on a bonded interface, etc.), all of it is subject to pre-migration investigation. We added a checklist item: "Are there IPs on this cluster that Kubernetes doesn't know about?"

4) A vanished process — monitoring doesn't know about the CNI

Every server in the customer's cluster had a host-based security monitoring tool installed. And that tool had flanneld and kube-proxy registered as Kubernetes-network watch processes.

After the switch, the monitoring system started alerting:

Watch process DOWN (/usr/local/bin/kube-proxy --config=...)
Watch process DOWN (/opt/bin/flanneld --ip-masq --kube-subnet-mgr)

On every node, continuously.

A natural result: flannel is no longer used, and kube-proxy was removed as an unnecessary component. But from the customer's perspective, this was "security monitoring reporting failures on every node." The watch-target processes changed along with the move to Cilium, but updating them wasn't in the migration work items.

The response was to update the watch-process list — removing flanneld and kube-proxy and adding cilium-agent. One thing to note: Cilium doesn't run as a host system daemon the way flannel does. It runs as DaemonSet pods, so whether it even belongs in a host-process watch list had to be reconsidered too.

Lesson: The blast radius of replacing an infrastructure component includes the outside systems that were watching that component. Monitoring target lists, firewall policies, asset inventories — things we didn't build but that become wrong because of us. When you replace a component, the lists that reference it must be updated within the same job.

5) Cleaned up later — L2 announcements and firewall

Less dramatic than the four above, but items cleaned up over the weeks after the switch.

L2 announcement lease holder. When Cilium advertises a LoadBalancer IP over L2, it elects the advertising node via a lease. There was a case where the UI wasn't reachable from inside the cluster; it turned out a node that didn't hold the ingress IP range had been elected lease holder. Traffic reached that node but went no further.

kubectl get lease cilium-l2announce-kube-system-cilium-ingress -n kube-system \
  -o jsonpath='{.spec.holderIdentity}'

This one command shows "which node are we currently exposed through." We resolved it by excluding that node from the LB node list. It's a kind of check that didn't exist in the NGINX ingress days.

Master↔worker firewall ports. We cleaned up the ports Cilium requires after the fact. For air-gapped customers this is a mandatory pre-install item, but we had no document for it initially.

8472/UDP : VXLAN
4240/TCP : cilium-health
4244/TCP : Hubble
9962/TCP : cilium-agent metric
9964/TCP : cilium-envoy
9965/TCP : Hubble metric

The one thing running through all five

Looking back, the issues all had the same shape.

Four of the five were not "a setting that didn't get migrated" but "a setting we didn't even know existed." No one needed to look at the SNI because the two domains shared an IP. The VIP didn't need declaring because kube-proxy covered it automatically. No one wrote down as a work item that the watch-process list must change when the CNI changes. No one expected two annotation lines to scatter into three places.

Replacing a platform means that assumptions that held implicitly all start demanding explicit declaration at once. And by definition, those assumptions aren't in the documentation — if they were, they wouldn't have been implicit.

So writing a migration checklist as "did we migrate every existing setting?" gets you a checklist that's only half right. The other half is "what was the previous implementation doing on our behalf?" The latter has no setting to migrate at all, so no matter how meticulously you diff configs, it won't be caught.

What we fixed into the checklist

Items we baked into the install process as a result of this retrospective:

• Investigate IPs Kubernetes doesn't know about — a full sweep of IPs created outside the cluster (keepalived VIPs, secondary IPs) that receive service traffic. On kube-proxy removal, all of them are subject to explicit declaration.

• Split domain/certificate/SNI impact into a separate checklist — if an ingress change affects domain layout, fix pre-checking the SSO/proxy-layer SNI settings as an item.

• Include the three timeout values in the pre-install customer prerequisites — pin defaults at 300s, and negotiate longer values at install time if needed.

• Verify by measurement after applying — handover and application are separate. For timeouts, measure how many seconds it actually cuts at.

• Hand over operational procedures — document not just the config values but how to read and change them.

• Update lists that reference a component at the same time — include artifacts that reference the replaced component (watch-process lists, firewall port lists) in the migration work items.

Why this keeps happening — a personal thought

However tight you make the checklist, I think similar things will happen again. When I think about why, it's almost inevitable.

Every item we added to the checklist this time was a sentence you can only write after getting burned once. "Investigate what kube-proxy handled implicitly" is something you can only write once you know what kube-proxy was handling implicitly. And you usually learn that only after it's gone. The next migration will have a different implicit assumption, and it won't be in today's checklist.

The other thing I feel: the small things that happen during an install — the ones you don't record — all eventually disappear with time. That's exactly why I wrote each of these issues up as a troubleshooting doc. If you don't record them, you repeat the same trial and error next time without even knowing it's a repeat.

So I don't expect "this will never happen again." Instead, if an issue that took a day to diagnose this time takes half a day next time, I consider that a good enough improvement.

Closing

Installing and operating on-premise Kubernetes-based AI infrastructure across many customers, I feel it every time: the parts that break outside the plan create the real learning curve, more than the parts that go to plan. The Cilium migration was a technical success, and we genuinely gained on both performance and structure. The bill just arrived over the following ten days rather than on migration day.

Going forward, the AIPub engineering blog will keep telling stories centered on things we actually experienced.

This post is a reconstruction of an actual customer migration, focused on technical lessons, with customer-identifying information (company names, domains, IPs, internal config values) removed or replaced with example values.