VXLAN fabric: SONiC control plane, VPP data plane

The problem

Tenant VMs run on separate dedicated servers. The servers share no Layer 2 segment and have one public address each. Each tenant needs a private network that spans the servers, a gateway on every server, isolation from other tenants with controlled route leaking where it is wanted, and filtering on the traffic that is leaked. Forwarding happens in user space with VPP, not in the Linux kernel.

My role

I built and tested the two-server fabric and wrote an as-built guide that someone without a networking-lab background can follow, with the expected output and the fix for every step. I then extended the guide to a third server, marking every step that has not run for real.

How it works

• Each server runs two containers. sonicN is a real SONiC-VS image whose FRR runs iBGP to the other servers. vppN runs VPP only, with no routing protocol.
• VPP takes over the physical NIC through af-packet and mirrors it into Linux with the LCP plugin. The server’s single public address lives on that mirror, so BGP in SONiC and VXLAN in VPP share one address.
• For each tenant VRF, a Linux bridge connects the tenant VM to VPP through a veth pair. VPP holds a bridge domain, a static VXLAN tunnel to each peer and an anycast gateway: the same gateway address on every server, so a VM never needs to know which server it runs on.
• Route leaking between the two VRFs is configured in VPP, and ACLs on the gateway interfaces decide which leaked traffic passes.

Decisions and trade-offs

Keep BGP out of the data plane

BGP runs only in SONiC. VPP has no BGP session and no AS number. The EVPN address family is active, but no routes are expected (VXLAN lives in VPP, not in the kernel). As a result, a BGP session going down does not stop VM traffic between those servers, and the data plane is configured and debugged directly with vppctl.

Trade-off: there is no dynamic VTEP or MAC learning, and a static full mesh grows quadratically. Three nodes need 3 BGP sessions and one tunnel pair per VRF per server pair. The guide recommends a route-reflector or spine design before a fifth or sixth node.

One public address per server, shared through LCP

The servers have one address each, and it cannot be duplicated between SONiC and VPP. Putting it on the LCP mirror lets VPP source VXLAN from it automatically while BGP keeps using it.

Trade-off: the server’s own connectivity now depends on the VPP container. Removing the container removes the mirror and the address with it. The rollback procedure moves the address back to the physical NIC first.

ACL plugin, not NPOL, for filtering

VPP’s NPOL plugin has the richer policy CLI, so it was tested first. A deny policy on a real interface left ping running at 0% loss: NPOL’s decisions were never consulted on the forwarding path of this build. The ACL plugin is on that path, so it carries the policy: from VRF 1 to VRF 2, ICMP and HTTP are allowed, SSH is blocked and everything else (including HTTPS) is denied by default. Enforcement was verified with ACL hit counters, not just pings.

Gotcha written into the guide: never assume a new ACL lands at index 0. Use the index VPP prints.

Build the VPP image once

One image replaced per-server installs. Two snags came up. VPP’s package post-install runs sysctl, which fails inside an unprivileged docker build, so the image uses the package’s documented VPP_INSTALL_SKIP_SYSCTL switch (the fabric configures no hugepages, so nothing depends on those settings). The LCP plugin is not loaded by the default startup configuration, so its enable stanza is baked into the image.

Incident: one interface name, one lost SSH session

The first version of the guide hardcoded server 1’s NIC name. When the same block ran on server 2, which has a different NIC name:

1. Creating the VPP host interface for a NIC that does not exist still returned success, creating a phantom interface.
2. LCP created a Linux mirror of the phantom.
3. Removing the address from the nonexistent NIC failed silently, so the real address stayed where it was.
4. Adding a default route through the mirror, via server 1’s gateway, created a second default route at the same metric. Once the kernel sent traffic through it, the SSH session to a live production server dropped.

I recovered through the out-of-band console by removing the bad route and checking that the original route was intact.

What changed: every command now uses $IP, $INTERFACE and $GATEWAY, set once per server from ip a and ip r on that host. They are never copied from another server’s row or terminal history. The rollback section also records two traps: deleting a bridge leaves its veth members behind, and removing VPP before moving the address back cuts the server off.

Verification

Run from each VM’s console with ping -c 3 on the two-server build:

All 13 passed. The ACL policy was then validated with a scenario test and hit counters on the same build.