# IPsec Tunneler VM — Support Guide Operational reference for engineers triaging or supporting a deployed IPsec Tunneler bridge VM. Describes what's installed on the host, how the host is configured to bridge traffic between strongSwan (IPsec) and ziti-router (OpenZiti), and where to look for metrics and logs. This document describes the **VM internals** (what the `ipsecBridgeV7` SDS role installs and configures). It is the companion to `IpsecTunnelerAwsSecurity.md`, which covers the surrounding AWS networking. ## What is on the VM The tunneler runs on Ubuntu (currently 24.04 LTS via NetFoundry's marketplace AMI). The `ipsecBridgeV7` Ansible role installed by SDS lays down the following software stack: | Component | Source | Service unit | |-------------------------|---------------------------------|----------------------------------| | strongSwan + swanctl | `strongswan` + `strongswan-swanctl` apt packages | `strongswan-starter.service` | | OpenZiti router (tunneler enabled) | `ziti` binary bundle from the network-versions endpoint | `ziti-router.service` | | nftables | `nftables` apt package | `nftables.service` | | VTI interface bring-up | Role-supplied unit | `nf-vti.service` | | dnsmasq | `dnsmasq` apt package | `dnsmasq.service` | | ipsec_exporter | `LarsTi/ipsec_exporter` built from source on-host | `ipsec-exporter.service` | | Filebeat + Metricbeat | Installed by the `monitoringV7` role (separate, runs after `ipsecBridgeV7`) | `filebeat.service`, `metricbeat.service` | The bridge runs a **private, tunneler-enabled Edge Router** — created by core-management as a real Edge Router resource on the network (named `-router`), enrolled with a router enrollment JWT, and running with no edge/link listeners reachable from outside (the AWS SG admits only IKE/NAT-T from the customer peer plus operator SSH). The router's embedded tunneler runs in **tproxy** mode and carries both traffic directions. dnsmasq is part of the ziti-router egress lockdown (see [its own section](#ziti-router-egress-lockdown) below). systemd-resolved owns top-level DNS resolution on the box; dnsmasq only handles `*.netfoundry.io` lookups so its `nftset` directive can populate the lockdown's allowlist. Every component listed above is installed and managed exclusively by Ansible. There is no manual configuration on the box. Re-running the `ipsecBridgeV7` role over an existing tunneler is safe — it is idempotent and is also the documented recovery path ("replay the playbook on an existing tunneler"). ## Network configuration The bridge terminates both data planes on the same host: | Interface | Type | Purpose | |-----------|------|---------| | `eth0` (or the host's `default_ipv4` device) | physical ENI | Public IP-bound interface. Receives encrypted IPsec traffic from the customer peer and sends encrypted traffic back. Also carries the router's fabric/controller connections. | | `vti0` | kernel VTI (Virtual Tunnel Interface) | The route-based IPsec "decrypt side". Packets that exit `vti0` are post-encryption (egress) or post-decryption (ingress) from the customer's perspective. | There is no tun device: the router's tproxy-mode tunneler intercepts traffic with iptables TPROXY rules and terminates it in-process, so both directions are local deliveries rather than forwards. ### Why VTI and not XFRM interfaces The role uses VTI because the marketplace AMI's kernel/iproute2 does not expose `ip link add ... type xfrm`. VTI is the same architectural family — route-based IPsec with traffic-selector matching via firewall marks — and is supported on every kernel the bridge runs on. If a future AMI exposes XFRM interfaces this would be a clean swap. ### `nf-vti.service` A oneshot systemd unit brings up `vti0` before strongSwan starts. The unit: - Creates `vti0` with `local ` / `remote ` and `ikey/okey = 42` (the firewall mark strongSwan stamps via `mark_in` / `mark_out` in `swanctl.conf`). - Disables XFRM policy on `vti0` itself (`net.ipv4.conf.vti0.disable_policy=1`). Without this, decrypted packets get policy-checked twice and drop. - Sets the interface MTU to 1400 (overhead room for IPsec + Ziti encapsulation). - Assigns a link-local `/32` (`169.254.42.1/32`) to `vti0`. The interface needs **some** IP address — without one, the kernel's `fib_validate_source` takes its `no_addr → last_resort` branch and drops every forwarded packet, even with `rp_filter` set to loose mode. - Brings the interface up. ### Sysctls `ipsecBridgeV7` writes `/etc/sysctl.d/02-netfoundry-ipsec.conf` AND pins the routing-critical knobs in `/etc/sysctl.conf` (the latter wins on every boot regardless of cloud-init drop-in ordering): | Key | Value | Why | |-----|-------|-----| | `net.ipv4.ip_forward` | `1` | Required — bridge is a forwarding host. | | `net.ipv4.conf.all.rp_filter` | `0` | Must be strict zero. `vti0` has no real IP, so even loose mode (`2`) drops every forwarded packet. | | `net.ipv4.conf.default.rp_filter` | `0` | Same reason — effective value is `max(all, iface)`. | | `net.ipv4.conf.vti0.disable_policy` | `1` | Avoid double XFRM policy check on packets exiting `vti0`. | | `net.ipv4.conf.vti0.rp_filter` | `2` | Loose mode is acceptable on `vti0` itself once `all`+`default` are 0 and `disable_policy=1`. | | `net.ipv4.conf.*.accept_redirects` | `0` | Hygiene. | ### `ip rule` and routing A custom IP rule and a set of routes wire customer-bound packets through `vti0`: ``` ip rule add iif vti0 lookup main pref 100 ip rule add to lookup main pref 50 ip route replace dev vti0 src ``` `bridge_internal_ip` is the bridge's own `eth0` IP. It is pinned as the `src` of the route so that **outbound** connections that the Ziti tunneler originates toward the customer (ziti → IPsec → service) leave the bridge with a source address that matches the IPsec policy's `local_ts` selector. Without the `src` pin, the kernel would pick `vti0`'s link-local `169.254.x.x` and the IPsec policy match would fail on the customer side. The pref-50 rules force customer-destined traffic into the main table ahead of the ziti tunneler's per-overlay routing table (pref 220); the pref-100 `iif vti0` rule forces packets arriving on `vti0` to consult the main table immediately. Both rule sets are reinstalled by `nf-vti.service`'s ExecStartPost on every unit start and by the networkd-dispatcher `routable.d` hook on every networkd reset (e.g. a systemd reexec during unattended-upgrades). ### nftables — `inet nf-bridge` table Loaded from `/etc/nftables.d/nf-bridge.nft`. Two chains do all the work. **`chain input` (priority `filter - 100`, policy accept):** locks down what can be delivered to the bridge VM itself from `vti0`. Anything from `vti0` other than the allow-list cases is dropped: ``` iifname "vti0" ct state established,related accept iifname "vti0" ip daddr 100.64.0.0/10 accept # Ziti intercept range (TPROXY'd to the router) iifname "vti0" limit rate 10/minute log prefix "nf-bridge-in-drop: " level info iifname "vti0" drop iifname != "lo" tcp dport 9814 drop # ipsec_exporter loopback-only ``` The conntrack allowance covers replies to connections the router originated toward the customer (hosting direction). The intercept-CIDR allowance covers customer packets destined for Ziti service intercept addresses: TPROXY delivers them locally with their original destination address intact, so they traverse INPUT, not FORWARD. **`chain forward` (priority `filter`, policy DROP):** nothing legitimately forwards through this host — both directions terminate in the local router process. The chain keeps an explicit default-drop with a rate-limited `nf-bridge-fwd-drop:` log line so any unexpected forwarding attempt is visible. `nf-bridge.nft` is included from `/etc/nftables.conf`, so the `nftables.service` reload picks it up on every boot. ## ziti-router egress lockdown A second nftables table, `inet nf-ziti-egress`, constrains every packet originated by the ziti-router process. It exists because the router's tunneler hosts services — it terminates Ziti circuits and originates fresh TCP connections to whatever destination the Ziti service config dictates. Without the lockdown the bridge VM is usable as an arbitrary egress proxy onto the public internet for anyone with a circuit on the Ziti overlay. ### What's permitted | Egress destination | Why allowed | |---------------------------------------------|----------------------------------------------| | `oifname vti0` (IPsec tunnel) | Customer-side traffic, the primary data plane. | | `ip daddr` ∈ `customer_cidrs` set | Fallback for bridges where customer CIDR isn't routed via vti0 explicitly — XFRM policy still engages. | | TCP/443 or TCP/6262 to `nf_allow_ips` set | NetFoundry-hosted control plane: Ziti controller (ctrl + edge on 443) and fabric router links. The set is populated dynamically by dnsmasq. | | Loopback UDP/TCP port 53 | DNS lookups to the local dnsmasq (required to populate `nf_allow_ips`). | | UDP sport 53 | Responses from the router's tproxy DNS resolver (bound to the primary interface address). | | `ct state established,related` | Stateful return path for flows already accepted — including replies on TPROXY'd customer connections. | Everything else from the ziti-router cgroup drops with a rate-limited `nf-ziti-drop:` kernel log line. All other processes on the VM (sshd, charon, salt-minion, metricbeat, ipsec_exporter, etc.) are unaffected. ### How the cgroup match works ``` chain output { type filter hook output priority filter; policy accept; socket cgroupv2 level 2 "system.slice/ziti-router.service" jump ziti_filter } ``` `socket cgroupv2 level N "path"` resolves the path to a cgroup **inode** at rule-load time. Every `systemctl restart` of ziti-router destroys the cgroup and creates a new one with a different inode, so the rules must be re-loaded on every service start. The unit file (`ziti-router.service`) takes care of this: ``` ExecStartPost=-+/usr/sbin/nft -f /etc/nftables.d/nf-ziti-egress.nft ``` The leading `-` makes systemd ignore the exit code so a missing nft file on first boot (before the role has dropped it) doesn't fail the unit; the `+` runs the command with full privileges because the main process runs as the unprivileged `ziti` user. ### Dynamic `nf_allow_ips` set via dnsmasq dnsmasq listens on `127.0.0.1:53` with one operationally meaningful directive: ``` nftset=/.netfoundry.io/4#inet#nf-ziti-egress#nf_allow_ips ``` Every A record dnsmasq returns for a name ending in `.netfoundry.io` is added to the nftables set with a 1h timeout. `systemd-resolved` is configured to route `~netfoundry.io` lookups specifically to `127.0.0.1` (see `/etc/systemd/resolved.conf.d/nf-bridge-allow.conf`); all other DNS continues through whatever upstream `resolved` has configured. The 1h timeout means an unused entry expires. The next time ziti-router looks up that hostname, the entry reappears. There's a small window between expiry and the next lookup where a connection could be dropped — observable as a `nf-ziti-drop:` log line followed by a successful retry. Bump the timeout in the role's `nf-ziti-egress.nft.j2` template if this proves disruptive. ### Diagnostic commands ```bash # Inspect the current allowlist (NF infrastructure IPs). sudo nft list set inet nf-ziti-egress nf_allow_ips # Per-rule counters — which rule is accepting traffic, how much. sudo nft list chain inet nf-ziti-egress ziti_filter # Live tail of drops (rate-limited to 10/min). sudo journalctl -k -g 'nf-ziti-drop' -f # Confirm DNS routing — *.netfoundry.io should hit dnsmasq, not the upstream. resolvectl status | grep -A1 'DNS Domain' ``` If ziti-router can't reach the controller after a fresh install or a hostname change, the failure mode is almost always one of: - `*.netfoundry.io` lookups not flowing through dnsmasq (resolved drop-in missing or `Domains=~netfoundry.io` typo). - dnsmasq not running (its systemd unit failed; check `systemctl status dnsmasq`). - The `socket cgroupv2 level 2 "..."` resolution failed at rule-load (the cgroup doesn't exist because ziti-router isn't running yet — usually self-heals on the next service start via the ExecStartPost). ## strongSwan configuration | File | Purpose | |------|---------| | `/etc/swanctl/swanctl.conf` | Top-level swanctl config; includes `conf.d/*.conf` and `secrets.d/*.conf`. | | `/etc/swanctl/conf.d/nf-bridge.conf` | The IKE/CHILD SA connection block for this bridge. Rendered from `swanctl.conf.j2`. | | `/etc/swanctl/secrets.d/nf-bridge.conf` | PSK secret. Mode `0600`, root-only. | | `/etc/strongswan.d/charon-logging.conf` | Stamps the bridge id into every charon log line. | | `/etc/systemd/system/strongswan-starter.service.d/10-bridge-id.conf` | `SyslogIdentifier=ipsec-bridge-` — every journald line for the daemon is searchable by tunneler id. | | `/etc/systemd/system/strongswan-starter.service.d/20-load-swanctl.conf` | `ExecStartPost=swanctl --load-all` — the Debian/Ubuntu `strongswan-starter` unit runs `/usr/sbin/ipsec start`, which loads only `/etc/ipsec.conf`. Without this drop-in, charon comes up with zero connections after a reboot. | The connection block in `swanctl.conf` is route-based via the `mark_in` / `mark_out` mechanism. swanctl stamps every encrypted packet with mark `42`; the kernel routes based on that mark via `vti0`. The CHILD SA is configured with `start_action = trap` — a trap policy is installed at load-time and a CHILD SA is auto-established the first time real traffic matches the policy. ## ziti-router The `ziti` binary bundle is downloaded from the network-versions endpoint (`/core/v2/network-versions?networkVersion=`) so the bridge runs the ziti version matching the network. Install root is `/opt/netfoundry/ziti/ziti-router` (config.yml, certs, enrollment marker); service unit is `ziti-router.service`, running as the unprivileged `ziti` user with `CAP_NET_ADMIN` + `CAP_NET_BIND_SERVICE` ambient capabilities (tproxy iptables management + the port-53 resolver). `config.yml` is rendered from the role-owned bridge-router template (`ziti-router-config.yml.j2` — the canonical zitiRouter template cannot render IP-based CSR SANs, and the bridge advertises by public IP) with the bridge profile: - `ctrl.endpoint = tls::443` - link **dialers only** — no link listener, no wss listener (private router) - edge listener advertises the bridge's public IP (CSR SANs are IP-based; the AWS SG does not admit inbound 443, so nothing connects to it externally) - tunnel binding in `tproxy` mode with the resolver on the primary interface address and `lanIf` set to the primary interface The enrollment JWT is delivered via Ansible extraVar and only persisted to disk for the duration of the `ziti router enroll` call — it is removed immediately after enrollment so a snapshot of the disk does not retain the credential. The `.is_registered` marker file records a completed enrollment; supplying a fresh JWT wipes the certs + marker and re-enrolls. ## Traffic flow ### Customer → service (ingress / intercept) ``` Customer peer ↓ ESP/UDP-encapsulated [eth0] → strongSwan/charon → (decrypt) → [vti0] ↓ ip rule iif vti0 → main table ↓ daddr ∈ 100.64.0.0/10 (service intercept address) [vti0] → iptables TPROXY (router-managed) → ziti-router tunneler (local delivery) ↓ terminates locally, dials the Ziti service Ziti fabric → service host ``` The router's tproxy tunneler intercepts customer packets addressed to Ziti service intercept IPs and dials the corresponding service over the fabric. Service intercept addresses must stay inside `100.64.0.0/10` — both the nf-bridge input chain and the swanctl `local_ts` selector are scoped to it. ### Ziti-overlay → customer (hosting) ``` Ziti overlay (a service dialer) ↓ ziti circuit over the fabric ziti-router tunneler (hosts the service) → terminates the circuit ↓ originates TCP from bridge_internal_ip ↓ ip rule pref 50 → main table → route via vti0 [vti0] → strongSwan/charon → (encrypt) → [eth0] ↓ ESP/UDP-encapsulated Customer peer ``` The router hosts services (host.v1 configs bound to the router's tunneler identity) and originates a fresh TCP connection from `bridge_internal_ip` toward the customer. Replies return via `vti0` and pass the input chain's conntrack allowance. ## Service ordering and lifecycle systemd ordering, baked into the unit `Before=` / `After=` directives: ``` network-online.target ↓ nf-vti.service (brings up vti0) ↓ strongswan-starter.service (charon comes up, swanctl --load-all from ExecStartPost) ↓ ziti-router.service (also After=network-online.target, Requires=nf-vti) ``` There is no startup dependency from any of these on the customer peer being reachable. If the customer is down at boot, charon comes up with the trap policy installed but no established SA; the SA establishes the first time real traffic matches the policy. `ipsec-exporter.service` has no ordering dependency on the others — it polls charon VICI and either gets stale state or no state until charon is running. ## Diagnostics — first places to look | Symptom | First check | |---------|-------------| | No tunnel at all | `swanctl --list-sas` (should show an IKE_SA + at least one CHILD_SA when traffic has flowed). `journalctl -u strongswan-starter -n 200`. | | Tunnel up but no traffic | `ip route` (CIDRs to `vti0`), `ip rule` (pref-50 + pref-100 rules), `nft list ruleset` (input chain loaded), `iptables -t mangle -L -n` (router's TPROXY intercept rules present when intercept services exist). | | Customer→Ziti intercepts dead, hosting fine | Check the router's tproxy rules (`iptables -t mangle -L NF-INTERCEPT -n` or current chain name) and that the service intercept address is inside `100.64.0.0/10` — outside it, the nf-bridge input chain drops before TPROXY delivery. | | Ziti circuit fails but customer side looks healthy | Likely the [egress lockdown](#ziti-router-egress-lockdown). `journalctl -k -g nf-ziti-drop` shows what destination was rejected; `sudo nft list set inet nf-ziti-egress nf_allow_ips` shows what dnsmasq has resolved. | | Controller unreachable after reboot | The cgroupv2 socket match resolves an inode at rule-load. If ziti-router wasn't running at boot when nftables loaded, the rule won't catch the unit later. Service restart re-loads via ExecStartPost — `systemctl restart ziti-router`. | | Customer-side reachability flapping | Look for "established" / "deleted" lines in `journalctl -t ipsec-bridge-`. | | PMTU black hole | The `nf-pmtu-probe` systemd timer runs periodic probes against the customer peer via `vti0` — `journalctl -u nf-pmtu-probe`. | | Source/dest check (returns black-holed at AWS) | EC2 instance attribute `sourceDestCheck` should be **false**. `AwsLaunchInstance` sets this automatically post-launch; if it's true, the launch path failed silently. | ## Metrics, monitoring, and log shipping The bridge ships metrics and logs via the Beats agents installed by the `monitoringV7` role. `ipsecBridgeV7` runs first and lays down the inputs `monitoringV7` consumes. ### `ipsec_exporter` (Prometheus exporter for charon) [`LarsTi/ipsec_exporter`](https://github.com/LarsTi/ipsec_exporter), a Go binary that talks to charon's VICI socket and exposes per-IKE/per-CHILD SA metrics at `127.0.0.1:9814/metrics`. The upstream publishes no prebuilt binaries, so the role builds it from source on-host using a pinned Go toolchain (1.25.10) fetched from `go.dev`, then removes the toolchain to keep the steady-state image small. The exporter binds `0.0.0.0:9814` (upstream hardcoded, no flag to change). nftables on the bridge drops inbound from anything but loopback, so the AWS SG is not the only thing keeping it private. Exported metrics include: - IKE SA state (established / connecting / ...) - CHILD SA state and lifetime - Bytes / packets in and out per CHILD - Rekey timers ### ziti-router metrics The router reports fabric metrics (link state, circuit counts, utilization) through the Ziti controller like every other Edge Router on the network — visible via the standard router events/metrics pipelines, keyed by the router name `-router`. Local journald output from `ziti-router.service` is shipped by filebeat (below). ### Beats — Metricbeat `monitoringV7` configures Metricbeat to scrape `127.0.0.1:9814/metrics` and forward the parsed Prometheus metrics to the platform Elasticsearch cluster. The scrape stays on loopback because of the nftables drop rule above. ### Beats — Filebeat `monitoringV7` configures Filebeat journald inputs filtered by `_SYSTEMD_UNIT` for: - `strongswan-starter.service` - `ziti-router.service` Both flows are stamped with the bridge correlation id (the `ipsec_tunneler_id` UUID issued by core-management) via two mechanisms: 1. **strongSwan journal lines** carry `SyslogIdentifier=ipsec-bridge-` (the systemd drop-in described in the strongSwan section). Search journald with `journalctl -t ipsec-bridge-`. 2. **Beats events** are stamped with `ipsec_tunneler_id` as a top-level field by `common_fields.yaml.j2` from `monitoringV7`, reading the value out of the `/etc/ansible/facts.d/ipsec_bridge.fact` file that `ipsecBridgeV7` drops: ```ini [bridge] ipsec_tunneler_id = ipsec_exporter_port = 9814 ``` ### Searching in Elasticsearch / Kibana Every metric and log event from a given bridge can be filtered by the `ipsec_tunneler_id` keyword field. The Elasticsearch index template gating this field as a `keyword` (so it's exact-match searchable rather than analysed) is owned by the platform team and managed outside this role. ## Lifecycle operations available from SDS The role exposes three named operations beyond the initial install path. These are invoked from core-management via dedicated SDS wrapper methods and run only when the corresponding Ansible tag is passed: | Operation | Tag | When | |-----------|-----|------| | Full install | `ipsec_bridge` (default) | First-time provisioning. Idempotent — safe to replay on an existing bridge. | | Restart SA | `restart-sa` | Operator action; tears down the current CHILD SA and lets the trap policy re-establish on the next packet. | | Restart services | `restart-services` | Operator action; bounces `strongswan-starter` and `ziti-router` in order (nf-vti is left alone — kernel device, not user-space). | | Uninstall | `uninstall` | Stand-alone teardown without VM destruction. Not invoked from the standard delete path (the VM is destroyed unconditionally there). | The standard playbook replay is **safe**: the enrollment guard in `install_tunneler.yml` detects an existing enrolled router (`.is_registered` + certs) and skips re-enrollment unless a fresh JWT is explicitly supplied. This makes "replay the role on a half-installed bridge" a first-class recovery action.