Skip to main content

Troubleshooting sandbox network timeouts on a single-server install

Written by Aron Day

An API pod restarting with connection timeouts, or workflow steps failing to reach a service that is up, can point to sandbox networking. This guide covers one possible cause and a workaround.

Check the symptoms

Run these commands on the k3s server:

sudo k3s kubectl -n 3b get pods
sudo k3s kubectl -n 3b get endpoints
sudo k3s kubectl -n 3b logs deploy/api -c api --tail=100

Look for repeated connection timeouts while the destination service has a ready pod and endpoint. For worker failures, use deploy/worker -c worker in the log command. If a container is restarting, add --previous to read its last run. A timeout alone does not confirm this cause.

Why this happens

The API and worker use sandboxes with their own network settings. They mark outbound TCP traffic so a local proxy can handle it. If Linux reverse-path filtering also uses that mark, it can drop the traffic before it reaches the proxy. The connection then times out even though the destination service is healthy.

We’ve seen Tailscale change host network settings in a way that can trigger this issue. Other network software may do the same. Check the sandbox setting directly to see whether this cause applies to your server.

Check the sandbox setting

Find the affected sandboxd process. The API uses sandboxd run --id api, and the worker uses sandboxd serve:

sudo ps -eo pid,comm,args | grep -E '[s]andboxd (run --id api|serve)'

Use the PID on the line where comm is sandboxd, rather than its tini parent. Replace 12345 below with that PID:

sudo nsenter -t 12345 --user --net -- sysctl \
  net.ipv4.conf.all.rp_filter \
  net.ipv4.conf.vsl0.rp_filter \
  net.ipv4.conf.all.src_valid_mark

On the affected server, the worker sandbox had both rp_filter values at 2 and all.src_valid_mark at 1. If your values differ, send the pod status, logs, and settings to Tines support before applying this workaround.

Workaround

For matching settings, set src_valid_mark to 0 in the affected process’s network namespace. This restored connectivity on the affected worker:

sudo nsenter -t 12345 --user --net -- sysctl -w net.ipv4.conf.all.src_valid_mark=0

Check whether the API becomes ready and its timeout stops:

sudo k3s kubectl -n 3b get pods
sudo k3s kubectl -n 3b logs deploy/api -c api --tail=100

For a worker, check deploy/worker -c worker and retry the affected workflow step. The namespace change is lost when its pod is recreated.

To set the value for new sandboxes, save it on the k3s host:

sudo tee /etc/sysctl.d/99-3b-sandbox-network.conf >/dev/null <<'EOF'
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.default.src_valid_mark = 0
EOF
sudo sysctl -p /etc/sysctl.d/99-3b-sandbox-network.conf

The file survives a reboot, but another service can change the live value afterward. Check sudo sysctl net.ipv4.conf.all.src_valid_mark after network services start. If it is 1, reapply the file before recreating the affected pods. Confirm you can still reach the server after changing its network setting, especially if you connect through a VPN. Contact Tines support if the timeout persists.

Did this answer your question?