Understanding Linux Routing Tables¶

Whenever your host sends a packet, the kernel asks one question: which interface, and which next hop, for this destination? The answer comes from the routing table. Most "I can reach the LAN but not the internet" and "traffic leaves through the wrong NIC" problems are solved by reading that table correctly.

How it works¶

Reading ip route¶

$ ip route
default via 192.0.2.1 dev eth0
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.10
198.51.100.0/24 dev eth1 proto kernel scope link src 198.51.100.10

Each line is one route: destination prefix, then how to reach it.

• 192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.10: a connected route. The kernel added it by itself when 192.0.2.10/24 was assigned to eth0 (proto kernel). scope link means the destination is directly on the wire, so no gateway is needed; packets go out of eth0 with source address 192.0.2.10.
• default via 192.0.2.1 dev eth0: the default route (0.0.0.0/0). It matches anything no more specific route matches, and forwards it to the gateway 192.0.2.1, which must itself be reachable through a connected route.

Longest prefix wins¶

The kernel picks the most specific matching route, meaning the one with the longest prefix length. Only if several routes have the same prefix does the metric break the tie (lower wins). Add two more routes:

ip route add 203.0.113.0/24 via 198.51.100.1 dev eth1
ip route add 203.0.113.128/25 via 192.0.2.254 dev eth0 metric 50

$ ip route show
default via 192.0.2.1 dev eth0
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.10
198.51.100.0/24 dev eth1 proto kernel scope link src 198.51.100.10
203.0.113.0/24 via 198.51.100.1 dev eth1
203.0.113.128/25 via 192.0.2.254 dev eth0 metric 50

Ask the kernel what it would do for a given destination with ip route get:

$ ip route get 203.0.113.5
203.0.113.5 via 198.51.100.1 dev eth1 src 198.51.100.10 uid 0

$ ip route get 203.0.113.200
203.0.113.200 via 192.0.2.254 dev eth0 src 192.0.2.10 uid 0

$ ip route get 8.8.8.8
8.8.8.8 via 192.0.2.1 dev eth0 src 192.0.2.10 uid 0

$ ip route get 192.0.2.77
192.0.2.77 dev eth0 src 192.0.2.10 uid 0

Why these results:

• 203.0.113.5 is inside the /24 but not the /25, so the /24 route wins: gateway 198.51.100.1 out of eth1.
• 203.0.113.200 is in both; the /25 is longer, so it wins even though its metric (50) is higher than the default (0). Prefix length is compared first, metric only breaks ties.
• 8.8.8.8 matches only the default route.
• 192.0.2.77 matches the connected route: no via, because it is on-link. The kernel will ARP for the host directly.

ip route get is the most useful single command in this article: it shows the decision, the chosen source address and the outgoing interface, instead of making you work it out by eye.

What "unreachable" looks like¶

Without a default route and with no matching prefix:

$ ip route get 8.8.8.8
RTNETLINK answers: Network is unreachable

The packet is dropped by the sending host and never reaches the network. A gateway that is not inside any connected subnet is rejected when you try to add the route:

$ ip route add 10.9.9.0/24 via 172.16.0.1
Error: Nexthop has invalid gateway.

The next hop has to be on a directly connected network, because the kernel must ARP for it.

Persistence and the other tables¶

ip route add changes only the running kernel; it is lost at reboot. Make routes permanent in the network configuration tool of your distribution (see below).

The table shown by ip route is the main table. Linux can have several tables selected by policy rules (ip rule show), used for multi-homed servers, VPNs and source-based routing. If ip route get disagrees with what you see in ip route, check ip rule show and ip route show table all.

For IPv6 the equivalent is ip -6 route. Forwarding between interfaces (acting as a router) is a separate switch, net.ipv4.ip_forward; a host with several interfaces does not route packets for others unless it is set to 1.

Ubuntu/Debian vs. RHEL-family differences¶

The ip route output and semantics are identical everywhere. Only the persistence layer differs. Prefer netplan try, which rolls back after a timeout if you lose connectivity.

Practical exercise¶

Reproduce the captures above without touching your real network. This needs no root: the interfaces live in a private network namespace and disappear with it.

unshare -rn bash                       # new shell in a private network namespace
ip link set lo up
ip link add eth0 type dummy; ip link add eth1 type dummy
ip addr add 192.0.2.10/24 dev eth0; ip addr add 198.51.100.10/24 dev eth1
ip link set eth0 up; ip link set eth1 up
ip route add default via 192.0.2.1 dev eth0
ip route add 203.0.113.0/24 via 198.51.100.1 dev eth1
ip route add 203.0.113.128/25 via 192.0.2.254 dev eth0 metric 50
ip route show
ip route get 203.0.113.5; ip route get 203.0.113.200
ip route del default
ip route get 8.8.8.8

Cleanup: exit the shell. The namespace, interfaces and routes are destroyed with it. If unshare is refused (unprivileged user namespaces disabled), do the same as root on a spare VM with sudo ip netns add lab and sudo ip netns exec lab bash, then sudo ip netns del lab.

How to verify you did it correctly¶

• ip route show lists five routes, including both connected /24 routes with proto kernel.
• ip route get 203.0.113.5 goes via 198.51.100.1 dev eth1; ip route get 203.0.113.200 goes via 192.0.2.254 dev eth0.
• After ip route del default, ip route get 8.8.8.8 prints RTNETLINK answers: Network is unreachable.

Common mistakes¶

• Adding a route with a gateway that is not on a connected network. The kernel replies Nexthop has invalid gateway. Fix the address/subnet first, then the route.
• Two default routes with the same metric on different NICs. Which one is used is not something you should rely on. Give them different metrics so the preference is explicit.
• Expecting ip route add to survive a reboot.
• Reading only ip route when policy routing is in use. Check ip rule show.
• Debugging with ping alone. A reply path can differ from the send path (asymmetric routing); ip route get plus tcpdump on both interfaces shows what actually happens.

⚠ Dangerous commands¶

• ip route del default or ip route replace default ... on a remote host over SSH can cut your own session if the new route does not work. Test the new path first with ip route get, keep a console/out-of-band access available, and prefer netplan try or a scheduled revert (( sleep 60; ip route replace default via <old-gw> ) &).
• ip route flush all (or flush table main) wipes every route including connected ones; the host is cut off until networking is restarted.
• Enabling net.ipv4.ip_forward=1 turns a server into a router. Do it deliberately and pair it with firewall rules, or hosts on one side will be able to reach the other side freely.

Related articles¶

• TCP/IP Networking and the ip Command Suite
• DNS Resolution and Troubleshooting: when the route is fine but names do not resolve
• Linux Firewall with nftables and iptables: filtering what you forward

Review questions¶

1. Two routes match a destination: 10.0.0.0/8 with metric 0 and 10.1.0.0/16 with metric 100. Which does the kernel use, and why?
2. Why does a connected route have no via?
3. What does Network is unreachable tell you, compared with a ping timeout?
4. How do you check which route and source address the kernel will use for a destination?
5. Why does a route added with ip route add disappear after a reboot, and how do you make it stay?