[Babel-users] Multiple internet gateways in the mesh

[Mark Shuttleworth]

On 07/11/2021 23:47, Juliusz Chroboczek wrote:
>> I have a hack which pings from each site to see if its own internet
>> connection is up. If it is, it installs a default route.
> How do the pings reach the Internet before a default route has been
> installed?


:)

I have an "isp" route table which has that default route, and a rule to 
use that table for packets which originate from the external interface 
of the internet connection:

$ ip route show table isp
default via 169.255.147.129 dev bond-net
169.255.147.128/29 dev bond-net scope link

$ ip rule show
0:    from all lookup local
32765:    from 169.255.147.134 lookup isp
32766:    from all lookup main
32767:    from all lookup default

The hack is then to ping the internet (pick a famously up site :)) 
setting the source fo the pings to that external interface IP. If 
packets return, I deem that link to be good and install a default route 
as usual, with a  metric that reflects my preference for that link.

The idea was then that Babel would install additional default routes for 
the "other" links which are up elsewhere in the mesh, and if their 
metric was lower than the metric of this link, they would "win" for this 
site.


>> Babel will not put its own proto route into the table, if there is
>> a static route already configured.
> That's not quite right.  Babeld will refuse to install a route that has the
> same destination as a route that it redistributes, but will blindly ignore
> any routes that it's not redistributing.  So the solution is to make sure
> that Babeld does not see your default route.
>
>    1. Make sure that the static route is not visible to babeld:
>
>       redistribute ip 0.0.0.0/0 proto xxx allow
>       redistribute ip :::/0 proto xxx allow
>       
>       where xxx is *not* the protocol of the static route.  (This is the
>       difference with your configuration, where you ask Babeld to
>       redistribute routes for all protocols, including the one of the
>       static route.)
>
>    2. When the route has been shown to be routable, have your reachability
>       daemon install a route tagged with the protocol that Babeld redistributes:
>
>       ip route add ... proto xxx

What's not clear to me then is how to get the local default route to 
propagate to other sites in the mesh.

Do I want to install TWO default routes with my script, one to be used 
and one to be propagated?

Here's the issue. Each site is constantly checking its own link. If that 
link is good, it needs to put a route in with a relevant metric for that 
link, *and* propagate that route to other sites. I'm not sure how that 
route can be simultaneously redistributed and ignored.

What I would like for most sites is that the daemon-installed route has 
a high metric, and the Babel-installed route has a low metric, so that 
traffic flows away from that site in the mesh, and out through the 
preferred link.

To answer Dave's question about link preference, I have two ISPs but the 
deal with one is that it is a backup and traffic should only flow to 
them if it doesn't flow to the first one.

Mark