[Babel-users] Multiple internet gateways in the mesh

[Mark Shuttleworth]

On 09/11/2021 12:18, Juliusz Chroboczek wrote:
> In that case, you need a dedicated router to handle the uplink.  In the
> following, "border router" is the router that handles the uplink, and
> "access router" are the other routers at the local site.
>
> On the border router, say
>
>    out ip 0.0.0.0/0 le 0 metric 16384
>
> The border router will still use the local uplink, but it will announce
> a route with a very high metric to its neighbours, which will cause the
> access routers to prefer a different route if one is available.
>
> I don't see a good way to achieve that if the border router is also the
> access router.  Let me know if that's one of your requirements, and I'll
> see if I can work out a solution.

Very much appreciate the help! This personal project was designed to 
teach me a little about routing and it has definitely served that 
purpose well :)  But perhaps learning a little about routing isn't the 
same as being very good at routing!

To give a little more insight, at each location I have two "access 
routers" with VRRP for redundancy. So there are a total of six routers, 
two at each site, and they are in a mesh because every router has direct 
independent fiber to every other one. That's why I thought Babel would 
be the right tool, I think of these routers as working in a mesh. 
Nevertheless, for internet connectivity, they work in pairs.

Unfortunately, because I don't control the ISP routers, I don't think I 
have a border router by this definition. I could go and get some NUCs 
and set those up as border routers. Or I think I could spin up a VM on 
each of the access routers, and have that be the border router. But this 
mesh is in a reasonably remote location for me most of the year, so I'm 
not sure I want connectivity dependent on that many experiments in series.

I used to use a central database to coordinate where the "main route 
out" was installed, but that was a little vulnerable to issues "in the 
center". I was hoping to have a more dynamic approach, with Babel 
installing routes for all the paths out, and route metrics/priority 
determining which one is actually used for a given site.

My current best guess is that I should be smarter about installing a 
default route out of a location only if its the best way out, by pinging 
"the internet" and then pinging the better ways out "from the internet". 
If a site can get out, but it can't see the other better sites "from the 
internet", then its the best way out. What a hack :)

As a final thought, I think what I want is effectively 
"install-duplicate-if-lowest-metric". In other words, imagine I had a 
local default route out with a metric of 128 and a proto of 250, and 
babel config like this:

redistribute 0.0.0.0/0 eq 0 proto 250 metric 128
redistribute 0.0.0.0/0 eq 0 proto 42 allow
in 0.0.0.0/0 install-lowest-metric

So this would say "go ahead and redistribute the local (proto 250) route 
with a metric 128 if it is installed, redistribute any other default 
routes that come from the mesh just incrementing metrics as usual, and 
allow a duplicate default route to be installed if it is the lowest 
metric option". So the "best" (lowest metric) route to a destination 
could be installed even if it was a duplicate.

The other thing I can imagine would be integrating something like 
babel-pinger, so that if a node in the mesh believed it had the best 
default route it could add it to the node via babel, meaning that it 
would only be added if it were better than the other options coming from 
the mesh.

Waving hands furiously :)

For now I'm going to play with a smarter approach to determining, node 
by node, if it has the best way out of mesh.

Mark