[Babel-users] Babel with VPP

[Pim van Pelt]

Hoi colleagues,

Juliusz, feel free to use my site as a backref for your research group. 
I will probably write one more article, once the in flight changes have 
been merged, and I roll out Babel with VPP in production [fingers 
crossed!]. I'll update the list once I do.

Thank you very much for engaging with me! I'll collate a few answers in 
one reply.

On 11.03.24 12:02, Juliusz Chroboczek wrote:
> I find it interesting that you find Babel useful on a carrier network.
> Since I've been working under the impression that IS-IS is absolutely
> perfect (TM) in carrier networks, we have no experience whatsoever with
> that case.
Perhaps I should clarify. My network AS8298 is built using point to 
point carrier ethernet links provided by AS25091. They have physical 
(DWDM, dark fiber, etc) links, and use OSPF and LDP to signal ethernet 
pseudowires for me. If there is an inter-city link that fails, their 
OSPF will reroute, and my traffic will go via a different path.

The problem is this: AS8298's IGP typically doesn't notice this. We run 
BFD with reasonably lax timeout of 3.0s and the convergence of the 
underlying network is pretty quick. Also, if a backbone link at AS25091 
goes down, that typically means nothing for my ethernet link to their 
routers -- in other words: my IGP stays up, fully converged, but one 
link all of the sudden goes from 5ms (frankfurt-zurich) to 35ms 
(frankfurt-amsterdam-paris,geneva,zurich).

I think Babel, for me at AS8298, will address this issue, and move 
traffic away from this now-high-latency link.

On 11.03.24 12:51, Daniel Gröber wrote:
>> 1) Is there any advice you could offer for rtt cost/min/max/decay values
>> when using Bird2 ?
> The defaults should be fine honestly. While I have recommended changing
> them in the past I fear I don't (yet) fully understand the impact of that
> change on stability so best to leave them as is for now.
If I go this route, what I think I will need to know is the normally 
expected city-to-city latency (using my AS25091 provided point to point 
ethernet VPWS), and the alternate latency (when AS25091 needs to 
reroute), and force a higher cost when this is the case. I realize a 
goal ought to be minimizing the changes of the costs and topology, 
because on top of the IGP I will have a full table at 950K/200K 
prefixes, as these routers are in the DFZ. One cool thing is that VPP 
will consume a full table in about 7 seconds, including programming the FIB.

> 1) Bird's proto/babel doesn't have good policy controls right now. Ff you
> need any sort of control over your IGP announcements for TE or what have
> you things might get tricky. I do have a patch ready to begin fixing that
> but unfortunately it's in limbo until BIRD v3 shakes out or we find some
> funding/motivation to push a port to v3 forward.
Understood. Luckily, I don't do traffic engineering with OSPF currently, 
and I'm OK leaving that off for now.
> Babeld does have (most of) the knobs I think you'd need but it's just not
> suitable for 24/7 operation outside of toy networks without major rework
> (sorry Juliusz!).
I think I will only be using Bird2 at AS8298. It is a production network 
after all :)
> 2) When a prefix is no longer reachable babel will insert an unreachable
> route for it until some timeout expires. I don't recall the details off
> hand but I'm sure Juliusz will jump in here ;)
That's acceptable for VPP. It picks up unreachable (and blackhole) 
routes and programs them correctly in the FIB. Thank you for pointing it 
out though!

On 11.03.24 23:09, Juliusz Chroboczek wrote:
>> 1) Is there any advice you could offer for rtt cost/min/max/decay values when
>> using Bird2 ?
> 1. RTT-MIN
>
> In the ideal case, your network consists of a number of interconnected
> clusters.  For example, if you have routers in Berlin, Paris and Warsaw,
> then each of the cities constitutes a cluster.  Within each cluster, it
> doesn't make sense to chose routes based on RTT, since small RTT values
> tend to be noisy and cause instability.
Agreed. Within the metro, reroutes are "free of charge" latency wise.
> Rtt-min should ba a value that is more than the intra-cluster latency but
> less than the inter-cluster latency.  For example, if latency within each
> cluster is on the order of 5ms, and the inter-cluster latency is 20ms,
> then the deault value of 10ms is fine.
I think this is what I'm looking for. Once the latency from ZRH-FRA is 
established at 5ms, but link failure drives that up to 30ms, I can play 
with a rtt-min of >>5ms (to account for jitter and variance) but <30ms, 
so that the cost raises only when strictly necessary.
> Large values of rtt-min improve stability in the presence of bufferbloat.
>
>
> 2. RTT-MAX
>
> Symmetrically to rtt-min, rtt-max is the value above which links are
> considered bad.  It should be slightly larger than the largest RTT in your
> network.  Set it as small as possible in your network, since it has
> a dramatic effect on stability in the presence of bufferbloat.
>
> The default is 120ms, which is very conservative, but already has
> a big effect on improving stability in bufferbloated networks.
I think for most all pairs of router adjacencies, 120ms will rarely if 
ever be reached. To confirm though, I am free to take different values 
of rtt-min and rtt-max per interface, right? I have a router in 
California at 150ms normal latency, so here rtt-min would be 170 and 
rtt-max might want to be 250ms or something higher like that.
> 3. MAX-RTT-PENALTY (rtt cost in BIRD)
>
> This is the maximum cost penalty that will be applied to high-RTT links.
> The default (96) is rather conservative, it will cause one high-RTT link
> to be equivalent to two low-RTT links.
Perhaps you can confirm my understanding. Considering a ring of routers 
ZRH-FRA-AMS-LIL-PAR-GVA-{ZRH}, any given link here will force traffic to 
go the other direction. So say ZRH-FRA fails, the traffic that used to 
be cost 96 would now become larger than 5x96 for ZRH-GVA-PAR-LIL-AMS-FRA 
to win. So I think my max penalty cost should be 500 so that the ZRH-FRA 
link will lose out on the alternate of 480. Is that how you see it as well?

On 11.03.24 13:26, Dave Taht wrote:
> He also did a nice writeup of an inexpensive 32x100Gbit switch
> recently, running... debian.
>
> https://www.linkedin.com/pulse/debian-mellanox-100g-switch-pim-van-pelt-3pivf/
Thank you for the plug, Dave :) If you're not on Linkedin, that article 
(and others) is primarily published on:
https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html

That particular one was fun for me because I really just read their own 
docs and published my findings after playing around with the switch and 
Debian and Switchdev, but they were suitably enamored that they gave me 
a call to meet with the silicon, ethernet, and switchdev teams :-)

And if you're still reading, an update from me:
- I had a conversation with the VPP developers to accept two patches to 
VPP, one of which allows the original premise of my article (ipv4-less 
transit networks, using loopback /32 only) to work.
- The other is to enable/allow point to point links over ethernet, to 
reply to ARP (which is what I see most platforms I am familiar with 
already do). It sparked a bit of discussion, but I'm hopeful that it can 
be merged.
- I also toyed a bit more with IPv4-less OSPFv2 (not quite there yet 
with VPP), but since that's probably more a topic for bird-users, I'll 
spare you the gory details.


groet,
Pim


-- 

Pim van Pelt<pim at ipng.ch>
PBVP1-RIPE -https://ipng.ch/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/babel-users/attachments/20240315/84a4a46b/attachment.htm>