[Babel-users] Route selection using smoothed metrics

[Dave Taht]

> Babel is designed to react very fast to topology changes -- after
> a route is lost, Babel is supposed to switch to an alternate route after
> a time that, in the absence of packet lost, is at most 20ms per hop to
> the source.

I am mostly writing this message as a heads up for those "doing
science" and expecting similar results on babel between openwrt, the
mainline linux kernel, and cerowrt, over time.

1) In the real world, last year I observed delays under load well in
excess of 30 seconds over wifi on some drivers. Among other problems,
several drivers "featured" infinite retry - and qdisc queue depths
were set to defaults of 1000 packets. The two offending drivers I know
of have been fixed (ath9k aug of last year, bcm a few months ago)

A change to openwrt was committed last year to reduce wifi txqueuelen
to 30 packets, this has helped somewhat.

2) Multicast on wifi can be delayed by 200ms by power save. Multicast
is sent via the CAB queue (content after beacon) on DTIM intervals.
(CAB is also referred to colloquially as "crap after beacon"). In many
situations distributing some data over unicast on wifi in a routing
protocol will help reduce latency as the rate differences between
multicast and wifi are enormous.

3) The CS6 marking of babel makes sense on ethernet. On linux wifi (I
have no data on others) it tosses babel into the qdisc VO queue, which
is ok on a current default setup as the hw and qdisc VO queues are not
presently treated specially.

The CS6 change to babel in the fall also helped jump the qdisc queues
using the pfifo_fast qdisc.

Now I wouldn't be mentioning all this if some changes in openwrt and
more changes in cerowrt weren't in the pipeline also.

1) Felix Feitkau just committed (r32589) a change to openwrt that
makes the ath9k hardware queues changeable from userspace.

2) fq_codel is enabled on all wifi queues in cerowrt. This is
definately the "wrong thing" [1] - but it is on to gather some data on
what the right thing might ultimately be, and to get a feel for how
fq_codel reacts to the rapid bandwidth changes in wifi.

I will be changing the hardware wifi VO queue in cerowrt 3.3.8-10 to
be sane (2 packets rather than 128). I would kind of expect that
change to become an ath9k default in the Linux kernel and openwrt,
frankly.

Reducing the hw BE queue on one end to 4 (from the default 128) in one
(fairly long distance, mcs rate jumping between 3 and 7) test I ran
reduced average latency under load from 138ms to 20ms, and fq_codel
did engage with both drops and ecn marking. the same test also reduced
throughput by about 40% (from >10Mbit to ~6.5Mbit).

I plan to repeat these tests more fully on both ends of the connection
in the upcoming cerowrt release as I'm hoping that multi-stream
utilization will be closer to the underlying bandwidth.

I do not presently know what I will set the other hw queues to by
default, but < 32 is certain.

3) Cerowrt will be moving diffserv CS3-CS7 and a variety of other
markings (IMM, AF*) out of the hw/qdisc VO queues and into the VI
queues, leaving EF only for the VO queue,
via a patch to the mac80211 layer.

It would be my hope that the bulk of these changes would result in
much faster babel route propagation as well, but: my primary intent is
to gain lower latency for sparser forms of streams (dhcp, dns, ra,
etc) over wifi, observe fq_codel's  and to exercise the VI queue.

Lastly:

with *your permission* I'd like to be marking babel packets in cerowrt
as CS6 + ECN capable. What to do with ECT(0) set by fq_codel in a
routing daemon is a good question. At this time I merely wish to track
how often - or even IF - fq_codel will end up marking babel packets at
all, in cerowrt.  Distinguishing between load based packet loss and
interference/distance mobility  based packet loss seems like a useful
idea, but I have no idea what to do with it. I merely wish to observe
fq_codel's behavior under load.

...

[1] http://www.bufferbloat.net/projects/cerowrt/wiki/Fq_Codel_on_Wireless



-- 
Dave Täht
http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out
with fq_codel!"