[Cerowrt-devel] Invisibility of bufferbloat and its remedies

[d***@deepplum.com]

https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/

It's distressing how little the tech press understands the real problem.

Of course, cable companies like Charter and ATT who have mostly DOCSIS 2 gear deployed can't admit to their plant being bloat-causing.

In fact it protects their cable business against cord cutters.

And the solution is needed in the CMTS once neighbors all start becoming heavier users, because it is a shared buffering pool with no fairness or bloat protection.

Still, routers with queue management that reduce bloat would help a lot, if "buffering" is seen frequently under load.

So why isn't anyone talking about this problem after at least a decade of knowing it, and knowing how to fix it?

I blame IETF members, individually and collectively. If ietf exists for any reason other than as a boondoggle for world travel, it's for resolving issues like this one.

[Dave Taht]

Post by d***@deepplum.com
https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.

It still remains a field of active academic research:

https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

Post by d***@deepplum.com
Of course, cable companies like Charter and ATT who have mostly DOCSIS 2 gear deployed can't admit to their plant being bloat-causing.
In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.

What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.

fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

Post by d***@deepplum.com
And the solution is needed in the CMTS once neighbors all start becoming heavier users, because it is a shared buffering pool with no fairness or bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.

I was sad to see recently a cisco white paper that even ignored their
own work on pie.

Post by d***@deepplum.com
Still, routers with queue management that reduce bloat would help a lot, if "buffering" is seen frequently under load.
So why isn't anyone talking about this problem after at least a decade of knowing it, and knowing how to fix it?
I blame IETF members, individually and collectively. If ietf exists for any reason other than as a boondoggle for world travel, it's for resolving issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.

I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.

[d***@deepplum.com]

I have been one of the most prominent advocates of network neutrality. I'm constantly informing my friends and the press about "buffering" not being related to neutrality at all.

I think that's the best we can do.

Packet neutrality is actually a key part of the Internet's design, pushing control mechanisms to the edge, with a minimum of "intelligence" in the routers/switches/gateways. In particular, "content-based" and "endpoint-address-based" targeted throttling was never how the Internet Protocol layer was supposed to work. That's a fundamental result of the "end-to-end argument" applied to congestion management. I've spent a lot of time on that issue technically. The original discussions going back before Van Jacobson's early work, up to RED and ECN, all are based on providing congestion signalling sufficient to cause endpoints competing for resources to adapt their behavior cooperatively in real time, while maintaining minimal latency under load.

Latency under load is the crucial metric across the entire IP layer, endpoints and routers. It's clear that minimizing latency under load has to be achieved by avoiding buffering insite the network, moving it to the source (and destination).

I've given this lecture to policy people a lot. In fact, deliberate creation of "bloat" is a technical strategy that has been used in the past to destroy VoIP and other real-time communicaitons. Microsoft was caught doing it decades ago, as were some other conflicted communicaitons providers. They could selectively delay small packets using DPI, while letting FTPs get full speed. That's one of the reasons we coined the idea of Network Neutrality.

But radical right wingers of the sort that blossom in the paranoid world of the dark net started arguing that the routers should have political freedom to do whatever they damn well pleased with packets, because routers are people just like corporations, and a "free market" is the solution to everything.

Well, technically, the Internet doesn't work if their is not some mechanism for eliminiating lag under load by eliminating queueing delay in bottleneck nodes.

That's ultimately what Network Neutrality is about. There's a lot of other crap being pushed by folks who pile on to the Network Neutrality discussion. People want to "fix prices" for example, arguing that profits are bad. Those guys are not the problem.

The problem is that the vertically integrated monopolists want to claim that the Internet should be subject to Deep Packet Inspection at every router, designed to charge rents based on content of the packets and who is the original sender or destination of the packet - that is, charging Netflix or NBC Universal packets nothing, and charging IPFS packets 100x as much.

So, no, the Network Neutrality people are NOT the problem with Bufferbloat.

Comcast, on the other hand, has been slow-rolling DOCSIS 3.0, because their customers on DOCSIS 2.0 are just ordering faster service tiers to overcome the Bufferbloat in their DOCSIS 2 CMTS's.
-----Original Message-----
From: "Dave Taht" <***@gmail.com>
Sent: Monday, June 18, 2018 3:07pm
To: "***@deepplum.com" <***@deepplum.com>
Cc: cerowrt-***@lists.bufferbloat.net, "bloat" <***@lists.bufferbloat.net>
Subject: Re: Invisibility of bufferbloat and its remedies

Post by d***@deepplum.com
https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.

It still remains a field of active academic research:

https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

Post by d***@deepplum.com
Of course, cable companies like Charter and ATT who have mostly DOCSIS 2 gear deployed can't admit to their plant being bloat-causing.
In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.

What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.

fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

Post by d***@deepplum.com
And the solution is needed in the CMTS once neighbors all start becoming heavier users, because it is a shared buffering pool with no fairness or bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.

I was sad to see recently a cisco white paper that even ignored their
own work on pie.

Post by d***@deepplum.com
Still, routers with queue management that reduce bloat would help a lot, if "buffering" is seen frequently under load.
So why isn't anyone talking about this problem after at least a decade of knowing it, and knowing how to fix it?
I blame IETF members, individually and collectively. If ietf exists for any reason other than as a boondoggle for world travel, it's for resolving issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.

I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.

[Jonathan Morton]

Post by d***@deepplum.com
So, no, the Network Neutrality people are NOT the problem with Bufferbloat.

No, but I think it's fair to point towards corporate greed and political ignorance as common causes of both problems.

- Jonathan Morton

[d***@deepplum.com]

No doubt.

However, sometimes one can get powerful economic forces to support good ideas.

The entire telecom industry was going after the Internet as a concept fiercely in the days of "dialup" Internet access. Trying to get the government to allow them to price it out of existence, trying to argue that ISDN couldn't be deployed, trying to argue that "selective content" (AOL) was better than the dangerous open Internet full of kiddy porn and drug cartels, whereas Ma Bell et al. would deliver clean and wholesome content only, if the government would just allow them to build the National Inofmration Superhighway the way it "should be engineered".

Yet, the Internet community routed around all of this, by showing hugely valuable new ideas that were available instantly, and a vibrant ecosystem of innovators working for users, not for the big companies.

It's still reasonable to continute that path. But it is worth remembering that when Venture Capital joined in, things started to go awry. @Home (done by Milo Medin - funded by Kleiner Perkins, now at Google) was conceived as a closed, walled garden, instituting "web caching" that was supposedly "good for users", whie at the same time breaking the WWW protocols needed for evolution of the Internet, and instituting systematic port blocking that prevented anyone from creating servers.

But Medin and Kleiner were failures, not getting that openness was at the core of the Internet.

-----Original Message-----
From: "Jonathan Morton" <***@gmail.com>
Sent: Monday, June 18, 2018 7:17pm
To: "***@deepplum.com" <***@deepplum.com>
Cc: "Dave Taht" <***@gmail.com>, cerowrt-***@lists.bufferbloat.net, "bloat" <***@lists.bufferbloat.net>
Subject: Re: [Bloat] Invisibility of bufferbloat and its remedies

Post by d***@deepplum.com
So, no, the Network Neutrality people are NOT the problem with Bufferbloat.

No, but I think it's fair to point towards corporate greed and political ignorance as common causes of both problems.

- Jonathan Morton

[Dave Taht]

I will be in D.C., presenting https://www.cs.kau.se/tohojo/cake/ - next week.

If there are people to see, asses to kick or kiss, I'm up for it, let
me know. Presently I just plan to give my talk and get the heck out of
dodge.

One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you
just set it to the same thing and get vastly superior performance to
docsis 3.1, pie, or the sqm-scripts.

Post by d***@deepplum.com
I have been one of the most prominent advocates of network neutrality. I'm
constantly informing my friends and the press about "buffering" not being
related to neutrality at all.
I think that's the best we can do.
Packet neutrality is actually a key part of the Internet's design, pushing
control mechanisms to the edge, with a minimum of "intelligence" in the
routers/switches/gateways. In particular, "content-based" and
"endpoint-address-based" targeted throttling was never how the Internet
Protocol layer was supposed to work. That's a fundamental result of the
"end-to-end argument" applied to congestion management. I've spent a lot of
time on that issue technically. The original discussions going back before
Van Jacobson's early work, up to RED and ECN, all are based on providing
congestion signalling sufficient to cause endpoints competing for resources
to adapt their behavior cooperatively in real time, while maintaining
minimal latency under load.
Latency under load is the crucial metric across the entire IP layer,
endpoints and routers. It's clear that minimizing latency under load has to
be achieved by avoiding buffering insite the network, moving it to the
source (and destination).
I've given this lecture to policy people a lot. In fact, deliberate creation
of "bloat" is a technical strategy that has been used in the past to destroy
VoIP and other real-time communicaitons. Microsoft was caught doing it
decades ago, as were some other conflicted communicaitons providers. They
could selectively delay small packets using DPI, while letting FTPs get full
speed. That's one of the reasons we coined the idea of Network Neutrality.
But radical right wingers of the sort that blossom in the paranoid world of
the dark net started arguing that the routers should have political freedom
to do whatever they damn well pleased with packets, because routers are
people just like corporations, and a "free market" is the solution to
everything.
Well, technically, the Internet doesn't work if their is not some mechanism
for eliminiating lag under load by eliminating queueing delay in bottleneck
nodes.
That's ultimately what Network Neutrality is about. There's a lot of other
crap being pushed by folks who pile on to the Network Neutrality discussion.
People want to "fix prices" for example, arguing that profits are bad. Those
guys are not the problem.
The problem is that the vertically integrated monopolists want to claim that
the Internet should be subject to Deep Packet Inspection at every router,
designed to charge rents based on content of the packets and who is the
original sender or destination of the packet - that is, charging Netflix or
NBC Universal packets nothing, and charging IPFS packets 100x as much.
So, no, the Network Neutrality people are NOT the problem with Bufferbloat.
Comcast, on the other hand, has been slow-rolling DOCSIS 3.0, because their
customers on DOCSIS 2.0 are just ordering faster service tiers to overcome
the Bufferbloat in their DOCSIS 2 CMTS's.
-----Original Message-----
Sent: Monday, June 18, 2018 3:07pm
Subject: Re: Invisibility of bufferbloat and its remedies

Post by d***@deepplum.com
https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.
https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

Post by d***@deepplum.com
Of course, cable companies like Charter and ATT who have mostly DOCSIS 2
gear deployed can't admit to their plant being bloat-causing.
In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.
What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.
fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

Post by d***@deepplum.com
And the solution is needed in the CMTS once neighbors all start becoming
heavier users, because it is a shared buffering pool with no fairness or
bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.
I was sad to see recently a cisco white paper that even ignored their
own work on pie.

Post by d***@deepplum.com
Still, routers with queue management that reduce bloat would help a lot,
if "buffering" is seen frequently under load.
So why isn't anyone talking about this problem after at least a decade of
knowing it, and knowing how to fix it?
I blame IETF members, individually and collectively. If ietf exists for
any reason other than as a boondoggle for world travel, it's for resolving
issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.
I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: [phone]

[Dave Taht]

Well, I ranted: http://blog.cerowrt.org/post/net_neutrality_isps/

I am thinking a string of shorter pieces aimed directly at customers,
reviewers, politicians, etc might do a bit more good than the
gargantuan things jim tends to do.

Post by Dave Taht
I will be in D.C., presenting https://www.cs.kau.se/tohojo/cake/ - next week.
If there are people to see, asses to kick or kiss, I'm up for it, let
me know. Presently I just plan to give my talk and get the heck out of
dodge.
One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you
just set it to the same thing and get vastly superior performance to
docsis 3.1, pie, or the sqm-scripts.

Post by d***@deepplum.com
I have been one of the most prominent advocates of network neutrality. I'm
constantly informing my friends and the press about "buffering" not being
related to neutrality at all.
I think that's the best we can do.
Packet neutrality is actually a key part of the Internet's design, pushing
control mechanisms to the edge, with a minimum of "intelligence" in the
routers/switches/gateways. In particular, "content-based" and
"endpoint-address-based" targeted throttling was never how the Internet
Protocol layer was supposed to work. That's a fundamental result of the
"end-to-end argument" applied to congestion management. I've spent a lot of
time on that issue technically. The original discussions going back before
Van Jacobson's early work, up to RED and ECN, all are based on providing
congestion signalling sufficient to cause endpoints competing for resources
to adapt their behavior cooperatively in real time, while maintaining
minimal latency under load.
Latency under load is the crucial metric across the entire IP layer,
endpoints and routers. It's clear that minimizing latency under load has to
be achieved by avoiding buffering insite the network, moving it to the
source (and destination).
I've given this lecture to policy people a lot. In fact, deliberate creation
of "bloat" is a technical strategy that has been used in the past to destroy
VoIP and other real-time communicaitons. Microsoft was caught doing it
decades ago, as were some other conflicted communicaitons providers. They
could selectively delay small packets using DPI, while letting FTPs get full
speed. That's one of the reasons we coined the idea of Network Neutrality.
But radical right wingers of the sort that blossom in the paranoid world of
the dark net started arguing that the routers should have political freedom
to do whatever they damn well pleased with packets, because routers are
people just like corporations, and a "free market" is the solution to
everything.
Well, technically, the Internet doesn't work if their is not some mechanism
for eliminiating lag under load by eliminating queueing delay in bottleneck
nodes.
That's ultimately what Network Neutrality is about. There's a lot of other
crap being pushed by folks who pile on to the Network Neutrality discussion.
People want to "fix prices" for example, arguing that profits are bad. Those
guys are not the problem.
The problem is that the vertically integrated monopolists want to claim that
the Internet should be subject to Deep Packet Inspection at every router,
designed to charge rents based on content of the packets and who is the
original sender or destination of the packet - that is, charging Netflix or
NBC Universal packets nothing, and charging IPFS packets 100x as much.
So, no, the Network Neutrality people are NOT the problem with Bufferbloat.
Comcast, on the other hand, has been slow-rolling DOCSIS 3.0, because their
customers on DOCSIS 2.0 are just ordering faster service tiers to overcome
the Bufferbloat in their DOCSIS 2 CMTS's.
-----Original Message-----
Sent: Monday, June 18, 2018 3:07pm
Subject: Re: Invisibility of bufferbloat and its remedies

Post by d***@deepplum.com
https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.
https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

Post by d***@deepplum.com
Of course, cable companies like Charter and ATT who have mostly DOCSIS 2
gear deployed can't admit to their plant being bloat-causing.
In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.
What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.
fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

Post by d***@deepplum.com
And the solution is needed in the CMTS once neighbors all start becoming
heavier users, because it is a shared buffering pool with no fairness or
bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.
I was sad to see recently a cisco white paper that even ignored their
own work on pie.

Post by d***@deepplum.com
Still, routers with queue management that reduce bloat would help a lot,
if "buffering" is seen frequently under load.
So why isn't anyone talking about this problem after at least a decade of
knowing it, and knowing how to fix it?
I blame IETF members, individually and collectively. If ietf exists for
any reason other than as a boondoggle for world travel, it's for resolving
issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.
I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: [phone]

--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: [phone]

[d***@deepplum.com]

good rant!

-----Original Message-----
From: "Dave Taht" <***@gmail.com>
Sent: Tuesday, June 19, 2018 3:33pm
To: "***@deepplum.com" <***@deepplum.com>
Cc: cerowrt-***@lists.bufferbloat.net, "bloat" <***@lists.bufferbloat.net>
Subject: Re: Invisibility of bufferbloat and its remedies



Well, I ranted: http://blog.cerowrt.org/post/net_neutrality_isps/

I am thinking a string of shorter pieces aimed directly at customers,
reviewers, politicians, etc might do a bit more good than the
gargantuan things jim tends to do.

Post by Dave Taht
I will be in D.C., presenting https://www.cs.kau.se/tohojo/cake/ - next week.
If there are people to see, asses to kick or kiss, I'm up for it, let
me know. Presently I just plan to give my talk and get the heck out of
dodge.
One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you
just set it to the same thing and get vastly superior performance to
docsis 3.1, pie, or the sqm-scripts.

Post by d***@deepplum.com
I have been one of the most prominent advocates of network neutrality. I'm
constantly informing my friends and the press about "buffering" not being
related to neutrality at all.
I think that's the best we can do.
Packet neutrality is actually a key part of the Internet's design, pushing
control mechanisms to the edge, with a minimum of "intelligence" in the
routers/switches/gateways. In particular, "content-based" and
"endpoint-address-based" targeted throttling was never how the Internet
Protocol layer was supposed to work. That's a fundamental result of the
"end-to-end argument" applied to congestion management. I've spent a lot of
time on that issue technically. The original discussions going back before
Van Jacobson's early work, up to RED and ECN, all are based on providing
congestion signalling sufficient to cause endpoints competing for resources
to adapt their behavior cooperatively in real time, while maintaining
minimal latency under load.
Latency under load is the crucial metric across the entire IP layer,
endpoints and routers. It's clear that minimizing latency under load has to
be achieved by avoiding buffering insite the network, moving it to the
source (and destination).
I've given this lecture to policy people a lot. In fact, deliberate creation
of "bloat" is a technical strategy that has been used in the past to destroy
VoIP and other real-time communicaitons. Microsoft was caught doing it
decades ago, as were some other conflicted communicaitons providers. They
could selectively delay small packets using DPI, while letting FTPs get full
speed. That's one of the reasons we coined the idea of Network Neutrality.
But radical right wingers of the sort that blossom in the paranoid world of
the dark net started arguing that the routers should have political freedom
to do whatever they damn well pleased with packets, because routers are
people just like corporations, and a "free market" is the solution to
everything.
Well, technically, the Internet doesn't work if their is not some mechanism
for eliminiating lag under load by eliminating queueing delay in bottleneck
nodes.
That's ultimately what Network Neutrality is about. There's a lot of other
crap being pushed by folks who pile on to the Network Neutrality discussion.
People want to "fix prices" for example, arguing that profits are bad. Those
guys are not the problem.
The problem is that the vertically integrated monopolists want to claim that
the Internet should be subject to Deep Packet Inspection at every router,
designed to charge rents based on content of the packets and who is the
original sender or destination of the packet - that is, charging Netflix or
NBC Universal packets nothing, and charging IPFS packets 100x as much.
So, no, the Network Neutrality people are NOT the problem with Bufferbloat.
Comcast, on the other hand, has been slow-rolling DOCSIS 3.0, because their
customers on DOCSIS 2.0 are just ordering faster service tiers to overcome
the Bufferbloat in their DOCSIS 2 CMTS's.
-----Original Message-----
Sent: Monday, June 18, 2018 3:07pm
Subject: Re: Invisibility of bufferbloat and its remedies

Post by d***@deepplum.com
https://www.cordcuttersnews.com/3-easy-tips-to-fix-constant-buffering/
It's distressing how little the tech press understands the real problem.

Yea, that one is pretty sad.
https://scholar.google.com/scholar?as_ylo=2018&q=bufferbloat&hl=en&as_sdt=0,5

Post by d***@deepplum.com
Of course, cable companies like Charter and ATT who have mostly DOCSIS 2
gear deployed can't admit to their plant being bloat-causing.
In fact it protects their cable business against cord cutters.

Lacking competition in general, doesn't help.
What I am actually more frustrated about is the network neutrality
advocates A) conflating "buffering" with malfeasance, rather than a
technical problem
and B) Using politics rather than technology to attempt to achieve
their goals. If *only* a few prominent members of that side of affairs
"got" that some better technology, deployed, might solve some of their
problems and make the internet better for everyone, we'd make more
progress.
fq_codel is a uniquely "american" algorithm, in that it gives the
"little guy" a little boost until it achieves parity with everyone
else.

Post by d***@deepplum.com
And the solution is needed in the CMTS once neighbors all start becoming
heavier users, because it is a shared buffering pool with no fairness or
bloat protection.

My principle observation is that with the changes in traffic patterns
in the last decade, and the predominance of application-rate limited
streaming, that most
folk are merely forced into a bandwidth tier that is less rarely
annoying. This does not of course solve the corporate gateway problems
very well, nor does it truly kill it dead, but until that day when
"the right stuff" is readily available, and more informed demand
exists.
I was sad to see recently a cisco white paper that even ignored their
own work on pie.

Post by d***@deepplum.com
Still, routers with queue management that reduce bloat would help a lot,
if "buffering" is seen frequently under load.
So why isn't anyone talking about this problem after at least a decade of
knowing it, and knowing how to fix it?
I blame IETF members, individually and collectively. If ietf exists for
any reason other than as a boondoggle for world travel, it's for resolving
issues like this one.

Heh. I have essentially abandoned the IETF as the inmates are running
the asylum, and trying to continue to make our points there was
seemingly fruitless
- and out of my budget. I'd rather stay home and get better code out
the door. Or come up with some other set of orgs to annoy into paying
attention.
I would not mind going to another IETF meeting to give a preso (on,
say, cake), but I'm unwilling to front the funds or time anymore.
--
Dave TÀht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: [phone]

--
Dave TÀht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: [phone]

[v***@vt.edu]

Post by Dave Taht
One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you just set it to the same thing and get vastly superior performance to
docsis 3.1, pie, or the sqm-scripts.

Do we have a good cookbook on how to determine the set-rate?

[Sebastian Moeller]

Well,

Post by v***@vt.edu

Post by Dave Taht
One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you just set it to the same thing and get vastly superior performance

to

Post by Dave Taht
docsis 3.1, pie, or the sqm-scripts.

Do we have a good cookbook on how to determine the set-rate?

With cable <DOCSIS 3.1 one can Snoop the management frames that will also carry the bandwidth definitions. Or with a compromised cable modem one could dump the DOCSIS config file that also contains that information. Or ask the ISP. Short of any of that one could run a umber of speedtests over a 24 hour period and simply extrapolate from the measured goodput to the required 'gross' DOCSIS shaper rate:

Measured TCP/ipv4 goodput * ((1518)/(1500-20-20)) = lower bound gross bandwidth estimate

DOCSIS employs a shaper to limit user's exceeding the contracted rates, that per DOCSIS standard assumes 1518 bytes of accountable raw frame size. Tangent that 1518 obviously is a 'lie' in that it does not include all per packet overhead in use on a DOCSIS carrier, but from an end-user perspective that difference is immaterial...

I realize that you probably wanted something simpler and more accurate, sorry to disappoint.

Best Regards
Sebastian

Post by v***@vt.edu
_______________________________________________
Cerowrt-devel mailing list
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Sebastian Moeller]

Hi all,

Post by Sebastian Moeller
Well,

Post by v***@vt.edu

Post by Dave Taht
One of cake's "minor" features is the *perfect* defeat of the htb
based shaper in cable modems. If you know the set-rate on the modem,
you just set it to the same thing and get vastly superior performance

to

Post by Dave Taht
docsis 3.1, pie, or the sqm-scripts.

Do we have a good cookbook on how to determine the set-rate?

Measured TCP/ipv4 goodput * ((1518)/(1500-20-20)) = lower bound gross bandwidth estimate

Addendum: when running speedtests on cable for the purpose of estimating the "true" docsis shaper goodput one needs to take care to not be fooled by transient bandwidth allowances like powerboost but rather one needs to find the sustainable stable maximum bandwidth; so fun all around. (And one needs to account for the correct overhead in the equation above so in the IPv6 case the (1500-20-20) needs to be replaced by (1500-40-20))

Best Regards
Sebastian

Post by Sebastian Moeller
DOCSIS employs a shaper to limit user's exceeding the contracted rates, that per DOCSIS standard assumes 1518 bytes of accountable raw frame size. Tangent that 1518 obviously is a 'lie' in that it does not include all per packet overhead in use on a DOCSIS carrier, but from an end-user perspective that difference is immaterial...
I realize that you probably wanted something simpler and more accurate, sorry to disappoint.
Best Regards
Sebastian

Post by v***@vt.edu
_______________________________________________
Cerowrt-devel mailing list
https://lists.bufferbloat.net/listinfo/cerowrt-devel

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Cerowrt-devel mailing list
https://lists.bufferbloat.net/listinfo/cerowrt-devel

[Jonathan Morton]

Post by v***@vt.edu
Do we have a good cookbook on how to determine the set-rate?

On DSL, the sync rates in each direction should usually be readable from the modem; they are typically reported on the router's status page. The advertised rate is less reliable, to say the least.

On wireless links, all bets are off - even with stationary endpoints, link capacity varies wildly over time. This needs to be solved in the radio-modems.

If it's wifi, however, a link-rate-independent solution now exists for certain hardware, and there's nothing theoretically stopping something similar being put into future HardMAC implementations. If we get the choice of hardware, naturally we choose wisely.

- Jonathan Morton

[Sebastian Moeller]

Post by Jonathan Morton

Post by v***@vt.edu
Do we have a good cookbook on how to determine the set-rate?

On DSL, the sync rates in each direction should usually be readable
from the modem; they are typically reported on the router's status
page. The advertised rate is less reliable, to say the least.

Well many ISPs nowadays employ additional traffic shapers upstream of the dslam/MSAN, and it is this shaper's set-rate we would need to have. All we know is that this needs to be <= sync but the details differ. Also we need information about per packet overhead which again is not easily available for end-users. IMHO our best bet would be to get regulatory agancies to force ISPs to make this information easily available to end-users so that customers can actually compare different offers realistically, but I am not holding my breath for this to happen in my lifetime...

Best Regards
Sebastian

Post by Jonathan Morton
On wireless links, all bets are off - even with stationary endpoints,
link capacity varies wildly over time. This needs to be solved in the
radio-modems.
If it's wifi, however, a link-rate-independent solution now exists for
certain hardware, and there's nothing theoretically stopping something
similar being put into future HardMAC implementations. If we get the
choice of hardware, naturally we choose wisely.
- Jonathan Morton
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat

[Kevin Darbyshire-Bryant]

Post by Jonathan Morton

Post by v***@vt.edu
Do we have a good cookbook on how to determine the set-rate?

On DSL, the sync rates in each direction should usually be readable from the modem; they are typically reported on the router's status page. The advertised rate is less reliable, to say the least.

I’ve been experimenting with this hack in sqm-scripts to do just that https://github.com/ldir-EDB0/sqm-scripts/commit/a71ab1c9acd75d6e9254360e832ac7b6f9514e88 originally on my parents DGN3500 and currently on my on BT HomeHub5a test line.

It fails to work if the ISP does rate banding (BT 20CN does this, BT 21CN doesn’t) where downstream is limited to a rate below downstream sync rate. I guess a lookup table could be implemented.

Kevin

[Sebastian Moeller]

Hi Kevin,

Post by Jonathan Morton

Post by v***@vt.edu
Do we have a good cookbook on how to determine the set-rate?

On DSL, the sync rates in each direction should usually be readable from the modem; they are typically reported on the router's status page. The advertised rate is less reliable, to say the least.

I’ve been experimenting with this hack in sqm-scripts to do just that https://github.com/ldir-EDB0/sqm-scripts/commit/a71ab1c9acd75d6e9254360e832ac7b6f9514e88 originally on my parents DGN3500 and currently on my on BT HomeHub5a test line.

Clever (I see how you chiseled data_rates() into shape here, respect)! Even though I believe that this is not pppoa specific and should probably check whether /lib/functions/lantiq_dsl.sh exists. Actually this code will also work on VDSL2 links... (and we should also be able to extract the encapsulation atm or ptm).

if [ "pppoa-wan" = "$IFACE" ]; then
if [ -f "/lib/functions/lantiq_dsl.sh" ] : then
...
fi
fi

But would it not be simpler to call /etc/init.d/dsl_control status | grep -e "Data Rate:" or somesuch? Cureently openwrt only supports lantiq/intel modems, but if broadcom modems should ever be supported I venture a guess they will not use /lib/functions/lantiq_dsl.sh to generate the stats output ;) (not that there is a guarantee that dsl_control would exost and generate compatible output).


But I believe that this is not that helpful as a mode to automatically set the bandwidth*, as I assume that most ISPs will shape the downstream bandwidth upstream of the DSLAM (if just to avoid having a DDOS against one user taking down the whole DSLAM). In my case my ISP even shapes the upstream, which is somewhat more puzzling. It looks like a cool way to deal with variable sync (either after a re-sync due to say DLM action or due to SRA) so how about polishing this a bit and including this as pure informational line in the log?


*) if this is to be made automatic by say allowing to scrape bandwidth from the modem we would need additional setting for setting the shaper percentage. I wonder whether all of this is worth it though, given that the number of users running sqm on devices in control of the dsl-modem is going to be miniscule, no?

It fails to work if the ISP does rate banding (BT 20CN does this, BT 21CN doesn’t) where downstream is limited to a rate below downstream sync rate. I guess a lookup table could be implemented.

I predict that a lookup table is going to be constantly out of date, especially since at least my ISP is a moving target in both the shaper settings as well as the actual overheads (on the plus side they started to send the applicable net bandwidth as part of the pppoe negotiations (but failed to document how those rates are actually to be interpreted ;) win some loose some)).


Final thought, how about just using this on the luci side to give hints about the sync bandwidth in the GUI (like displaying the value of either sync for xdsl or the speed for ethernet devices (speed in ethtool parlance, so 10Mb/s, 100Mb/s, ...)) that will not be as smooth as your solution, but should also be more robust against doing the wrmg thing automatically? Or am I overcomplicating things again.

Final final thought ;) since lantig_dsl.sh is quite openwrt specific, maybe we should do the automatic mode inside of the equally openwrt specific /usr/lib/sqm/run instead of in start-sqm that will might also be executed on other platforms?


Best Regards
Sebastian

Kevin
_______________________________________________
Bloat mailing list
https://lists.bufferbloat.net/listinfo/bloat

[Kevin Darbyshire-Bryant]

Post by Sebastian Moeller
Hi Kevin,

Post by Kevin Darbyshire-Bryant

Post by Jonathan Morton

Post by v***@vt.edu
Do we have a good cookbook on how to determine the set-rate?

On DSL, the sync rates in each direction should usually be readable from the modem; they are typically reported on the router's status page. The advertised rate is less reliable, to say the least.

I’ve been experimenting with this hack in sqm-scripts to do just that https://github.com/ldir-EDB0/sqm-scripts/commit/a71ab1c9acd75d6e9254360e832ac7b6f9514e88 originally on my parents DGN3500 and currently on my on BT HomeHub5a test line.

Clever (I see how you chiseled data_rates() into shape here, respect)! Even though I believe that this is not pppoa specific and should probably check whether /lib/functions/lantiq_dsl.sh exists. Actually this code will also work on VDSL2 links... (and we should also be able to extract the encapsulation atm or ptm).
if [ "pppoa-wan" = "$IFACE" ]; then
if [ -f "/lib/functions/lantiq_dsl.sh" ] : then
...
fi
fi
But would it not be simpler to call /etc/init.d/dsl_control status | grep -e "Data Rate:" or somesuch? Cureently openwrt only supports lantiq/intel modems, but if broadcom modems should ever be supported I venture a guess they will not use /lib/functions/lantiq_dsl.sh to generate the stats output ;) (not that there is a guarantee that dsl_control would exost and generate compatible output).
But I believe that this is not that helpful as a mode to automatically set the bandwidth*, as I assume that most ISPs will shape the downstream bandwidth upstream of the DSLAM (if just to avoid having a DDOS against one user taking down the whole DSLAM). In my case my ISP even shapes the upstream, which is somewhat more puzzling. It looks like a cool way to deal with variable sync (either after a re-sync due to say DLM action or due to SRA) so how about polishing this a bit and including this as pure informational line in the log?
*) if this is to be made automatic by say allowing to scrape bandwidth from the modem we would need additional setting for setting the shaper percentage. I wonder whether all of this is worth it though, given that the number of users running sqm on devices in control of the dsl-modem is going to be miniscule, no?

Post by Kevin Darbyshire-Bryant
It fails to work if the ISP does rate banding (BT 20CN does this, BT 21CN doesn’t) where downstream is limited to a rate below downstream sync rate. I guess a lookup table could be implemented.

I predict that a lookup table is going to be constantly out of date, especially since at least my ISP is a moving target in both the shaper settings as well as the actual overheads (on the plus side they started to send the applicable net bandwidth as part of the pppoe negotiations (but failed to document how those rates are actually to be interpreted ;) win some loose some)).
Final thought, how about just using this on the luci side to give hints about the sync bandwidth in the GUI (like displaying the value of either sync for xdsl or the speed for ethernet devices (speed in ethtool parlance, so 10Mb/s, 100Mb/s, ...)) that will not be as smooth as your solution, but should also be more robust against doing the wrmg thing automatically? Or am I overcomplicating things again.

What part of ‘hack’ didn’t you understand? ;-)

Luci is a ‘bad’ idea in that setting it there effectively makes the rate fixed again
. and line resyncs do not remain static, so it needs to be a hotplug type solution.

The commit isn’t perfect
it was never meant to make it into the wild
I explored 2 years or so ago to see if something like that could be done but was foiled by a BT 20CN banding plan. But the seed of an idea is there. To grow it needs watering by someone who can code.

Incidentally luci sqm scripts typo (and possibly worse/not intended) "Create log file for this SQM instance under /var/run/sqm/${Inerface_name}.debug.log. Make sure to delete log files manually.”

‘Inerface_name’. - and I’m not sure if that isn’t supposed to be expanded to the actual interface name in use - I tried fixing the typo but when the name expansion still didn’t work and I had no idea how to fix that
. I got distracted by something else :-)

[Sebastian Moeller]

Hi Kevin,

Post by Sebastian Moeller
Hi Kevin,

Post by Jonathan Morton

Post by v***@vt.edu
Do we have a good cookbook on how to determine the set-rate?

On DSL, the sync rates in each direction should usually be readable from the modem; they are typically reported on the router's status page. The advertised rate is less reliable, to say the least.

I’ve been experimenting with this hack in sqm-scripts to do just that https://github.com/ldir-EDB0/sqm-scripts/commit/a71ab1c9acd75d6e9254360e832ac7b6f9514e88 originally on my parents DGN3500 and currently on my on BT HomeHub5a test line.

Clever (I see how you chiseled data_rates() into shape here, respect)! Even though I believe that this is not pppoa specific and should probably check whether /lib/functions/lantiq_dsl.sh exists. Actually this code will also work on VDSL2 links... (and we should also be able to extract the encapsulation atm or ptm).
if [ "pppoa-wan" = "$IFACE" ]; then
if [ -f "/lib/functions/lantiq_dsl.sh" ] : then
...
fi
fi
But would it not be simpler to call /etc/init.d/dsl_control status | grep -e "Data Rate:" or somesuch? Cureently openwrt only supports lantiq/intel modems, but if broadcom modems should ever be supported I venture a guess they will not use /lib/functions/lantiq_dsl.sh to generate the stats output ;) (not that there is a guarantee that dsl_control would exost and generate compatible output).
But I believe that this is not that helpful as a mode to automatically set the bandwidth*, as I assume that most ISPs will shape the downstream bandwidth upstream of the DSLAM (if just to avoid having a DDOS against one user taking down the whole DSLAM). In my case my ISP even shapes the upstream, which is somewhat more puzzling. It looks like a cool way to deal with variable sync (either after a re-sync due to say DLM action or due to SRA) so how about polishing this a bit and including this as pure informational line in the log?
*) if this is to be made automatic by say allowing to scrape bandwidth from the modem we would need additional setting for setting the shaper percentage. I wonder whether all of this is worth it though, given that the number of users running sqm on devices in control of the dsl-modem is going to be miniscule, no?

It fails to work if the ISP does rate banding (BT 20CN does this, BT 21CN doesn’t) where downstream is limited to a rate below downstream sync rate. I guess a lookup table could be implemented.

I predict that a lookup table is going to be constantly out of date, especially since at least my ISP is a moving target in both the shaper settings as well as the actual overheads (on the plus side they started to send the applicable net bandwidth as part of the pppoe negotiations (but failed to document how those rates are actually to be interpreted ;) win some loose some)).
Final thought, how about just using this on the luci side to give hints about the sync bandwidth in the GUI (like displaying the value of either sync for xdsl or the speed for ethernet devices (speed in ethtool parlance, so 10Mb/s, 100Mb/s, ...)) that will not be as smooth as your solution, but should also be more robust against doing the wrmg thing automatically? Or am I overcomplicating things again.

What part of ‘hack’ didn’t you understand? ;-)

I guess your hack have a higher quality than my release patches ;)

Luci is a ‘bad’ idea in that setting it there effectively makes the rate fixed again….

Sure.

and line resyncs do not remain static, so it needs to be a hotplug type solution.

from my experience sync rates are typically quite robust, not on the kbit but close enough given the typical reduction in bandwidth for the shaper to be active. But here we are just introducing DLM, so I guess I have a somewhat naive and optimistic view based on my ISPs conservative profiles and enforced high SNR-margins...

The commit isn’t perfect…it was never meant to make it into the wild…I explored 2 years or so ago to see if something like that could be done but was foiled by a BT 20CN banding plan. But the seed of an idea is there. To grow it needs watering by someone who can code.

Nah, there are always folks that will fix the implementation (whom am I kidding, it typically is Toke who picks up the slack ;) ) the challenge is to get the algorithm right. Between your UK link(s?) and my single DTAG link we might be able to come up with something. The issue remains though that this still leaves the overhead to be accounted for and that the number of users that will have a chance to exercise this feature will be small.
On the other hand maybe changing the GUI to set bandwidth and shaper-percentage as two independent values might be more in line with our recommendations (set bandwidth to 90% of the contracted bandwidth), what do you think?

Incidentally luci sqm scripts typo (and possibly worse/not intended) "Create log file for this SQM instance under /var/run/sqm/${Inerface_name}.debug.log. Make sure to delete log files manually.”

Oops, thanks that is my typo, fixes and pushed upstream.

‘Inerface_name’. - and I’m not sure if that isn’t supposed to be expanded to the actual interface name in use - I tried fixing the typo but when the name expansion still didn’t work and I had no idea how to fix that…. I got distracted by something else :-)

No, that is not supposed to auto expand, I just happen to like the shell syntax to denote that something is a variable...

Best Regards
Sebastian

[Michael Richardson]

Post by d***@deepplum.com
I blame IETF members, individually and collectively. If ietf exists for
any reason other than as a boondoggle for world travel, it's for
resolving issues like this one.

Slightly fair.

A thing that I tried to get ISOC's "deploy360" to do was to create a wall of
shame on bufferbloat five or eight years ago. It was before the AQM WG did
any work, I think. I wanted ISOC to collect the data, because I wanted it
visible at the CTO level.

At the very very least, I wanted a series of curated links to statements from
various equipment vendors about the problem.
I.e. www.example.com/bufferbloat.html for example in
{cisco,juniper,huawei,calix,...}

I asked many of my IETF contacts at these places if they knew who at their
company was dealing with it. I also asked salespeople that I dealt with,
hoping to put pressure the other way. I got nowhere.

I wanted, at the very least, to get them to acknowledge that there was a
problem, even if they didn't have an estimate on a solution, at least I could
point the salesperson at the page on their site, and say, "I'll buy when you
get me a delivery date".
A major client of mine lost three large (VoIP) customers because of hidden
bufferbloat in Bell Canada's LAN extension service... which "never lost any
packets", the sales people explained. It was at a time when we
weren't quite using the term bufferbloat, and we didn't quite know how to
measure it in convincing ways.

I still think that this is a good idea.

--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | network architect [
] ***@sandelman.ca http://www.sandelman.ca/ | ruby on rails [

[Dave Taht]

From a ton of memes at: https://me.me/t/buffering

"Dear youtube:

I can deal with ads
I can deal with buffer
But when ads buffer, I suffer"

and innumerable others, some NSFW

https://me.me/i/16133421

Post by Michael Richardson

Post by d***@deepplum.com
I blame IETF members, individually and collectively. If ietf exists for
any reason other than as a boondoggle for world travel, it's for
resolving issues like this one.

Slightly fair.
A thing that I tried to get ISOC's "deploy360" to do was to create a wall of
shame on bufferbloat five or eight years ago. It was before the AQM WG did
any work, I think. I wanted ISOC to collect the data, because I wanted it
visible at the CTO level.
At the very very least, I wanted a series of curated links to statements from
various equipment vendors about the problem.
I.e. www.example.com/bufferbloat.html for example in
{cisco,juniper,huawei,calix,...}
I asked many of my IETF contacts at these places if they knew who at their
company was dealing with it. I also asked salespeople that I dealt with,
hoping to put pressure the other way. I got nowhere.
I wanted, at the very least, to get them to acknowledge that there was a
problem, even if they didn't have an estimate on a solution, at least I could
point the salesperson at the page on their site, and say, "I'll buy when you
get me a delivery date".
A major client of mine lost three large (VoIP) customers because of hidden
bufferbloat in Bell Canada's LAN extension service... which "never lost any
packets", the sales people explained. It was at a time when we
weren't quite using the term bufferbloat, and we didn't quite know how to
measure it in convincing ways.
I still think that this is a good idea.
--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | network architect [