Tomato firmware w/reduced bufferbloat
A while ago I read about a problem called "bufferbloat." In short, it's the continued use of extra large network buffers to try and give the illusion of high bandwidth connectivity. Your network card, the software, your router, your modem, the routers out on the internet, and the servers, are all responsible for adding too many buffers.
Over time this has led to increasing latency whenever we try and do something interactive online. While gaming we call it "lagging" and while using an ssh session I notice my input takes seconds to respond. Furthermore, websites take longer to load even though they are usually pretty small compared to our available bandwidth.
You can read about the technical details, and potential patches to the problem here: bufferbloat.net
Also check out this Google TechTalk if you are into the geeky details. Warning: I admit it's a bit of a dry conversation.
At home I took steps to eliminate the bufferbloat from my own network. Reduced my network card buffers. Disable things like "Interrupt Moderation" and offloading if it hurts latency. Turned on ECN. And then I looked at my home router with this benchmark:
It turns out my WRT54GL, with a custom firmware build of Tomato, was getting 1800ms average latency when my uplink was full with the SFQ scheduler! That's with bandwidth shaping through QoS. With the pfifo scheduler, it was even worse, about 3600ms of latency on average.
Why so much latency even with QoS enabled?
It turns out the popular builds of Tomato, and probably other custom firmwares like DD-WRT and OpenWRT, have 128 packets of buffering for the SFQ scheduler. And 256 packets for pfifo. Then there's the network card's own buffer, aka, txqueuelen. It's set to 1000!
So all of those extra buffers meant that my router never dropped packets, but buffered them, even if my lag got really bad. Let's do some simple math:
(256 packets * (1500 bytes)) / (768 Kbps) = 3.90625 seconds
So a typical upload of 768Kbps could potentially give you over 3 seconds of lag if you happened to be using it. Unacceptable. You would also see this problem if you didn't have properly set up QoS rules. Bittorrent for example would cause the rest of your network to lag and slow down.
Thanks to TomatoUSB and its guides: tomatousb.org I downloaded the Victek's RAF build of the firmware and hacked the linux kernel just a bit.
Changes to the kernel:
pfifo 256 packet limit -> 2 packets sfq 128 packet limit -> 2 packets sfq 10 second perturbation time -> 15 seconds enable tcp_ecn enable tcp_low_latency Wireless driver 512 packet limit -> 8 packets. WRT54GL only. b43 based driver.
I also highly recommend you open your router administration scripts page, and add this to the init tab:
ifconfig eth0 txqueuelen 2 ifconfig eth1 txqueuelen 2
Save and reboot your router after you add the above. You can also telnet or ssh into the router to set the txqueuelen.
These changes reduce our theoretical maximum latency greatly:
(2 packets * (1500 bytes)) / (768 Kbps) = 30.5175781 milliseconds
If you are interested in some benchmarks of changing these settings, I recommend this excellent blog post: coverfire.com/archives/2011/02/21/network-latency-experiments/
I also built two other builds, with 10 packet limits, and 20 packet limits. These are for people with really fast upload speeds (greater than 1Mbit/s). Do the bit of math above (in Google) to figure out which build is right for you.
You might think 2 packets isn't enough, but in my testing, it rarely dropped packets unless you had lots of bittorrent traffic. And this is the expected behavior! It should drop packets if your line is overloaded, otherwise the TCP protocols can't properly measure the speed of your line. Remember, you should still have separate priority queues set up for QoS. Place your latency sensitive traffic at highest priority. Put P2P and downloads much lower.
So here's the custom builds. USE AT YOUR OWN RISK. They work fine on my WRT54GL v1.1, but I make no guarantees. These are based on Tomato RAF's branch. Check the file names for the packet limits.
Mips R1 - WRT54GL (4MB flash)
The build is based upon Tomato RAF. It has IPv6 and VPN enabled.
Mips R2 (8MB)
The MIPSR2 build is based upon Tomato RAF. It has USB, IPv6, and VPN enabled.
Let me know how it goes for you. In my own testing, my network is more responsive under load.
FAQ:
Q: Do I need QoS enabled?
A: Yes! The most important changes to this build of Tomato is to the QoS limits. I've made a few tweaks elsewhere, but they don't impact latency as much.
Q: But I don't want to use QoS because it might slow down my REALLY fast connection.
A: Try it anyways, I think you'll find it doesn't slow down your connection that much. However, if you don't want any QoS, then make sure you use an appropriate txqueuelen value for your connections speed. This will help in general.
Q: What version do I need?
A: Check out this page of Tomato builds. Most older routers like the WRT54GL need a MIPS R1 build that fits in 4MB of flash. Newer routers like the Asus RT-N16 use a MIPS R2 build that fits within 8MB of flash.
Be careful, the Linksys E3000, etc, need custom builds. If there is interest in custom builds, ask me to compile it.
Q: Do I need to clear my NVRAM?
A: Not if you are just moving from another version of Tomato. Yes if you are coming from a different firmware.
Q: I used Netalyzr, and even with QoS enabled and prioritized, it says I still have lots of buffers.
A: Your modem or gateway might have lots of buffers. Try reducing your maximum outgoing and incoming bandwidth to 90% of your maximum tested speed. That way your modem's buffers never completely fill up. You might even have to reduce the maximum even more if you care about latency.
Q: I used Netalyzr, and it says my connection looks fine and isn't buffered.
A: Sometimes Netalyzr isn't fast enough to fill up your bandwidth, so your buffers never fill up. Try downloading/uploading something and run Netalyzr again. Perhaps you are just lucky and don't have bufferbloat in your case.
Q: What about Windows. Does it have buffers?
A: Yes! Check out your network cards drivers' properties. Most of the time, you can lower the amount of buffers on it. Also look out for something called "Interrupt Moderation" and turn it off.
Q: What about Linux. Does it have buffers?
A: Yes! Try changing your network cards txqueuelen value. Lower is usually better, unless you have a very high speed connection. Also checkout its hardware buffers. Use ethtool -g to view it. Use ethtool -G to set it.
Q: What packet limit do I want?
A: The smaller, the better, for most people. If you have a very fast connection (greater than 1Mb/s upload) then try a higher packet limit build.
Q: How can I tell if it's working?
A: Log into your router via ssh or telnet. Run
tc -s qdisc
to show the queue stats. It will tell you how many packets have been dropped in each class.