Join me as continue my series on cloud fundamentals by examining the last mile of the cloud, including how exciting it was to move away from Comcast, how a distribution of Linux with a potentially offensive name solved my home connectivity issues, why I chose to use a region I never otherwise use when setting up my home network, the real reason why latency affects applications (hint: it’s not latency from a DNS server or latency that stems from geographical distances), the party that’s really responsible for network performance, and more.
Episode Show Notes & Transcript
About Corey Quinn
Over the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.
Corey: Hello and welcome to our Networking in the Cloud, mini series sponsored by ThousandEyes. That's right. There may be just one of you, but there are a thousand eyes on a more serious note. ThousandEyes has sponsored their cloud performance benchmarking report for 2019 at the end of last year. Talking about what it looks like when you race various cloud providers. They looked at all the big cloud providers and determined what does performance look like from an end user perspective? What does the user experience look like among and between different cloud providers? To get your copy of this report, you can visit snark.cloud/realclouds. Why real clouds? Well, because they raced AWS, Azure, GCP, IBM Cloud and Alibaba, all of which are real clouds.
They did not include Oracle Cloud because once again they are real clouds. Check out your copy of the report at snark.cloud/realclouds. It's interesting that that report focuses on the end user experience because as this mini series begins to wind down, we're talking today about the last mile and its impact on what perceived cloud performance looks like. And I will admit that even having given this entire mini series and having a bit of a network engineering background, once upon a time, I still wind up in a fun world of always defaulting to blaming my local crappy ISP.
Now today, my local ISP is amazing. I use Sonic in San Francisco. I get Symmetric Gigabit. It's the exact opposite of Comcast who was my last provider until Sonic came to my neighborhood and it was fun that day because I looked up and down the block and saw no fewer than six Sonic trucks ripping Comcast out by the short and curlies. Which let's not kid ourselves, is something we all wish we could do and I was the happiest boy in town the day I got to do it. Now, the hard part is figuring out that yes, it is in fact a local ISP problem because it isn't always. This is also fortuitous because I spent the last month or so fixing my own local internet situation and today I'd like to tell you a little bit more about that as well as how and why.
Originally when I first moved into my roughly, we'll call it 2,800 square foot house, it's spread across three stories, I wound up getting EEROs, that's E-E-R-O. They're a mesh network set up that was acquired by Amazon after I'd purchased them. These are generation one. The wireless environment in San Francisco is challenging and in certain parts of my house, the reception as a result, wound up being a steaming bowl of horse crap. The big challenge was figuring out that, that's what the problem was. With weird dropouts and handoff issues, it was interesting. This one area that caused immediate improvement was not having these things talk to each other wirelessly as most full mesh systems will do, but instead making sure that they were cabled up appropriately to a switch, the central patch panel and then hooked them in. Now you have to be careful with switches because a lot of stuff won't do anything approaching full throughput because that can get expensive and a lot of consumer gear is crap.
This was a managed HP pro curved device back in the days that HP made networking equipment. That was great. And it's still crap, but it is crap that works at full line rate. So there's that. Next I wound up figuring that ... all right, it's time to take this seriously. So I did some research and talked to people I know who are actually good at things, instead of sounding on the internet like they're good at things. And I figured the next step was to buy some Ubiquiti Networks style stuff. Great. We go ahead and trot some of that out. It's an enterprise gear. It's full mesh. I of course now have a guest wifi that you have to pay for to use the hotspot. It's called Toss a coin to your wifi for an SS ID because of course it is. I have problems. And it's fun and I can play these stupid games, but suddenly every weird internet problem I had in my house started getting better as a result.
And it's astonishing how that changed my perception of various third party services. None of whom, by the way, had anything to do with my actual problem. But there were still some perceptual differences. And this impacts the cloud in a number of subtle ways and that's what I want to talk about today. So one of the biggest impacts is DNS. And I don't mean that in the sense of big cloud provider DNS, we've already talked about how DNS works in a previous episode. But rather what resolver you wind up using yourself. One of the things that I did as a part of this upgrade, is I rolled out a distribution of Linux called Pi-hole, which sounds incredibly insulting as applied to people as in, you know what, you should shut? Your Pi-hole. However, it's designed to run on top of Raspberry Pi and provide a DNS server that creatively blocks ads.
And that's super neat. I liked the idea of just blocking ad servers, but you have to trust whatever you're using for a DNS resolver because of a few specific use cases that I stumbled over as I went down this process. One, it turns out that having access to every website you'd care to visit as far as a list of things you've been doing, is not really the most privacy conscious thing in the universe. Now, for some reason, the internet collectively decided, you know who we trust with all the things that we look at on the internet and have no worries about giving that information to? That's right. Freaking Google. So eight dot eight dot eight dot eight, was a famously to remember open resolver and it works super well. It's quick. It returns everything. The problem is, is that Google's primary business model is very clearly surveillance and I don't do anything particularly interesting.
If you look at my DNS history, you're going to find a lot of things that you'd think you could use to blackmail me, but it turns out you actually can't because I talk about them on podcasts. That's right. I use Route 53 as a database. What of it? And it's all very strange just as far as even without anything to hide, I still feel this sense of pervasive creepiness at the idea that a giant company can look at this. Can look at my previous browsing history. So blocking things like that are of interest to me. So okay, instead, if I run Pi-hole that acts as my own resolver but then it winds up passing queries on to an upstream provider. I mean I could run my own, but that has other latency concerns and DNS latency when you're making requests is super indicative because the entire internet has gone collectively dumb. And decided to display a simple static webpage, You need to make 30 distinct DNS request in series and wait for them all to come back and other ridiculous nonsense that is the modern web today.
What makes this extra special is I figured out, okay, I'm not going to go with Google or CloudFlare because it has other problematic aspects to its business that we need not go into here, but okay, we'll pick Level 3 as another example. And Level 3 was recently sold to CenturyLink and they are, to be polite the devil because they break DNS in horrifying and obscene ways. Namely, whenever something isn't found, it returns a result to its own personalized search engine, which means that anything that you have that is depending upon certain ... an X domain or no resolution available results, suddenly has a result returned. Well, you can set a cookie in your browser to wind up avoiding that behavior so we don't see what the problem is.
The problem jack hole is that I can't set cookies in a DNS system that is querying it from a Daymond perspective. A whole bunch of email block lists software runs based upon DNS results and when you suddenly have results returning for everything, they either block nothing or they block everything to taste. And that winds up being obnoxious and difficult. Now, I'll talk to you a little bit more about how this impacts the cloud in just a second.
But first I would like to thank once again ThousandEyes for making this entire ridiculous rant possible. In addition to their cloud performance benchmark report, ThousandEyes winds up giving companies insight into what's going on, on the broader internet routing issues, provider failures upstream, different companies having different problems. It more or less is a real time traffic meets weather map for the internet. This helps companies who use them wind up getting a better perspective of what the current end user experience is and begin routing around provider failures, yelling at providers, et cetera, Ideally before those errors become evident to customers.
To learn more, visit ThousandEyes.com and tell them Cory sent you. In fact, they may very well say something like, "Wow, you heard about us from Corey and you still looked us up. What a Testament to how awesome our product is." My thanks again to ThousandEyes for putting up with my ridiculous nonsense to sponsor this ridiculous podcast. Now I hate running hardware mostly because I have a knack for breaking things, so when I ran the Pi-hole, sorry, I still can't take that name seriously. I didn't want to run it at an actual Raspberry or anything locally myself that I could accidentally spill water into. If you take a look at my previous history with laptops and liquids, you can understand why I have that perspective. So instead I decided I wanted to just run it in an EC2 instance because "Hey, those credits aren't going to spend themselves."
This is odd because normally when I'm trying to build something in the cloud, I put it in either Oregon or Virginia. For this thing, I parked it in the more expensive capacity constrained region known as US West One because it is close to Northern California where I personally call home. The latency was far lower, which means for every round trip to talk to a DNS server over a VPN because I'm not an irresponsible moron, it takes less time. Which leads to a faster perceived user experience. Now, am I saying that automatically having a closer DNS server is going to lead to a faster user experience? Not necessarily. And depending entirely upon the profile of what application you're interacting with, you're going to see mixed results, but generally speaking, a faster responding DNS server is going to have a positive overall experience on what it's like to be using online web applications for whoever you happen to be and whatever you tend to be accessing.
So having a resolver that is closer to me was important, not important enough for me to actually have a Raspberry PI or something like that here that I have to feed, maintain, watch get compromised, turn into an attack platform, and destroy my television slash scale which are currently busy monitoring a DDoS attack against the entire internet infrastructure because everything is terrible. Now this is an optimization that can lead to a better experience on the end user side, but ... and this is where I think a lot of cloud providers get things hilariously wrong, it's usually not latency from a DNS server or latency between where your customer is and the resource that they're speaking to. It's that most applications, be they web app or otherwise are designed like complete crap. They have a series of different connections to different hosts that need to all be contacted, resolved correctly, and transfer data to render a page correctly.
One of the best examples we saw of this was when GDPR first came out and rather than a justice for folks who are admitting from Europe, a bunch of websites just dropped all the tracking, all the garbage, and you could see for a while, sites like the Los Angeles Times would load in 20k and fractions of a second instead of the 30 megabyte monstrosities we see now. If you take a look at any given webpage that loads today, it's enormous and it's crappy and having faster DNS resolution times is helpful, sure. But because you're straining raw sewage with your teeth when dealing with the internet, it's worse than just showing you ads. It's tracking behavior, it's malware shoved under ad networks. If you take a look at any given website property that I have with last week and AWS with this podcast, et cetera, yes, I sell ads.
That's normal. What I don't do is track the living hell out of my customers. If you wind up visiting the website from a browser that has significant ad blocking turned on like I do, you don't get a degraded experience. You don't see a whole host of things getting destroyed and it's still not perfect. I still think we're pulling CSS from third party hosted sites, which is monstrous and obnoxious to me, but I'm not a web designer. I pay other people for that problem. In conclusion, yes, improving your local environment and your crappy ISP to a better ISP is going to lead to better outcomes. But there's so much else that can be done before getting to that point. And I think collectively, we missed the boat a lot when we talk about how performance is a cloud providers ultimate responsibility, yada, yada, yada. No, it's ours.
Stop designing things like crap and start treating customers with a modicum of respect and I suspect that things will become a lot better for folks who are creating interesting content and sharing useful things. Sure, the rent seekers are going to have different problems, but I don't really care what happens to them. I don't mean to be unkind, but there we have it. In conclusion, the last mile matters a lot, partially because it is the experience filter through which everything we do on the internet matters to customers and partially because so many of these things treat it completely like garbage. That's it for this episode. I am Corey Quinn. This is our Networking in the Cloud mini series of the AWS morning brief. For comments, feedback, et cetera, you can find me on Twitter @Quinnypig. Q-U-I, double N-Y pig. Or make sure to leave a five star review for this episode regardless of what you actually thought of it and tell me exactly what my problem is in the comments. Thanks again to ThousandEyes for their continuing and ongoing support of this ridiculous podcast.
Announcer: This has been a HumblePod production. Stay Humble.