The Man Behind the Curtain at Zoph with Victor Grenu

Victor is an Independent Senior Cloud Infrastructure Architect working mainly on Amazon Web Services (AWS), designing: secure, scalable, reliable, and cost-effective cloud architectures, dealing with large-scale and mission-critical distributed systems. He also has a long experience in Cloud Operations, Security Advisory, Security Hardening (DevSecOps), Modern Applications Design, Micro-services and Serverless, Infrastructure Refactoring, Cost Saving (FinOps).

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: This episode is brought to us in part by our friends at Datadog. Datadog's SaaS monitoring and security platform that enables full stack observability for developers, IT operations, security, and business teams in the cloud age. Datadog's platform, along with 500 plus vendor integrations, allows you to correlate metrics, traces, logs, and security signals across your applications, infrastructure, and third party services in a single pane of glass.

Combine these with drag and drop dashboards and machine learning based alerts to help teams troubleshoot and collaborate more effectively, prevent downtime, and enhance performance and reliability. Try Datadog in your environment today with a free 14 day trial and get a complimentary T-shirt when you install the agent.

To learn more, visit datadoghq.com/screaminginthecloud to get. That's www.datadoghq.com/screaminginthecloud

Corey: Managing shards. Maintenance windows. Overprovisioning. ElastiCache bills. I know, I know. It's a spooky season and you're already shaking. It's time for caching to be simpler. Momento Serverless Cache lets you forget the backend to focus on good code and great user experiences. With true autoscaling and a pay-per-use pricing model, it makes caching easy. No matter your cloud provider, get going for free at gomomento.co/screaming That's GO M-O-M-E-N-T-O dot co slash screaming

Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. One of the best parts about running a podcast like this and trolling the internet of AWS things is every once in a while, I get to learn something radically different than what I expected. For a long time, there’s been this sort of persona or brand in the AWS space, specifically the security side of it, going by Zoph—that’s Z-O-P-H—and I just assumed it was a collective or a whole bunch of people working on things, and it turns out that nope, it is just one person. And that one person is my guest today. Victor Grenu is an independent AWS architect. Victor, thank you for joining me.

Victor: Hey, Corey, thank you for having me. It’s a pleasure to be here.

Corey: So, I want to start by diving into the thing that first really put you on my radar, though I didn’t realize it was you at the time. You have what can only be described as an army of Twitter bots around the AWS ecosystem. And I don’t even know that I’m necessarily following all of them, but what are these bots and what do they do?

Victor: Yeah. I have a few bots on Twitter that I push some notification, some tweets, when things happen on AWS security space, especially when the AWS managed policies are updated from AWS. And it comes from an initial project from Scott Piper. He was running a Git command on his own laptop to push the history of AWS managed policy. And it told me that I can automate this thing using a deployment pipeline and so on, and to tweet every time a new change is detected from AWS. So, the idea is to monitor every change on these policies.

Corey: It’s kind of wild because I built a number of somewhat similar Twitter bots, only instead of trying to make them into something useful, I’d make them into something more than a little bit horrifying and extraordinarily obnoxious. Like there’s a Cloud Boomer Twitter account that winds up tweeting every time Azure tweets something only it quote-tweets them in all caps and says something insulting. I have an AWS releases bot called AWS Cwoud—so that’s C-W-O-U-D—and that winds up converting it to OwO speak. It’s like, “Yay a new auto-scawowing growp.” That sort of thing is obnoxious and offensive, but it makes me laugh.

Yours, on the other hand, are things that I have notifications turned on for just because when they announce something, it’s generally fairly important. The first one that I discovered was your IAM changes bot. And I found some terrifying things coming out of that from time to time. What’s the data source for that? Because I’m just grabbing other people’s Twitter feeds or RSS feeds; you’re clearly going deeper than that.

Victor: Yeah, the data source is the official AWS managed policy. In fact, I run AWS CLI in the background and I’m doing just a list policy, the list policy command, and with this list I’m doing git of each policy that is returned, so I can enter it in a git repository to get the full history of the time. And I also craft a list of deprecated policy, and I also run, like, a dog-food initiative, the policy analysis, validation analysis from AWS tools to validate the consistency and the accuracy of the own policies. So, there is a policy validation with their own tool. [laugh].

Corey: You would think that wouldn’t turn up anything because their policy validator effectively acts as a linter, so if it throws an error, of course, you wouldn’t wind up pushing that. And yet, somehow the fact that you have bothered to hook that up and have findings from it indicates that that’s not how the real world works.

Victor: Yeah, there is some, let’s say, some false positive because we are running the policy validation with their own linter then own policies, but this is something that is documented from AWS. So, there is an official page where you can find why the linter is not working on each policy and why. There is a an explanation for each findings. I thinking of [unintelligible 00:05:05] managed policy, which is too long, and policy analyzer is crashing because the policy is too long.

Corey: Excellent. It’s odd to me that you have gone down this path because it’s easy enough to look at this and assume that, oh, this must just be something you do for fun or as an aspect of your day job. So, I did a little digging into what your day job is, and this rings very familiar to me: you are an independent AWS consultant, only you’re based out of Paris, whereas I was doing this from San Francisco, due to an escalatingly poor series of life choices on my part. What do you focus on in the AWS consulting world?

Victor: Yeah. I’m running an AWS consulting boutique in Paris and I’m working for a large customer in France. And I’m doing mostly infrastructure stuff, infrastructure design for cloud-native application, and I’m also doing some security audits and [unintelligible 00:06:07] mediation for my customer.

Corey: It seems to me that there’s a definite divide as far as how people find the AWS consulting experience to be. And I’m not trying to cast judgment here, but the stories that I hear tend to fall into one of two categories. One of them is the story that you have, where you’re doing this independently, you’ve been on your own for a while working specifically on this, and then there’s the stories of, “Oh, yeah, I work for a 500 person consultancy and we do everything as long as they’ll pay us money. If they’ve got money, we’ll do it. Why not?”

And it always seems to me—not to be overly judgy—but the independent consultants just seem happier about it because for better or worse, we get to choose what we focus on in a way that I don’t think you do at a larger company.

Victor: Yeah. It’s the same in France or in Europe; there is a lot of consulting firms. But with the pandemic and with the market where we are working, in the cloud, in the cloud-native solution and so on, that there is a lot of demands. And the natural path is to start by working for a consulting firm and then when you are ready, when you have many AWS certification, when you have the experience of the customer, when you have a network of well-known customer, and you gain trust from your customer, I think it’s natural to go by yourself, to be independent and to choose your own project and your own customer.

Corey: I’m curious to get your take on what your perception of being an AWS consultant is when you’re based in Paris versus, in my case, being based in the West Coast of the United States. And I know that’s a bit of a strange question, but even when I travel, for example, over to the East Coast, suddenly, my own newsletter sends out three hours later in the day than I expect it to and that throws me for a loop. The AWS announcements don’t come out at two or three in the afternoon; they come out at dinnertime. And for you, it must be in the middle of the night when a lot of those things wind up dropping. The AWS stuff, not my newsletter. I imagine you’re not excitedly waiting on tenterhooks to see what this week’s issue of Last Week in AWS talks about like I am.

But I’m curious is that even beyond that, how do you experience the market? From what you’re perceiving people in the United States talking about as AWS consultants versus what you see in Paris?

Victor: It’s difficult, but in fact, I don’t have so much information about the independent in the US. I know that there is a lot, but I think it’s more common in Europe. And yeah, it’s an advantage to whoever ten-hour time [unintelligible 00:08:56] from the US because a lot of stuff happen on the Pacific time, on the Seattle timezone, on San Francisco timezone. So, for example, for this podcast, my Monday is over right now, so, so yeah, I have some advantage in time, but yeah.

Corey: This is potentially an odd question for you. But I find an awful lot of the AWS documentation to be challenging, we’ll call it. I don’t always understand exactly what it’s trying to tell me, and it’s not at all clear that the person writing the documentation about a service in some cases has ever used the service. And in everything I just said, there is no language barrier. This documentation was written—theoretically—in English and I, most days, can stumble through a sentence in English and almost no other language. You obviously speak French as a first language. Given that you live in Paris, it seems to be a relatively common affliction. How do you find interacting with AWS in French goes? Or is it just a complete nonstarter, and it all has to happen in English for you?

Victor: No, in fact, the consultants in Europe, I think—in fact, in my part, I’m using my laptop in English, I’m using my phone in English, I’m using the AWS console in English, and so on. So, the documentation for me is a switch on English first because for the other language, there is sometimes some automated translation that is very dangerous sometimes, so we all keep the documentation and the materials in English.

Corey: It’s wild to me just looking at how challenging so much of the stuff is. Having to then work in a second language on top of that, it just seems almost insurmountable to me. It’s good they have automated translation for a lot of this stuff, but that falls down in often hilariously disastrous ways, sometimes. It’s wild to me that even taking most programming languages that folks have ever heard of, even if you program and speak no English, which happens in a large part of the world, you’re still using if statements even if the term ‘if’ doesn’t mean anything to you localized in your language. It really is, in many respects, an English-centric industry.

Victor: Yeah. Completely. Even in French for our large French customer, I’m writing the PowerPoint presentation in English, some emails are in English, even if all the folks in the thread are French. So yeah.

Corey: One other area that I wanted to explore with you a bit is that you are very clearly focused on security as a primary area of interest. Does that manifest in the work that you do as well? Do you find that your consulting engagements tend to have a high degree of focus on security?

Victor: Yeah. In my design, when I’m doing some AWS architecture, my main objective is to design some security architecture and security patterns that apply best practices and least privilege. But often, I’m working for engagement on security audits, for startups, for internal customer, for diverse company, and then doing some accommodation after all. And to run my audit, I’m using some open-source tooling, some custom scripts, and so on. I have a methodology that I’m running for each customer. And the goal is to sometime to prepare some certification, PCI DSS or so on, or maybe to ensure that the best practice are correctly applied on a workload or before go-live or, yeah.

Corey: One of the weird things about this to me is that I’ve said for a long time that cost and security tend to be inextricably linked, as far as being a sort of trailing reactive afterthought for an awful lot of companies. They care about both of those things right after they failed to adequately care about those things. At least in the cloud economic space, it’s only money as opposed to, “Oops, we accidentally lost our customers’ data.” So, I always found that I find myself drifting in a security direction if I don’t stop myself, just based upon a lot of the cost work I do. Conversely, it seems that you have come from the security side and you find yourself drifting in a costing direction.

Your side project is a SaaS offering called unusd.cloud, that’s U-N-U-S-D dot cloud. And when you first mentioned this to me, my immediate reaction was, “Oh, great. Another SaaS platform for costing. Let’s tear this one apart, too.” Except I actually like what you’re building. Tell me about it.

Victor: Yeah, and unusd.cloud is a side project for me and I was working since, let’s say one year. It was a project that I’ve deployed for some of my customer on their local account, and it was very useful. And so, I was thinking that it could be a SaaS project. So, I’ve worked at [unintelligible 00:14:21] so yeah, a few months on shifting the product to assess [unintelligible 00:14:27].

The product aim to detect the worst on AWS account on all AWS region, and it scan all your AWS accounts and all your region, and you try to detect and use the EC2, LDS, Glue [unintelligible 00:14:45], SageMaker, and so on, and attach a EBS and so on. I don’t craft a new dashboard, a new Cost Explorer, and so on. It’s it just cost awareness, it’s just a notification on email or Slack or Microsoft Teams. And you just add your AWS account on the project and you schedule, let’s say, once a day, and it scan, and it send you a cost of wellness, a [unintelligible 00:15:17] detection, and you can act by turning off what is not used.

Corey: What I like about this is it cuts at the number one rule of cloud economics, which is turn that shit off if you’re not using it. You wouldn’t think that I would need to say that except that everyone seems to be missing that, on some level. And it’s easy to do. When you need to spin something up and it’s not there, you’re very highly incentivized to spin that thing up. When you’re not using it, you have to remember that thing exists, otherwise it just sort of sits there forever and doesn’t do anything.

It just costs money and doesn’t generate any value in return for that. What you got right is you’ve also eviscerated my most common complaint about tools that claim to do this, which is you build in either a explicit rule of ignore this resource or ignore resources with the following tags. The benefit there is that you’re not constantly giving me useless advice, like, “Oh, yeah, turn off this idle thing.” It’s, yeah, that’s there for a reason, maybe it’s my dev box, maybe it’s my backup site, maybe it’s the entire DR environment that I’m going to need at little notice. It solves for that problem beautifully. And though a lot of tools out there claim to do stuff like this, most of them really failed to deliver on that promise.

Victor: Yeah, I just want to keep it simple. I don’t want to add an additional console and so on. And you are correct. You can apply a simple tag on your asset, let’s say an EC2 instances, you apply the tag in use and the value of, and then the alerting is disabled for this asset. And the detection is based on the CPU [unintelligible 00:17:01] and the network health metrics, so when the instances is not used in the last seven days, with a low CPU every [unintelligible 00:17:10] and low network out, it comes as a suspect. [laugh].

[midroll 00:17:17]

Corey: One thing that I like about what you’ve done, but also have some reservations about it is that you have not done with so many of these tools do which is, “Oh, just give us all the access in your account. It’ll be fine. You can trust us. Don’t you want to save money?” And yeah, but I also still want to have a company left when all sudden done.

You are very specific on what it is that you’re allowed to access, and it’s great. I would argue, on some level, it’s almost too restrictive. For example, you have the ability to look at EC2, Glue, IAM—just to look at account aliases, great—RDS, Redshift, and SageMaker. And all of these are simply list and describe. There’s no gets in there other than in Cost Explorer, which makes sense. You’re not able to go rummaging through my data and see what’s there. But that also bounds you, on some level, to being able to look only at particular types of resources. Is that accurate or are you using a lot of the CloudWatch stuff and Cost Explorer stuff to see other areas?

Victor: In fact, it’s the least privilege and read-only permission because I don’t want too much question for the security team. So, it’s full read-only permission. And I’ve only added the detection that I’m currently supports. Then if in some weeks, in some months, I’m adding a new detection, let’s say for Snapshot, for example, I will need to update, so I will ask my customer to update their template. There is a mechanisms inside the project to tell them that the template is obsolete, but it’s not a breaking change.

So, the detection will continue, but without the new detection, the new snapshot detection, let’s say. So yeah, it’s least privilege, and all I need is the get-metric-statistics from CloudWatch to detect unused assets. And also checking [unintelligible 00:19:16] Elastic IP or [unintelligible 00:19:19] EBS volume. So, there is no CloudWatching in this detection.

Corey: Also, to be clear, I am not suggesting that what you have done is at all a mistake, even if you bound it to those resources right now. But just because everyone loves to talk about these exciting, amazing, high-level services that AWS has put up there, for example, oh, what about DocumentDB or all these other—you know, Amazon Basics MongoDB; same thing—or all of these other things that they wind up offering, but you take a look at where customers are spending money and where they’re surprised to be spending money, it’s EC2, it’s a bit of RDS, occasionally it’s S3, but that’s a lot harder to detect automatically whether that data is unused. It’s, “You haven’t been using this data very much.” It’s, “Well, you see how the bucket is labeled ‘Archive Backups’ or ‘Regulatory Logs?’” imagine that. What a ridiculous concept.

Yeah. Whereas an idle EC2 instance sort of can wind up being useful on this. I am curious whether you encounter in the wild in your customer base, folks who are having idle-looking EC2 instances, but are in fact, for example, using a whole bunch of RAM, which you can’t tell from the outside without custom CloudWatch agents.

Victor: Yeah, I’m not detecting this behavior for larger usage of RAM, for example, or for maybe there is some custom application that is low in CPU and don’t talk to any other services using the network, but with this detection, with the current state of the detection, I’m covering large majority of waste because what I see from my customer is that there is some teams, some data scientists or data teams who are experimenting a lot with SageMaker with Glue, with Endpoint and so on. And this is very expensive at the end of the day because they don’t turn off the light at the end of the day, on Friday evening. So, what I’m trying to solve here is to notify the team—so on Slack—when they forgot to turn off the most common waste on AWS, so EC2, LTS, Redshift.

Corey: I just now wound up installing it while we’ve been talking on my dedicated shitposting account, and sure enough, it already spat out a single instance it found, which yeah was running an EC2 instance on the East Coast when I was just there, so that I had a DNS server that was a little bit more local. Okay, great. And it’s a T4g.micro, so it’s not exactly a whole lot of money, but it does exactly what it says on the tin. It didn’t wind up nailing the other instances I have in that account that I’m using for a variety of different things, which is good.

And it further didn’t wind up falling into the trap that so many things do, which is the, “Oh, it’s costing you zero and your spend this month is zero because this account is where I dump all of my AWS credit codes.” So, many things say, “Oh, well, it’s not costing you anything, so what’s the problem?” And then that’s how you accidentally lose $100,000 in activate credits because someone left something running way too long. It does a lot of the right things that I would hope and expect it to do, and the fact that you don’t do that is kind of amazing.

Victor: Yeah. It was a need from my customer and an opportunity. It’s a small bet for me because I’m trying to do some small bets, you know, the small bets approach, so the idea is to try a new thing. It’s also an excuse for me to learn something new because building a SaaS is a challenging.

Corey: One thing that I am curious about, in this account, I’m also running the controller for my home WiFi environment. And that’s not huge. It’s T3.small, but it is still something out there that it sits there because I need it to exist. But it’s relatively bored.

If I go back and look over the last week of CloudWatch metrics, for example, it doesn’t look like it’s usually busy. I’m sure there’s some network traffic in and out as it updates itself and whatnot, but the CPU peeks out at a little under 2% used. It didn’t warn on this and it got it right. I’m just curious as to how you did that. What is it looking for to determine whether this instance is unused or not?

Victor: It’s the magic [laugh]. There is some intelligence artif—no, I’m just kidding. It just statistics. And I’m getting two metrics, the superior average from the last seven days and the network out. And I’m getting the average on those metrics and I’m doing some assumption that this EC2, this specific EC2 is not used because of these metrics, this server average.

Corey: Yeah, it is wild to me just that this is working as well as it is. It’s just… like, it does exactly what I would expect it to do. It’s clear that—and this is going to sound weird, but I’m going to say it anyway—that this was built from someone who was looking to answer the question themselves and not from the perspective of, “Well, we need to build a product and we have access to all of this data from the API. How can we slice and dice it and add some value as we go?” I really liked the approach that you’ve taken on this. I don’t say that often or lightly, particularly when it comes to cloud costing stuff, but this is something I’ll be using in some of my own nonsense.

Victor: Thanks. I appreciate it.

Corey: So, I really want to thank you for taking as much time as you have to talk about who you are and what you’re up to. If people want to learn more, where can they find you?

Victor: Mainly on Twitter, my handle is @zoph [laugh]. And, you know, on LinkedIn or on my company website, as zoph.io.

Corey: And we will, of course, put links to that in the [show notes 00:25:23]. Thank you so much for your time today. I really appreciate it.

Victor: Thank you, Corey, for having me. It was a pleasure to chat with you.

Corey: Victor Grenu, independent AWS architect. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that is going to cost you an absolute arm and a leg because invariably, you’re going to forget to turn it off when you’re done.

Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Announcer: This has been a HumblePod production. Stay humble.