• Skip to primary navigation
  • Skip to main content
Last Week in AWS Logo
Master your cloud costs
  • About
    • Community
    • Contact
  • Blog
  • Newsletter
  • Podcasts
    • Last Week in AWS Podcast
    • Screaming in the Cloud
    • Nominate a Guest
  • Merch
  • Resources
    • AWS Network Map

Screaming in the Cloud

Episode34

Episode 34: Slack and the Safety Dance of Chaos Engineering

10.30.2018
FacebookXLinkedInReddit
Episode 34: Slack and the Safety Dance of Chaos EngineeringListen to Episode 34: Slack and the Safety Dance of Chaos Engineering
Corey Quinn HeadshotAbout the Author

Corey is the Chief Cloud Economist at Duckbill, where he specializes in helping companies improve their AWS bills by making them smaller and less horrifying. He also hosts the "Screaming in the Cloud" and "AWS Morning Brief" podcasts; and curates "Last Week in AWS," a weekly newsletter summarizing the latest in AWS news, blogs, and tools, sprinkled with snark and thoughtful analysis in roughly equal measure.

Search Screaming in the Cloud
Listen on Apple Podcasts
Listen on Overcast
Listen on Pocket Casts
Listen on Spotify
Get the RSS Feed

Episode Summary

In the early days, angry nerd corners on the Internet viewed Slack and some of its predecessors as, “Oh, it’s just IRC. Now, you pay someone for it.” Many fell into that trap of wondering about what value such systems offered.The big differentiator? Slack is built as a collaborative business tool. Today, we’re talking to Holly Allen, who helped make government software better while serving as the director of engineering at 18F. Now, she’s a senior engineering manager at Slack, a collaborative chat program where you can do most of your work through a rich platform of integrations. Holly enjoys taking a weird set of skills that make a computer do things and convincing people who know how to make computers do things do things. Some of the highlights of the show include: Safety engineering brings chaos and resilience engineering, incident management, and post-mortem processes together for resiliency and reliability Slack strives to move really fast while being in complete control Slack is primarily on AWS, but is working on a multi-Cloud strategy because if AWS is down, Slack still needs to work Slack has a close relationship with AWS and is a collaborative company; it has immediate access to AWS staff anytime there’s a problem Slack uses Terraform and Chef and working to determine if its production workflows in Kubernetes would be worthwhile Disasterpiece Theater: Real scenario that might happen and surmise what will happen; don’t cause production issues, but teach Slack employees Slack hires collaborative, empathetic people to create a collaborative environment where everyone works together toward a goal Slack was firmly in a centralized operations model, but is transforming toward development teams to increase responsibility and service ownership Slack doesn’t encourage remote work because it’s not in a position to put in that investment; day-to-day work happens in hallways and between desks Slack sees itself as an enterprise software company; an enterprise software company must have enterprise software reliability, stability, and processes Slack has thousands of servers, so events and disruptions happen more often; system needs to respond, react, and repair itself without human intervention Links: Holly Allen on Twitter 18F Slack Freenode IRC HipChat AWS Kubernetes Terraform Chef QCon Datadog

Episode Show Notes

In the early days, angry nerd corners on the Internet viewed Slack and some of its predecessors as, “Oh, it’s just IRC. Now, you pay someone for it.” Many fell into that trap of wondering about what value such systems offered.The big differentiator? Slack is built as a collaborative business tool.

Today, we’re talking to Holly Allen, who helped make government software better while  serving as the director of engineering at 18F. Now, she’s a senior engineering manager at Slack, a collaborative chat program where you can do most of your work through a rich platform of integrations. Holly enjoys taking a weird set of skills that make a computer do things and convincing people who know how to make computers do things do things.

Some of the highlights of the show include:

  • Safety engineering brings chaos and resilience engineering, incident management, and post-mortem processes together for resiliency and reliability
  • Slack strives to move really fast while being in complete control
  • Slack is primarily on AWS, but is working on a multi-Cloud strategy because if AWS is down, Slack still needs to work
  • Slack has a close relationship with AWS and is a collaborative company; it has immediate access to AWS staff anytime there’s a problem
  • Slack uses Terraform and Chef and working to determine if its production workflows in Kubernetes would be worthwhile
  • Disasterpiece Theater: Real scenario that might happen and surmise what will happen; don’t cause production issues, but teach Slack employees
  • Slack hires collaborative, empathetic people to create a collaborative environment where everyone works together toward a goal
  • Slack was firmly in a centralized operations model, but is transforming toward development teams to increase responsibility and service ownership
  • Slack doesn’t encourage remote work because it’s not in a position to put in that investment; day-to-day work happens in hallways and between desks
  • Slack sees itself as an enterprise software company; an enterprise software company must have enterprise software reliability, stability, and processes
  • Slack has thousands of servers, so events and disruptions happen more often; system needs to respond, react, and repair itself without human intervention

Links:

  • Holly Allen on Twitter
  • 18F
  • Slack
  • Freenode IRC
  • HipChat
  • AWS
  • Kubernetes
  • Terraform
  • Chef
  • QCon
  • Datadog

.

You might also like

More Podcast Episodes

AI Should Solve the Problem, Not Deflect the Customer

Screaming in the Cloud
10.06.2026
31 Minutes
Play Episode

Open Source and the Future of Databases with German Eichberger

Screaming in the Cloud
09.24.2026
24 Minutes
Play Episode

The Invisible Network Powering AWS with Matt Rehder

Screaming in the Cloud
09.10.2026
37 Minutes
Play Episode

Get the Newsletter

Reach over 30,000 discerning engineers, managers, enthusiasts who actually care about the state of Amazon’s cloud ecosystems.

Gnarly cloud cost questions?

Good news: we’ve got answers (and coffee). Meet the Duckbill team for personalized advice on your thorniest AWS challenges.

Sign up for Office Hours
You’re faced with tough questions about cloud costs. You should have confident answers.
Get in touch.
  • Newsletter
  • Podcasts
  • Blog
  • Merch
  • About
  • Contact
  • Disclosures
Billie Footer
footprint-orange
© 2026 L9 Labs, Inc. All Rights Reserved.
Terms of Service Privacy Policy Cookie Policy