Chaos Engineering with Gremlin

• Schedule a game day (testing when systems go down)
• Develop a hypothesis of what might happen.
• Communicate plan
• Execute during the day, make sure all teams are available
• Document results (Restore system if/when things fail, document those)
• Learn from mistakes and make the system better

Resources:

• stress-ng, comcast, vegeta for Linux load testing

Avoiding Shiny New Object Syndrome

• Communicate
• Research and Understand new products
• Use a check list to test durability before deploying to production
• Work together to build better technology products

Documentation the Underdog

• It's easy to get used to chaos
• Write docs not code
  • Interractions become, "Have you checked the Docs?"
  • quick turn around to create quick value
• Create reference architecture
• Different types of documentation:
  • User guide
  • Technical Guide
  • contributing guid
• Get feedback on documentation from users