Did the Chaos Test Pass?

People used to ask me all the time how to figure out if their chaos test has “passed,” and I’d always say “well, that’s a loaded question.” To confirm that a chaos test “passed,” we need to do verification of hypotheses - sometimes you’re trying to prove some system behavior occurred in response to a stimulus, while other times you’re trying to prove the absence of a change in system behavior. Take this already nebulous concept, and now think about making it generic enough that the core validation logic can be re-used by any engineer running any kind of experiment on any one of our products. Then, try to do all of this in a complex distributed technical environment where it’s hard enough just to determine whether an application was healthy in the first place! That’s exactly the problem that the chaos engineering team at Vanguard has been tackling with the recent addition of automated assertions to the internal chaos tooling. In this talk, you’ll learn about when it’s appropriate to define “pass” and “fail” for a chaos experiment, and when it might not be, and you’ll get to take a peek under the hood at the way that Vanguard engineers are automatically verifying their hypotheses in the context of chaos experiments.