Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critical Failures
Search
Caitie McCaffrey
June 14, 2016
Technology
8
400
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
210
Argus Papers We Love
caitiem20
13
1.1k
The Verification of a Distributed System
caitiem20
22
2.1k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
710
The Verification of a Distributed System
caitiem20
12
680
The Verification of a Distributed System
caitiem20
6
690
A Brief History of Distributed Programming: RPC
caitiem20
31
6.3k
Building Scalable Stateful Services
caitiem20
12
1.3k
Other Decks in Technology
See All in Technology
Prisma ORMを2年運用して培ったノウハウを共有する
tockn
19
5.1k
Password cracking: past, present, future
openwall
0
270
シンプルなHITL機械学習と様々なタスクにおけるHITL機械学習
naohachi89
0
320
Step by Stepで学ぶ、ADT(代数的データ型)、モナドからEffect-TSまで
leveragestech
1
3k
グイグイ系QAエンジニアでやっていくよ!
____rina____
0
770
令和版ソフトウェアエンジニアの情報収集術 PHPカンファレンス香川2024
ysknsid25
4
890
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
15
36k
Laboratories in Science and Technology: Deep Neural Networks
keio_smilab
PRO
3
150
大規模言語モデル (LLM)における低精度数値表現
pfn
PRO
3
830
ハードウェアを動かすTypeScriptの世界
9wick
3
1.2k
エムスリーQAチーム紹介資料 / Introduction of M3 QA Team
m3_engineering
1
330
拓展QA日常工作的邊界
line_developers_tw
PRO
0
550
Featured
See All Featured
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
51k
Agile that works and the tools we love
rasmusluckow
325
20k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
21
2k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
67
14k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
41
4.5k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
188
16k
Statistics for Hackers
jakevdp
790
220k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
26
2.3k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
7k
Raft: Consensus for Rubyists
vanstee
133
6.3k
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie