Continuous Delivery at scale - social and technical practices for fast flow - Conflux - Agile Manchester 2023

Fast flow delivers vital business outcomes:

* Improved efficiency due to removed dependencies and reduced rework
* Shorter time from hypothesis to user value
* Greater transparency and traceability in service development and operation
* Quicker problem diagnosis and service restoration
* Happier teams

But what are the social and technical practices that enable fast flow?

At the heart of fast flow is the practice of Continuous Delivery (CD). CD practices have increasingly become “table stakes” for effective software delivery since the publication of the book ‘Continuous Delivery’ by Jez Humble and Dave Farley in 2010. The combination of sound technical practices and a scientific approach to testing and feature development has led to significant success with software delivery with organizations around the world.

However, in large organizations with many teams and many different suppliers, there are also many ideas about what Continuous Delivery is. This variety of approaches and assumptions can lead to conflicts around deployments, testing, releasing, and operations, resulting in a macro-level sub-optimal flow of change.

In this talk, Matthew Skelton will share some insights from his time spent as Engineering Lead at a large GOV.UK department during most of 2018. There he championed Continuous Delivery practices across 70+ teams and 7 locations, helping to raise standards for software operability, Developer Experience (DevEx), testing, deployments, and inter-team communications. He will share some practical techniques for getting Continuous Delivery working at scale.

---
From a keynote talk at Agile Manchester conference on 10 May 2023 - https://agilemanchester.net/programme/continuous-delivery-scale-social-and-technical-practices-fast-flow

Transcript

1. Continuous Delivery at scale Social and technical practices for fast flow Matthew Skelton, Founder at Conflux K03 Keynote at Agile Manchester | 10 May 2023
2. Photo goes here Matthew Skelton Founder at Conflux Co-author of Team Topologies 2
3. Team Topologies Organizing business and technology teams for fast flow Matthew Skelton & Manuel Pais IT Revolution Press, September 2019 Order via stores worldwide: teamtopologies.com/book 3
5. 5 Fast flow: multiple ongoing streams of changes to software-enriched services done safely and sustainably https://confluxhq.com/fast-flow
6. Fast flow delivers vital business outcomes 6
7. 7 Improved efficiency due to removed dependencies and reduced rework
8. 8 Shorter time from hypothesis to user value
9. 9 Greater transparency and traceability in service development and operation
10. 10 Quicker problem diagnosis and service restoration
11. 11 Happier teams
12. 12 Improved efficiency due to removed dependencies and reduced rework Shorter time from hypothesis to user value Greater transparency and traceability in service development and operation Quicker problem diagnosis and service restoration Happier teams
13. What social and technical practices are needed for fast flow at scale? 13
14. 14 Make space for learning/sharing Define the Platform Address operability early on Good logging is foundational Empower teams to improve
15. Case study: context & challenges Guiding principles for fast flow What we did to help fast flow Results and key takeaways 15
16. Case study: context & challenges 16
17. Case Study UK Home Office (GOV.UK) 17
18. Case Study 18
19. Case Study 19 700+ people in total 70+ engineering teams, 7 locations Time-critical delivery (#Brexit) Most of 2018
20. Case Study 20 Increase speed & safety of delivery Multi-year programme New Brexit-related work Track & control infra costs
21. Case Study 21 (Traditional DC + VMs + JVM) AWS + Kubernetes + JVM ELK for logs and transaction traces [ ]
22. Case Study Scale existing approach to 7x size 22
23. Case Study Spread awareness of effective practices 23
24. Case Study Cross internal and external boundaries 24
25. Case Study “Continuous Delivery” 25
26. Case Study Avoid “one-size-fits-all” 26
27. Case Study Define the Platform 27
28. Case Study Improve logging quality 28
29. Case Study Multi-supplier delivery model 29
30. What do we aim for in the context of fast flow? 30
31. 31 Continuously ‘untangle’ business concepts
32. 32 Find and adjust team & system boundaries for flow
33. 33 Minimize hand-offs
34. 34 Avoid blocking dependencies
35. 35 Move some decision-making to teams
36. 36 ➔ Untangle business concepts ➔ Adjust boundaries for flow ➔ Minimize hand-offs ➔ Remove blocking dependencies ➔ Move decision-making to teams
37. 37 Example: deployment pipeline architecture
42. 42 The architecture of deployment pipelines is a key consideration for fast flow
43. 43 “Show me your deployment pipelines and I will predict your team boundaries” 💥
44. Guiding principles for fast flow 45
45. 46 Reliable Software Releases Through Build, Test, and Deployment Automation Jez Humble & Dave Farley, 2010 Continuous Delivery
46. Continuous Delivery with Windows and .NET 47 Matthew Skelton & Chris O’Dell O’Reilly, 2016 cdwithwindows.net ** Now a key text for MSc in Software Engineering at UCL **
47. Team Topologies Organizing business and technology teams for fast flow Matthew Skelton & Manuel Pais IT Revolution Press, September 2019 Order via stores worldwide: teamtopologies.com/book 48
48. 49 Re-aligned architecture
49. 50 Fast feedback via deployment pipelines
50. 51 Good technical practices (TDD, …)
51. 52 Team ownership of software & services
52. 53 Configuration in version control (Git)
53. 54 Cloud-native: transparent in operation
54. 55 Cloud-native: designed for automation
55. 56 Continuous testing performance scanning deployment monitoring right-sizing integration
56. Team Topologies Organizing business and technology teams for fast flow Matthew Skelton & Manuel Pais IT Revolution Press, September 2019 Order via stores worldwide: teamtopologies.com/book 57
57. What we did to help fast flow 58
58. Case Study New flow-based operating model 59
59. 60 Flow of change
60. Case Study Focus on the "operator experience" 61
61. Case Study Define the platform 62
62. Case Study Engineering Guild + Engineering Working Group 64
63. Case Study Weekly lunchtime tech talks 65
64. Case Study ‘Continuous Delivery for people who don’t write code’ Slides: https://www.slideshare.net/matthewskelton/continuous-delivery-for-p eople-who-do-not-write-code-matthew-skelton-conflux 66
65. Case Study Team-first techniques Example: Run Book Dialogue Sheet 67
66. 68 runbooktemplate.info Run Book dialogue sheets
67. Case Study Engineering Standards 69
69. Engineering Standards 71 Minimal but important constraints Dynamic for reviewing and updating Emphasise team engagement
70. Case Study Engineering Assessments 72
71. Engineering Assessments 73 6 dimensions [in 2018]: Team Health, Deployment, Continuous Delivery, Flow, Operability, Testing SoftwareDeliveryAssessment.com
72. Engineering Assessments 74 Criteria taken from existing sources: Spotify Team Health Check, DevOps for the Modern Enterprise, Continuous Delivery, Accelerate, Team Guide to Software Operability, Team Guide to Software Testability
76. Engineering Assessments 78 Team self-assessment Facilitated to guide discussions “Viral” facilitation model Results collated across teams
78. 80 SoftwareDeliveryAssessment.com
79. Many of these principles and practices appear in the Team Topologies book 💡 82
80. Team Topologies Organizing business and technology teams for fast flow Matthew Skelton & Manuel Pais IT Revolution Press, September 2019 Order via stores worldwide: teamtopologies.com/book 83
81. Results and key takeaways 84
82. Case Study 85 Software releases From 2 successful release candidate builds per week to 8 builds per day Multiple, independent routes to live
83. Case Study 86 Operability Ops teams love the new "operator experience" with logging & dashboards No major operational problems with new releases
84. Case Study 87 Collaboration Logging beginning to be seen as a key observability tool Better cross-team awareness of how systems work through telemetry
85. Case Study Moving towards Continuous Delivery at DDaT See public blog post at https://hodigital.blog.gov.uk/2018/08/10/moving-tow ards-continuous-delivery-in-ddat/ 88
86. Case Study 89 (Many other people were also involved)
87. Case Study 90 Ideas and patterns from Team Topologies now being used in this department
88. What social and technical practices are needed for fast flow at scale? 91
89. 92 Make space for learning/sharing Promote good work Help to develop skills in speaking Lunchtime talks (food) Interplay of working group + guild
90. 93 Define the Platform Service Level Expectations Who owns what? What's missing? Use DevEx to drive improvements Thinnest Viable Platform
91. 94 Address operability early on Operational aspects are also features Add operational checks into the deployment pipeline Team-first: RunBookTemplate.info
92. 95 Good logging is foundational Use a well-defined event space Searchable EventIDs increase context Enables collaboration between teams Invest in logging infra (Saas/people/&c.)
93. 96 Empower teams to improve Co-create the engineering standards Look for organisational blockers “No-blame” is paramount SoftwareDeliveryAssessment.com
94. 98 The architecture of deployment pipelines is a key consideration for fast flow
95. Team Topologies Organizing business and technology teams for fast flow Matthew Skelton & Manuel Pais IT Revolution Press, September 2019 Order via stores worldwide: teamtopologies.com/book 100
96. expert-led group learning ❏ P602 - Software Delivery Assessment ❏ P616 - software reliability for product managers ❏ P630 - Find possible streams with Independent Service Heuristics ❏ P634 - Peer Learning - Writing Tech Talks ❏ P641f - Preparing to adopt Team Topologies ❏ P641i - Skills paths for fast flow - leaders and HR 101 online workshops for up to 20 people
97. 102 expert-led group learning for adopting fast flow and Team Topologies confluxhq.com
98. thank you confluxhq.com Copyright (c) 2017-2023 Conflux group of companies. All Rights Reserved. The name “Conflux” and the filled C device are Registered Trademarks ® in multiple jurisdictions. 103