Notes from Enterprise DevOps Summit 2017 - San Francisco
I was privileged to be sponsored by Akamai to attend the Enterprise DevOps Summit in San Francisco. Here are the notes from the sessions I attended.
All the resources for the conferences were already shared. Here are the links:
Day 1: 11/13/2017
Keynotes
More Culture, More Engineering, Less Duct Tape
Tranformation methodology is very similar to Improvment Kata
Augmenting the Org for DevOps
- significant outage in 2015. Driver and accelerator for DevOps movement
- application had over 200 hops on the internal network for a single interaction - this was the application that had to move to cloud
Methodology: Monolith to 3 tier app, re-skilling org to Agile processes
Critical Success factors:
- get CXX support
- metrics based
- changing team structured from contractors to employees => end to end engg rather than narrow focus.
Manual testing was the primary bottleneck (20 hrs). Automation reduced it to 12 mins.
Automated regression testing
- grass roots bottom-up approach
- know how to pitch upwards (eg: talk about reduced lead times instead of no of deploys a week)
Containergeddon - everything is moving to containers.
Used circuit breakers and Netflix hysterix code
DevOps value: During an outage, 10 UI changes were released in 3 days. Even the CEO made statements on the success of DevOps movement.
Using DevOps to Build Your Learning Organization
- Columbia Sportswear - Family owned business.
- DevOps movement was implemented without a CXX sponsor / support.
- DevOps born out of the need to stay relevant and to “do the right thing”
Challenge: Microsoft shop with vendor lock-in. Very UI driven and lack of automation.
You are either building a learning organization or losing to someone who is. - Andrey Clay Shaffer
Be -> Know -> Do model of leadership
- Version control was a big game changer
- Go for small wins using the Plan-Do-Check-Act (PDCA) model.
Referenced a velocity talk about Ops in one of the roles:
- rockstars
- builders
- janitors
“Weekly retrospective” added a huge value and gave a lot of insight on how to improve from grass-roots.
ChatOps - game changer
Anti-pattern to most others: If some task is not common, then automate it. Since it is not repeated often, potential to make an error is high.
Transformational Leadership and DevOps - Beyond the Research
Cultural change is the hardest change. Success rate of just 19%. But, ability to change is the core competency of organizations.
Why do transformations fail? Confusion -> Frustration -> Anxiety -> Fear (now people are at the lowest tier of Maslow’s pyramid) -> Resistance
Books referenced:
- The Startup Way
- Turn the ship around
Middle managers are the ones most impacted by change. However,
- They know the processes,
- have the network and
- can point to places where skeletons are buried. This is very important in transformation so that mistakes aren’t repeated.
Breakout sessions
What Does it Mean to Lead IT?
Historically, IT department was an “add-on”. Business went on fine w/o IT. Streotypes created for IT:
- uninterested in business
- don’t care about timelines
- can’t even speak in English (as in can’t speak in terms understood by others)
So, IT had to be provided with exact requirements. IT had to respond with goals, timelines and budget. Business would control IT and extract value. IT was never considered an integral part of the org. ==> Waterfall model.
“IT’s” customer is business mentality. As though, IT did not contribute anything to the actual product.
CIO’s office created to:
- talk English to business
- come up with measurable goals, even if Biz did not understand what the goals meant
- enforce timelines and budget
No other department in the org is “controlled”. They have freedom to operate and do the necessary things to help business succeed. Not IT.
Even with Agile and DevOps, the attitude hasn’t changed much. It is about business controlling IT.
New model
Instead of requirements, provide high level objectives. Let IT be a part of the strategy discussion. Since teams are already multi-functional, no need to control.
Role of the leadership:
- provide clear objectives
- remove impediments
- monitor the progress towards business objectives
Continuous Chaos in DevOps
Build chaos engineering tools as a foundational service. Every application has to “subscribe” to the tools and specify the components. This will identify the right failure services to use. “Failure as a service” model.
DevOps Handboook Experiments in Accelerating Delivery
Lessons implemented from the DevOps hand book:
Instead of “presenting” to management, rely on “Show and Tell”. It provides better grasp to management.
One of main benefits of DevOps - automation testing reduced the lead time.
Architecting Your App And Your Pipeline for Continuous Delivery - 10 DOs for Successful DevOps
One of the best predictors of high performing organizations - “how much do they fear deployments”?
Another one: Binary artifact repo - indicates maturity for devOps success.
Very technical deck. Extremely useful for development teams / architects. Speaks about software patterns to build code and for the pipelines.
Some good resources from microservices.io
Follow the @honest_update for some fun status messages.
Day 2: 11/14/2017
Keynotes
Better Governance - Banking on Continuous Delivery
Goals for 2017 and beyond:
- slayTheMonolith
- noFearRelease
- youBuildItYouOwnIt
NoFearRelease
Typical fears are:
- fear of speed
- fear of breakdown
- fear of being out of control
- fear of being non-compliant
Compliance is a bad word. Governance is better. It implies awareness and ownership of risk.
3 lines of defense:
- 1st line: Who owns the risk?
- 2nd line: policies and processes
- 3rd line: independent assurance
Minimum set of controls:
- Two sets of eyes (peer review
- Least privilege
- Unauthorized change monitoring
Being bank, the question from auditors were: How to assure that the pipeline itself is not tampered, the process is not broken or the pre-commit checks were not relaxed for a driven deployment.
In terms of deployments, Kubernetes / containers provide immutable servers. And no human access is provided to production systems.
Opensource tools: CapitalOne
The Making of Amazon Prime Now
i had to work so could not focus much
Making Digital Magic
Digital transformation involved:
- Technology
- Leadership
- Community very similar to the People-Process-Technology model in Agile
Mesos - what is it?
Technology for Movies Anywhere
- cloud native
- ops deeply embedded into design
- Kubernetes with Kops
- Terraform
- helm, vault
- Lambda functions used for monitoring
- PagerDuty, NewRelic and Data dog for monitoring
Reference to book - What Got You Here Won’t Get You There
Breakouts
Intel’s Journey to Build Quality In: How QA and Test Automation Drive DevOps Transformation
Initial model push and pray. 85% of code was not covered.
Parameterization of tests and using environment variables made a big difference in the usabilty of tests.
On-demand is the key. Environment needs to be spun up and down with no tickets and wait times.
Cloud Native Architectures
Just download the presentation. It is simply too good. Try to get the book as well. Slides
Starting and Scaling DevOps in the Enterprise
Very informative and down to earth presentation on DevOps transformation. Download slides for future use.
CAB is the “change resistance board”!
Today’s approach to DevOps:
I am a millenial. My mom still cleans my room and ops will clean up after me.
In today’s deployments, more time is taken to setup the environment and run the checks than the actual coding.
Repeated manual tests: (3 Ms )
- waste of time (Muda)
- inconsitancy (Mura)
- burden on testing team (Muri)
Book - Jeff Morgan on testing
Action: Sign up for newsletter and get the book free.
DevOps: From Analyst Inquiry to Organizational Action
Trends
- Last year questions were about - “What is DevOps”?
- This year, DevOps has reached escape velocity.
- Trends for this year:
- Dev and Ops are now working together.
- Security needs to fit in.
- Data / database is still outside of the scope. This will increasingly become the bottleneck and requires newer architectures.
- In last 90 days, Business is coming back and saying that deployment are too fast. Net result: Deployment frequency is being re-defined as “on demand deployments”
- Technologies: Gitlab, Jenkins, Kubernetes and Jira - 80% of organizations use them. However, what’s missing is end-to-end tool chains.
- Insurance, Banking and FinTech - DevOps is smoking hot.
Again - download slide for future use.
DevSecOps: It’s Not Me or You, It’s WE!
Panel discussion..
Security is traditionally “command and control”. Sec teams tend to say - “here’s the code / process. Just follow it.” No longer the right approach.
Security companies still stuck in last century of request/response or client/server model.
Follow NIST security framework instead of ISO. ISO is too burdensome.
Approach for security:
- what security incidents were handled
- what security incidents were failures
- Pen Test and take action items and fix the bugs.
Closing tips:
- Break silos between security and devops teams
- Practice often: War games, incident response
- Security should be “built-in”. It should be part of the product design and part of the build pipeline. Security team needs to understand the business objectives.
- Know your bill of materials, i.e., Configuration Items should be clearly known. This will help address issues like struts vulnerability.
Finally, aim for “security as code”. Although not currently a feasible solution, this is the end goal.
The Key to High Performance: What Data Says
All about the State of DevOps reports. Trends:
- 2014: DevOps Works!
- 2015: IT goes lean
- 2016: Shift left
- 2017: DevOps works for all kinds of organizations.
Other notes:
- Measure Cycle time: code commit to code deployment.
- Throughput and stability are possible without any trade offs.
- Maturity models don’t work in this industry. “What is a mature org?” - metrics itself keep changing very fst.
- Architecture matters - technology stack doesn’t
- Westrum’s organizational model and transformational leadership - very good predictors of DevOps successes.
Read the HBR article by Besson on Real Reason Superstar firms are racing ahead.
Coping With Complexity: Resilience Engineering Research In DevOps
Action item: Download and share the stellar report.
incidents of yesterday inform and the architecture and rule of tomorrow.
Check out adaptivecapatictylabs.com
Watch the video again. It is amazing!
Day 3: 11/15/2017
Keynotes
The human factor: Inspiring the pursuit of success and averting drift into failure
Session by Dr Sidney Dekker
- the company / airline that reports the most incidents has the least number of catastrophic accidents ==> open culture and less fear of being honest
- sweet spot of honesty exists for organizations.. if you stress beyond it, catastrophic results occur
- award for speaking up is generally unclear and delayed. award for not speaking up is generally immediate
What’s difference between organizations that screw up and those that don’t screw up? It is the presence of positive capacities and not in the absence of negative capacities. This follows John Allspaw’s talk. When trying to find the root cause, analyze why the incident had not occurred more times - what causes the lack of incidents.
Understand how things go right to figure out processes on when things go wrong.
Fear does not exist in the dojo - a devops journey with a competitive twist
DoJo - creating an internal training environment. Process cross-pollinated to Verizon from Target.
6 weeks workshop:
- 2 day Sprint
- 12 sprints
- Clear outcomes of the effort
- Up skilling of employees to new ways of working, esp in global org Note to self: Can our consulting work-shop be modeled on this?
Internal DevOpsDays - a very good way to engage, encourage and learn.
Gamification of DevOps and get teams excited.
There is No Finish Line
No “devOps” team - everyone has to adopt it.
Breakouts
The Case for Value Stream Architecture
Competition is not across the board. It can be at specific value stream. Case point -
DevOps is big and there are a lot of startups
DevOps / Lean tries to take the analogy of assembly line. Although accurate to some extent, it does not capture the complexities of software developement:
- Car assembly lines can stay stable for about a year. Software keeps changing.
- Car manufacturing can follow linear steps. Software release is not all that linear.
- Software value stream looks more like an airline network with some hubs seeing more dense connections than others
- Replace airports by tools and process - this is a more accurate representation
Waterfall and very narrowly focused DevOps movement is trying to force linearity where linearity doesn’t really exist.
Sample work-flow at Nationwide:
Download Value stream architecture eBook
Download book - Thinking Environments
Enterprise DevOps and Unicorns
Fear / change approval is a mind killer.
Process-as-Code: Real-World Examples that Scale
Treat your automation process as a product / software artifact. So it goes through rigorous testing, validation, enhacements and bug fixes.
Follow-up:
- what is groovy
- what is Domain Specific Language (DSL)
- what is vagrant
Baking Security into your Pipeline: Start Here
Embed security experts in SCRUM meetings.
Hacked pipeline is very scary
- when plugins are used in Jenkins, who checks for vulnerabilities in them?
Credentials should be in vault and not hard-coded or in plain-text configuration files.
Work on reducing blast radius by reducing the impact of a compromise.
What is SonarQube?
80% of official (latest) docker images had at least 1 severe vulnerability
Configuration item + auto patch
Keep testing your process. Process as code means treat it as:
- version control
- test
- log
- audit trail
Microservices for the Enterprise: Myths vs. Reality
This session seemed to be put together without any real agenda or focus. Very boring :-(
Convergence of Safety Culture and Lean: Lessons from the Leaders
Panel Discussion between Gene Kim, Dr Sidney Dekker, Dr Steven Spear and Dr Richard Cook.
Leaders -> enable team. Not control it. Leaders need to say:
- I don’t know
- I need help If this is not seen by org, then learning culture will never take effect.
Most important learning from Toyota Production System - Andon Chord ==> recognize that the known processes is not able to cope up with current work processes.
Guidelines to Community
- Tell stories, esp when things go wrong. It helps sensitize community on what can go wrong.
- DevOps is about building community of users who work on devOps. Mentor others.
- Share stores of failures and near misses.. Success comes from experience and experience comes from failures.
- You are all experts and you have moral responsibility to share it with the community.
- Identify yourself as one who practices DevOps instead of identifying as someone working for a particular role in a company
- Share stories of uncertainty
Closing Keynotes
Data and DevOps: Breaking Down the Silos
Data is still in silos.
- DBA
- Data Engineer
- Data Scientist
Read up on 12 factor app
Cloud Native architecture for data.
The Yin and Yang of Speed and Control
Read up on Cynefin framework
Lessons at Barclays bank:
- Don’t scale work. Create smaller teams and batches first.
- Don’t impose an Agile transformation. Focus on outcomes - start with why
It is easier to act your way to a new way of thinking rather than think your way to a new way of acting.
Impediments is not in the path but impediments is the path.
Every requirement should follow a Hypothesis Drive Development by @barryoreilly
Books referenced
- The Startup Way
- Turn the ship around
- What Got You Here Won’t Get You There
- Cucumber and Cheese - A Tester’s Workshop
- Cloud Native
Key trends
- DevOps is in full-swing accross industries
- Cultural transformation is the hardest but, it is necessary.
- Focus is shifting from plain “deployment frequency” to being ready for “on-demand” deployments.
- Security, esp security of code and the pipeline is gaining attention.
- Maturity in-terms of expanding the DevOps to mean more teams like audit, marketing, finance and legal teams
- Data is the hardest problem to move to cloud based architecture. This is going be to an ongoing item for investigation.