DevOps and CI/CD are often used interchangeably, but in reality, DevOps is much broader than a single tactic or tool. CI/CD refers specifically to continuous integration and continuous delivery of code. But DevOps is truly rooted in continuous everything — continuous improvement of continuous planning, building, testing, deploying, and monitoring. Ultimately, when you adopt a DevOps mindset and methodology, you’re posed to continually improve both the quality of your work and the ways in which you work.
Here’s the catch. Most companies don’t approach DevOps with the goal of achieving continuous improvement. Rather, they simply seek to implement CI/CD. But those who build on CI/CD with process improvements, alignment across teams, agile adoption, testing automation and more, are the companies who maximize their ROI and outpace their competition.
Over the past five years, Cox Automotive has embraced continuous improvement to mature their DevOps processes and increase their business results. They’re expansion beyond build and deploy tactics has helped them establish a single source of truth for Salesforce, improve feature quality, and standardize processes across 8 agile teams.
|Results in 2017-18||Results in 2021|
The increased productivity and quality not only make for a better development process, they also improve team morale and create additional business value. The process standardization across Cox Automotive’s 5 Salesforce orgs has upped team confidence. Developers can now work seamlessly across development teams without overwriting each other’s work.
“Before Copado, we were losing development hours every month, and we had zero confidence in our releases. Now, we’re pushing new features into production almost everyday.” — Todd Bussey, DevOps Enablement Manager, Cox Automotive
These results speak for themselves, but if you’re just getting started with DevOps, thinking beyond the first step can be overwhelming. Still, planning for the next phases of DevOps helps manage expectations and build toward long-term goals. Let’s take a closer look at how Cox Automotive matured their DevOps practices over time.
The Beginning: Streamlining Release at Cox Automotive
When Cox Automotive came to Copado in 2017, their challenge was clear — they’re release processes were manual and error-prone. They needed a structured approach that allowed them to move away from changesets and stop making updates directly in production in order to increase the quality of their deployments.
For Cox Automotive, the solution was a combination of Copado’s user story-centric approach and admin-friendly, configurable deployments.
- Copado’s User Story records allow teams to track metadata changes directly to an item of work. This relationship between stories and metadata changes helped Cox reduce overwrites and increase visibility into where changes were being made.
- Since changesets aren’t tied directly to a User Story, Cox had struggled with deploying partial functionality when team members would accidentally forget to add metadata changes to the release. Copado Continuous Delivery allows Cox to ensure all metadata changes for each user story are deployed between environments, eliminating both lost work and rework.
- Lastly, the updates occurring in production were leading to overwrites and syncing challenges that resulted in bugs and lost work. Copado Pipelines made it easy for Cox to configure their deployments. And the native Salesforce UI/UX allowed both low-code and pro-code team members to follow the process rather than making changes directly in production.
This streamlined approach helped Cox unlock more confidence and consistency in their releases. But they didn’t want to stop there. They knew if they continued improving how they work, they’d improve both their quality and velocity. And in 2019, they set out to make four critical improvements.
Improvement 1: Make GitHub the Single Source of Truth
Like many Salesforce customers, prior to adopting DevOps, Cox Automotive’s production org was their source of truth. The challenge with this is that your Dev, QA and PreProd orgs are always “behind” production, so when you deploy from a staging sandbox to production, changes that have been made directly in production get overwritten. Additionally, using your production org as your source of truth severely limits your ability to rollback changes. To tackle this improvement, Cox Automotive took a 3-step approach:
- Governance: They implemented a policy that required all admins and developers capture all changes in a User Story, and then promote that story via the Copado pipeline.
- Version Control: The Copado pipeline was updated to sync all changes to GitHub, ensuring all changes were tracked and stored with a full audit trail.
- Release Versioning: They then introduced Copado Release Versioning, which places a tag in the GitHub branch every time a promotion to production occurs.
Together, these 3 updates allowed Cox Automotive to shift their source of truth from their production org to GitHub, enabling rollbacks and greatly reducing overwrites.
Improvement 2: Implement Continuous Integration
After GitHub was established as the single source of truth, the next step was to set up a branching strategy to ensure teams were building on the most up-to-date code. Their strategy was two-pronged:
- Sandbox Synchronization: Each sandbox needed to mirror the GitHub repository.
- User-story Branches: When work began on a new user story, the developer needed to be working from the most recent code.
Copado’s Continuous Integration tool allowed Cox Automotive to tackle both of these requirements. Copado back-promotions automatically migrate work between developer sandboxes. The feature also allows admins and developers to create a developer sandbox with the most recent version of code directly from a user story. This mechanism closely mirrors branch management used by open-source tools while also working within the best practices of the Salesforce platform.
Ultimately, implementing Continuous Integration prevented lost work and reduced the amount of rework the teams were doing. Cox Automotive has multiple teams working in parallel within the same orgs, so the ability to both work from and commit to a centralized repository helped the team become more efficient and produce more value for the business.
Improvement 3: Code & Feature Quality
The concept of continuous improvement in DevOps goes deeper than CI/CD. Just as version control and CI allows companies to build and deploy earlier in the development process, it also allows businesses to shift testing left. And when development teams test earlier in the process, they catch and rectify more functional and code-quality issues, ultimately streamlining upstream deployments and resulting in more usable features.
Following on key improvements to their build and deploy processes, it was time for Cox Automotive to tackle quality. The first prong of their approach involved configuring two quality gates between their Dev and QA orgs in their Copado pipeline:
- Apex Testing: They now enforce Apex unit testing before each promotion, and they’ve set coverage minimums to help them track and enforce this standard.
- Copado Selenium Testing: They’ve built pre-identified tests into their pipeline that are run between QA and PreProd, helping stealime final testing and production deployments.
Together, the quality gates and testing improvements have helped the team go from frequent overwrites to daily accurate production releases. And they’re planning to build on these efforts with automation. Next, their team will look to automate test triggers, particularly between QA and PreProd with the goal of optimizing quality and reducing lead time for new feature requests.
Improvement 4: Process Standardization
The version control, continuous integration and quality improvements came together to create a process that spanned their full dev lifecycle, and it was time to set up governance across all 8 agile teams.
Their process is rooted in 5 tenants:
- All team members — low-code and pro-code — must use Copado User Stories to promote all changes from dev orgs to production
- All changes must be committed to GitHub.
- Quality gates for promoting between Dev & QA are both required and built into the Copado pipeline
- Changes must pass Selenium tests to be promoted between QA & PrePrd
- No More Sandbox Refreshes: Copado back-promotions are used exclusively
“One of the biggest wins with Copado has been standardizing tools and processes across all our scrum teams. Process standardization has made DevOps seamless across all our teams and orgs.” — Kevin Mest, Director of Software Engineering, Cox Automotive