Finding CI/CD Nirvana Using Jenkins

Recently, FINE DevOps Engineer Charlie was asked to meet with the team at CircleCI to answer questions about how we use Jenkins, and how we leverage CI/CD to improve our dev processes, all in an effort for them to better help customers who are making the transition from Jenkins to CircleCI. Which got us thinking about how improving our CI/CD process and code has accelerated our own development process.

While we don't build complex micro-services, and aren’t fully cloud-native, we do build in Ruby on Rails using our open source CMS, Fae, and we host hundreds of sites. Which means our version of "scale" is the number of sites we host, and not the standard metric of requests, or the number of users.

We need to push to development, staging, and production environments on any one of hundreds of repositories multiple times a day, with as little friction as possible. Though it’s simple enough, we also need to ensure the jobs stay generic, with little to no drift between builds or repositories. Code needs to make it to the servers. Like, a lot of it. So the process has to be smooth and cost effective. The dream, right?

Making the Choice
With most hosted CI/CD services, they charge based on the number of repositories you build, build minutes you use, or the number of build nodes you need. None of these options really worked for our purposes. But, then there was Jenkins. Since we were already making the server transition to AWS so it could be run on EC2, scaled easily, and support build slaves with Docker, we opted for a plan that cost us roughly $76/month, with up to eight concurrent builds.

We used to only have a dozen or so projects using CI/CD, but now it’s every client, all of our internal tasks, Docker image builds, Puppet testing, and various other clean-up jobs. Our current record is 140 builds in 24 hours across all repositories and environments.

We can go from feature branch all the way to production in less 15 minutes, if it’s moved up the environment chain. All of this didn’t happen overnight. Or even in a couple months. It took many iterations to see this level of improvement.

Befriend Incremental Improvements
Each of our developers has ownership over the code they write, and if they write any bad code that makes it into production — or any other environment — they’ll see alerts, or “failed build,” and be responsible for fixing it. We also use a pull request process for feature adds to help avoid this, but our developers are always responsible to see those changes through after the pull request is approved.

We’ve gone through a few iterations over the last three years, and each has been an incremental improvement to our process. Whether you choose CircleCI, Jenkins, or some other service, you can get a huge boost from putting time and effort into good CI/CD — like shifting your process from manual work into deployments-as-code.

Our main shared library code is now open sourced for all to use or fork as they wish. Each time we add a new piece of functionality, or modify the current setup to make it better, it’s one step closer to CI/CD nirvana. How much you gain from getting more code tested, integrated, and deployed faster than you did yesterday is really the measurement of success.

So thanks to CircleCI for taking the time to talk about our CI/CD challenges and achievements. We’ve learned a lot. Like how it’s a community effort, no matter the brand or software.