Deployment is a defining moment for engineering teams.
Not only are the stakes the highest, but the act itself is a test of confidence and a leap of faith. As such, deployment offers the most authentic peek into a team’s culture.
Built In Chicago asked four local engineering leaders about their advice, rituals and superstitions on deployment day to get a sense of what their teams are like. While some engineering teams take a straight-laced approach to game day, others are more irreverent, reveling in the potential for disaster and cracking jokes to ease the tension.
“Before a production deployment, it’s absolutely crucial to prepare your body and mind for the impending trials,” cracked Yello Senior Software Engineer Dan Borden. “To start your day, eat a healthy breakfast, go for a run and be sure to wear your bespoke deployment underwear to ensure that you are set up for success.”
Joking aside, deployment is the culmination of the hard work of development. While it can be both exciting and stressful, four Chicago managers have different tried-and-true strategies to make launches successful — or at least to bring levity to the situation.
These featured engineering teams all approach deployment differently — with celebratory patches, potted plants, moral support and level-headedness. Read on to hear about the key ingredients to a successful deployment and the variety of ways these engineering teams commemorate the occasion.
Yello is a talent acquisition software.
What are the key ingredients that go into a successful deployment into production?
A successful deployment requires clear processes, clear minds and a mature and well-designed software development life cycle. We’ve taken an iterative approach to our SDLC and gone from “engineer pushing code from a laptop” to a full continuous integration and continuous delivery suite with automated testing.
When performing a mission-critical and highly sensitive procedure like a production deployment, it’s important to consult your well-maintained and thorough documentation every time. Don’t have documentation? Create your own checklist and build it out over time. When someone new wants to start deploying you’ll have a great start.
In addition, building and maintaining a full software development life cycle is critical to setting yourself up for successful deployments. When processes break down, our customers are the ones who will pay for our sloppiness. At Yello, that means a valuable candidate might not get hired.
Finally, a production deployment should not be performed without the watchful eye of a skilled test engineer. Having someone involved with the deployment who is not motivated to release features is a crucial check on engineers who want to move fast.
Having someone involved with the deployment who is not motivated to release features is a crucial check on engineers who want to move fast.”
What are the most common mistakes engineers make when deploying code?
When I was but a Padawan, I remember wanting to get involved in the deployment process. It seemed like a good way to stand out. I was so excited to be a part of the deployment that I forgot to merge our release code down to main before deploying. The deployment was flawless — since no new code was introduced — and so I patted myself on the back for my own competence. Needless to say, a few days later our clients started to ask us when we were going to release our promised features. I stood out from my peers, but not the way I hoped.
Software developers are motivated to get their code into production as quickly as possible, which can lead to carelessness or sloppiness. A single missed step can often lead to problems that are not immediately apparent. The missed step can be minor, like logging a release, or major, like not properly merging code prior to deployment.
A more subtle mistake that is relatively common is when individual engineers subvert the software development life cycle for the sake of speed. These problems are particularly hard to catch and correct because they can result in code problems that do not immediately appear to be related to deployment.
Outside of concrete preparation, do you or your team have any superstitions or rituals that you observe before deploying code?
I’ll add on to what I said earlier: Once you’ve logged your first successful deployment, make sure to wear your bespoke deployment underwear (BDU) every time and never wash them for any reason.
TackleAI builds technology to help businesses extract, validate and classify unstructured data.
What are the key ingredients that go into a successful deployment into production?
Production code should be clean, modularized and efficient in its task. From the time of development to deployment, the code should go through a series of quality and optimization steps.
We have automated pipelines that accelerate our deployment lifecycle called continuous integration continuous deployment. A runner, or a computer process, takes the code committed for production and containerizes it to be served and tested on any computer.
Having a machine handle a bit of our deployment saves time but the machine doesn’t know if the code is fully functional. After the code is working as expected in the development environment, a pull request is created where at least a second pair of eyes in the team approves of the quality and function of the code. The code is then ready for production.
What are the most common mistakes engineers make when deploying code?
Some mistakes that engineers make during deploying code can be easily fixed and reduced through proper software development practices. Here are a few mistakes that can be found in the wild.
- Pushing code to production on a Friday evening. Never push code to production on a Friday unless you really want to work over the weekend. If something breaks you either have to revert to a previous version or dig deep into the code in debugging mode.
- Hardcoded configurations in code. Having a configuration file where all constant values are stored is a must. One location for configurations really helps programmers change constant dependencies and saves time excavating values from a sea of code.
- Unused dependencies in the requirements file. Having a requirements file with a lot of unused dependencies slows the CI/CD build process down. Being able to make changes quickly and test the build really speeds up the debugging process while getting ready for production.
After taking the services down, the whole team gathers around the laptop deploying the code to production.”
Outside of concrete preparation, do you or your team have any superstitions or rituals that you observe before deploying code?
Nothing prepares you or the team for the day of deployment. It’s a whole-team-on-alert operation where it all depends on how the team has produced clean and efficient code with logging during development. Notifying all other teams in the company that the production server will be offline for upgrades always comes with a wave of anxiety. One hopes no one is pointing their applications to that server. At that point, multiple pull requests have been merged and the staged production space is in a state of limbo. After taking the services down, the whole team gathers around the laptop deploying the code to production. Waiting for someone to push the button, someone then volunteers by mustering up the courage and pushing to production.
Mastery Logistics is a transportation management system.
What are the key ingredients that go into a successful deployment into production?
I like to use an analogy that a successful deployment is like putting a puzzle together. It’s important to focus on the placement of the individual puzzle pieces. At the same time you need to understand the big picture of the finished puzzle. To get a successful release out, it takes many different teams to all do their part. The individual teams must understand how their work contributes to the overall release and how their portion impacts the work other teams must do. It’s all about teamwork and communication.
What are the most common mistakes engineers make when deploying code?
It’s important to avoid a mindset that the job is done because code has been deployed. Another common mistake I have seen in my career is making assumptions and not asking enough questions. People think and do things differently so asking questions is imperative.
It’s important to avoid a mindset that the job is done because code has been deployed.”
Outside of concrete preparation, do you or your team have any superstitions or rituals that you observe before deploying code?
If you’re a sports fan, there are times when you are confident your favorite team will win but for some reason they don’t. For me, this carries over to releases. One thing I have learned is to avoid feeling good or bad about a specific release. Have expectations, be prepared, but expect the unexpected. Games have to be played and releases have to be released. Things may go wrong and we learn and grow from them.
fuboTV is a global live TV streaming platform. Fubo Gaming, their newest division, is the home of Fubo Sportsbook, an interactive sports wagering platform.
What are the key ingredients that go into a successful deployment into production?
For us it’s communication: what gets into the release branches, when it gets shipped, who needs to know about it and when do they need to know about it. It all comes down to having efficient and consistent communication around releases.
What are the most common mistakes engineers make when deploying code?
Aside from any technical mistakes, not communicating business impact by the code that’s released can be a common mistake. Requirements are gathered and solutions are built in code. It’s important to communicate what that code actually means for the business and our shared goals. Celebrate and communicate your team’s successes.
We celebrate our launches in each state with some token for the release.”
Outside of concrete preparation, do you or your team have any superstitions or rituals that you observe before deploying code?
There’s no gong ringing or anything like that involved, but we celebrate our launches in each state with some token for the release. In Arizona, everyone was presented with a cactus.
For our first software launch we presented the team with a launch patch similar to a NASA mission patch.
