Tackling Tech Debt with the "Frontend Guild"

This post originally appeared on the Constant Contact Tech Blog.

I love working at Constant Contact. But like any engineering group — especially at a mid-sized, 20-year old startup-gone-public — technology can easily lag over time, drift between teams, and generally become a hassle. This time last year I wanted to do something about it, specifically for frontend tooling since that’s what I work with every day.

Your team trying to keep up with tech debt

Fast-forward one year. I’ve been leading a group of fellow engineers that we call the Frontend Guild. Together we’re making tech debt reduction and tooling improvements a full-time effort, not an afterthought. We still have a long way to go, but our work is already impacting teams across the company. Keeping up with changing technology is tough, but we’re more ready for it than ever.

Leading and organizing this group has been an incredible learning experience. Here’s the short version of how I ran the group, the successes we had, and the challenges we faced.

Before The Guild

It all began with some friendly complaining between coworkers. When overall you enjoy your work environment the negatives really stand out. For us it was frontend tech debt that we continued to gripe about, the stuff that kept us from doing our best work.

Shouting alone doesn't get us anywhere

After talking it through a number of times, we identified some key issues: * How do we keep frontend tech up-to-date? We’re not talking about hopping on the latest framework hotness. We mean tools and techniques that can simplify writing and shipping JavaScript: ES2015+, Babel, Webpack, modular code, etc. * How do we bridge the gap between teams? We’re pretty good at coordinating work across teams. But our build tools became out-of-date and ownerless, so no team had stepped up to fix them in a few years. * How do we do this with as few resources as possible? We were still adjusting from some major company changes last year. So having the engineering group drop everything to support a tooling modernization wasn’t gonna happen.

We’ve formed specialized groups before. For instance, Constant Contact has a “Continuous Delivery” team who develops tooling and standards, but their expertise is in backend apps and deploy processes. By this point the frontend teams that cared about dev tooling had all separately begun to upgrade their build setups independently. This of course resulted in several different implementations instead of a single standard.

Previous attempts to get teams working together on tech debt had fizzled out without making progress. But for whatever reason, my coworkers and I thought we could do better! I talked to my manager and our director about forming another group to tackle our tech debt. They both supported the idea — it was a low-stakes way of changing and improving a very busy engineering organization.

With management’s blessing and a quorum of eager engineers on board, we could begin. We called ourselves the “Frontend Guild,” held our first meeting, and wrote up a mission statement:

The Frontend Guild is a group of engineers who want to push for Stability, Maintainability, and Modernization on frontend projects across the organization. We will do this by providing shared tools, patterns, and resources for teams across the company. Together we will make tech enablement a dedicated effort at Constant Contact.

Right away…

Knowledge Gathering

And so the Guild came into existence. There was no shortage of challenges, but we didn’t know where to begin. How would we know what issues were really important?

Simple: just ask! We came up with a “Tech Inventory” survey for all teams to fill out, with about 25 questions regarding the state of their projects and tooling: library versions (jQuery, underscore, etc.), Node versions (for tooling), ES6 or ES5, etc. We also gathered opinions such as “how happy are you with your build setup” or “do you find your UI automation reliable.”

After some nagging we got nearly 30 responses, representing most frontend apps created over the last 10 years. Even if the Guild idea were to fall apart here we would have done something useful — in taking a rough inventory of technology across the company, we can provide awareness for both engineering and management.

We learned a lot, very quickly

We followed this up by asking each team to give a presentation about their projects to the Guild to get some more in-depth info than the survey. They reiterated their tech stack, shared some “pride points” (things they do well), and listed some major technical challenges or blockers.

We kept our meetings bi-weekly so as not to overwhelm anyone. Each team was represented and so awareness of the Guild spread. Attendance was good in these early days, too, since everyone came out to support their team’s speaker. The presentations rounded out our knowledge from the survey and got us ready for the next exciting phase…

In the first month…

Backlog Grooming

Okay, no one really finds backlog grooming exciting. Our meeting attendance began to slip pretty heavily, from a rotating group of 20+ people to a core group of around 8. But we had sufficient cross-team representation that one team wouldn’t be making decisions for the whole company (or lead to unnecessary fights later on). And someone had to turn all our gathered knowledge into something useful!

Backlog grooming? Let's look at cats instead!

After pouring over the survey we came up with a bunch of categories such as: * Enable ES6, Sass, ESLint, ESDoc, etc.: Upgrade our build tooling to enable the latest tech for developing and testing. * Webpack and “Modularize” Code: Enable proper dependency resolution, simplify code bundling, and enable the development of smaller, sharable code modules. * App/Component Generators: Allow developers to create new projects - and upgrade existing ones - using a command-line tool.

Potential topics for the Guild to tackle

Using these categories as a guide we began to form them into epics, each with several specific implementation stories. A pattern began to emerge: demo a build tool, integrate it into a new project, test it in an existing project, and wait for adoption. One tool at a time, it seemed like we could slowly bring about a large number of improvements. But moving slowly doesn’t get developers excited, and it certainly doesn’t improve meeting attendance.

In the first three months…

The Task Force

And so a backlog was forming but we had no one to do the work! We tried asking for a full-time engineer to work on it, but that idea got no traction with management this early on. And I didn’t want to rely on the Guild being just a side project - when push comes to shove sprint work will always come first, leaving Guild work to fall to the wayside.

We discussed this predicament during Guild meetings, and we came up with a compromise: a rotating “Task Force” consisting of two volunteers from different teams for one month at a time. This would provide a constant work force without sacrificing too much from any single team. By pairing, they would be able to solve problems together. And if they come from different teams, knowledge from the Guild would be able to spread to more teams more quickly.

How well we hoped Task Force members would work together

This plan quickly fell apart. We started with a short list of interested engineers, enough to last a few months. But some engineers felt uncomfortable leaving their teams to make up for them during a busy sprint. Others simply lost interest and wanted to focus on their own work. After some lineup changes we only ever got one pair of engineers together, but we did get four back-to-back months of volunteers, mostly one engineer at a time.

Those of us not actively volunteering helped by reviewing code. We required two reviewers — both from different teams — to approve a pull request before merging. This was to ensure a wider variety of opinion, since we’ve found teammates tend to think somewhat alike (for better or worse). The small group of developers who provided code review continued the core of our meetings and discussions as well.

After about five months…

A Yeoman Generator

Our first engineer on the Task Force made the smart choice to focus on the “App Generator” work first. This would be the vehicle for all other tooling changes. And so he made a custom Yeoman generator for scaffolding new JavaScript projects, both “apps” (single-page applications) and “libraries” (smaller, sharable components). He set it up to create new projects built using Webpack with plugins for Babel, Sass, EJS templates, and more. We continued to use Grunt to tie together all our tools and commands, including Webpack.

CTCT <3's Yeoman

The next few Task Force engineers built upon the initial progress - unit testing (with code coverage) via Karma, UI automation testing with Protractor, sample Marionette code for new projects, and more. Most importantly, we built in the ability to “extend” core config files - package.json, .eslintrc, and Webpack configs to name a few. This would enable us to deliver a “base” config that could change over time while maintaining the ability for teams to provide overrides to those files.

# "base" config lives here, should be overwritten by the Generator
config
  grunt/
    aliases.js
    ...
  webpack/
    webpack.dev.config.js
    webpack.prod.config.js
  .eslintrc.js
  karma.conf.js
# Overrides go in top-level files, won't get overwritten on Generator re-run
grunt/
  aliases.js
  ...
# All source code - JS, CSS, HTML, etc
src/
# All unit tests
tests/
# "top-level" config files, also for overrides
.eslintrc.js
karma.conf.js
webpack.dev.config.js
webpack.prod.config.js

Some of the directory structure produced by the Generator

In order to further spread the word about our work, I had begun sending out monthly emails (made with Constant Contact, of course!) describing our progress on the Generator and other news from our meetings. Beyond information, I encouraged people to stay involved by coming to Guild meetings. We also used our HipChat room as a place for tech support, which our regular members graciously helped provide.

After about seven months…

Showing Off

We had run out of willing volunteers, and a few weeks went by with no active Task Force. I had been running meetings and doing code review thus far, so I stepped up to do active development on the Generator for a short time. A few final PRs later and we were ready to show off work!

It's OK to show off once in a while

Along with the Task Force volunteers, we presented our progress to the company over several engineering meetings. We outlined the new “company standards” for frontend tech — Babel, Webpack, ESLint, etc. — and touted the benefits of modular code. This and a little in-person badgering led to a few teams adopting the Generator for new projects. Early reviews were positive, especially in regards to daily development — with the Webpack dev server, changes built nearly instantly and local pages would auto-reload!

Now that teams were beginning to use our tool our meetings had become weekly. Though attendance remained fairly regular at 6-10 people, this helped us keep up with bug reports and remain as open to the organization as possible. We had to find time to do the work in between normal sprint work, but our managers were supportive — The Generator and the modular approach it supports were key to some upcoming initiatives.

After about ten months…

A Real Full-Time Effort

Coming up on a year since starting the Guild a lot had changed. We were making a big push for new product features, with several new projects and redesigns of existing ones underway (and lots of new positions open to get the work done). The tooling provided by the Generator simplified the setup and continued development of those projects. Senior engineers had continued to push for broad frontend architecture changes to reduce costs and minimize future tech debt. The Generator was central to this plan and the standardization system and delivery mechanism for future changes. And the Guild’s ability to groom a backlog and get work done demonstrated that this group effort was supported by teams and sustainable if given the resources. But of course without a dedicated Task Force, our ability to multitask would eventually falter.

Resources were getting stretched a bit thin...

Luckily management had taken notice of our good work and we finally got what we needed: a full-time engineering position for the Guild! One of the past Task Force volunteers, a relatively recent hire, transitioned into the new role. Embedded in our Continuous Delivery team he began to integrate the Generator more closely with our existing Jenkins-based tooling for building and deploying JavaScript apps, in addition to feature development. The Guild continued to own the backlog of work, which we happily struggled to keep up with at our meetings.

After about a year…

Further Progress

The Generator is out in the wild but our work is far from over. Of course there are many more improvements to make, especially in ensuring the quality of the Generator itself. We primarily test by generating new apps, so identifying issues when upgrading from any old version to the latest is tricky. We’re looking into doing more manual regression testing before each release to try and catch more bugs before our users do. If the manual process helps, we’ll try to automate the process to speed things up and test a wider variety of cases.

As more teams adopt the Generator — and therefore Webpack and modular JavaScript — we’ll begin testing the limits of our approach. For each individual project, how do we best ensure that they rerun the generator regularly to stay up to date? As projects become intertwined, how will we deal with dependency conflicts? How will we properly split up our testing? How will these new apps play with our legacy projects? We have some answers and guesses for now, and we’ll keep working towards the best solutions.

We still have lots of questions

Whatever changes may come there will be a lot of work coming up. After only three months we’re seeing that a single full-time engineer might not be enough to keep up with bug fixes, features, and tech support. Though the “Task Force” never officially ended, finding new volunteers has been a fruitless effort after the first group that stepped up. We’re starting to discuss whether to formalize the program — have managers assign engineers every so often — or hire another engineer, but it’s too soon to tell what we’ll end up with.

We know our immediate future lies in the Generator, but I also want to get an idea of what might come after. And so we’ve also begun a second round of knowledge gathering. Instead of getting every single team involved, we’re targeting project leads and critical apps/services. Our goal this time is to identify the next “big thing” we can tackle — after the Generator, what other ways can we improve the frontend development experience at Constant Contact?

Conclusion

I certainly put a lot of work into the Frontend Guild: hosting meetings, taking notes, planning work, finding volunteers, giving presentations, reviewing pull requests, contributing code… And all of that in between normal sprint work, too! But I was far from going it alone. I had support from my manager on how best to organize the Guild. I received technical guidance from senior engineers on the goals of the tooling and in seeing the big picture. And if it wasn’t for my generous coworkers who come to Guild meetings, volunteer for the Task Force, review code, and more, all of this would never have happened!

Thanks to everyone for the help along the way!

But all that hard work has paid off. I’m seeing the benefits of our work every time I work on my project. Most other teams have one or more projects using the Generator as well. Now that most teams have the same, up-to-date base setup we can more easily help each other out. And when we solve a problem for one project, we’re really enabling all teams at once. Not only has tackling tech debt become a serious focus for us, we’re dealing with it faster and better than ever before.