Engineers for Designers: The Engineer’s Dilemmas

Engineers have various concerns to balance while writing code.

(This is the fourth and finalĀ in a 4 part series about engineers for designers. This series is based on a talk originally presented as part of Whitespace.)

Throughout this series, we have touched upon some of the motives and reasoning of engineers. I believe that understanding these underlying concerns creates empathy that might not exist otherwise when conversations or decisions are taken at face value. As such, this post will detail some of the major tradeoffs that engineers face in their work: these abstract decisions distinguish an engineer’s work from “just” writing some code to spec. They are the choices that come back to haunt engineers months or years down the line.

As with the rest of the series, I use some terms loosely. In general, these posts are targeted towards software engineers and graphic designers in web application startups, though you may find applications elsewhere as well.

Waterfall versus Iteration

Silicon Valley loves Agile methodologies. There are many subtypes and derivatives that exist, but generally, we believe in a culture of rapid iteration, failing fast, and continuous improvement. The pros and cons of this approach become outwardly apparent as the product can change hourly, though perhaps at the expense of cohesiveness. For an engineer’s perspective, however, I want to step back from the product and think about the effect from a code perspective.

At one extreme, consider a waterfall method, where engineers are provided with a complete specification for the desired product. Not only does it include the behavior now, but it also details how the system will be expected to evolve over the coming months. Assuming that this spec is correct, engineers should be very happy: they know how to build the system not only for now, but also to provide the right affordances for the future. The code can allow for flexibility when they know that this field will one day have to accommodate international clients and not only domestic clients. On the other hand, they know that this workflow is truly a special case and can be treated as an exception rather than a general case that needs to grow later. When provided with a master plan for the present and future, engineers can build to it.

However, I’m assuming you noted that caveat that we are “assuming that this spec is correct.” We tend to lean towards rapid iteration because the future is unknowable, and we have to react accordingly. Engineers embraced Agile because it meant that they weren’t over-engineering systems and seeing their work go to waste when the spec ended up being wrong. It does, however, make it harder to know how to write code correctly. Engineers have to question how extensible a feature should be: if it isn’t flexible enough, then they will have to rewrite the code in a few months to accommodate more cases. If it is too flexible, then it’s lost time upfront. For engineers, this problem is particularly painful since incorrect assumptions can make hard work completely meaningless.

With experience, engineers do learn to build the right abstractions and flexibility in the system, but none of us have crystal balls with which to read the future. Knowing when to ask for more details to get the system right or when to just run with the current understanding is a difficult balancing act for engineers to maintain.

Responsiveness versus Technical Debt

Closely related to the last point is the idea of technical debt. Technical debt is the accumulation of architectural decisions and changes that build up in a codebase. Without taking the time to rewrite existing code, technical debt can overwhelm engineers and cause new development go more slowly. As an analogy, think of code decisions as household items. As you gather more things in your codebase or house, it becomes harder to fit all of those new purchases onto the bookshelf or closet, and they pile up on the ground or table. If you don’t reorganize or give things away, you will become a hoarder, and it becomes hard to use any particular item or even walk through your house. Don’t be a hoarder.

Like impulse purchasing decisions, however, there is usually a good immediate reason for the code change. When engineers say they can’t handle some specific case, the truth is that they probably can: most, very specific problems can be handled with an “if” statement to alter the behavior for just that case.

Just now, the engineers reading this post reeled in disgust, which is correct. Engineers hate this type of code design: when we extend this logic to the countless use cases that exist, the codebase becomes an endless stream of special cases that make the system nigh impossible to understand or modify without breaking other cases. That is a classic example of technical debt.

Oftentimes, the right way to do it is to understand the single case and accommodate it in a generalized fashion in the code. However, this takes more time to write, and in our culture of rapid iteration, this response often goes over poorly with others. Engineers can hack in the changes now and pay for it later, or spend the time upfront to do it right. Oddly, engineers can even go too far in building overly generalized code so dissimilar to the actual use case that the code also becomes difficult to understand. The right decision is largely subject to context, but there is usually enough uncertainty that rational engineers can disagree on the right approach and complexity.

User interface needs to be designed towards the correct audience, whether that be casual one-time visitors or power users. Similarly, a code base needs to be written for the use case, whether it’s little widget or a master system for a huge organization. Knowing how to build for the right level of complexity is hard. If the code is too simple, then everything is a special case because the system can’t accommodate anything. If the code is too complex, then it is impossible understand without holistic knowledge. Knowing when to be fast and dirty versus slow and deliberate is hard.

System Complexity versus User Complexity

The final tradeoff to discuss is where that complexity manifests itself in a system. Engineers like hard problems. Software is great because it can codify complex interactions into patterns for people to use. Not all applications or domains are hard, but most of the interesting ones are. When a task is hard, who needs to deal with it?

For example, let’s say that you are letting users search through chairs on your site, and you have some filters to help them find it. You come up with a few basic properties, like the number of legs, what material its made out of, and how sturdy it is. Unfortunately, this doesn’t quite cover how complex and messy the world of chairs is. If it has a mesh seat and a metal frame, what material is it made out of? Does a stool count as a chair? And if it is a stool with a single support on a platform, does that count as 0 or 1 leg?

The world is complex, and maybe the right thing is to have the system deal with the complexity. Engineers are smart and can come up with good abstractions to handle these problems. If a user searches for “metal”, then it will find both full-metal chairs and metal-legged chairs. That way, the mess is contained to the system. This architecture, however, can leave users mystified as to how the system works, which could be quite frustrating.

On the other hand, we could let the users handle all of the complexity themselves. They get long forms that allow them to perfectly pick the composition of the parts of the chair. However, forcing users to be completely explicit is frustrating for many simple use cases. Presumably, the complexity should be split between the user and the system, but where is that balance?

Of course, much of this difficult can be minimized with help from design and product. The core issue I want to get at, however, is that the world is a very messy place. Codebases, however, are completely explicit, and it’s hard to know how much of that mess should be addressed and represented in the system.

Conclusion

I think many of the ideas above are relatable for people in other roles as well, including design and product. The output of engineers happens to be code, but the rest are general concerns about working with complex systems, and these concerns should be shared and understood throughout an organization.

Although most of these posts have been directed towards engineers as people and how they interact with them, I want to end with a final point about codebases.

A codebase has to be complete, logical, consistent, and understandable. Code can’t be fuzzy like people: it has to do the same thing every time if the inputs are the same, and it can’t improvise when unexpected things happen. This process is quite different from how most people think, and although engineers typically have some inclination towards it, coding is still a task of translating, wrangling, and architecting.

Hopefully this series of posts has helped you better understand software engineers. And hopefully that knowledge can help you work better with them in the future. For the designers out there, I would love to get the opposite perspective of “Designers for Engineers”, and maybe then, we might just understand each other.