Premature Scaling

The most common startup engineering mistake is to scale prematurely.

Premature scaling means building for large scale when you’re still small. Premature scaling trades immediate productivity for far off, and often never realized, benefits.

Premature scaling troubles are everywhere in startups. In the early 2010s, SQL databases were shunned partly for the (perceived) infinite scalability of NoSQL databases like MongoDB and Cassandra. The startup productivity benefits of monoliths was lost in the excitement around microservices. For years, Rails has been called unfit for startups because it is slower than other frameworks.

Premature scaling wastes engineering resources. Scaling before reaching product-market fit is devastating for early startups who have little money and few engineers but huge feature requirements and the constant need to iterate.

Why does premature scaling occur so often if it is the number one startup killer?

First, scaling is fun. The initial version of Twitter was a simple monolithic CRUD application that any bootcamp engineer can now build. Just a few years in, Twitter had a panoply of interesting problems: large amounts of data to query, a huge cost to downtime, large usage spikes, a huge userbase. Adding in the scaled technologies to address these needs made the job more exciting and more attractive to those joining. The excitement makes early scaling an easy trap for engineering teams to fall into.

Second, building performant systems seems rational. Some engineers are appalled by “hacky” systems as if they are a moral failing. This aversion to potentially inelegant solutions is especially an issue for startup engineers who worked for large tech companies where everything must be done at scale. “Do things that don’t scale” is a regular missive in Y Combinator, and it applies in engineering as much as it does in business processes.

Technical debt kills less early stage startups than you think. If you’re successful, you’ll often have the money to fix all the engineering mistakes you’ve made and shortcuts you’ve taken.

Third, engineering roadmaps and tools are built around hyper-optimistic views of the future. Startup leaders are idealistic about how their business grows because, let’s face it, they wouldn’t have started the company otherwise. Even once you hit product-market fit — a critical milestone when engineering decisions need to be reassessed — the danger is that you overestimate the level of scale you’ll need.

Trying to anticipate future needs before they arise takes valuable resources away from your immediate goal — building a product that people need.

Using Untested Technology

Startups hurt themselves when they rely too much on shiny or newtechnology.

Shiny technology is technology that software engineers clamor for. Shiny technology often makes an engineer’s life easier and more enjoyable, especially in the extreme short term. But relying on the newest tech misses what will be most productive for the broader team in the medium term.

For example, MongoDB’s Javascript-like DSL and JSON data store made writing code an easier experience for Javascript developers compared to using relational SQL databases. But ease of use should not be the primary metric when choosing a database.

Source: Karpov, Valeri. The MEAN Stack: MongoDB, ExpressJS, AngularJS and Node.js (Slideshare)

During an interview, a software engineer told me that he’d never work at a company that didn’t use CoffeeScript (today, the debate might be about using Elixir). His refusal to even consider other tools promised problems if the right solution was at odds with his preference.

New technology has its own issues. The failure states of new tools are poorly understood, so it’s hard to anticipate how things will break. New languages/frameworks don’t have libraries or many engineers who can write in them. This lack of resources increases the development effort and makes recruiting challenging. It also means committing your startup’s scarce engineering resources to learning something new when you could be focusing on creating features that your users value.

Part of the problem is that engineers — especially non-founders — want to implement technology that makes them seem relevant in the market. That should worry a startup. In spaces like front-end engineering, having engineers chase the current hype cycle might mean rewriting the stack every 6–12 months.

Developers early in their career are especially susceptible to using new and shiny technology instead of tools that make the most sense for the project at hand. They haven’t been through a few hype cycles or seen the downsides of choosing the latest technology. They can miss all the marketing behind what they’re seeing when reading Hacker News or attending hackathons/conferences.

Worse, these developers don’t realize that just because a technology is hyped that doesn’t make it appropriate in a startup environment. It’s common to take a tech stack of a well known startup or large company and transplant its stack, without assessing the appropriateness of these choices. When teams are small, developers don’t always have the mentorship from seasoned engineers to counteract the external media they see.

Startup developers cite Paul Graham’s famous post, The Python Paradox, where Python increased startup productivity compared to Java. Graham’s post is often used to justify implementing every latest framework and language. However, Graham’s real point was that software engineers should pick tools that maximize startup productivity, rather than reflexively favor the newest technology. In fact, after seeing cohorts of startups at Y Combinator in 2013, Graham was asked about ideal languages. He noted, “I mean, we have had startups writing their code in PHP — and that worries me a little bit. But not as much as other things worry me.”

As an engineer you’re taking a huge risk joining a startup. You shouldn’t add the risk of hyped, but unnecessary technology to the mix.

Hiring the wrong engineer

Many startup engineering problems stem from hiring the wrong engineers.

Startups often look for “rock stars” or “ninjas.” They want gaudy educational credentials and the pixie dust that comes from being part of successful companies. In Silicon Valley, startup job posts filter for candidates who use new and shiny technology rather than favoring pragmatic engineers who can learn necessary tools quickly. Once a company starts with perceived “ninjas” and “rock stars,” it sets the tone for every subsequent hire.

Startup engineering is rarely taught. The vast number of software engineers work for non-startups. As such, most engineers who join startups aren’t trained in startup engineering and don’t have the skillset or attitude to succeed in the startup environment. Consequently, startups often must rely on engineers who aren’t a perfect fit for the project.

Each of these engineers has their own risk for a startup:

• Large company engineers are often exposed to scaled systems and may not feel comfortable with hacky code. They are much less likely to be exposed to their users.
• Elite computer scientists get bored by the plumbing-type engineering that typifies early startup tasks, leading to overengineering.
• Junior engineers, who are often attracted to startups, may inject too much new technology and poorly architect their systems.
• Development shops often favor technologies that let them be productive across many clients. These shops have no incentive to learn new tools for a particular client, and they also don’t have the long-term skin in the game needed to make critical early tech decisions.

By contrast, ideal early startup engineers are excited about the company’s mission. This ensures they’ll stay when the company inevitably hits challenges along the journey to product-market fit.

They have a minimum viable product mindset and a comfort with constant iteration. The best startup engineers often have a few years of experience in the workforce or on open source projects so that they’ve learned how to maintain systems, not just build them.

Rather than seeing particular technologies as silver bullets, they favor productive tools that meet the medium term needs of the organization. Great early startup engineers see technology as a means to solving a problem. They think about the problem space (the user’s needs) and how software can be used to solve the problem, rather than looking for a problem that a new technology can solve.

They need to have a can-do attitude. They rise to the herculean challenges in front of them while moderating, but not naysaying, the optimism of the product owners. These engineers need empathy and intuition for their users because they often play role of product when iterating.

One cautionary note. Even among startup engineers, the experience gained at each stage of the startup is radically different. A startup engineer who is effective at five engineers may be terrible at 25. For startups hiring talent, seeing an engineer who was part of a successful startup often overwhelms the nuance of what that person actually did there.

Startups should look beyond the name brands and hire based on fit for the current project.

Problems with Product and Management

So many startup mistakes stem from product and management, not engineering.

Startup founders are wildly optimistic. The product roadmap then usually exceeds the capabilities of their small teams. This optimism leads to too much hiring, too much fundraising, and eventually, burnout. For management, this cycle begins to look like “the engineers aren’t performing,” when it actually means that management didn’t set a clear and narrow vision.

The always be pitching startup mentality of founders also gives a mistaken sense of certainty to the engineering team about how the system should be built, despite the uncertainty that exists in the business model. A better mindset is to support optimistic engineers and develop (internally) pessimistic business leaders, with both sides working together to find product-market fit.

Another problem for new startups is that management often finds engineering advisors — engineers at other successful startups — to guide the new company. Importing supposed “experts” from potentially unrelated companies or fields raises problems when these advisors transplant what worked at their particular company to a new startup that they have spent only a few hours trying to understand.

Finally, management needs to recognize that its engineers are a critical component of any major product decision. Too often, a company’s engineers are seen as a service organization that can be steamrolled when important business decisions arise (ensuring a healthy product-engineering relationship is one more reason it’s critical to have hired the right engineers).

Don’t Shoot Your Startup in the Foot

Even if you avoid these startup engineering pitfalls, you’ll still be faced with challenges aplenty. But by steering clear of these traps, you’ll worry less about self-inflicted wounds.

Follow me on Twitter — or read more at nemil.com/musings. You can also sign up for my mailing list.