The runaway proof of concept

Ah, the proof of concept.

There are few other ideas in software development that are more vague, more abused and yet less valuable than the proof of concept.

It starts out so innocently. Instead of talking for hours about whether a technical approach will work, or whether some perceived impediment to shipping the product is real, let’s try it out! Let’s try and integrate those two complex systems using quick and dirty methods, just to show that it’s possible.

The motivation here is legitimate. We want to reduce risk in software development, and we want to do that as cheaply and quickly as possible, without our normal production-quality standards getting in the way.

But in practice, the proof of concept often fails in this primary goal of reducing risk.

The first big problem is that the term is used to denote different goals, which can get conflated very easily.

Entrepreneurs and business owners might consider the purpose of a proof of concept as cheaply assessing the market potential of a product. The Minimum Viable Product (MVP) from lean takes this idea and runs with it. Before the MVP was popularised, the term proof of concept was used to denote similar goals of market acceptance.

Software developers will usually talk about a proof of concept as something that addresses technical feasibility. Can this thing even be built in practice?

Those are both very important, but very different concerns. When someone suggests running a proof of concept, it’s very important to be clear on what the goal is.

The second big problem is that proofs of concept have a nasty habit of growing like a weed and crapping all over our carefully laid development plans.

For example, we might set out to discover whether integrating a new modern web app with a legacy system is possible by building an intermediate API service. Initially, our question is ‘Can it be done at all?’, so we build a really rough system in a development environment to check that data can be retrieved and persisted to a cut-down version of the legacy system.

But as we work on this proof of concept, we unearth more questions than answers. Can we make this particular big data query performant enough? Will underpowered mobile devices be able to handle this big data set? How do we scale this out to all our customers without rebuilding the legacy systems?

The temptation with all these new questions is to keep going using the existing proof of concept we’ve built. Before we know it, we have a large system, albeit one that is nowhere near production quality.

We’ve deliberately used quick and dirty methods, haven’t written tests, and haven’t thought about performance, scalability, usability or accessibility, except in patches. Developers working on these concepts might even have used languages and technologies that they wouldn’t have used for production systems, purely for the benefits of familiarity or development speed (or for the opportunity to try out something new).

We might then be put under pressure to ‘productionize’ a proof of concept that we inadvertently made larger than intended. Business stakeholders may only see the visible features that have been built, and assume that a bit of spit and polish will make it ready to put in front of customers.

And through all of this, at no point have we actually tested what we set out to test. Because the question of feasibility is not ‘Can we build a crappy, untested, unusable, unmaintainable product that integrates with this legacy system and scales only to 20% of our users?’

Tests of feasibility need to include all the qualities of a production system, because these concerns rightly have an active impact on system design.

For me, proofs of concept are experiments. So they should be run as experiments. And you should probably call them experiments too.

This means you formulate a very narrow and explicit hypothesis that represents a high degree of risk in the project. Then you focus on disproving it. And then you dismantle your test apparatus.

Tomorrow we’ll look at an alternative framework for reducing risk, the walking skeleton.

In the meantime, I’d love to hear a story of a proof of concept that accidentally became the final product. Hit reply and tell me about it. I promise you’ll feel better for venting.

All the best,

– Jim