How to run a software proof of concept

When a proof of concept is worth it

A proof of concept costs real time on both sides, so it is not the first step. Use it when three things are true: the product has already passed a demo, a specific question is still open, and the answer to that question would change the decision.

The open question nearly always comes from the list of what a demo hides: will our data import cleanly, does the integration with our other system hold up, is it fast enough at our volume, can our permission rules be expressed. If you cannot name the question, you are not ready for a proof of concept. You want a longer look, and a sandbox or trial is the cheaper way to get one.

Proof of concept, pilot, trial

The three get used as if they meant the same thing. They do not, and agreeing which one you are running avoids most disappointments.

The plan, on one page

Write this before any work starts and send it to the vendor. If it does not fit on a page, the scope is too large.

1. The question. One sentence, answerable yes or no. "Can we import our open orders and have the quantities match our current system?" is a question. "Evaluate the import" is not.
2. The success criteria. Written now, with numbers where numbers are possible. "All 400 sample orders import; fewer than ten need manual correction; totals match to the unit." Criteria written after the results are in are not criteria.
3. The data. A real sample, chosen to include the difficult cases, not the clean ones. A few hundred records is usually enough to find the problems. Decide who is allowed to see it and remove personal data that the test does not need.
4. The time limit. Two to three weeks, with the end date in the plan. A proof of concept with no end date turns into a pilot nobody agreed to.
5. The owner on each side. One named person at your company and one at the vendor. Group ownership means nobody checks whether the criteria were met.
6. What each side provides. The vendor: an environment, access, a person who answers questions. You: the data, the time of someone who knows it, and a decision by the end date.
7. What happens afterwards. What a pass leads to, what a fail leads to, and what happens to your data in either case.

Questions to settle with the vendor first

• Is there a charge? Some vendors run a proof of concept at no cost, some charge and credit the fee against a purchase, some charge outright. All three are legitimate. Know which before you start.
• Whose environment is it? A dedicated test environment is normal. Check that it is separate from other customers and from any shared demo.
• What paper is needed? If real data is going in, a confidentiality agreement at least, and a data processing agreement if it contains personal data. Demo data and privacy covers what to check.
• Who does the configuration? If the vendor's engineers set everything up, you have tested their engineers. Ask to do part of it yourself, or to watch.
• What happens to the environment and the data at the end? Deleted, by when, and confirmed how.

Running it

Keep a short log as you go: date, what was tried, what happened. Three weeks later nobody remembers which import attempt failed for which reason.

• Start with the hardest part. If the open question is the integration, do not spend the first week on screens you already saw in the demo.
• Count the vendor's help. Every problem their team fixed for you is a problem you will meet again after signing. Note what was a setting, what was a fix to the product, and what was a workaround.
• Do not widen the scope. New questions will come up. Write them down for later. Adding them is how two weeks becomes two months.
• Check in at the midpoint. One short call: are we going to be able to answer the question by the end date? If not, say so then.

The written result

End with a document of a page or less, whatever the outcome:

• The question, as written at the start.
• Each success criterion, marked met, partly met or not met, with the evidence.
• What had to be changed, configured or worked around to get there, and by whom.
• What was learned that was not being tested.
• What is still unknown.

Share it with the vendor. A good one will correct factual errors and leave the conclusions alone. Then carry the result into your scorecard: a line that was "told" or "unknown" becomes "did it" or "does not".

How proofs of concept fail

• No question. "Let us try it properly" produces three weeks of browsing.
• Clean data. The sample was tidied before loading, so the import worked and the real one will not.
• The vendor did it all. The result shows what their best engineer can do in your account, which is not what you are buying.
• Passing by redefinition. The criteria were loosened once the results were in. If the bar moves, write down that it moved and why.
• No decision. The test ended and nobody was responsible for saying yes or no. Name that person in the plan.