Why the standard demo tells you little
A sales engineer gives the same demo many times a week. It runs on an environment built for it, along a route where every screen is loaded with flattering data and every click works. None of that is dishonest. It is simply a rehearsal, and you are watching a performance of the product, not the product.
The fix is not to distrust the presenter. It is to change who writes the script.
Before the call: send your scenarios
Write three or four short scenarios from your own work and send them a few days ahead, with a line saying you would like the demo built around them. A scenario is a small story with a beginning and an awkward middle:
- "A supplier delivers 60 of 100 units. Two are damaged. Show me receiving it, and what the purchase order looks like afterwards."
- "A user who should only see their own site tries to open another site's record."
- "We find a mistake in a record that was approved last week. Show me correcting it and what the history says."
How the vendor responds is your first piece of evidence. Some will build the demo around your scenarios. Some will say a scenario is not something the product does, which saves you a meeting. Some will ignore the email and give the standard demo; note that too.
During the call
- Ask for the product, not the slides. Company history and customer logos can be sent afterwards. You have a limited number of minutes with someone who can operate the software.
- Ask them to slow down and narrate the clicks. A practiced presenter moves faster than a user can. Count the steps out loud if you need to.
- Ask to drive. Have them share control, or tell them what to click. Hesitation here is information: an environment that only works along one path is a fragile one.
- Ask for the mistake. "Now enter it wrong." Watching the product refuse bad input, or fail to, takes thirty seconds and no slide covers it.
- Ask to see where it is configured. When something looks right, ask to see the setting that makes it so. You learn whether it is a checkbox, a services project or custom code.
- Write down every "yes, it can do that". A yes with no screen behind it goes in the "told" column, with the name of the person who said it.
Questions worth the time
These get specific answers because they are specific. Vague questions get the brochure back.
About what you just saw
- Is this environment the same version customers run today, or does it include unreleased work?
- Which of the things you showed are in the standard product, and which are extra modules or configured for this demo?
- What did you leave out of this demo because it does not show well?
About the first ninety days
- Who does the setup: us, you, or a partner? What does a customer of our size usually pay for it?
- What do you need from us before go-live, and what most often delays it?
- How does our existing data get in, and who cleans it?
About the bad days
- When did you last have an outage, how long was it, and where did you publish what happened?
- If we report a fault, who answers, in what hours, and what is the commitment in the contract as opposed to the target on the website?
- What is one thing customers ask for that you have decided not to build?
About leaving
- If we cancel, what do we receive, in what format, and for how long is it available?
- Does the export include history and attachments, or only current records?
- What notice is needed, and does the contract renew by itself?
About price
- What is the price based on: users, records, sites, transactions? Which of those is most likely to grow for us?
- What is not included in the quote that customers usually end up buying?
- What happens to the price at renewal?
After the call
Within the hour, while you still remember which things you saw and which you were told:
- Mark each scenario as shown, partly shown or not shown.
- Send the vendor your list of "yes, it can" answers and ask them to confirm it in writing. This is polite, normal, and it sorts firm answers from hopeful ones quickly.
- Ask for access to the environment you were shown, even for a day. If it exists for the demo, it can be lent.
- Transfer the result to your scorecard before the next vendor's demo overwrites your memory of this one.
"That is coming next quarter" is an answer about a different product from the one you can buy today. Score what exists. If a planned feature decides the purchase, ask for the delivery date as a term of the contract and see whether the answer changes.
If you can get a sandbox as well
A sales demo and a self-serve demo answer different questions, and the best evaluations use both: the sandbox first, so that your call is spent on what you could not test yourself, and the checklist to make sure the hour in the sandbox was used. If a vendor has no sandbox, ask whether one can be set up. It is a reasonable request, and each format has its place.