The test they set us

They had hidden a non-conformity in one of their own past bids. They did not tell us where.

From the field · Associate, onboarding · 6 September 2026

We do a half-day session with prospective customers using one of their own past tenders rather than a demonstration on ours. It is more useful for them and considerably more exposing for us.

A managed connectivity provider took that further than anyone had before. Before we arrived, they took a bid they had submitted previously, inserted a non-conformity into their own response, and handed us the tender pack and their submission history.

The question was simple: would the system find it?

What that actually tests

It is a better test than it first appears, and worth understanding why.

It is not testing whether the system can write. It is testing whether it can check — whether it reads the client’s requirements as a set of obligations, compares them against what was actually submitted, and reports the gap rather than summarising the document.

It is also blind. We did not know what had been changed, where, or how subtle it was. There was no opportunity to steer towards the answer.

And they chose the material. Their tender, their response, their alteration. Nothing about the setup favoured us.

It found it

Inside the half day.

The output also matched, fairly closely, the formal feedback they had received on that submission when it was originally evaluated — which was not something we knew going in, and which they were able to check against a document they already had.

Test passed, in their words.

Why I keep telling this story

Because it settles an argument that otherwise runs in circles.

The most common question we get is some version of “how is this different from the AI assistant we already have?” It is a fair question and it is hard to answer persuasively, because both things produce fluent text and the difference sounds like marketing.

This makes the difference concrete. A general-purpose assistant given that same response would have read it, found it competent, and said so. It had no basis on which to know that a requirement in the tender was no longer satisfied, because nobody handed it the tender as a body of obligations to check against.

The gap is not writing quality. It is whether anything is checking your answer against what was actually asked for.

The bit I would steal if I were you

Regardless of what you buy, or whether you buy anything: set your own exam question.

Take a bid you have already submitted. Change something in the response — remove a commitment, weaken a compliance statement, break something that was previously satisfied. Then hand the pack and the altered response to whatever process or tool you are evaluating, and see whether it notices.

Any vendor confident in what they have will agree to it. The ones who steer you back towards a scripted demonstration are telling you something.