Why we score answers instead of marking them up

A customer explained this to us better than we had explained it to ourselves.

From the workshop · Platform team · 6 September 2026

When we built the review function, the obvious design was mark-up. Show what is weak, suggest the improvement, let the writer act on it. Every tool that reviews writing works that way, and it is what a human reviewer does.

We built scoring instead — every answer gets a number before it goes out. At the time we could articulate a technical justification for it, but not a very good one about why it mattered.

An investment manager at a PPP infrastructure investor gave us the real reason:

“It’s half psychological and half functional. If you come back with a score of two out of five, somebody’s going to be on the case. You can’t submit a four out of ten answer.”

What he was describing

A reviewer confronted with a page of mark-up has options. Some of the comments are stylistic. Some are arguable. Under deadline pressure, the reasonable response is to accept the easy changes, push back on two, and move on. The answer improves slightly and ships.

A number removes those options. Two out of five is not a matter of opinion about phrasing. It is a statement about the answer that somebody now has to either fix or defend, in front of colleagues, with their name against the decision.

The mark-up asks the writer to agree. The score asks the organisation to decide.

Why that changed what we built next

Once we understood that, several things followed that we would not otherwise have prioritised.

The score has to be explainable. A number nobody trusts is worse than no number, because it invites an argument about the scoring rather than about the answer. Every score has to be traceable to what is missing — the requirement not covered, the evidence not cited — or the mechanism collapses on first contact with a sceptical reviewer.

It has to be consistent across writers. The value is comparability. If the same answer scores differently depending on who submitted it for review, it is a mark-up with extra steps.

It must not be gameable in a way that damages the bid. If a writer learns that a particular phrasing lifts the score without improving the response, we have built something actively harmful. That constrains what the score is allowed to reward.

What it is not

It is not a prediction of what the evaluator will give you. We do not know their moderation, their panel, or how they weight in practice, and a system that implied otherwise would be lying in a way that eventually gets found out.

It is a measure of completeness against what was asked — which is the part that is knowable from the documents, and which is where most avoidable marks are lost.

The general lesson

We had built the right thing for a reason we could not articulate. A customer articulated it, and the articulation changed the roadmap.

That is not a story about being clever. It is a case for asking customers why something works rather than whether they like it — the answers are consistently more useful, and occasionally they are better than the reasoning that produced the feature.