Benchmark method
How Cuesta benchmarked Agnes
Cuesta Partners tested Agnes against ChatGPT and Claude on 10 of its own business questions, before bringing Agnes to its clients. This page sets out the questions, the setups, how every answer was graded and what the result can and cannot tell you.
The questions
Ten questions, fixed in advance.
Cuesta fixed the questions and the scoring rubric before any system saw its data. Each question tests one capability the firm needs, using its own proposals, project files and live business data. Client names are left out of every question shown here.
| # | Capability | What it asks for |
|---|---|---|
| 1 | Structured metrics (live data) | Live business metrics. |
| 2 | Precedent recall | Finding every past engagement of one type. |
| 3 | Closest match | Finding the closest past project for a prospect. |
| 4 | Synthesis | |
| 5 | Specific facts | |
| 6 | Ambiguity | Spotting a question with two meanings and saying so. |
| 7 | Provenance | Showing where an answer came from. |
| 8 | Negative case | A question with no right answer. |
| 9 | Counting | Counting across many folders. |
| 10 | Entity resolution | How two companies are related. |
Three of the questions, word for word
Finding the closest past project
“A prospect is a building-products manufacturer, PE-owned, and wants to know if we've done this kind of rapid AI roadmap work before. What's our closest precedent and why is it close?”
Hard because: The right precedent is filed under a different service name.
Counting across many folders
“Which PE sponsor has driven the most of our AI assessment engagements this year, and across all of those engagements, what industries have we touched?”
Hard because: The answer spans dozens of folders and a CRM field that is often blank.
Spotting a question with two meanings
“Show me our manufacturing work.”
Hard because: It can fairly mean two different things. A good answer says so.
The setups
Five setups, same questions.
Every setup received the questions word for word. The four setups with data ran them three times each and every answer was graded. Plain Claude, with no data, is the baseline and is not graded.
Baseline, not graded
Claude, no data
Claude with no access to the firm’s files or data.
Graded, 3 runs per question
Agnes
Connected company data and knowledge.
Graded, 3 runs per question
ChatGPT + SharePoint
Document connector to the firm's SharePoint.
Graded, 3 runs per question
Claude + SharePoint
Document connector to the firm's SharePoint.
Graded, 3 runs per question
Claude + SharePoint + Cuesta's context pack
Document connector plus Cuesta's own curated definitions.
Grading
Graded by Cuesta, answer by answer.
Cuesta graded every answer itself, against its own rubric of seven weighted scoring dimensions. Each answer got one of four verdicts.
- Pass
- Counts toward the setup's score.
- Partial
- Partly right. Does not count as a pass.
- Fail
- Does not count as a pass.
- No access
- The setup could not reach the data the question needs, so the question could not be run on it. It is not graded.
Cuesta also checked that restricted data stayed with the people allowed to see it. That access check is reported on the story as “0 leaks”; it is separate from the seven graded dimensions and not part of the scores below.
Scoring basis
30 graded answers for Agnes, 27 for the others.
A setup’s score is its passes divided by its graded answers. Agnes was graded on all 30: 10 questions, 3 runs each. The other three graded setups were graded on 27, because the structured metrics (live data) question could not be run on them: they had no access to the firm’s live data.
| Setup | Passed | Graded answers | Score | Questions passed on every run |
|---|---|---|---|---|
| Agnes | 30 | 30 | 100% | 10 of 10 |
| ChatGPT + SharePoint | 21 | 27 | 78% | 6 of 9 |
| Claude + SharePoint | 19 | 27 | 70% | 4 of 9 |
| Claude + SharePoint + Cuesta's context pack | 16 | 27 | 59% | 5 of 9 |
Counted over all 30 attempts instead, the number of passes does not change for any setup: a question that could not be run is never a pass.
Limits
What the result can tell you.
One firm, its own questions, graded by Cuesta. It is a test of Cuesta’s job, not a general ranking of AI models. Read it as directional: it shows how these setups did on this firm’s documents and data, not how they would do on yours.