Skip to content
Agnes

Benchmark method

How Cuesta benchmarked Agnes

Cuesta Partners tested Agnes against ChatGPT and Claude on 10 of its own business questions, before bringing Agnes to its clients. This page sets out the questions, the setups, how every answer was graded and what the result can and cannot tell you.

The questions

Ten questions, fixed in advance.

Cuesta fixed the questions and the scoring rubric before any system saw its data. Each question tests one capability the firm needs, using its own proposals, project files and live business data. Client names are left out of every question shown here.

The ten capabilities Cuesta tested
#CapabilityWhat it asks for
1Structured metrics (live data)Live business metrics.
2Precedent recallFinding every past engagement of one type.
3Closest matchFinding the closest past project for a prospect.
4Synthesis
5Specific facts
6AmbiguitySpotting a question with two meanings and saying so.
7ProvenanceShowing where an answer came from.
8Negative caseA question with no right answer.
9CountingCounting across many folders.
10Entity resolutionHow two companies are related.

Three of the questions, word for word

  • Finding the closest past project

    “A prospect is a building-products manufacturer, PE-owned, and wants to know if we've done this kind of rapid AI roadmap work before. What's our closest precedent and why is it close?”

    Hard because: The right precedent is filed under a different service name.

  • Counting across many folders

    “Which PE sponsor has driven the most of our AI assessment engagements this year, and across all of those engagements, what industries have we touched?”

    Hard because: The answer spans dozens of folders and a CRM field that is often blank.

  • Spotting a question with two meanings

    “Show me our manufacturing work.”

    Hard because: It can fairly mean two different things. A good answer says so.

The setups

Five setups, same questions.

Every setup received the questions word for word. The four setups with data ran them three times each and every answer was graded. Plain Claude, with no data, is the baseline and is not graded.

  1. Baseline, not graded

    Claude, no data

    Claude with no access to the firm’s files or data.

  2. Graded, 3 runs per question

    Agnes

    Connected company data and knowledge.

  3. Graded, 3 runs per question

    ChatGPT + SharePoint

    Document connector to the firm's SharePoint.

  4. Graded, 3 runs per question

    Claude + SharePoint

    Document connector to the firm's SharePoint.

  5. Graded, 3 runs per question

    Claude + SharePoint + Cuesta's context pack

    Document connector plus Cuesta's own curated definitions.

Grading

Graded by Cuesta, answer by answer.

Cuesta graded every answer itself, against its own rubric of seven weighted scoring dimensions. Each answer got one of four verdicts.

Pass
Counts toward the setup's score.
Partial
Partly right. Does not count as a pass.
Fail
Does not count as a pass.
No access
The setup could not reach the data the question needs, so the question could not be run on it. It is not graded.

Cuesta also checked that restricted data stayed with the people allowed to see it. That access check is reported on the story as “0 leaks”; it is separate from the seven graded dimensions and not part of the scores below.

Scoring basis

30 graded answers for Agnes, 27 for the others.

A setup’s score is its passes divided by its graded answers. Agnes was graded on all 30: 10 questions, 3 runs each. The other three graded setups were graded on 27, because the structured metrics (live data) question could not be run on them: they had no access to the firm’s live data.

Passes, graded answers and score per setup
SetupPassedGraded answersScoreQuestions passed on every run
Agnes3030100%10 of 10
ChatGPT + SharePoint212778%6 of 9
Claude + SharePoint192770%4 of 9
Claude + SharePoint + Cuesta's context pack162759%5 of 9

Counted over all 30 attempts instead, the number of passes does not change for any setup: a question that could not be run is never a pass.

Limits

What the result can tell you.

One firm, its own questions, graded by Cuesta. It is a test of Cuesta’s job, not a general ranking of AI models. Read it as directional: it shows how these setups did on this firm’s documents and data, not how they would do on yours.