Sample data · read-only
A document-AI change, reviewed.
A support assistant answers questions from two policy documents. Its model was swapped for a larger one. This is the approved report for that change: the same 12 cases, both sets of answers, and the source behind each case.
- Harbor Lane Appliances is invented. The two source documents and the twelve cases were written by Valcairn for this sample.
- The answers are real outputs from two open-weight models run locally, each given both documents and the same instruction. They were imported as files; Valcairn did not run the models.
- Scoring is normalized exact match only. No person reviewed these answers, so an answer that is right but worded differently counts as a fail here.
- The twelve cases are development cases chosen by the author. They are not a protected test set and say nothing about other documents.
Why the decision is fail. 1 critical case did not pass: warranty-espresso (fail). The rule was set before the comparison: every critical case must pass, and at least 90% of all cases must pass.
Open the failing cases and compare each answer with its source. Exact match cannot credit an answer that is right but worded differently; in the product a reviewer records a judgment on those cases, and the report keeps automatic and human results apart.
Cases
FixedAbove what order value is standard delivery free?minor · fees
- Expected answer
$80
- Baseline: Smaller local model
$6.
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
$80
Pass · token overlap 100% (diagnostic only)- Source the case cites
Delivery is free for orders over $80.
service-and-fees.md
FixedHow much does out-of-warranty repair diagnosis cost?minor · fees
- Expected answer
$25
- Baseline: Smaller local model
$25.
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
$25
Pass · token overlap 100% (diagnostic only)- Source the case cites
Out-of-warranty repair diagnosis costs $25.
service-and-fees.md
Still failingHow much does it cost to repair a toaster out of warranty?major · fees
- Expected answer
Not stated
- Baseline: Smaller local model
$25.
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
$25
Fail · token overlap 0% (diagnostic only)- Source the case cites
Out-of-warranty repair diagnosis costs $25. The fee is waived if the customer goes ahead with the repair.
service-and-fees.md
Still failingWhat time does the service desk close on Saturday?minor · hours
- Expected answer
13:00
- Baseline: Smaller local model
8:00 to 18:00.
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
9:00 to 13:00
Fail · token overlap 33% (diagnostic only)- Source the case cites
Saturday: 9:00 to 13:00.
service-and-fees.md
Still failingWhat time does the service desk close on Sunday?major · hours
- Expected answer
Closed on Sunday
- Baseline: Smaller local model
8:00 to 18:00.
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
The service desk is closed on Sunday.
Fail · token overlap 25% (diagnostic only)- Source the case cites
The service desk is closed on Sunday and on public holidays.
service-and-fees.md
FixedWithin how many business days is a refund paid after the item reaches the warehouse?major · returns
- Expected answer
5 business days
- Baseline: Smaller local model
5 business days.
Fail · token overlap 50% (diagnostic only)- Candidate: Larger local model
5 business days
Pass · token overlap 100% (diagnostic only)- Source the case cites
Refunds are paid to the original payment method within 5 business days of the item reaching our warehouse.
returns-and-warranty.md
FixedCan a clearance item be returned? Answer yes or no.major · returns
- Expected answer
No
- Baseline: Smaller local model
yes
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
No
Pass · token overlap 100% (diagnostic only)- Source the case cites
Clearance items cannot be returned.
returns-and-warranty.md
FixedWhat restocking fee applies when an opened item is returned?critical · returns
- Expected answer
15%
- Baseline: Smaller local model
15% restocking fee.
Fail · token overlap 33% (diagnostic only)- Candidate: Larger local model
15%
Pass · token overlap 100% (diagnostic only)- Source the case cites
A 15% restocking fee is deducted from the refund for an opened item.
returns-and-warranty.md
FixedHow many days does a customer have to return an unopened item?critical · returns
- Expected answer
30 days
- Baseline: Smaller local model
30
Fail · token overlap 50% (diagnostic only)- Candidate: Larger local model
30 days
Pass · token overlap 100% (diagnostic only)- Source the case cites
An unopened item can be returned within 30 days of delivery for a full refund.
returns-and-warranty.md
Still failingWhat must the agent ask for before offering a replacement?major · warranty
- Expected answer
The order number
- Baseline: Smaller local model
warranty claim the order number.
Fail · token overlap 33% (diagnostic only)- Candidate: Larger local model
order number
Fail · token overlap 67% (diagnostic only)- Source the case cites
For a warranty claim the agent must ask for the order number before offering a replacement.
returns-and-warranty.md
Still failingHow long is the warranty on an espresso machine?critical · warranty
- Expected answer
3 years
- Baseline: Smaller local model
3
Fail · token overlap 50% (diagnostic only)- Candidate: Larger local model
3-year
Fail · token overlap 0% (diagnostic only)- Source the case cites
Espresso machines carry a 3-year warranty from the delivery date.
returns-and-warranty.md
FixedDoes the warranty cover limescale damage? Answer yes or no.major · warranty
- Expected answer
No
- Baseline: Smaller local model
yes
Fail · token overlap 0% (diagnostic only)- Candidate: Larger local model
No
Pass · token overlap 100% (diagnostic only)- Source the case cites
It does not cover limescale damage or accidental drops.
returns-and-warranty.md
Source documents
returns-and-warranty.md · SHA-256 587113665bea…
# Harbor Lane Appliances: returns and warranty policy (synthetic) This document is invented for the Valcairn sample. Harbor Lane Appliances is not a real company. ## Returns An unopened item can be returned within 30 days of delivery for a full refund. An opened item can be returned within 14 days of delivery. A 15% restocking fee is deducted from the refund for an opened item. Clearance items cannot be returned. Refunds are paid to the original payment method within 5 business days of the item reaching our warehouse. The customer pays return shipping unless the item arrived damaged. ## Warranty Kettles and toasters carry a 2-year warranty from the delivery date. Espresso machines carry a 3-year warranty from the delivery date. The warranty covers manufacturing defects. It does not cover limescale damage or accidental drops. For a warranty claim the agent must ask for the order number before offering a replacement. A replacement is shipped within 3 business days after the claim is approved.
service-and-fees.md · SHA-256 b57fd85a9dca…
# Harbor Lane Appliances: service desk hours and fees (synthetic) This document is invented for the Valcairn sample. Harbor Lane Appliances is not a real company. ## Service desk hours Monday to Friday: 8:00 to 18:00. Saturday: 9:00 to 13:00. The service desk is closed on Sunday and on public holidays. Email support is answered within 1 business day. ## Fees Standard delivery costs $6. Delivery is free for orders over $80. Express delivery costs $14 and arrives the next business day. Out-of-warranty repair diagnosis costs $25. The fee is waived if the customer goes ahead with the repair. Descaling service for an espresso machine costs $40.
What the report records
- Case set version 1, content hash
9a37669259f3959f… - Report revision 1, content hash
420e4c7f26e1a3d4…, approved by Sample owner (synthetic). - Baseline: Open-weight local model, about 0.75B parameters, prompt v1. Candidate: Open-weight local model, about 4B parameters, prompt v1. Both are imported answer files.
- Scorer normalized-exact-v1: normalized exact string match only.
- Imported predictions are customer-supplied results. A batch does not prove which model ran, or its latency, cost, configuration or training-data separation.
- Normalized exact match compares strings after the documented normalization. It is not semantic correctness. Token overlap is a diagnostic and never a pass or a fail.
- The labels development and final assessment identify how the team uses the set. Neither label means the set is secret, leakage-free or representative of future traffic.
- Counts use every eligible case in the frozen set as the denominator. They describe this set only.
- This report is not a certification and does not say the assistant is safe for production.
Bring one document-AI failure.
We set up the same review on your documents, cases and answers. Early access is limited and available by request.