ValcairnRequest access

Sample data · read-only

A document-AI change, reviewed.

A support assistant answers questions from two policy documents. Its model was swapped for a larger one. This is the approved report for that change: the same 12 cases, both sets of answers, and the source behind each case.

  • Harbor Lane Appliances is invented. The two source documents and the twelve cases were written by Valcairn for this sample.
  • The answers are real outputs from two open-weight models run locally, each given both documents and the same instruction. They were imported as files; Valcairn did not run the models.
  • Scoring is normalized exact match only. No person reviewed these answers, so an answer that is right but worded differently counts as a fail here.
  • The twelve cases are development cases chosen by the author. They are not a protected test set and say nothing about other documents.
FailDecision
7 of 12Candidate cases passing
7Fixed
0Regressed
5Still failing

Why the decision is fail. 1 critical case did not pass: warranty-espresso (fail). The rule was set before the comparison: every critical case must pass, and at least 90% of all cases must pass.

Open the failing cases and compare each answer with its source. Exact match cannot credit an answer that is right but worded differently; in the product a reviewer records a judgment on those cases, and the report keeps automatic and human results apart.

Cases

  • FixedAbove what order value is standard delivery free?minor · fees
    Expected answer
    $80
    Baseline: Smaller local model
    $6.Fail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    $80Pass · token overlap 100% (diagnostic only)
    Source the case cites
    Delivery is free for orders over $80.
    service-and-fees.md
  • FixedHow much does out-of-warranty repair diagnosis cost?minor · fees
    Expected answer
    $25
    Baseline: Smaller local model
    $25.Fail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    $25Pass · token overlap 100% (diagnostic only)
    Source the case cites
    Out-of-warranty repair diagnosis costs $25.
    service-and-fees.md
  • Still failingHow much does it cost to repair a toaster out of warranty?major · fees
    Expected answer
    Not stated
    Baseline: Smaller local model
    $25.Fail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    $25Fail · token overlap 0% (diagnostic only)
    Source the case cites
    Out-of-warranty repair diagnosis costs $25. The fee is waived if the customer goes ahead with the repair.
    service-and-fees.md
  • Still failingWhat time does the service desk close on Saturday?minor · hours
    Expected answer
    13:00
    Baseline: Smaller local model
    8:00 to 18:00.Fail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    9:00 to 13:00Fail · token overlap 33% (diagnostic only)
    Source the case cites
    Saturday: 9:00 to 13:00.
    service-and-fees.md
  • Still failingWhat time does the service desk close on Sunday?major · hours
    Expected answer
    Closed on Sunday
    Baseline: Smaller local model
    8:00 to 18:00.Fail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    The service desk is closed on Sunday.Fail · token overlap 25% (diagnostic only)
    Source the case cites
    The service desk is closed on Sunday and on public holidays.
    service-and-fees.md
  • FixedWithin how many business days is a refund paid after the item reaches the warehouse?major · returns
    Expected answer
    5 business days
    Baseline: Smaller local model
    5 business days.Fail · token overlap 50% (diagnostic only)
    Candidate: Larger local model
    5 business daysPass · token overlap 100% (diagnostic only)
    Source the case cites
    Refunds are paid to the original payment method within 5 business days of the item reaching our warehouse.
    returns-and-warranty.md
  • FixedCan a clearance item be returned? Answer yes or no.major · returns
    Expected answer
    No
    Baseline: Smaller local model
    yesFail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    NoPass · token overlap 100% (diagnostic only)
    Source the case cites
    Clearance items cannot be returned.
    returns-and-warranty.md
  • FixedWhat restocking fee applies when an opened item is returned?critical · returns
    Expected answer
    15%
    Baseline: Smaller local model
    15% restocking fee.Fail · token overlap 33% (diagnostic only)
    Candidate: Larger local model
    15%Pass · token overlap 100% (diagnostic only)
    Source the case cites
    A 15% restocking fee is deducted from the refund for an opened item.
    returns-and-warranty.md
  • FixedHow many days does a customer have to return an unopened item?critical · returns
    Expected answer
    30 days
    Baseline: Smaller local model
    30Fail · token overlap 50% (diagnostic only)
    Candidate: Larger local model
    30 daysPass · token overlap 100% (diagnostic only)
    Source the case cites
    An unopened item can be returned within 30 days of delivery for a full refund.
    returns-and-warranty.md
  • Still failingWhat must the agent ask for before offering a replacement?major · warranty
    Expected answer
    The order number
    Baseline: Smaller local model
    warranty claim the order number.Fail · token overlap 33% (diagnostic only)
    Candidate: Larger local model
    order numberFail · token overlap 67% (diagnostic only)
    Source the case cites
    For a warranty claim the agent must ask for the order number before offering a replacement.
    returns-and-warranty.md
  • Still failingHow long is the warranty on an espresso machine?critical · warranty
    Expected answer
    3 years
    Baseline: Smaller local model
    3Fail · token overlap 50% (diagnostic only)
    Candidate: Larger local model
    3-yearFail · token overlap 0% (diagnostic only)
    Source the case cites
    Espresso machines carry a 3-year warranty from the delivery date.
    returns-and-warranty.md
  • FixedDoes the warranty cover limescale damage? Answer yes or no.major · warranty
    Expected answer
    No
    Baseline: Smaller local model
    yesFail · token overlap 0% (diagnostic only)
    Candidate: Larger local model
    NoPass · token overlap 100% (diagnostic only)
    Source the case cites
    It does not cover limescale damage or accidental drops.
    returns-and-warranty.md

Source documents

returns-and-warranty.md · SHA-256 587113665bea…
# Harbor Lane Appliances: returns and warranty policy (synthetic)

This document is invented for the Valcairn sample. Harbor Lane Appliances is not a real company.

## Returns

An unopened item can be returned within 30 days of delivery for a full refund.
An opened item can be returned within 14 days of delivery. A 15% restocking fee is deducted from the refund for an opened item.
Clearance items cannot be returned.
Refunds are paid to the original payment method within 5 business days of the item reaching our warehouse.
The customer pays return shipping unless the item arrived damaged.

## Warranty

Kettles and toasters carry a 2-year warranty from the delivery date.
Espresso machines carry a 3-year warranty from the delivery date.
The warranty covers manufacturing defects. It does not cover limescale damage or accidental drops.
For a warranty claim the agent must ask for the order number before offering a replacement.
A replacement is shipped within 3 business days after the claim is approved.
service-and-fees.md · SHA-256 b57fd85a9dca…
# Harbor Lane Appliances: service desk hours and fees (synthetic)

This document is invented for the Valcairn sample. Harbor Lane Appliances is not a real company.

## Service desk hours

Monday to Friday: 8:00 to 18:00.
Saturday: 9:00 to 13:00.
The service desk is closed on Sunday and on public holidays.
Email support is answered within 1 business day.

## Fees

Standard delivery costs $6. Delivery is free for orders over $80.
Express delivery costs $14 and arrives the next business day.
Out-of-warranty repair diagnosis costs $25. The fee is waived if the customer goes ahead with the repair.
Descaling service for an espresso machine costs $40.

What the report records

  • Case set version 1, content hash 9a37669259f3959f…
  • Report revision 1, content hash 420e4c7f26e1a3d4…, approved by Sample owner (synthetic).
  • Baseline: Open-weight local model, about 0.75B parameters, prompt v1. Candidate: Open-weight local model, about 4B parameters, prompt v1. Both are imported answer files.
  • Scorer normalized-exact-v1: normalized exact string match only.
  • Imported predictions are customer-supplied results. A batch does not prove which model ran, or its latency, cost, configuration or training-data separation.
  • Normalized exact match compares strings after the documented normalization. It is not semantic correctness. Token overlap is a diagnostic and never a pass or a fail.
  • The labels development and final assessment identify how the team uses the set. Neither label means the set is secret, leakage-free or representative of future traffic.
  • Counts use every eligible case in the frozen set as the denominator. They describe this set only.
  • This report is not a certification and does not say the assistant is safe for production.

Bring one document-AI failure.

We set up the same review on your documents, cases and answers. Early access is limited and available by request.

Request an early-access review