A fair test for catalog AI on your own data
Run every catalog AI option on the same fixed sample of your own records, with the correct answers agreed before any run. A worked example shows why a clean result on a small sample still leaves room for errors.
Published by Youzu · Editorial policy
- Published
- Reading time
- 5 min read
Key points
- Compare catalog AI on one fixed sample of your own records, with the answers agreed before any run.
- Check what a small sample can show: a clean result on 100 records still allows about 3 percent errors.
- Keep the sample and rubric the same for every option, and count each decision type on its own.
In this article
Most catalog AI evaluations go wrong before the first record is processed. The sample is chosen by whoever is selling, the answers are decided after the results arrive, and a handful of friendly examples stand in for a catalog of thousands. A fair test is not complicated, but it has to be set up in a fixed order. This article walks through that order with one worked example you can copy.
Fix the sample before anything runs
Start with a fixed sample: a list of records drawn from your own catalog, written down, and not changed once the first option has been run. Draw it from the whole catalog, not from the products you are proud of. Include the records that cost your team time today: thin supplier data, products with several photos, items that look like duplicates, and listings that break a policy. If every option sees exactly the same records, differences between options come from the options and not from the draw.
100 records is a practical size for a first round. It is small enough that 2 people can agree on the right answer for every row in a day, and large enough to show which option is clearly weaker. It is not large enough to separate 2 options that are close, as the next section shows.
Write the rubric first
The rubric is a table with one row per record and one column per decision you care about: category, each required attribute, whether the record is a duplicate of another, and whether it passes your listing rules. Fill in the correct answer by hand, before any option sees the data. Leave a value empty when the source does not support it; a blank that stays blank is a correct answer. Data quality standards such as ISO/IEC 25012 give useful names for what you are checking, among them accuracy, completeness and consistency, so that 2 reviewers mean the same thing by "wrong".[2]
- One row per record, with a stable record id.
- One column per decision type, so that category errors are not mixed with attribute errors.
- An agreed answer for each cell, including "leave empty" where the source is silent.
- A named person who settles disagreements, decided before the run.
A worked example with round numbers
Suppose your rubric has 100 records and you are comparing 2 hypothetical options on one decision, the product category. Option A returns the wrong category for 0 of the 100 records. Option B returns the wrong category for 5 of the 100.
| Option | Wrong category | Observed error rate | Plausible error rate, 95 percent upper bound |
|---|---|---|---|
| Option A | 0 of 100 | 0 percent | about 3 percent |
| Option B | 5 of 100 | 5 percent | about 10 percent |
Option A looks perfect, but zero errors in 100 does not mean the true error rate is zero. The rule of three says that when nothing goes wrong in n trials, the upper 95 percent bound on the rate is about 3 divided by n, here 3 percent.[1] A fair reading is that Option A is probably better, that nobody can yet say it has no errors, and that a bigger sample is worth running before you commit. If the 2 options had scored 2 and 4 errors, the sample could not tell them apart at all.
Count each decision type on its own
One headline accuracy number hides the failures that matter. A tool that gets categories right and invents attribute values is a different risk from one that leaves attributes empty. Report each column of the rubric separately, and report empty-where-correct as a success. Then read the failures themselves: for 10 wrong rows, ask whether a reviewer would have caught them, and how long the fix takes.
Include your current process and a model API
Your current process is an option. Run it on the same sample, record the time it takes, and score it with the same rubric. If a general-purpose model API is on the shortlist, test it the same way, and write down what you would still have to build around it: review, partial results, failures and maintenance. The comparison is on the whole workflow, not on a single answer.
Keep the test honest
3 habits protect the result. Do not tune any option on the sample before the scored run; if a vendor or a team member sees the answers, draw a new sample. Do not drop records that an option found hard, because the hard records are the point of the test. And record how long each option took to reach an accepted answer, including the time reviewers spent correcting it, since a result that needs a day of fixing is a different result from one that does not.
What to write down at the end
- The sample: how it was drawn, how many records, and the date it was fixed.
- The rubric file, unchanged from before the first run.
- Per option, per decision type: correct, wrong, and correctly left empty.
- The records each option got wrong, with the reviewer's note.
- What the next, larger sample should test.
The catalog benchmark page is linked below.
Questions buyers ask
Large enough for 2 people to agree on every answer in a day, and large enough to show a clear gap between options. With 100 records and no errors, the real error rate can still be up to about 3 percent, so close results call for a bigger sample before anyone decides.
Yes. Evaluations start with a sample of your own catalog and expand to more workflows once the results are reviewed. We start with a sample of your own catalog, and we publish customer results only with the customer's approval. Fix the sample and the rubric first.
Yes, if it is on your shortlist, and with the same sample and rubric as every other option. Compare Youzu with in-house builds, model APIs, PIMs and search tools on the full workflow using your data: identity, variants and offers, moderation, review, partial and failed states, and maintenance.
Sources
If nothing goes wrong, is everything all right? Interpreting zero numerators, Hanley and Lippman-Hand, JAMA 249(13), 1983 (doi 10.1001/jama.1983.03330370053031). doi.org/10.1001/jama.1983.03330370053031. Accessed . Back
ISO/IEC 25012: data quality model, ISO and IEC. iso.org/standard/35736.html. Accessed . Back
Statements about other vendors describe what each vendor documents on the access date shown.
Related
Keep reading
Evaluation
Benchmark Youzu on a sample of your own catalog
We run a sample of your catalog and report results against your own labels. You set the acceptance criteria; critical errors are counted separately.
Comparison
AI product data enrichment tools compared
How to compare AI product data enrichment options: PIM suites, feed tools, AI writers, model APIs, in-house builds and Youzu, tested on your own sample.
Article
How to Evaluate AI Product Data Enrichment: A Benchmark for Retailers
A practical framework for comparing AI product-data tools on accepted decisions: attributes, taxonomy, variants, duplicates and review workload.