Skip to main content

Guide Listings and moderation

How to test a moderation rule before you switch it on.

In short

Test a moderation rule on a reference set of listings before it goes live. Count the false flags and the misses apart, fix their causes, then activate one exact revision of the ruleset.

Published by Youzu · Editorial policy

Published
Reading time
8 min read

Key points

  • Start with a reference set of listings before you write the rule, not after.
  • Place in it listings that must be flagged, must pass and are borderline.
  • Compare the rule's flags with your reviewers' labels, and count each kind of error.
  • Use one exact revision when you activate, never the latest.
In this guide

Why test a rule before you switch it on

A moderation rule is a prediction about which listings break a policy. Until it has been run against real listings, nobody knows how many it would flag, how many of those are right and how many it would miss. A rule that flags too much floods reviewers and annoys honest sellers; a rule that flags too little lets the listings it was written for through.

Testing does not need special tools. It needs a set of listings, your reviewers' labels for them and a way to compare what the rule would have done with what the labels say. A spreadsheet is enough for the first rounds, and the work is the same whether the rule is written by a person or proposed by software. What matters is that the comparison is written down, so that the next person can repeat it and reach the same count.

Build a reference set

A reference set is a list of listings with a decision beside each: should this listing be flagged under the policy, yes or no. Take the listings from your own marketplace, including old ones that reviewers already decided, and write down who decided and why.

What a reference set contains
GroupWhat it holdsWhat it tells you
Must flagListings that clearly break the ruleWhether the rule catches what it was written for
Must passListings that look similar but are allowedWhether the rule flags honest sellers
BorderlineListings your reviewers disagree aboutWhere the policy wording needs work

Keep the set small enough that a person can read every listing in it, and large enough to hold a few examples of each group, including the odd cases that only a marketplace like yours attracts. A set of about 100 listings, with at least 20 in each group, is a workable start. Add a listing every time an appeal or a reviewer shows a case the rule handled badly, so the set grows from real mistakes and not from guesses about what might go wrong.

Run the rule and compare

Apply the rule to every listing in the set and record whether it would flag it. Then compare with the labels. Each listing falls into one of 4 cells: flagged and should be, flagged and should not be, passed and should be flagged, passed and rightly so.

The 4 outcomes of one rule on a reference set
Policy says flagPolicy says pass
Rule flagsCorrect flagFalse positive: an honest listing is flagged
Rule passesMissed listing: a breach gets throughCorrect pass

Count the false positives and the missed listings apart, and read every one. Most rules show a pattern within a few cases: a keyword that appears in allowed listings, a scope that includes images the rule was not meant to check, or a severity set higher than the harm. Fix the cause, and run the set again.

Worksheet

Sketch 3 rules for your own policy

  1. Would flag 2 of 8 listings.

  2. Would flag 2 of 8 listings.

  3. Would flag 2 of 8 listings.

  4. The 8 sample listings and the rules that would flag them
    ListingWhat a reviewer seesFlagged by
    1. Oak dining tableA table photographed in a living room.Not flagged
    2. City bike, good conditionThe description asks buyers to phone the seller directly.Rule 1
    3. Leather sofaA phone number is written on a card in the photo.Rule 1
    4. Replica sword, decorativeThe title names a replica weapon, and the photo shows it.Rule 2
    5. Camping setThe photo shows a hunting knife beside the tent.Rule 2
    6. Designer floor lampThe photo is copied from a shop's product page.Rule 3
    7. Wool rugThe photo carries another seller's watermark.Rule 3
    8. Kids' bike, half price todayThe title carries a price offer.Not flagged
Read as text

Worked example: 3 rules sketched against 8 made-up listings.

  1. Contact details outside the platform, high severity, text and image, without a reference image.
  2. Weapons and replicas, high severity, image, without a reference image.
  3. Watermarked or borrowed photos, medium severity, image, with a reference image.
  • Oak dining table: not flagged.
  • City bike, good condition: flagged by rule 1.
  • Leather sofa: flagged by rule 1.
  • Replica sword, decorative: flagged by rule 2.
  • Camping set: flagged by rule 2.
  • Designer floor lamp: flagged by rule 3.
  • Wool rug: flagged by rule 3.
  • Kids' bike, half price today: not flagged.

The copied lamp photo is found only because rule 3 has a reference image; the kids' bike would need a text rule about prices in titles.

A planning worksheet: the topics and the 8 listings are made up for this page. Change a rule and see which listings it would flag. Nothing you choose is sent or stored.Policy topics like these appear in public seller policies, for example: Prohibited and restricted items, eBay Help, read on .

Severity, scope and revisions

Decide the severity before activation, because it decides what happens to a flagged listing. A high-severity rule might send a listing to a person before it goes live; a low-severity one might only add a note for the seller. The same finding can be right for one severity and wrong for another.

Write down the scope as well: text only, text and the first image, or every image. Scope changes the result more than wording does, because a rule that reads images flags listings a text-only rule never sees. Test each scope you intend to use.

Save each change as a new revision of the ruleset and activate an exact revision, never 'the latest'. A decision can then always be traced to the rule text that produced it, and an earlier revision can be restored if a new one turns out to be too broad. Public marketplace policies and regulation both point the same way: say which rule applied, and keep a record of it.[1][2]

A worked example: a rule about contact details

Take a policy line that says listings may not put phone numbers or messaging handles in the title, because contact outside the platform removes the buyer's protection. The rule flags titles that contain a run of digits shaped like a phone number, or a handle written after an at sign.

Run it on a reference set of 60 listings: 20 that must be flagged, 20 that must pass and 20 borderline. The first run flags 18 of the 20 that must be flagged and 4 of the 20 that must pass. The 4 are listings whose titles include a model number or a measurement, such as '5000 mAh 2021', which the rule read as a phone number.

One rule on a synthetic reference set of 60 listings
GroupListingsFlagged by the ruleReading
Must flag20182 missed: handles written with spaces
Must pass2044 false positives: model numbers read as phones
Borderline209Reviewers disagree on 7 of the 9

The numbers are round values made up for this page, to show how a result is read, not a measurement of any tool. The fix follows from the cases: tighten the pattern so that it needs a leading plus sign or a recognised digit grouping, add the spaced handles to the pattern and run the same set again. The borderline group shows something else: reviewers disagree, so the policy wording needs a sentence about model numbers.

Who reviews, and how often

Assign the reference set to a named owner, usually the policy owner or the lead reviewer, and decide who may add to it. A set that anyone can edit stops being a fixed yardstick, and a set that no one owns goes stale as the marketplace changes.

Use 2 reviewers for the borderline group and record both labels. Where they differ, the difference is the finding: it is a gap in the policy text that a rule cannot close. Write the clarified policy line, relabel the listing and keep the disagreement in the log.

Re-run the full set after every revision and after any change to the policy. A short run takes minutes, and it catches the common regression in which a fix for one false positive reopens an old miss.

After activation

Testing before activation does not end the work. Watch the first days: the share of listings flagged, the share of flags that reviewers overturn and the appeals that follow. A sharp change in any of them is a reason to look at the revision again.

  • Review a sample of flagged listings each week and record the overturn rate.
  • Read every appeal as a candidate for the reference set.
  • Never let the active rules change on their own; a refinement is a draft that a person reviews and activates.
  • Keep a short log of each revision: what changed, why and who approved it.

The guide on classifieds listing moderation covers how the rules, the queue and the appeals fit together, and the glossary entry on a moderation ruleset defines the terms used here.

Boundaries

What Youzu does not do

  • Youzu does not make a marketplace compliant. You define the rules; Youzu applies them and routes exceptions to your reviewers.
  • Youzu rules do not change by themselves. False-positive feedback can produce a draft rule refinement that you review and activate.
  • Youzu publishes no accuracy figures. Measure accuracy on your own sample against your own labels, with critical errors reported separately.
How an evaluation runs

Checklist

Moderation rule test worksheet

A sheet for a reference set: one row per listing, your reviewers' decision, whether the rule flagged it and the cause of each error.

Download the worksheet (CSV)

A CSV file of 5 rows, named youzu-moderation-rule-test-worksheet.csv. No form to fill in first.

Moderation rule test worksheet, scrollable
ListingGroupPolicy saysRule flaggedCause of error
Listing 1Must flagFlag
Listing 2Must passPass
Listing 3BorderlineReviewers disagree
Listing 4Must flagFlag
Listing 5Must passPass

FAQ

Questions buyers ask

Short answers to the questions teams ask about this topic.

Yes. With Youzu, you define moderation rules as versioned rulesets with severity levels, image scope and reference images, then activate an exact revision. The wording and the severity levels are yours, because the rules state your own policy and your reviewers decide the borderline cases.

No. Active rules never change automatically. A refinement can be suggested for you to review as a draft, but the revision in force stays exactly the same until a person on your team deliberately activates another revision, so you can always say which rules applied to a listing.

No. A tested rule records your own policy and how it applied. Whether a process meets a regulation such as the Digital Services Act is a legal question for your counsel, and the article cited here is context for the idea of giving reasons, not a claim about any rule.

Start with about 100 listings, with at least 20 each that must be flagged, must pass and are borderline. Small enough that a person reads every one, large enough that each group has examples. Grow it with each appeal that shows a case the rule handled badly.

The false positives, because they hurt honest sellers and show the rule's cause most quickly: a keyword that is also in allowed listings, a scope that is too wide or a severity that is too high. Then the missed listings, which show what the wording fails to describe.

Last reviewed

Sources

  1. Regulation (EU) 2022/2065 (Digital Services Act), Article 17: statement of reasons, EUR-Lex. eur-lex.europa.eu/eli/reg/2022/2065/oj. Accessed . Back

  2. Commerce Policies, Meta Transparency Center. facebook.com/policies_center/commerce. Accessed . Back

About this guide

Published by Youzu · Editorial policy

Published

Every guide defines its terms, works through one example and cites the public sources it relies on, with the date each source was checked.

Guides explain methods. They do not rank Youzu above other approaches, and an example is never presented as a customer result.

More in all guides and the catalog glossary.

Next step

Automate Commerce. Make shopping personal.

Bring your catalog, listing flow or shopping idea. Let's find the right place to start.

Book a demo

Tell us about your catalog and we will get back to you.