Guide Listings and moderation
How to test a moderation rule before you switch it on.
Test a moderation rule on a reference set of listings before it goes live. Count the false flags and the misses apart, fix their causes, then activate one exact revision of the ruleset.
Published by Youzu · Editorial policy
- Published
- Reading time
- 8 min read
Key points
- Start with a reference set of listings before you write the rule, not after.
- Place in it listings that must be flagged, must pass and are borderline.
- Compare the rule's flags with your reviewers' labels, and count each kind of error.
- Use one exact revision when you activate, never the latest.
In this guide
Why test a rule before you switch it on
A moderation rule is a prediction about which listings break a policy. Until it has been run against real listings, nobody knows how many it would flag, how many of those are right and how many it would miss. A rule that flags too much floods reviewers and annoys honest sellers; a rule that flags too little lets the listings it was written for through.
Testing does not need special tools. It needs a set of listings, your reviewers' labels for them and a way to compare what the rule would have done with what the labels say. A spreadsheet is enough for the first rounds, and the work is the same whether the rule is written by a person or proposed by software. What matters is that the comparison is written down, so that the next person can repeat it and reach the same count.
Build a reference set
A reference set is a list of listings with a decision beside each: should this listing be flagged under the policy, yes or no. Take the listings from your own marketplace, including old ones that reviewers already decided, and write down who decided and why.
| Group | What it holds | What it tells you |
|---|---|---|
| Must flag | Listings that clearly break the rule | Whether the rule catches what it was written for |
| Must pass | Listings that look similar but are allowed | Whether the rule flags honest sellers |
| Borderline | Listings your reviewers disagree about | Where the policy wording needs work |
Keep the set small enough that a person can read every listing in it, and large enough to hold a few examples of each group, including the odd cases that only a marketplace like yours attracts. A set of about 100 listings, with at least 20 in each group, is a workable start. Add a listing every time an appeal or a reviewer shows a case the rule handled badly, so the set grows from real mistakes and not from guesses about what might go wrong.
Run the rule and compare
Apply the rule to every listing in the set and record whether it would flag it. Then compare with the labels. Each listing falls into one of 4 cells: flagged and should be, flagged and should not be, passed and should be flagged, passed and rightly so.
| Policy says flag | Policy says pass | |
|---|---|---|
| Rule flags | Correct flag | False positive: an honest listing is flagged |
| Rule passes | Missed listing: a breach gets through | Correct pass |
Count the false positives and the missed listings apart, and read every one. Most rules show a pattern within a few cases: a keyword that appears in allowed listings, a scope that includes images the rule was not meant to check, or a severity set higher than the harm. Fix the cause, and run the set again.
Worksheet
Sketch 3 rules for your own policy
Would flag 2 of 8 listings.
Would flag 2 of 8 listings.
Would flag 2 of 8 listings.
The 8 sample listings and the rules that would flag them Listing What a reviewer sees Flagged by 1. Oak dining table A table photographed in a living room. Not flagged 2. City bike, good condition The description asks buyers to phone the seller directly. Rule 1 3. Leather sofa A phone number is written on a card in the photo. Rule 1 4. Replica sword, decorative The title names a replica weapon, and the photo shows it. Rule 2 5. Camping set The photo shows a hunting knife beside the tent. Rule 2 6. Designer floor lamp The photo is copied from a shop's product page. Rule 3 7. Wool rug The photo carries another seller's watermark. Rule 3 8. Kids' bike, half price today The title carries a price offer. Not flagged
Read as text
Severity, scope and revisions
Decide the severity before activation, because it decides what happens to a flagged listing. A high-severity rule might send a listing to a person before it goes live; a low-severity one might only add a note for the seller. The same finding can be right for one severity and wrong for another.
Write down the scope as well: text only, text and the first image, or every image. Scope changes the result more than wording does, because a rule that reads images flags listings a text-only rule never sees. Test each scope you intend to use.
Save each change as a new revision of the ruleset and activate an exact revision, never 'the latest'. A decision can then always be traced to the rule text that produced it, and an earlier revision can be restored if a new one turns out to be too broad. Public marketplace policies and regulation both point the same way: say which rule applied, and keep a record of it.[1][2]
A worked example: a rule about contact details
Take a policy line that says listings may not put phone numbers or messaging handles in the title, because contact outside the platform removes the buyer's protection. The rule flags titles that contain a run of digits shaped like a phone number, or a handle written after an at sign.
Run it on a reference set of 60 listings: 20 that must be flagged, 20 that must pass and 20 borderline. The first run flags 18 of the 20 that must be flagged and 4 of the 20 that must pass. The 4 are listings whose titles include a model number or a measurement, such as '5000 mAh 2021', which the rule read as a phone number.
| Group | Listings | Flagged by the rule | Reading |
|---|---|---|---|
| Must flag | 20 | 18 | 2 missed: handles written with spaces |
| Must pass | 20 | 4 | 4 false positives: model numbers read as phones |
| Borderline | 20 | 9 | Reviewers disagree on 7 of the 9 |
The numbers are round values made up for this page, to show how a result is read, not a measurement of any tool. The fix follows from the cases: tighten the pattern so that it needs a leading plus sign or a recognised digit grouping, add the spaced handles to the pattern and run the same set again. The borderline group shows something else: reviewers disagree, so the policy wording needs a sentence about model numbers.
Who reviews, and how often
Assign the reference set to a named owner, usually the policy owner or the lead reviewer, and decide who may add to it. A set that anyone can edit stops being a fixed yardstick, and a set that no one owns goes stale as the marketplace changes.
Use 2 reviewers for the borderline group and record both labels. Where they differ, the difference is the finding: it is a gap in the policy text that a rule cannot close. Write the clarified policy line, relabel the listing and keep the disagreement in the log.
Re-run the full set after every revision and after any change to the policy. A short run takes minutes, and it catches the common regression in which a fix for one false positive reopens an old miss.
After activation
Testing before activation does not end the work. Watch the first days: the share of listings flagged, the share of flags that reviewers overturn and the appeals that follow. A sharp change in any of them is a reason to look at the revision again.
- Review a sample of flagged listings each week and record the overturn rate.
- Read every appeal as a candidate for the reference set.
- Never let the active rules change on their own; a refinement is a draft that a person reviews and activates.
- Keep a short log of each revision: what changed, why and who approved it.
The guide on classifieds listing moderation covers how the rules, the queue and the appeals fit together, and the glossary entry on a moderation ruleset defines the terms used here.
Boundaries
What Youzu does not do
- Youzu does not make a marketplace compliant. You define the rules; Youzu applies them and routes exceptions to your reviewers.
- Youzu rules do not change by themselves. False-positive feedback can produce a draft rule refinement that you review and activate.
- Youzu publishes no accuracy figures. Measure accuracy on your own sample against your own labels, with critical errors reported separately.
Checklist
Moderation rule test worksheet
A sheet for a reference set: one row per listing, your reviewers' decision, whether the rule flagged it and the cause of each error.
A CSV file of 5 rows, named youzu-moderation-rule-test-worksheet.csv. No form to fill in first.
| Listing | Group | Policy says | Rule flagged | Cause of error |
|---|---|---|---|---|
| Listing 1 | Must flag | Flag | ||
| Listing 2 | Must pass | Pass | ||
| Listing 3 | Borderline | Reviewers disagree | ||
| Listing 4 | Must flag | Flag | ||
| Listing 5 | Must pass | Pass |
FAQ
Questions buyers ask
Short answers to the questions teams ask about this topic.
Yes. With Youzu, you define moderation rules as versioned rulesets with severity levels, image scope and reference images, then activate an exact revision. The wording and the severity levels are yours, because the rules state your own policy and your reviewers decide the borderline cases.
No. Active rules never change automatically. A refinement can be suggested for you to review as a draft, but the revision in force stays exactly the same until a person on your team deliberately activates another revision, so you can always say which rules applied to a listing.
No. A tested rule records your own policy and how it applied. Whether a process meets a regulation such as the Digital Services Act is a legal question for your counsel, and the article cited here is context for the idea of giving reasons, not a claim about any rule.
Start with about 100 listings, with at least 20 each that must be flagged, must pass and are borderline. Small enough that a person reads every one, large enough that each group has examples. Grow it with each appeal that shows a case the rule handled badly.
The false positives, because they hurt honest sellers and show the rule's cause most quickly: a keyword that is also in allowed listings, a scope that is too wide or a severity that is too high. Then the missed listings, which show what the wording fails to describe.
Last reviewed
Sources
Regulation (EU) 2022/2065 (Digital Services Act), Article 17: statement of reasons, EUR-Lex. eur-lex.europa.eu/eli/reg/2022/2065/oj. Accessed . Back
Commerce Policies, Meta Transparency Center. facebook.com/policies_center/commerce. Accessed . Back
About this guide
Published by Youzu · Editorial policy
- Published
Every guide defines its terms, works through one example and cites the public sources it relies on, with the date each source was checked.
Guides explain methods. They do not rank Youzu above other approaches, and an example is never presented as a customer result.
More in all guides and the catalog glossary.
Related
Keep reading
Product
Listing moderation for marketplaces and classifieds
Define moderation rules with severity, image scope and reference images, test them, preview their impact and review flagged products in a queue.
Glossary
What is a moderation ruleset?
Moderation ruleset defined in plain words, with a worked example, common mistakes and links to the guides and Youzu products that apply it.
Guide
Classifieds listing moderation: policies, reason codes and review queues
How to design listing moderation for classifieds and marketplaces: policy rules per category, reason codes, review queues, testing and appeals.