Guide Shopper experiences
Visual search for e-commerce: how it works and how to evaluate it.
Visual search lets a shopper start from an image. A system detects the product in the photo, converts it to an embedding and returns exact and visually similar items from your catalog. Quality depends on clean crops, catalog attributes and ranking, so this Youzu guide shows how to evaluate it on your own images.
Published by Youzu · Editorial policy
- Published
- Reading time
- 9 min read
Key points
- See visual search as 5 steps: detect, crop, embed, retrieve and rank.
- Use exact, similar and broad labels so shoppers know what each result is.
- Check catalog images and attributes first; search quality depends on them.
- Compare vendors on your own query photos and your own catalog, never on a demo set.
- Keep text search for shoppers who know what they want; add images where words fail.
In this guide
How visual search works: detect, crop, embed, retrieve, rank
Visual search lets a shopper start from a picture instead of words: a photo of a chair seen in a café, a screenshot, an item in a room. Published descriptions of large production systems follow the same broad steps, even when their models differ.[1][2]
Diagram
From a photo to ranked products
Diagram
Detect and crop: one scene, 5 crops

- Crop 1: the lemon tree in its terracotta planter.
- Crop 2: a white folding chair.
- Crop 3: the round bistro table.
- Crop 4: a folding chair with a striped cushion.
- Crop 5: the rattan lantern.
- Crop 1: the lemon tree in its terracotta planter.
- Crop 2: a white folding chair.
- Crop 3: the round bistro table.
- Crop 4: a folding chair with a striped cushion.
- Crop 5: the rattan lantern.
- Detect. A model finds the items in the photo: a sofa, an armchair, a coffee table, a lamp.
- Crop. The chosen item is cut out of the photo, or the shopper adjusts the frame around it. A tight crop removes the floor, the wall and the other furniture that would otherwise confuse the next step.
- Embed. The crop is turned into an embedding: a list of numbers that describes its appearance, so that items that look alike end up close together.
- Retrieve. The system finds the catalog images whose embeddings are nearest to the query's. This is why every product needs a usable image: a product without one cannot be found.
- Rank. The nearest images are ordered, often with help from catalog data such as category, so a sofa query does not return a sofa-colored cushion first.
Managed services package the same idea for retailers: you index your product images, and the service returns ranked, visually similar products for a shopper's photo.[3]
The steps also explain most failures. A wrong result usually traces back to one of them: the detector picked the wrong item, the crop included the background, the catalog image was missing or unusual, or the ranking ignored the category. Diagnosing step by step finds the cause sooner than retraining anything.
Exact, similar and broad matches
Shoppers use visual search for 2 different jobs. Some want the exact item in the photo; others want something like it. Mixing the 2 in one unlabeled list disappoints both. Labeling each result makes the difference visible:
- Exact: the same product as the one in the photo, from your catalog.
- Similar: a different product that shares the key visual features, such as shape, material and color.
- Broad: a product of the same type that shares some features, useful when nothing closer exists.
The worked example below shows 3 queries against a small example catalog of 8 furniture items: an image link of a navy sofa, a product photo of a terracotta armchair and a sofa picked out of a room photo. The lamp in that room returns nothing useful, because the example catalog has no lamps, and an honest empty result serves the shopper more than a random product would.
Worked example
4 photo queries, labeled matches
Image link
A photo of a navy velvet sofa, found online
- Velvet 3-seater, navySofa · Velvet · Navy · 3 seatsExact
- Linen 3-seaterSofa · Linen · Cream · 3 seatsSimilar
- Cream 3-seaterSofa · Cream · 3 seatsSimilar
- Velvet loveseat, plumSofa · Velvet · Plum · 2 seatsBroad
Product photo
A photo of a terracotta armchair
- Armchair, terracottaArmchair · Terracotta · Wooden legsExact
- Accent chair, plumArmchair · Plum · Wooden legsSimilar
- Velvet loveseat, plumSofa · Velvet · Plum · 2 seatsBroad
Room photo, sofa selected
The sofa picked out of a living room photo
- Linen 3-seaterSofa · Linen · Cream · 3 seatsSimilar
- Cream 3-seaterSofa · Cream · 3 seatsSimilar
- Velvet 3-seater, navySofa · Velvet · Navy · 3 seatsBroad
- Velvet loveseat, plumSofa · Velvet · Plum · 2 seatsBroad
Room photo, lamp selected
The floor lamp picked out of the same photo
We could not find a useful match in this assortment. Try a closer image or select a different item.
Labels also set expectations for empty results. When the catalog has nothing close, an honest empty state with a next step, such as a closer photo or a different item, keeps more trust than a page of unrelated products. The lamp query in the example shows exactly that.
What drives quality
Model choice matters less than most buyers expect. In practice, quality is driven by 4 things, 3 of which sit in your catalog:
| Driver | Why it matters | What to check |
|---|---|---|
| Catalog images | Only products with usable images can be retrieved | Every product has a clean image of the whole item |
| Crops | A loose crop matches the background, not the product | Detection boxes are tight, and shoppers can adjust them |
| Catalog attributes | Category and attributes keep ranking on the right product type | Category is a leaf, and color and material are typed fields |
| Assortment coverage | Visual search cannot return what you do not sell | Queries for missing types return an honest empty result |
The third driver connects visual search to catalog work. Ranking and filters can use attribute extraction (pulling structured values such as color, material or size out of text or images and storing them as fields on a product record) output, so a catalog with colors in titles and categories in catch-alls produces weaker results, however good the model is.
Image consistency matters as much as image count. Catalog photos on plain backgrounds compare well with each other but less well with shoppers' photos taken in rooms and on streets. Some teams add lifestyle or room images to the search index for that reason; whichever you choose, keep the product page's main image as the plain product photo.
Evaluating on your own images
Vendor demos use queries and catalogs chosen to look good. Your shoppers will use their own photos against your catalog, so evaluate on exactly that.
- Collect real queries. Use photos your shoppers or your team would actually take: rooms, screenshots, products on people, poor light.
- Write the expected results. For each query, list which catalog products are exact matches and which would count as similar.
- Run every option on the same set. Index the same catalog images and send the same queries to each vendor or prototype.
- Judge the top results. Check whether the exact match appears when one exists, whether similar results are actually similar, and how empty results are handled.
- Measure the shopper path. Count visits to the correct product page from search results, not only clicks on any result.
Include queries that should come back empty. A careful system returns nothing, or says it found only broad matches, when the catalog has no close product; a careless one fills the page with whatever happens to be nearest.
Image search against text search
Visual search does not replace text search. Shoppers who know what they want type it, and text search handles brand names, model numbers and specifications that images cannot carry. Visual search helps where words fail: a pattern, a shape, a color, an item seen somewhere else.
Most stores therefore offer both from the same search entry point. A camera icon in the search bar, a 'find similar' action on product images and an item picker on inspiration photos cover most of the ways shoppers start from a picture. Results from either route should land on the same product pages, with the same filters.
Measure the 2 routes separately. Shoppers who start from a picture behave differently from those who type: they browse more of the results, refine less and react more to the first row. Compare each route with itself over time rather than with the other.
From the same catalog, shoppers can search with a photo, shop the products in a scene, see products in a room photo, and see products on new backgrounds or in 3D. Text search stays in your existing search; Youzu adds photo search and cleaner product data.
Boundaries
What Youzu does not do
- Youzu does not replace text search. Text search stays in your existing search; Youzu adds photo search and cleaner product data.
- Youzu publishes no search speed figures. We measure search latency on your catalog during the evaluation.
- Youzu does not claim to work on every category out of the box. Supported categories are scoped with you.
Checklist
Visual search evaluation sheet
One row per query photo: what it shows, the products you expect as exact and similar matches, and empty columns for each option's results. The first rows use the furniture example from this guide.
A CSV file of 4 rows, named youzu-visual-search-evaluation-sheet.csv. No form to fill in first.
| Query | What it shows | Expected exact match | Expected similar matches | Option A results | Option B results | Notes |
|---|---|---|---|---|---|---|
| Image link | A navy velvet 3-seat sofa | Velvet 3-seater, navy | Linen 3-seater; Cream 3-seater | |||
| Product photo | A terracotta armchair | Armchair, terracotta | Accent chair, plum | |||
| Room photo, sofa selected | A sofa in a living room | None in this catalog | Linen 3-seater; Cream 3-seater | |||
| Room photo, lamp selected | A floor lamp | None in this catalog | None in this catalog |
FAQ
Questions buyers ask
Short answers to the questions teams ask about this topic.
No. Image recognition labels what is in a picture, such as a sofa. Visual search goes further: it finds the specific products in a catalog that look most like the item in the picture. It usually uses recognition as its first step, to find and crop the item before comparing it with catalog images.
No. Visual search adds a way to start from a picture; text search remains the main way most shoppers search, and it handles brands, model numbers and specifications. Text search stays in your existing search; Youzu adds photo search and cleaner product data. Many stores offer both from the same search bar.
Usable images for every product, ideally a clean shot of the whole item, plus a leaf category and typed attributes such as color and material. Products without images cannot be found by appearance, and weak catalog data makes ranking worse. Fixing images and attributes usually improves results more than changing the model.
Collect real query photos, write down the expected exact and similar products for each, and run every vendor on the same queries against the same catalog images. Judge the top results, how empty results are handled and how often shoppers reach the correct product page. We measure search latency on your catalog during the evaluation.
Youzu Lens lets shoppers search your catalog with a photo: an upload, an image URL or an object detected in a photo; matches can be exact, similar or broad. Visual search works from your catalog's own product images; products need images. Supported categories are scoped with you.
Last reviewed
Sources
Visual Search at Pinterest, Yushi Jing, David Liu and others, arXiv. arxiv.org/abs/1505.07647. Accessed . Back
Visual Search at eBay, Fan Yang, Ajinkya Kale and others, arXiv. arxiv.org/abs/1706.03154. Accessed . Back
Vision API Product Search documentation, Google Cloud. docs.cloud.google.com/vision/product-search/docs. Accessed . Back
About this guide
Published by Youzu · Editorial policy
- Published
Every guide defines its terms, works through one example and cites the public sources it relies on, with the date each source was checked.
Guides explain methods. They do not rank Youzu above other approaches, and an example is never presented as a customer result.
More in all guides and the catalog glossary.
Related
Keep reading
Product
Youzu Lens: visual search for e-commerce
Youzu Lens lets shoppers search your catalog with a photo, an image URL or an object detected in a photo; matches can be exact, similar or broad.
Product
Discover: product discovery that starts from an image
Discover lets shoppers search your catalog with a photo: Youzu Lens visual search, Shop the Look and visually similar products from your catalog.
Glossary
What is visual search?
Visual search defined in plain words, with a worked example, common mistakes and links to the guides and Youzu products that apply it.