Skip to main content

Guide Shopper experiences

Visual search for e-commerce: how it works and how to evaluate it.

In short

Visual search lets a shopper start from an image. A system detects the product in the photo, converts it to an embedding and returns exact and visually similar items from your catalog. Quality depends on clean crops, catalog attributes and ranking, so this Youzu guide shows how to evaluate it on your own images.

Published by Youzu · Editorial policy

Published
Reading time
9 min read

Key points

  • See visual search as 5 steps: detect, crop, embed, retrieve and rank.
  • Use exact, similar and broad labels so shoppers know what each result is.
  • Check catalog images and attributes first; search quality depends on them.
  • Compare vendors on your own query photos and your own catalog, never on a demo set.
  • Keep text search for shoppers who know what they want; add images where words fail.
In this guide

How visual search works: detect, crop, embed, retrieve, rank

Visual search lets a shopper start from a picture instead of words: a photo of a chair seen in a café, a screenshot, an item in a room. Published descriptions of large production systems follow the same broad steps, even when their models differ.[1][2]

Diagram

From a photo to ranked products

From a photo to ranked products7 stacked steps. Photo: a room, a product or an image link. Detect: find the items in it. Crop: cut out the chosen item. Embed: describe its look as numbers. Retrieve: find the nearest catalog images. Rank: order them, using the category. Results: labeled exact, similar or broad.PhotoA room, a product or an image linkDetectFind the items in itCropCut out the chosen itemEmbedDescribe its look as numbersRetrieveFind the nearest catalog imagesRankOrder them, using the categoryResultsLabeled exact, similar or broad
Detect, crop, embed, retrieve and rank. The last 2 depend on your catalog: retrieval needs a usable image of every product, and ranking uses its categories and attributes.

Diagram

Detect and crop: one scene, 5 crops

A balcony with a white bistro table and chairs, a lemon tree in a terracotta pot, a striped cushion and a rattan lantern.
The detect step finds each item in the photo; the crop step cuts each one out, so every crop can be searched on its own.
  1. Crop 1: the lemon tree in its terracotta planter.
  2. Crop 2: a white folding chair.
  3. Crop 3: the round bistro table.
  4. Crop 4: a folding chair with a striped cushion.
  5. Crop 5: the rattan lantern.
  1. Detect. A model finds the items in the photo: a sofa, an armchair, a coffee table, a lamp.
  2. Crop. The chosen item is cut out of the photo, or the shopper adjusts the frame around it. A tight crop removes the floor, the wall and the other furniture that would otherwise confuse the next step.
  3. Embed. The crop is turned into an embedding: a list of numbers that describes its appearance, so that items that look alike end up close together.
  4. Retrieve. The system finds the catalog images whose embeddings are nearest to the query's. This is why every product needs a usable image: a product without one cannot be found.
  5. Rank. The nearest images are ordered, often with help from catalog data such as category, so a sofa query does not return a sofa-colored cushion first.

Managed services package the same idea for retailers: you index your product images, and the service returns ranked, visually similar products for a shopper's photo.[3]

The steps also explain most failures. A wrong result usually traces back to one of them: the detector picked the wrong item, the crop included the background, the catalog image was missing or unusual, or the ranking ignored the category. Diagnosing step by step finds the cause sooner than retraining anything.

Exact, similar and broad matches

Shoppers use visual search for 2 different jobs. Some want the exact item in the photo; others want something like it. Mixing the 2 in one unlabeled list disappoints both. Labeling each result makes the difference visible:

  • Exact: the same product as the one in the photo, from your catalog.
  • Similar: a different product that shares the key visual features, such as shape, material and color.
  • Broad: a product of the same type that shares some features, useful when nothing closer exists.

The worked example below shows 3 queries against a small example catalog of 8 furniture items: an image link of a navy sofa, a product photo of a terracotta armchair and a sofa picked out of a room photo. The lamp in that room returns nothing useful, because the example catalog has no lamps, and an honest empty result serves the shopper more than a random product would.

Worked example

4 photo queries, labeled matches

  • Image link

    A photo of a navy velvet sofa, found online

    1. Velvet 3-seater, navySofa · Velvet · Navy · 3 seatsExact
    2. Linen 3-seaterSofa · Linen · Cream · 3 seatsSimilar
    3. Cream 3-seaterSofa · Cream · 3 seatsSimilar
    4. Velvet loveseat, plumSofa · Velvet · Plum · 2 seatsBroad
  • Product photo

    A photo of a terracotta armchair

    1. Armchair, terracottaArmchair · Terracotta · Wooden legsExact
    2. Accent chair, plumArmchair · Plum · Wooden legsSimilar
    3. Velvet loveseat, plumSofa · Velvet · Plum · 2 seatsBroad
  • Room photo, sofa selected

    The sofa picked out of a living room photo

    1. Linen 3-seaterSofa · Linen · Cream · 3 seatsSimilar
    2. Cream 3-seaterSofa · Cream · 3 seatsSimilar
    3. Velvet 3-seater, navySofa · Velvet · Navy · 3 seatsBroad
    4. Velvet loveseat, plumSofa · Velvet · Plum · 2 seatsBroad
  • Room photo, lamp selected

    The floor lamp picked out of the same photo

    We could not find a useful match in this assortment. Try a closer image or select a different item.

The example catalog holds 8 furniture items and no lamps, so the lamp query returns no useful match.

Labels also set expectations for empty results. When the catalog has nothing close, an honest empty state with a next step, such as a closer photo or a different item, keeps more trust than a page of unrelated products. The lamp query in the example shows exactly that.

What drives quality

Model choice matters less than most buyers expect. In practice, quality is driven by 4 things, 3 of which sit in your catalog:

4 drivers of visual search quality and what to check for each
DriverWhy it mattersWhat to check
Catalog imagesOnly products with usable images can be retrievedEvery product has a clean image of the whole item
CropsA loose crop matches the background, not the productDetection boxes are tight, and shoppers can adjust them
Catalog attributesCategory and attributes keep ranking on the right product typeCategory is a leaf, and color and material are typed fields
Assortment coverageVisual search cannot return what you do not sellQueries for missing types return an honest empty result

The third driver connects visual search to catalog work. Ranking and filters can use attribute extraction (pulling structured values such as color, material or size out of text or images and storing them as fields on a product record) output, so a catalog with colors in titles and categories in catch-alls produces weaker results, however good the model is.

Image consistency matters as much as image count. Catalog photos on plain backgrounds compare well with each other but less well with shoppers' photos taken in rooms and on streets. Some teams add lifestyle or room images to the search index for that reason; whichever you choose, keep the product page's main image as the plain product photo.

Evaluating on your own images

Vendor demos use queries and catalogs chosen to look good. Your shoppers will use their own photos against your catalog, so evaluate on exactly that.

  1. Collect real queries. Use photos your shoppers or your team would actually take: rooms, screenshots, products on people, poor light.
  2. Write the expected results. For each query, list which catalog products are exact matches and which would count as similar.
  3. Run every option on the same set. Index the same catalog images and send the same queries to each vendor or prototype.
  4. Judge the top results. Check whether the exact match appears when one exists, whether similar results are actually similar, and how empty results are handled.
  5. Measure the shopper path. Count visits to the correct product page from search results, not only clicks on any result.

Include queries that should come back empty. A careful system returns nothing, or says it found only broad matches, when the catalog has no close product; a careless one fills the page with whatever happens to be nearest.

Visual search does not replace text search. Shoppers who know what they want type it, and text search handles brand names, model numbers and specifications that images cannot carry. Visual search helps where words fail: a pattern, a shape, a color, an item seen somewhere else.

Most stores therefore offer both from the same search entry point. A camera icon in the search bar, a 'find similar' action on product images and an item picker on inspiration photos cover most of the ways shoppers start from a picture. Results from either route should land on the same product pages, with the same filters.

Measure the 2 routes separately. Shoppers who start from a picture behave differently from those who type: they browse more of the results, refine less and react more to the first row. Compare each route with itself over time rather than with the other.

From the same catalog, shoppers can search with a photo, shop the products in a scene, see products in a room photo, and see products on new backgrounds or in 3D. Text search stays in your existing search; Youzu adds photo search and cleaner product data.

Boundaries

What Youzu does not do

  • Youzu does not replace text search. Text search stays in your existing search; Youzu adds photo search and cleaner product data.
  • Youzu publishes no search speed figures. We measure search latency on your catalog during the evaluation.
  • Youzu does not claim to work on every category out of the box. Supported categories are scoped with you.
How an evaluation runs

Checklist

Visual search evaluation sheet

One row per query photo: what it shows, the products you expect as exact and similar matches, and empty columns for each option's results. The first rows use the furniture example from this guide.

Download the evaluation sheet (CSV)

A CSV file of 4 rows, named youzu-visual-search-evaluation-sheet.csv. No form to fill in first.

Visual search evaluation sheet, scrollable
QueryWhat it showsExpected exact matchExpected similar matchesOption A resultsOption B resultsNotes
Image linkA navy velvet 3-seat sofaVelvet 3-seater, navyLinen 3-seater; Cream 3-seater
Product photoA terracotta armchairArmchair, terracottaAccent chair, plum
Room photo, sofa selectedA sofa in a living roomNone in this catalogLinen 3-seater; Cream 3-seater
Room photo, lamp selectedA floor lampNone in this catalogNone in this catalog

FAQ

Questions buyers ask

Short answers to the questions teams ask about this topic.

No. Image recognition labels what is in a picture, such as a sofa. Visual search goes further: it finds the specific products in a catalog that look most like the item in the picture. It usually uses recognition as its first step, to find and crop the item before comparing it with catalog images.

No. Visual search adds a way to start from a picture; text search remains the main way most shoppers search, and it handles brands, model numbers and specifications. Text search stays in your existing search; Youzu adds photo search and cleaner product data. Many stores offer both from the same search bar.

Usable images for every product, ideally a clean shot of the whole item, plus a leaf category and typed attributes such as color and material. Products without images cannot be found by appearance, and weak catalog data makes ranking worse. Fixing images and attributes usually improves results more than changing the model.

Collect real query photos, write down the expected exact and similar products for each, and run every vendor on the same queries against the same catalog images. Judge the top results, how empty results are handled and how often shoppers reach the correct product page. We measure search latency on your catalog during the evaluation.

Youzu Lens lets shoppers search your catalog with a photo: an upload, an image URL or an object detected in a photo; matches can be exact, similar or broad. Visual search works from your catalog's own product images; products need images. Supported categories are scoped with you.

Last reviewed

Sources

  1. Visual Search at Pinterest, Yushi Jing, David Liu and others, arXiv. arxiv.org/abs/1505.07647. Accessed . Back

  2. Visual Search at eBay, Fan Yang, Ajinkya Kale and others, arXiv. arxiv.org/abs/1706.03154. Accessed . Back

  3. Vision API Product Search documentation, Google Cloud. docs.cloud.google.com/vision/product-search/docs. Accessed . Back

About this guide

Published by Youzu · Editorial policy

Published

Every guide defines its terms, works through one example and cites the public sources it relies on, with the date each source was checked.

Guides explain methods. They do not rank Youzu above other approaches, and an example is never presented as a customer result.

More in all guides and the catalog glossary.

Next step

Automate Commerce. Make shopping personal.

Bring your catalog, listing flow or shopping idea. Let's find the right place to start.

Book a demo

Tell us about your catalog and we will get back to you.