Guides

Can Google crawl, index and trust your online store?

A product page can be indexed only if robots.txt allows it, it returns 200 with no noindex and its canonical points to itself.

They can if each product page clears a short list of gates. robots.txt must let Googlebot and Bingbot fetch it. It must return 200 with no noindex. It needs one absolute canonical in <head> that points to itself, and it should be in your sitemap and linked from a category. Then Google reads your shipping policy, return policy and contact details to decide whether to trust the store.

Why aren't my product pages indexed?

Work through the gates in the order Google meets them, and stop at the first one that fails: robots.txt allows the URL, HTTP 200, real content after render, no noindex, a valid canonical, listed in a sitemap, linked internally, eligible for a snippet. The sitemap and internal links help Google find the page; the other gates decide whether it can be indexed at all.

  1. robots.txt blocks the path. Google can't fetch a disallowed URL, so it can't read your content, your noindex or your canonical. At best it indexes the bare URL with no snippet.
  2. The page doesn't return 200. Only 200 pages can be indexed and shown with a snippet. A client-side 404 page served with a 200 status is also a problem. Google's JavaScript SEO guide says to "use a meaningful status code" (as of 2026-03-04).
  3. A noindex you can't see. A noindex can sit in the meta robots tag or in an X-Robots-Tag HTTP header. The header is often set by a CDN, a plugin or a staging config left on after launch. View source won't show it, so check the response headers.
  4. The canonical points somewhere else. If the canonical names another URL, you are asking Google to index that URL instead.

The same gates decide AI features. As of 2025-12-10, Google says AI Overviews and AI Mode need no special markup: a page is eligible when it is indexed and can be shown with a snippet. So a product page that isn't indexed won't be quoted in AI Overviews either. Also watch for nosnippet or max-snippet:0. They let a page rank but stop Google quoting it in AI answers.

Should variant URLs have their own canonical?

It depends on how your variants are built. Google's product variants guide (as of 2026-09-08) describes two setups:

  • Single-page variants (one product page, a ?variant= or similar parameter picks the size or colour): every variant URL uses one canonical, the base product URL with no variant preselected.
  • Multi-page variants (each variant has its own page): each page canonicalises to itself and carries full, self-contained markup.

Whichever you use, the canonical tag has to follow Google's rules (as of 2026-07-10). It is only accepted in <head>. It should be an absolute URL. It belongs on the canonical page itself, not only on the duplicates. It should point to a URL that returns 200, is indexable and doesn't redirect. Google's JavaScript guide also says not to use JavaScript to change the canonical.

<head>
  <link rel="canonical" href="https://example.com/products/linen-shirt">
</head>

The same product often lives at more than one path, such as Shopify's /collections/shirts/products/linen-shirt and /products/linen-shirt. Shopify's docs call these "the same content on separate URLs". Point product grids at the canonical URL; on Shopify, remove | within: collection from the product card links.

What should I do with out-of-stock product pages?

Keep them live. Google's guidance (as of 2025-12-10) is: "It's better to still allow that page, and mark it out of stock."

  1. Keep a temporarily out-of-stock page at 200 and indexable.
  2. Set availability to https://schema.org/OutOfStock in the product markup. Use BackOrder or PreOrder, with a date, if that is what you offer.
  3. Show the stock state on the page so it matches the markup.
  4. Don't noindex the page, and don't redirect it to your home page or a category.
  5. When a product is discontinued for good, return 404 or 410, or 301 the URL to a close replacement.

Don't leave a sold-out page as an empty template that still returns 200. Google may treat it as a soft 404.

How do I stop filter and tracking URLs wasting crawl?

Category pages with filters can create thousands of URLs for the same products. Google's faceted navigation guide (as of 2025-12-18) says: "Use robots.txt to disallow crawling of faceted navigation URLs". You can also build filters as fragments or as controls that aren't links. Keep filter order consistent, and return 404 when a filter combination has no results. A rel=nofollow only works if every link to that URL has it.

Google's URL structure guide (as of 2025-12-10) also says to use ?key=value, not to repeat a parameter, and not to link internally to session IDs or tracking codes such as utm_.

Pagination is the exception: page 2 and beyond should be crawlable. Use real <a href> links with unique URLs such as ?page=2, and let each page canonicalise to itself. Google doesn't click "load more" buttons, and says not to use page 1 as the canonical for the whole series.

Does robots.txt let search crawlers fetch my products?

Check it directly. smbuild tests sampled product and image URLs against your robots.txt for Googlebot, Googlebot-Image, Storebot-Google and Bingbot. If robots.txt returns a 5xx error or can't be reached, crawlers treat that as "disallow everything".

A store robots.txt usually keeps products, collections and images open, blocks cart, checkout and filter parameters, and points to the sitemap:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /*?*filter.
Allow: /

Sitemap: https://example.com/sitemap.xml

Don't use robots.txt to hide duplicate URLs. Google says "Don't use the robots.txt file for canonicalization purposes" (as of 2026-07-10); use a canonical tag. On Shopify, keep the default robots.txt rules. Shopify warns that incorrect edits "can result in loss of all traffic".

Do I need a product sitemap?

Yes, unless every product is linked from your categories and you send a Merchant Center feed. Google says to link to every product you want indexed, and otherwise to "use a sitemap or a Google Merchant Center feed". It is also "strongly recommended to link to all products that you wish indexed" (both as of 2025-12-10).

A good product sitemap:

  1. Lists every live product URL, absolute and canonical, each returning 200.
  2. Keeps each file under 50,000 URLs and 50 MB uncompressed.
  3. Uses real lastmod dates. Google uses lastmod only when it is consistently and verifiably accurate, and ignores priority and changefreq (as of 2026-07-08).
  4. Is referenced from robots.txt with a Sitemap: line.

Product images need to be crawlable too. Google says "Google doesn't index CSS images" (as of 2026-03-02), so put product photos in <img> tags with descriptive alt text.

Does Google read my return policy and shipping policy?

Yes, from your pages and from markup. As of 2026-10-17, Google Merchant Center requires a website return policy for free listings. The policy has to be explicit even if you accept no returns: the process, the timeframe, who pays return shipping and when refunds arrive. Link it from the footer so it is one click from every page.

Shipping costs are required for free listings in about 30 countries, including the US, UK, Canada, Australia, Germany, France and Japan. Merchant-level shipping markup (Organization.hasShippingService) launched on 2025-11-12. It only needs to be on one page. Settings in Merchant Center or Search Console take precedence, so the markup is the fallback when you don't have them.

Add return policy markup once, on your Organization node. Google's return policy guide (as of 2026-09-08) accepts applicableCountry plus returnPolicyCategory (plus merchantReturnDays for a finite window), or a merchantReturnLink. Add returnPolicyCountry too: a 2025-03-14 changelog called it required, though the current page lists it as recommended.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "OnlineStore",
  "name": "Example Store",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "email": "help@example.com",
  "hasMerchantReturnPolicy": {
    "@type": "MerchantReturnPolicy",
    "applicableCountry": "US",
    "returnPolicyCountry": "US",
    "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
    "merchantReturnDays": 30,
    "returnMethod": "https://schema.org/ReturnByMail",
    "returnFees": "https://schema.org/FreeReturn",
    "merchantReturnLink": "https://example.com/policies/returns"
  }
}
</script>

The same Organization (ideally OnlineStore) node is how Google identifies the merchant: name, url, logo and at least one contact method. Google calls vatID "an important trust signal" (as of 2026-09-08). Merchant Center also requires contact information to be "clearly available", which smbuild reads as at least two of email, phone, address or a contact form. It also requires payment data to be collected on SSL-protected pages, so serve every page over HTTPS and redirect http:// to https://.

How smbuild checks this

smbuild fetches your home page, robots.txt, sitemaps, one or two category pages, up to 12 product pages and your shipping, returns and contact pages over plain HTTP, and reads the status codes, headers, canonicals and JSON-LD. What it can't see from outside, such as zero-result filter URLs or Merchant Center settings, is marked Not measured, not failed.

CheckTitle
IDX-01Faceted navigation bounded
IDX-02Internal link parameter hygiene
IDX-03Canonical tags on product pages
IDX-04One URL per product
IDX-05Out-of-stock pages handled correctly
IDX-06Category pagination crawlable
IDX-09Product pages indexable
IDX-10Products linked internally
ORG-10robots.txt lets search crawlers fetch products
ORG-11Product sitemap
ORG-12Product images crawlable (img tags, alt text)
ORG-01Organization / OnlineStore markup
ORG-02Shipping policy page and markup
ORG-03Return policy page and markup
ORG-05aHTTPS everywhere
ORG-05bContact information clearly available

Sources

Questions

Should out-of-stock product pages be noindexed?

No. Keep a temporarily out-of-stock page at 200 and indexable, mark it OutOfStock in the markup and show the stock state on the page. Only discontinued products should return 404 or 410, or 301 to a close replacement.

Should every product variant have its own canonical URL?

Only if each variant has its own page. If one product page handles all variants through a parameter, every variant URL should canonicalise to the base product URL with no variant preselected.

Can I use robots.txt to block duplicate product URLs?

Google says not to use robots.txt for canonicalisation. A blocked URL can't show Google its canonical tag, so use a canonical that points to the main product URL instead.

Why does my product page show noindex when the HTML doesn't have it?

The noindex is probably in an X-Robots-Tag HTTP header, often set by a CDN, a plugin or a leftover staging config. Check the page's response headers and remove it at the server or CDN.

Does Google require a return policy for free listings?

Yes. As of 2026-10-17, Merchant Center requires a return policy on your website for free listings, stated explicitly even if you accept no returns. Adding hasMerchantReturnPolicy markup to your Organization node lets Google read it.

Do I need special markup to appear in AI Overviews?

No. As of 2025-12-10, Google says a page is eligible for AI Overviews and AI Mode when it is indexed and can be shown with a snippet. Remove noindex, nosnippet and max-snippet:0 from product pages you want quoted.

Check your own site

Get a free snapshot to run these checks on your site, or book a 20-minute call and we’ll walk you through what we find.

More guides

By Nathan Tarasiuk, founder of smbuild. Updated . Markdown.