# Can Google crawl, index and trust your online store?

> A product page can be indexed only if robots.txt allows it, it returns 200 with no noindex and its canonical points to itself.

Updated 2026-10-21 · https://trysmbuild.com/guides/store-indexing-and-trust

They can if each product page clears a short list of gates. robots.txt must let Googlebot and Bingbot fetch it. It must return 200 with no noindex. It needs one absolute canonical in `<head>` that points to itself, and it should be in your sitemap and linked from a category. Then Google reads your shipping policy, return policy and contact details to decide whether to trust the store.

## Why aren't my product pages indexed?

Work through the gates in the order Google meets them, and stop at the first one that fails: robots.txt allows the URL, HTTP 200, real content after render, no noindex, a valid canonical, listed in a sitemap, linked internally, eligible for a snippet. The sitemap and internal links help Google find the page; the other gates decide whether it can be indexed at all.

1. **robots.txt blocks the path.** Google can't fetch a disallowed URL, so it can't read your content, your noindex or your canonical. At best it indexes the bare URL with no snippet.
2. **The page doesn't return 200.** Only 200 pages can be indexed and shown with a snippet. A client-side 404 page served with a 200 status is also a problem. Google's JavaScript SEO guide says to "use a meaningful status code" (as of 2026-03-04).
3. **A noindex you can't see.** A `noindex` can sit in the meta robots tag or in an `X-Robots-Tag` HTTP header. The header is often set by a CDN, a plugin or a staging config left on after launch. View source won't show it, so check the response headers.
4. **The canonical points somewhere else.** If the canonical names another URL, you are asking Google to index that URL instead.

The same gates decide AI features. As of 2025-12-10, Google says AI Overviews and AI Mode need no special markup: a page is eligible when it is indexed and can be shown with a snippet. So a product page that isn't indexed won't be quoted in AI Overviews either. Also watch for `nosnippet` or `max-snippet:0`. They let a page rank but stop Google quoting it in AI answers.

## Should variant URLs have their own canonical?

It depends on how your variants are built. Google's product variants guide (as of 2026-09-08) describes two setups:

- **Single-page variants** (one product page, a `?variant=` or similar parameter picks the size or colour): every variant URL uses one canonical, the base product URL with no variant preselected.
- **Multi-page variants** (each variant has its own page): each page canonicalises to itself and carries full, self-contained markup.

Whichever you use, the canonical tag has to follow Google's rules (as of 2026-07-10). It is only accepted in `<head>`. It should be an absolute URL. It belongs on the canonical page itself, not only on the duplicates. It should point to a URL that returns 200, is indexable and doesn't redirect. Google's JavaScript guide also says not to use JavaScript to change the canonical.

```html
<head>
  <link rel="canonical" href="https://example.com/products/linen-shirt">
</head>
```

The same product often lives at more than one path, such as Shopify's `/collections/shirts/products/linen-shirt` and `/products/linen-shirt`. Shopify's docs call these "the same content on separate URLs". Point product grids at the canonical URL; on Shopify, remove `| within: collection` from the product card links.

## What should I do with out-of-stock product pages?

Keep them live. Google's guidance (as of 2025-12-10) is: "It's better to still allow that page, and mark it out of stock."

1. Keep a temporarily out-of-stock page at 200 and indexable.
2. Set `availability` to `https://schema.org/OutOfStock` in the product markup. Use BackOrder or PreOrder, with a date, if that is what you offer.
3. Show the stock state on the page so it matches the markup.
4. Don't noindex the page, and don't redirect it to your home page or a category.
5. When a product is discontinued for good, return 404 or 410, or 301 the URL to a close replacement.

Don't leave a sold-out page as an empty template that still returns 200. Google may treat it as a soft 404.

## How do I stop filter and tracking URLs wasting crawl?

Category pages with filters can create thousands of URLs for the same products. Google's faceted navigation guide (as of 2025-12-18) says: "Use robots.txt to disallow crawling of faceted navigation URLs". You can also build filters as fragments or as controls that aren't links. Keep filter order consistent, and return 404 when a filter combination has no results. A `rel=nofollow` only works if every link to that URL has it.

Google's URL structure guide (as of 2025-12-10) also says to use `?key=value`, not to repeat a parameter, and not to link internally to session IDs or tracking codes such as `utm_`.

Pagination is the exception: page 2 and beyond should be crawlable. Use real `<a href>` links with unique URLs such as `?page=2`, and let each page canonicalise to itself. Google doesn't click "load more" buttons, and says not to use page 1 as the canonical for the whole series.

## Does robots.txt let search crawlers fetch my products?

Check it directly. smbuild tests sampled product and image URLs against your robots.txt for Googlebot, Googlebot-Image, Storebot-Google and Bingbot. If robots.txt returns a 5xx error or can't be reached, crawlers treat that as "disallow everything".

A store robots.txt usually keeps products, collections and images open, blocks cart, checkout and filter parameters, and points to the sitemap:

```text
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /*?*filter.
Allow: /

Sitemap: https://example.com/sitemap.xml
```

Don't use robots.txt to hide duplicate URLs. Google says "Don't use the robots.txt file for canonicalization purposes" (as of 2026-07-10); use a canonical tag. On Shopify, keep the default robots.txt rules. Shopify warns that incorrect edits "can result in loss of all traffic".

## Do I need a product sitemap?

Yes, unless every product is linked from your categories and you send a Merchant Center feed. Google says to link to every product you want indexed, and otherwise to "use a sitemap or a Google Merchant Center feed". It is also "strongly recommended to link to all products that you wish indexed" (both as of 2025-12-10).

A good product sitemap:

1. Lists every live product URL, absolute and canonical, each returning 200.
2. Keeps each file under 50,000 URLs and 50 MB uncompressed.
3. Uses real `lastmod` dates. Google uses lastmod only when it is consistently and verifiably accurate, and ignores priority and changefreq (as of 2026-07-08).
4. Is referenced from robots.txt with a `Sitemap:` line.

Product images need to be crawlable too. Google says "Google doesn't index CSS images" (as of 2026-03-02), so put product photos in `<img>` tags with descriptive alt text.

## Does Google read my return policy and shipping policy?

Yes, from your pages and from markup. As of 2026-10-17, Google Merchant Center requires a website return policy for free listings. The policy has to be explicit even if you accept no returns: the process, the timeframe, who pays return shipping and when refunds arrive. Link it from the footer so it is one click from every page.

Shipping costs are required for free listings in about 30 countries, including the US, UK, Canada, Australia, Germany, France and Japan. Merchant-level shipping markup (`Organization.hasShippingService`) launched on 2025-11-12. It only needs to be on one page. Settings in Merchant Center or Search Console take precedence, so the markup is the fallback when you don't have them.

Add return policy markup once, on your Organization node. Google's return policy guide (as of 2026-09-08) accepts `applicableCountry` plus `returnPolicyCategory` (plus `merchantReturnDays` for a finite window), or a `merchantReturnLink`. Add `returnPolicyCountry` too: a 2025-03-14 changelog called it required, though the current page lists it as recommended.

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "OnlineStore",
  "name": "Example Store",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "email": "help@example.com",
  "hasMerchantReturnPolicy": {
    "@type": "MerchantReturnPolicy",
    "applicableCountry": "US",
    "returnPolicyCountry": "US",
    "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
    "merchantReturnDays": 30,
    "returnMethod": "https://schema.org/ReturnByMail",
    "returnFees": "https://schema.org/FreeReturn",
    "merchantReturnLink": "https://example.com/policies/returns"
  }
}
</script>
```

The same Organization (ideally OnlineStore) node is how Google identifies the merchant: name, url, logo and at least one contact method. Google calls `vatID` "an important trust signal" (as of 2026-09-08). Merchant Center also requires contact information to be "clearly available", which smbuild reads as at least two of email, phone, address or a contact form. It also requires payment data to be collected on SSL-protected pages, so serve every page over HTTPS and redirect `http://` to `https://`.

## How smbuild checks this

smbuild fetches your home page, robots.txt, sitemaps, one or two category pages, up to 12 product pages and your shipping, returns and contact pages over plain HTTP, and reads the status codes, headers, canonicals and JSON-LD. What it can't see from outside, such as zero-result filter URLs or Merchant Center settings, is marked Not measured, not failed.

| Check | Title |
|---|---|
| IDX-01 | Faceted navigation bounded |
| IDX-02 | Internal link parameter hygiene |
| IDX-03 | Canonical tags on product pages |
| IDX-04 | One URL per product |
| IDX-05 | Out-of-stock pages handled correctly |
| IDX-06 | Category pagination crawlable |
| IDX-09 | Product pages indexable |
| IDX-10 | Products linked internally |
| ORG-10 | robots.txt lets search crawlers fetch products |
| ORG-11 | Product sitemap |
| ORG-12 | Product images crawlable (img tags, alt text) |
| ORG-01 | Organization / OnlineStore markup |
| ORG-02 | Shipping policy page and markup |
| ORG-03 | Return policy page and markup |
| ORG-05a | HTTPS everywhere |
| ORG-05b | Contact information clearly available |

## Sources

- [Google Search Central: Consolidate duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) (updated 2026-07-10)
- [Google Search Central: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) (updated 2026-03-04)
- [Google Search Central: Product variants structured data](https://developers.google.com/search/docs/appearance/structured-data/product-variants) (updated 2026-09-08)
- [Google Search Central: Pause or disable an online business](https://developers.google.com/search/docs/crawling-indexing/pause-online-business) (updated 2025-12-10)
- [Google Search Central: Faceted navigation](https://developers.google.com/crawling/docs/faceted-navigation) (updated 2025-12-18)
- [Google Search Central: Designing a URL structure for ecommerce sites](https://developers.google.com/search/docs/specialty/ecommerce/designing-a-url-structure-for-ecommerce-sites) (updated 2025-12-10)
- [Google Search Central: Pagination and incremental page loading](https://developers.google.com/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading) (updated 2025-12-10)
- [Google Search Central: Help Google understand your ecommerce site structure](https://developers.google.com/search/docs/specialty/ecommerce/help-google-understand-your-ecommerce-site-structure) (updated 2025-12-10)
- [Google Search Central: Robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro) (checked 2026-10-17)
- [Google Search Central: Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) (updated 2026-07-08)
- [Google Search Central: Google Images best practices](https://developers.google.com/search/docs/appearance/google-images) (updated 2026-03-02)
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) (effective 2025-12-10)
- [Google Search Central: Return policy structured data](https://developers.google.com/search/docs/appearance/structured-data/return-policy) (updated 2026-09-08)
- [Google Search Central: Shipping policy structured data](https://developers.google.com/search/docs/appearance/structured-data/shipping-policy) (updated 2026-09-08)
- [Google Search Central: Organization structured data](https://developers.google.com/search/docs/appearance/structured-data/organization) (updated 2026-09-08)
- [Google Merchant Center Help: Merchant Center requirements](https://support.google.com/merchants/answer/6363310) (checked 2026-10-17)
- [Google Merchant Center Help: Free listings requirements](https://support.google.com/merchants/answer/9199328) (checked 2026-10-17)
- [Shopify: Liquid `within` filter](https://shopify.dev/docs/api/liquid/filters/within) (checked 2026-10-17)
- [Shopify Help Center: Editing robots.txt.liquid](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt) (checked 2026-10-17)

## Questions

### Should out-of-stock product pages be noindexed?

No. Keep a temporarily out-of-stock page at 200 and indexable, mark it `OutOfStock` in the markup and show the stock state on the page. Only discontinued products should return 404 or 410, or 301 to a close replacement.

### Should every product variant have its own canonical URL?

Only if each variant has its own page. If one product page handles all variants through a parameter, every variant URL should canonicalise to the base product URL with no variant preselected.

### Can I use robots.txt to block duplicate product URLs?

Google says not to use robots.txt for canonicalisation. A blocked URL can't show Google its canonical tag, so use a canonical that points to the main product URL instead.

### Why does my product page show noindex when the HTML doesn't have it?

The noindex is probably in an `X-Robots-Tag` HTTP header, often set by a CDN, a plugin or a leftover staging config. Check the page's response headers and remove it at the server or CDN.

### Does Google require a return policy for free listings?

Yes. As of 2026-10-17, Merchant Center requires a return policy on your website for free listings, stated explicitly even if you accept no returns. Adding `hasMerchantReturnPolicy` markup to your Organization node lets Google read it.

### Do I need special markup to appear in AI Overviews?

No. As of 2025-12-10, Google says a page is eligible for AI Overviews and AI Mode when it is indexed and can be shown with a snippet. Remove noindex, `nosnippet` and `max-snippet:0` from product pages you want quoted.