User-agent: * Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ # Private/internal endpoints. Note: `/buy/` is the canonical # BuyAction target advertised in the Product JSON-LD schema (Google # Merchant Center's Storebot requires it to verify purchase intent). # It is intentionally CRAWLABLE (BuyPage carries `index,follow`), so # it is deliberately NOT listed in Disallow rules here. Disallow: /call # Legacy subcategory URLs no longer routed — 301'd to /products Disallow: /subcategory/ Disallow: /subcategory # Search results are user-generated and thin — noindex via robots Disallow: /search? # Faceted product URLs are duplicates of their canonical /category/{slug} # shelves. Canonical + rel=canonical already point Google there, but this # blocks the crawl before Google reaches the noindex/canonical hint on # the destination page — saves crawl budget for real SKUs. # 01 Oct 2026: four legacy Microsoft 365 filter URLs now serve a real # HTTP 301 to their commercial landing page. Allow Googlebot to reach # the redirect so link signals consolidate onto the canonical landing; # the generic Disallow below still blocks every other faceted URL. Allow: /products?category=Microsoft+365&subcategory=Enterprise Allow: /products?category=Microsoft+365&subcategory=Business Allow: /products?category=Microsoft+365&subcategory=Frontline Allow: /products?category=Microsoft+365$ Disallow: /products?subcategory= Disallow: /products?category= Disallow: /products?brand= Disallow: /products?query= # Cart / checkout / payment routes — never indexable transactional flows. Disallow: /checkout/ Disallow: /payment-return Disallow: /oauth/ # Backend API is never indexable HTML content — but image_links, sitemaps # and the GMC feed MUST remain crawlable for Google Merchant Center to # validate every product entry. Disallow: /api/ Allow: /api/gmc-icons/ Allow: /api/uploads/ # NOTE: /api/merchant-feed.xml and /merchant-feed.xml both 301-redirect to # /feeds/google-merchant.xml — no need to allow the legacy paths any more. Allow: /feeds/ # --------------------------------------------------------------- # Explicit Googlebot / Googlebot-Image / Bingbot groups. # Cloudflare's managed prefix injects its own `User-agent: *` group # above ours. Google/Bing merge same-agent groups per spec, but # stricter parsers only honour the first matching group they see — # so we duplicate the app rules here to guarantee coverage. # # 02 Sep 2026: Googlebot + Googlebot-Image are explicitly granted # full site crawl (all pages + all images). We only keep the # security-sensitive admin/login/api excludes — every product page, # category shelf, brand landing, family page and image asset is # fully crawlable. # --------------------------------------------------------------- User-agent: Googlebot Allow: / Allow: /images/ Allow: /brands/ Allow: /brand-logos/ Allow: /feeds/ Allow: /api/gmc-icons/ Allow: /api/uploads/ Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: Googlebot-Image Allow: / Allow: /images/ Allow: /brands/ Allow: /brand-logos/ Allow: /feeds/ Allow: /api/gmc-icons/ Allow: /api/uploads/ Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ User-agent: Googlebot-News Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: Googlebot-Video Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: AdsBot-Google Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ User-agent: Storebot-Google Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ User-agent: Bingbot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /call Disallow: /subcategory/ Disallow: /search? Allow: /products?category=Microsoft+365&subcategory=Enterprise Allow: /products?category=Microsoft+365&subcategory=Business Allow: /products?category=Microsoft+365&subcategory=Frontline Allow: /products?category=Microsoft+365$ Disallow: /products?subcategory= Disallow: /products?category= Disallow: /products?brand= Disallow: /products?query= Disallow: /api/ Sitemap: https://indiaenterprises.co/sitemap.xml # --------------------------------------------------------------- # AI assistants — LLM crawler policy # --------------------------------------------------------------- # India Enterprises WELCOMES retrieval-augmented answer engines # (ChatGPT, Claude, Perplexity, Gemini, Bing Copilot, etc.) — # accurate representation on these surfaces sends high-intent # buyers to /contact and /chaos, /microsoft, /adobe, etc. The # canonical AI-facing index is served at: # # https://indiaenterprises.co/llms.txt (spec: llmstxt.org) # https://indiaenterprises.co/llms-full.txt (expanded context) # # We explicitly ALLOW every major AI crawler so their policies # don't fall back to a restrictive default when Cloudflare's # managed prefix injects its own "User-agent: *" group above ours. # Each crawler is denied /admin, /login, /api and the faceted # duplicates — matching the Googlebot rules. # --------------------------------------------------------------- # OpenAI — ChatGPT + GPTBot (training + browse-on-behalf-of-user). User-agent: GPTBot Allow: / Allow: /llms.txt Allow: /llms-full.txt Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /buy/ Disallow: /call Disallow: /search? Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ # Anthropic — Claude web + retrieval crawler. User-agent: ClaudeBot Allow: / Allow: /llms.txt Allow: /llms-full.txt Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /buy/ Disallow: /call Disallow: /search? Disallow: /api/ User-agent: Claude-Web Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: anthropic-ai Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ # Perplexity — retrieval-only answer engine. User-agent: PerplexityBot Allow: / Allow: /llms.txt Allow: /llms-full.txt Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /buy/ Disallow: /call Disallow: /search? Disallow: /api/ # Google — AI Overviews / Gemini opt-in via Google-Extended. User-agent: Google-Extended Allow: / Allow: /llms.txt Allow: /llms-full.txt Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /buy/ Disallow: /call Disallow: /search? Disallow: /api/ # Common Crawl — powers many open LLM training corpora. User-agent: CCBot Allow: / Allow: /llms.txt Allow: /llms-full.txt Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /buy/ Disallow: /call Disallow: /search? Disallow: /api/ # Amazon Alexa + Bedrock retrieval crawler. User-agent: Amazonbot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ # Bytespider (TikTok / Doubao) + Meta AI. User-agent: Bytespider Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: FacebookBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: meta-externalagent Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: cohere-ai Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ # Mistral / You.com / DuckDuckGo AI-Assist / Diffbot / Applebot-Extended. User-agent: MistralAI-User Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: YouBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: DuckAssistBot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: Diffbot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /admin Disallow: /admin/ Disallow: /login Disallow: /login/ Disallow: /api/