Sitemap: https://pixelforce.com/sitemap.xml User-agent: * Disallow: /cpresources/ Disallow: /admin/ Disallow: /demo/ Disallow: /services/jsps/* Disallow: /service/jsps/* # /book/* and /batphone are deliberately NOT disallowed here. # # They must stay out of the search index, and they do - every page in the book # section plus /batphone serves # (see templates/_2026/partials/_head.twig). Blocking them here as well was # self-defeating: a crawler that is not allowed to fetch a page can never read # the noindex on it, so the block prevented the very deindexing it was meant to # achieve. Google's own guidance is explicit - for noindex to work the page must # NOT be blocked by robots.txt. Removed 12 August 2026. # # The old rule was `Disallow: /batphone/` with a trailing slash, which did not # match the real URL `/batphone` in any case. # # If you ever need a page kept out of search: give it noindex and leave it # crawlable. Only use Disallow for things that must not be FETCHED at all # (the control panel, asset internals, /demo/). # # /demo/ is the one place that carries BOTH: the Disallow above stops crawlers # fetching client prototypes, and every HTML page under /demo/ also serves # as a backstop for crawlers # that ignore robots.txt. Adding a page under /demo/? Give it that meta tag # (pixelforce-canvas output carries it already). The pairing is safe only # because no /demo/ URL has ever been indexed - verified with GSC URL # Inspection on 13 October 2026. If one ever shows up in Search Console, lift # the Disallow until Google has recrawled it and read the noindex. # # The rules sit in the `*` group, which covers AI crawlers (GPTBot, ClaudeBot, # Google-Extended and the rest). A crawler with its OWN User-agent group ignores # the `*` group entirely, so any group added here must repeat Disallow: /demo/.