# robots.txt for staffroom # # Policy: open by default to well-behaved crawlers, with a named blocklist. # # The governing principle is default-deny where it is enforceable, and # default-allow where it is only advisory. robots.txt is a request, not a lock — # it only ever governs crawlers that choose to obey it, and those are precisely # the ones staffroom wants reading its pages. Actual enforcement lives at the # API layer and at the edge, not here. # # Blocked below: crawlers that collect content solely to train models, with no # citation and no referral back. staffroom's review corpus is its only durable # asset; absorbed into model weights it can answer teachers' questions without # anyone ever reaching staffroom. Crawlers that quote and link are welcomed. # --------------------------------------------------------------------------- # Default — everything not named below may crawl the site. # Keeps staffroom reachable by AI search engines that do not exist yet. # --------------------------------------------------------------------------- User-agent: * Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification # --------------------------------------------------------------------------- # Search and answer engines that cite and link back — explicitly welcomed. # Named even though the default already allows them, so a future edit to the # default cannot silently remove staffroom from these surfaces. # # ⚠ EVERY GROUP BELOW MUST REPEAT THE DISALLOW LINES FROM THE `*` GROUP. # # A crawler obeys ONLY its most specific matching user-agent group and ignores # every other group, including `*`. From the day these named groups were added # until 21 Aug 2026 each contained nothing but `Allow: /`, so Googlebot, # Bingbot and all the rest were exempt from every Disallow on this page. # Confirmed in Google's own crawl log: it fetched /api/insights/career-stats # on 19 Aug 2026 and got a 200. # # If you add a Disallow to the `*` group, add it to all of these too. # --------------------------------------------------------------------------- User-agent: Googlebot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Googlebot-Image Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Google-InspectionTool Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Bingbot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: OAI-SearchBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: ChatGPT-User Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Claude-SearchBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Claude-User Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: PerplexityBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Perplexity-User Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Applebot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: DuckDuckBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: YandexBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification # --------------------------------------------------------------------------- # Link-preview crawlers. Roughly 84% of staffroom sessions arrive through # in-app browsers, and Instagram alone is ~60% of traffic. If these are ever # blocked, every shared staffroom link renders as bare text with no title or # image — a direct hit on the largest acquisition channel. Named deliberately. # --------------------------------------------------------------------------- User-agent: facebookexternalhit Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Twitterbot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: LinkedInBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: WhatsApp Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Slackbot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: TelegramBot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification User-agent: Discordbot Allow: / Allow: /api/og/ Disallow: /api/ Disallow: /dashboard/ Disallow: /dev/ Disallow: /profile Disallow: /my-applications Disallow: /apply/ Disallow: /whatsapp-verification # --------------------------------------------------------------------------- # Training-only crawlers — blocked. These read the corpus but never cite it, # never link back, and never send a teacher to staffroom. # --------------------------------------------------------------------------- # OpenAI model training. OAI-SearchBot above is allowed, so staffroom still # appears in ChatGPT search results with attribution. User-agent: GPTBot Disallow: / # Anthropic model training. Claude-SearchBot above is allowed. User-agent: ClaudeBot Disallow: / # Gemini model training and Gemini app grounding. Blocking this does NOT # affect Google Search, AI Overviews or AI Mode — those are served by # Googlebot, which is allowed above. Google's own documentation: "Google-Extended # does not impact a site's inclusion in Google Search nor is it used as a # ranking signal in Google Search." User-agent: Google-Extended Disallow: / # Meta model training (Llama, Meta AI). User-agent: meta-externalagent Disallow: / # Apple model training. Applebot above stays allowed, so Siri and Spotlight # can still surface staffroom. User-agent: Applebot-Extended Disallow: / # Amazon AI surfaces. User-agent: Amazonbot Disallow: / # --------------------------------------------------------------------------- # Dataset redistribution and resale — blocked. Common Crawl's output is the # standard bootstrap corpus for anyone assembling a competing dataset, and the # rest are commercial data brokers. # --------------------------------------------------------------------------- User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / # Sitemap location Sitemap: https://www.thestaffroom.in/sitemap.xml