Control Crawler Access With a Hand-Checked robots.txt
Get a commented robots.txt implementing your stance on search and AI crawlers, with the classic self-sabotage traps checked and test steps included.
When to use it: When you need staging or admin paths kept out of crawlers' way, or you've decided which AI bots are welcome, and one wrong line could hide your whole site from Google.
You are a technical SEO helping an Australian small business write a robots.txt they fully understand. You are blunt about what the file can and cannot do.
Site and platform: [SITE — e.g. example.com.au on WordPress]
Paths that shouldn't be crawled: [BLOCK — e.g. /wp-admin/, /cart/, /thank-you/]
My stance on AI crawlers: [AI STANCE — e.g. allow AI search assistants but block model-training bots / allow everything / block all AI bots]
Sitemap URL: [SITEMAP — e.g. https://example.com.au/sitemap.xml — or 'not sure']
Current robots.txt, if one exists: [CURRENT — paste it, or 'none']
Before writing the file, set expectations in three plain lines: robots.txt is a polite request honoured by well-behaved bots, not security — anything sensitive needs authentication, not a Disallow line (which actually advertises the path); blocking a page here doesn't remove it from the index if it's already there; and blocking CSS or JavaScript can wreck how Google renders the site.
Then deliver:
1. A complete robots.txt in one block, commented by section: search engine rules; my blocked paths; an AI-crawler section implementing my stance with per-bot user-agent groups, plus a comment that bot names change over time and this list reflects the common ones as of now — [VERIFY current bot list when you review this file].
2. The sitemap line if I gave one, or a note on finding my platform's sitemap URL.
3. A trap check against MY paste and platform: a lone 'Disallow: /' hiding the whole site, blocking asset folders, case mismatches, rules that conflict, and (if I pasted a current file) exactly what your version changes and why.
4. Where the file lives for my platform and how to publish it there.
5. Test steps: fetch domain/robots.txt in a browser, then verify in Google Search Console's checker that a normal page is allowed and a blocked path is blocked — with what a pass looks like.
6. A one-line change log comment format so future edits are dated.
Rules: use only the paths I listed — never invent paths; if my stance and my goals conflict (e.g. blocking bots I rely on for visibility), flag it before finalising.
Copy the block above straight into Any AI tool — anything in [BRACKETS] is yours to fill in.
Want it tuned to your business? Bring it to the free weekly call and we'll adapt it live.
Join the free callMore au business & compliance prompts
Plain-English Contract Summariser
Understand what you're signing before you sign it — and know what to ask a professional
Set Up Redirects and Caching on an Apache Site
Get a ready-to-paste .htaccess block with redirects, HTTPS forcing and caching, plus install, test and undo steps.
Script a Hard Conversation Before You Have It
Turn a dreaded workplace conversation into a short, direct script with openers, key lines and responses to likely reactions.