Code a polite scraper for a page structure you describe
Generates working scraper code with rate limits, honest headers and robots.txt respect built in from the first line.
You are a careful automation developer writing a small, polite web scraper for an Australian small business. Being a good citizen of the target site is a hard requirement, not a preference.
<context>
TARGET + PURPOSE: [e.g. "supplier's public price list, checked weekly for changes"]
PAGE STRUCTURE: [paste a sample of the HTML around the data, or describe the layout and selectors precisely]
DATA FIELDS WANTED: [e.g. "product name, SKU, price, stock flag"]
LANGUAGE/STACK: [e.g. "Python 3 + requests/BeautifulSoup" - or Playwright only if the page is JS-rendered]
VOLUME + FREQUENCY: [e.g. "about 40 pages, once a week"]
</context>
Before writing code: (a) state whether an official API, feed or export probably makes scraping unnecessary and how to check; (b) list what must be confirmed in robots.txt and the site's terms of use for these pages; (c) if the data includes personal information about individuals, stop - output instead a short list of questions to put to a privacy professional before proceeding.
Requirements:
1. Fetch politely: identifying User-Agent with a contact-email placeholder, one request at a time, randomised delay of [DELAY: e.g. 3-5 seconds], retry with exponential backoff on 429/5xx, hard stop after repeated failures.
2. Check robots.txt programmatically before each run.
3. Parse using only the provided page structure; fail loudly (logged, not silent) when selectors stop matching.
4. Never bypass logins, paywalls or CAPTCHAs; if one appears, exit with a clear message.
5. Write output to [FORMAT: CSV or JSON] with a scrape timestamp per row; make re-runs idempotent.
6. Keep it to one commented file a non-expert could maintain.
<output_format>
The complete runnable script, then setup and run instructions in 3-5 lines, then one maintenance note: what will break first and how to fix it.
</output_format>
Grounding: use only the described structure - no guessed selectors; unknowns become TODO comments with instructions.
Copy the block above straight into Claude — anything in [BRACKETS] is yours to fill in.
Want it tuned to your business? Bring it to the free weekly call and we'll adapt it live.
Join the free callMore coding & technical prompts
Keep a Two-Minute Daily Engineering Log That Pays Off Later
Turn end-of-day scraps into a structured log entry — decisions with their why, lessons, blockers — capped at 120 words and searchable months later.
Break a Stalled Decision With a Structured Tiebreak
Lay out the options, score them against your own criteria, price the delay, and get a verdict with a 48-hour commitment step — plus what would legitimately reopen it.
Engineer a Distraction-Proof Deep-Work Block
Set up a recurring focus block matched to your energy and team reality — device rules, a status script, an interruption protocol and a two-week bedding-in plan.