Scraping is a legitimate technique with real constraints. Handle those properly and you have a reliable data source; ignore them and you get blocked, or worse.

1. Check the boring things first

Is there an API? Many sites have one and it will be faster, stabler and permitted. Check that before writing a parser.

Read robots.txt and the terms of service. Rate limits and prohibited paths are stated there. And be clear about what you're collecting: personal data brings legal obligations regardless of it being publicly visible.

2. Be a good citizen

scraper-build.txt
Build a scraper for <site>, extracting <fields> from <pages>.

Politeness, non-negotiable:
- Respect robots.txt.
- One request at a time, with a delay between them. Say what delay.
- A real User-Agent identifying my tool with contact information.
- Exponential backoff on any error, and stop entirely on repeated 429
  or 403 rather than hammering.
- Cache pages locally so re-runs during development hit disk, not the
  site.

Extraction:
- Select on stable attributes, not deep CSS paths that break on a
  redesign.
- If an expected field is missing, log it loudly and skip the record —
  never write a partial row silently.

That caching point saves you from being the person who requests the same page four hundred times while debugging a selector.

3. Expect it to break

Sites change. Your scraper will break, and the dangerous failure is not crashing — it's quietly returning empty strings for a week.

So: validate what you extracted. If a field that's always present is missing, that's an error, not an empty value. Alert on a run that returns far fewer records than usual.

4. Store raw, parse separately

Save the raw HTML, then parse from that. When your extraction turns out to be wrong you can re-parse without re-fetching — which is both faster and much politer.

Set a hard cap on total requests per run. A pagination bug that loops forever is the fastest way to get your IP banned permanently.