# Broken Link Checker (Python, nothing to install)

> Crawl your whole site, check every link and image, and get an HTML report showing where each broken link lives. Python standard library only.

- Kind: Plan
- Page: https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/
- Licence: CC BY 4.0 (reuse with credit)

## Steps

### 1. List the links on one page

Start small: fetch one page and print every link it contains. Links are made absolute and duplicates are dropped.

Uses: CLAUDE.md

Prompt:

```text
Read CLAUDE.md. Create linkcheck.py with a fetch(url, method) helper (10 second timeout, our own User-Agent) and an extract_links(page_url, html) function using html.parser that collects a/link href and img/script/iframe src. Make each link absolute with urljoin, remove the #fragment, skip mailto:, tel:, javascript:, data: and pure #anchors, keep only http and https, and drop duplicates. Add `python linkcheck.py URL --list` that prints the links of that one page.
```

Done when: python linkcheck.py https://www.python.org/ --list prints one absolute link per line, with no mailto: lines and no duplicates

### 2. Check whether a link works

HEAD requests are cheap, but some servers refuse them, so fall back to GET. Busy servers get one retry.

Uses: CLAUDE.md

Prompt:

```text
Add check(url) that returns {url, status, ok, error, ms}. Send HEAD first; if it fails, or the server answers 403, 405, 501 or 5xx, try GET. If the answer is 429 or 5xx, wait 2 seconds and try once more. ok means a final status from 200 to 399. Connection errors and timeouts give status 0 and a short error text. Never raise.
```

Done when: check() says ok for https://www.python.org/, gives 404 for a made-up page on that site, and status 0 for http://127.0.0.1:9/

### 3. Crawl the whole site

Crawl pages on your own domain only. Outside links are collected and checked later, but never crawled.

Uses: CLAUDE.md

Prompt:

```text
Add crawl(start, max_pages=200) that walks the site breadth-first from the start URL. Only follow links on the same host, only parse responses that are 2xx HTML, and never visit a page twice. Read robots.txt with urllib.robotparser and skip pages it disallows. Record for every link the list of pages it was found on. Count only real HTML pages toward max_pages. Add an optional --delay in seconds between pages.
```

Done when: on your own site the crawl stops at --max-pages, skips paths in robots.txt Disallow, and knows which pages each link came from

### 4. Check every link in parallel

Checking hundreds of links one by one is slow. A small thread pool makes it fast without hammering anyone.

Uses: CLAUDE.md

Prompt:

```text
Add run(start) that crawls, then checks every collected link with a ThreadPoolExecutor (--workers, default 8). Add a repeatable --ignore PREFIX flag that skips links starting with that text (some big sites answer bots with errors). Print one line per broken link (status or error, the URL and the first page it is on), then a summary: pages crawled, links checked, broken count.
```

Done when: python linkcheck.py https://www.python.org/ --max-pages 3 finishes in about a minute and prints a summary line

### 5. Write the HTML and JSON report

The report is one self-contained HTML file you can open, share or upload, and a JSON file for other tools.

Uses: CLAUDE.md

Prompt:

```text
Write report.html (change with --html) and report.json (change with --json). The HTML shows the start URL, the counts, the time in UTC, and a table of broken links with status and every page they were found on, or a green "No broken links found." line. Escape every value with html.escape, wrap long URLs so the table fits a phone, and support dark mode with color-scheme. Exit with code 1 when anything is broken and 0 when all links work, so CI can fail the build.
```

Done when: report.html opens in a browser and lists the broken links, and echo %ERRORLEVEL% (or $? on Mac/Linux) shows 1 when one is broken

### 6. Test it against a fake site

The tests start a tiny local website with known good and broken links, so they run offline in seconds.

Uses: CLAUDE.md

Prompt:

```text
Add test_linkcheck.py using unittest only. Start a ThreadingHTTPServer on 127.0.0.1 port 0 with pages: / links to /about, /missing, a mailto, a #anchor, /logo.png, /about#team, /private/x and /<script>; /about links to /gone and http://127.0.0.1:9/ and answers HEAD with 405; robots.txt disallows /private/. Test that: the fragment duplicate is dropped, /missing and /gone are 404, /gone is found on /about, the dead host is reported without a crash, /about is ok through the GET fallback, /private/ is never crawled, the HTML escapes <script>, the exit code is 1, and --ignore removes a link from the report.
```

Done when: python -m unittest passes all tests with no network needed

## Files

- CLAUDE.md: https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/claude/ (Project rules for your AI tool. Put it in the empty project folder before step 1.)

## About

Build a small command-line tool that crawls your website, checks every link, image, script and stylesheet it finds, and writes a clean HTML report (plus JSON) listing each broken link with its status and the pages it appears on. It stays on your own domain while crawling, checks outside links without crawling them, respects robots.txt, retries busy servers once, and exits with code 1 when anything is broken so you can run it in CI.

Who it is for: anyone running a blog, docs site, portfolio or client sites who wants to catch dead links before visitors or search engines do.

What you need: Python 3.10 or newer and an AI coding tool. Nothing to install: it uses only Python's standard library (urllib, html.parser, urllib.robotparser, concurrent.futures).

How to use this plan: put CLAUDE.md (asset 1) in an empty folder, then paste each step's prompt into your AI tool in order. Check the "done when" line before moving on. We built and tested every step of this plan ourselves.

---
Also available as JSON: https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/?format=json
MCP server: https://ownlyplans.com/mcp