# Broken Link Checker (Python, nothing to install)

> Crawl your whole site, check every link and image, and get an HTML report showing where each broken link lives. Python standard library only.

- Category: Developer tools
- Version: 1.0
- Source: https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/

## Overview

Build a small command-line tool that crawls your website, checks every link, image, script and stylesheet it finds, and writes a clean HTML report (plus JSON) listing each broken link with its status and the pages it appears on. It stays on your own domain while crawling, checks outside links without crawling them, respects robots.txt, retries busy servers once, and exits with code 1 when anything is broken so you can run it in CI.

Who it is for: anyone running a blog, docs site, portfolio or client sites who wants to catch dead links before visitors or search engines do.

What you need: Python 3.10 or newer and an AI coding tool. Nothing to install: it uses only Python's standard library (urllib, html.parser, urllib.robotparser, concurrent.futures).

How to use this plan: put CLAUDE.md (asset 1) in an empty folder, then paste each step's prompt into your AI tool in order. Check the "done when" line before moving on. We built and tested every step of this plan ourselves.

## Tech stack

- HTML
- Python

## Build steps

### 1. List the links on one page

Start small: fetch one page and print every link it contains. Links are made absolute and duplicates are dropped.

Prompt:

```text
Read CLAUDE.md. Create linkcheck.py with a fetch(url, method) helper (10 second timeout, our own User-Agent) and an extract_links(page_url, html) function using html.parser that collects a/link href and img/script/iframe src. Make each link absolute with urljoin, remove the #fragment, skip mailto:, tel:, javascript:, data: and pure #anchors, keep only http and https, and drop duplicates. Add `python linkcheck.py URL --list` that prints the links of that one page.
```

**Done when:** python linkcheck.py https://www.python.org/ --list prints one absolute link per line, with no mailto: lines and no duplicates

### 2. Check whether a link works

HEAD requests are cheap, but some servers refuse them, so fall back to GET. Busy servers get one retry.

Prompt:

```text
Add check(url) that returns {url, status, ok, error, ms}. Send HEAD first; if it fails, or the server answers 403, 405, 501 or 5xx, try GET. If the answer is 429 or 5xx, wait 2 seconds and try once more. ok means a final status from 200 to 399. Connection errors and timeouts give status 0 and a short error text. Never raise.
```

**Done when:** check() says ok for https://www.python.org/, gives 404 for a made-up page on that site, and status 0 for http://127.0.0.1:9/

### 3. Crawl the whole site

Crawl pages on your own domain only. Outside links are collected and checked later, but never crawled.

Prompt:

```text
Add crawl(start, max_pages=200) that walks the site breadth-first from the start URL. Only follow links on the same host, only parse responses that are 2xx HTML, and never visit a page twice. Read robots.txt with urllib.robotparser and skip pages it disallows. Record for every link the list of pages it was found on. Count only real HTML pages toward max_pages. Add an optional --delay in seconds between pages.
```

**Done when:** on your own site the crawl stops at --max-pages, skips paths in robots.txt Disallow, and knows which pages each link came from

### 4. Check every link in parallel

Checking hundreds of links one by one is slow. A small thread pool makes it fast without hammering anyone.

Prompt:

```text
Add run(start) that crawls, then checks every collected link with a ThreadPoolExecutor (--workers, default 8). Add a repeatable --ignore PREFIX flag that skips links starting with that text (some big sites answer bots with errors). Print one line per broken link (status or error, the URL and the first page it is on), then a summary: pages crawled, links checked, broken count.
```

**Done when:** python linkcheck.py https://www.python.org/ --max-pages 3 finishes in about a minute and prints a summary line

### 5. Write the HTML and JSON report

The report is one self-contained HTML file you can open, share or upload, and a JSON file for other tools.

Prompt:

```text
Write report.html (change with --html) and report.json (change with --json). The HTML shows the start URL, the counts, the time in UTC, and a table of broken links with status and every page they were found on, or a green "No broken links found." line. Escape every value with html.escape, wrap long URLs so the table fits a phone, and support dark mode with color-scheme. Exit with code 1 when anything is broken and 0 when all links work, so CI can fail the build.
```

**Done when:** report.html opens in a browser and lists the broken links, and echo %ERRORLEVEL% (or $? on Mac/Linux) shows 1 when one is broken

### 6. Test it against a fake site

The tests start a tiny local website with known good and broken links, so they run offline in seconds.

Prompt:

```text
Add test_linkcheck.py using unittest only. Start a ThreadingHTTPServer on 127.0.0.1 port 0 with pages: / links to /about, /missing, a mailto, a #anchor, /logo.png, /about#team, /private/x and /<script>; /about links to /gone and http://127.0.0.1:9/ and answers HEAD with 405; robots.txt disallows /private/. Test that: the fragment duplicate is dropped, /missing and /gone are 404, /gone is found on /about, the dead host is reported without a crash, /about is ok through the GET fallback, /private/ is never crawled, the HTML escapes <script>, the exit code is 1, and --ignore removes a link from the report.
```

**Done when:** python -m unittest passes all tests with no network needed

## Assets

### CLAUDE.md — Project rules for your AI tool. Put it in the empty project folder before step 1.

```
# Broken link checker

## Rules
- One file: `linkcheck.py` (tests in `test_linkcheck.py`). Python 3.10+.
- Standard library only: urllib, html.parser, urllib.robotparser, concurrent.futures, json, html. No pip packages.
- Crawl only pages on the start URL's host. Check outside links, never crawl them.
- Respect robots.txt. Send our own User-Agent. Keep workers small (default 8) so we never hammer a site.
- A link is ok for a final status of 200-399. Errors and timeouts are status 0 with a short reason. Never crash on one bad link.
- Escape every value you put into HTML with `html.escape`.
- Exit code: 1 if any link is broken, 0 if all links work.
- If a site that opens in your browser shows CERTIFICATE_VERIFY_FAILED, fix the Python certificates (macOS: run "Install Certificates.command"; Windows: update Python from python.org). Never turn certificate checks off.

## Commands
- `python linkcheck.py URL --list`  links on one page
- `python linkcheck.py URL`  crawl and report (`--max-pages`, `--workers`, `--delay`, `--ignore PREFIX`, `--html`, `--json`)
- `python -m unittest`  run the tests
```


## Build instructions by tool

### Claude Code

Open a terminal in an empty folder, run `claude`, then paste the prompt.

```text
Build the "Broken Link Checker (Python, nothing to install)" app from the OwnlyPlans blueprint at https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/.
Stack: HTML, Python.
Follow the plan's steps in order: List the links on one page → Check whether a link works → Crawl the whole site → Check every link in parallel → Write the HTML and JSON report → Test it against a fake site.
Download the full plan (Markdown) from the link above and keep it in the project root as PLAN.md.
Ask me before adding anything not in the plan. (Prepared for Claude Code.)
```

### Cursor

Open an empty folder in Cursor, start an Agent chat, then paste the prompt.

```text
Build the "Broken Link Checker (Python, nothing to install)" app from the OwnlyPlans blueprint at https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/.
Stack: HTML, Python.
Follow the plan's steps in order: List the links on one page → Check whether a link works → Crawl the whole site → Check every link in parallel → Write the HTML and JSON report → Test it against a fake site.
Download the full plan (Markdown) from the link above and keep it in the project root as PLAN.md.
Ask me before adding anything not in the plan. (Prepared for Cursor.)
```

### Lovable

Start a new Lovable project and paste the prompt as your first message.

```text
Build the "Broken Link Checker (Python, nothing to install)" app from the OwnlyPlans blueprint at https://ownlyplans.com/bhaskar/broken-link-checker-python-nothing-to-install/.
Stack: HTML, Python.
Follow the plan's steps in order: List the links on one page → Check whether a link works → Crawl the whole site → Check every link in parallel → Write the HTML and JSON report → Test it against a fake site.
Download the full plan (Markdown) from the link above and keep it in the project root as PLAN.md.
Ask me before adding anything not in the plan. (Prepared for Lovable.)
```

## Definition of done

- Every build step is complete and the app runs locally.
- The main user flow works end to end.
- Share your build on OwnlyPlans with a demo link, build time and tool.
