SourceCited / URL cleanup & canonicals
Technical · URL hygiene
Dead and duplicate URLs waste crawl budget and keep stale versions of you in the index AI engines read. Use 301/410/canonical correctly, fix soft 404s, and deindex what shouldn't exist.
Old, duplicate, and junk URLs drag you down and can keep a stale identity in the index AI engines cite. The discipline is status-code honesty: 301 for moves, 410 Gone to permanently deindex what's gone for good, canonical tags to consolidate duplicates — and fixing soft 404s (dead pages that still return 200), the trap that keeps old URLs indexed. Don't block removed URLs in robots.txt, or crawlers never see the 410.
Old, duplicate, and junk URLs are not harmless. They waste crawl budget, split signals across near-identical pages, and — worst — keep a stale identity in the index long after you have moved on. AI answer engines read the index they are given; if half of what they can find about you is a retired version of your site, that is what they may cite. Cleaning up URLs is unglamorous and one of the highest-return technical jobs there is.
The whole discipline comes down to telling crawlers the truth with HTTP status codes. The common failure is a soft 404: a page that is gone but still answers 200 OK (often by serving your homepage), which keeps the dead URL indexed indefinitely.
| Situation | Use | Why |
|---|---|---|
| Page moved to a new URL | 301 redirect | Permanent move; passes signals to the new URL |
| Page gone for good, no replacement | 410 Gone | The clean, permanent “deleted, deindex it” signal — faster and clearer than 404 |
| Page temporarily missing | 404 | Not found; keep it in case it returns |
| Legacy URLs answering 200 with wrong content | Fix to 410 (or 301) | A soft 404 is the trap that keeps an old identity indexed |
| Duplicate/parameter versions of one page | canonical tag | Points engines to the one true URL (see below) |
Retired URLs that still return 200. A site that swapped its whole purpose left the old routes serving the homepage with a 200 — so the old identity stayed indexed and kept getting surfaced. A hard 410 on those routes is the permanent deindex signal. And do not block them in robots.txt to hide them: that stops crawlers from ever seeing the 410, which keeps them indexed longer.
Duplicate or near-duplicate URLs — ?ref= tracking params, print versions, http vs https, trailing-slash variants, faceted pages — split your signals and confuse which version to cite. The <link rel="canonical"> tag names the one authoritative URL so engines consolidate on it. Every important page should self-canonical to its clean URL; duplicates should point at the original.
canonical pointing to its own clean, absolute URL.Run this periodically, and always after a migration or a change of purpose:
I need to clean up my site's URLs. I'll paste my sitemap, a list of old/legacy URLs, and my Search Console index coverage. Help me: 1. Classify each URL: keep, 301-redirect (to where), 410 (gone for good), or canonicalize (to which URL). 2. Flag soft 404s — URLs that should be gone but likely return 200 — and give me the exact rule (e.g. a server/router condition) to return a hard 410 for them, WITHOUT touching my real routes and params. List near-miss params I must NOT match. 3. Give me the Search Console steps to speed up removal, and remind me why I should NOT block these URLs in robots.txt. Output as a table: url | action | destination/notes.
After deploying, verify with curl that legacy URLs return 410 and real pages still return 200 before submitting anything for removal.
404 means "not found" and signals the page might return, so engines keep checking. 410 means "Gone" — permanently deleted — and is the clean, fast signal to deindex. Use 410 when a page is gone for good and you want it out of the index; use 404 for temporary absence.
Return a hard 410 (or 301 if there's a replacement), make sure the URL is NOT blocked in robots.txt (or crawlers can't see the 410), and optionally use Search Console's Removals tool to speed it up. Verify with curl that the URL returns 410 while your real pages still return 200.
Usually a soft 404: the page is gone but still returns 200 OK — often by serving your homepage or a generic page — so engines think it's still live and keep it indexed. Return a real 410 (or 301) instead of a 200, and don't block the URL in robots.txt.
A canonical tag (rel="canonical") names the one authoritative URL among duplicates or near-duplicates — parameter versions, print pages, http/https and www variants — so engines consolidate signals on it instead of splitting them. Every page should self-canonical to its clean URL; don't canonicalize everything to the homepage.