Free tool

Free Robots.txt Validator & Sitemap Checker

Enter your domain. We fetch /robots.txt and your sitemap, then check for the mistakes that quietly keep pages out of Google.

Free sitemap and robots.txt validator. Checks both files exist, the sitemap is well-formed, and robots.txt isn't silently blocking your own pages. No signup.

All SEO Mini Apps →

  • Domain

Validate

Example

robots.txt found · sitemap reachable · 2 conflicts

5/7

Checks passed

Status Fix conflicts

How it works

  1. 1

    Enter your domain

    Just the domain. We fetch /robots.txt and the sitemap it points to, or /sitemap.xml if it lists none.

  2. 2

    Press validate

    We run six checks: both files reachable, the sitemap well-formed, no blanket Disallow, and no sitemap URL blocked by robots.txt.

  3. 3

    Fix what failed

    Each check passes or fails with a short explanation, plus a prompt you can paste into an AI to write the fix.

What gets checked

Six checks that catch the mistakes that quietly keep whole sections of a site out of search results.

robots.txt exists and is reachable.
robots.txt isn't blocking the entire site with a blanket Disallow.
robots.txt references a sitemap.
The sitemap is reachable and well-formed XML.
No sitemap URLs are also blocked by a Disallow rule (a common, silent conflict).
lastmod dates aren't all stale.

Why this trips founders up

A migration, a new CMS, or a well-meaning "block staging paths" rule in robots.txt can end up blocking real pages that are still listed in your sitemap — Google follows robots.txt, so those pages get skipped even though you're telling search engines they exist.

What this does not do

  • A regex-based check, not a full XML schema validator — it catches the common mistakes, not every possible malformed edge case.
  • Only the first sitemap referenced in robots.txt (or the conventional /sitemap.xml) is fully checked. A sitemap index is followed one level deep, into its first listed sitemap only.
  • Disallow-vs-sitemap conflicts only check the `*` and Googlebot user-agent groups.
  • "Stale" means the URL's own lastmod date is older than 180 days — that is a heuristic threshold, not a Google requirement.

Frequently asked questions

Is this validator free?
Yes, completely free — no signup. Enter your domain and get a checklist in seconds.
What is a Disallow/sitemap conflict?
It's when a URL is listed in your sitemap (telling crawlers "index this") but also blocked by a Disallow rule in robots.txt (telling crawlers "don't crawl this"). Crawlers follow the block, so the page silently gets skipped.
My sitemap is a sitemap index — will it check everything?
It follows the first listed sitemap in the index one level deep and checks that one. If you have many child sitemaps, run this per key section if you want full coverage for now.
What if I don't have a robots.txt?
That's flagged as a warning, not a hard fail — crawlers assume everything is allowed with no robots.txt. It's still good practice to add one that at least references your sitemap.
What counts as a stale lastmod?
Any sitemap entry whose <lastmod> date is older than 180 days. It's a heuristic flag to review, not something Google requires.
Does it fix anything for me?
No — it's a diagnostic tool. It tells you exactly what's wrong (and which URLs) so you can fix robots.txt or your sitemap generator yourself.

Other free tools

  • Free XML Sitemap Generator

    Free XML sitemap generator. Paste a domain, we discover public URLs, and you download sitemap.xml to host at the root. No signup.

  • Free Broken Link Checker

    Free broken link checker. Paste a page and we check every link on it for 4xx, 5xx, timeouts and redirects. No signup.

  • Free llms.txt Generator

    Free llms.txt generator. Paste a URL and download llms.txt, llms-full.txt, and llms-developers.txt for AI crawlers. No signup.

  • SaaS SEO Checklist for Founders

    Free SaaS SEO checklist for founders. Paste your site, get a Critical / Important / Recommended plan — including a per-page on-page scorecard and AI search track.

  • SaaS SEO Agent Skills for Founders

    Free SaaS SEO agent skills for founders. Paste your site and download a customized SKILL.md pack for Cursor, Claude Code, or ChatGPT — crawl, product-page, intent, authority, and AI visibility.

  • JavaScript SEO Checker

    Free SPA shell checker. Paste your website to see if crawlers get real HTML or an empty JavaScript shell. No signup.

List your product when the page is ready for founders to find it.