Check the files at your site’s root before they bite WELLKNOWN
The files at a site’s root are edited once and forgotten, and each has a mistake that stays invisible until it costs something. Paste one and see what it actually says, and what is wrong with it.
The file
What it says, and what is wrong
How it works
Why these three
robots.txt, sitemap.xml and security.txt are hand-edited, deployed once and never looked at again, until a bad rule hides the site from Google, a sitemap of dead links wastes crawl budget, or a researcher finds an expired security.txt and gives up. WELLKNOWN reads each back plainly and flags the specific fault. It checks the text you paste and never crawls your site; to build a correct SPF/DMARC/DKIM record set or a security.txt from scratch, see DMARC.
The guide: What each robots.txt line blocks
Questions people ask about WELLKNOWN
What does Disallow: / do in robots.txt?
Under User-agent: * it asks every crawler to stay out of every path on the site, and it is flagged here as an error. It is often left over from a staging site and copied to production.
Two things robots.txt is not. It is not access control: the file is public, and a crawler that ignores it can still fetch everything. And it is not a way to keep a page out of search: a blocked page can still be listed by its address if other sites link to it, because the crawler is never allowed to read the noindex on it.
Why does it call my robots.txt line an unknown directive?
Because it is not one crawlers act on. Recognised here are User-agent, Allow, Disallow and Sitemap, plus Crawl-delay, Host and Clean-param, which some crawlers read and Google ignores. Anything else, such as Noindex:, which Google stopped honouring in 2019, is reported with its line number. An Allow or Disallow that comes before any User-agent line is also flagged, because it belongs to no group.
Which fields does a security.txt need?
RFC 9116 requires two: at least one Contact: (a mailto: or https: address for reports) and exactly one Expires:, an ISO 8601 date after which the file should be treated as stale. The RFC recommends an expiry less than a year ahead. The file is served at /.well-known/security.txt.
This checks both required fields, that Expires has not passed and appears only once, and lists which of the optional Encryption, Policy, Canonical and Preferred-Languages are missing. To write one from a few answers, use DMARC.
Why must every URL in a sitemap be absolute?
Because the sitemaps protocol says so: each <loc> is the full address, protocol included, such as https://example.com/page. A crawler reads the sitemap on its own and has no page to resolve /page against. Every <loc> that does not begin http:// or https:// is counted here, with the first one quoted. A <lastmod> that does not read as a date at all is flagged too; write it as 2026-01-31 or a full W3C timestamp.
How many URLs can one sitemap hold?
50,000 URLs, and no more than 50 MB uncompressed. Past either limit, split the URLs over several sitemaps and list those in a sitemap index, which has the same limits on the number of child sitemaps. Both limits are checked here, and an index is recognised and counted as child sitemaps rather than pages. The first 100 entries are listed so you can see what was read.
Does it check that the pages in my sitemap work?
No. It reads only the text you paste and fetches nothing, so a dead link, a redirect or a page marked noindex in the sitemap is not found here. The robots.txt rules it can test: put an address and a crawler’s name under the file, and it says whether that crawler may fetch it, by which group and which rule, with * and $ read as RFC 9309 reads them.
How does it know which of the three files I pasted?
From the contents. XML, a <urlset> or a <sitemapindex> is a sitemap; robots directives with no Contact: line make a robots.txt; a Contact: line makes a security.txt. Anything it cannot place is read as robots.txt. If it guesses wrong, choose the file under Read as.
Doing this often? OBSCURA Umbra edits the site's files where they are and previews the pages, with search and replace across the folder. Everything is US$5 a month or US$48 a year.