Robots Exclusion Checker
Robots Exclusion Checker is designed to visually indicate whether any robots exclusions are preventing your page from being crawled or indexed by Search Engines.
The extension reports on 3 elements:
2. Meta Robots in the HTML <head>
3. X-robots-tag in the HTTP header response
If the extension detects that the current URL is affected by an exclusion, the robot icon will turn Red. If nothing is detected the icon will turn Green. The extension has been designed primarily for ‘googlebot’ & ‘googlebot-news’ user-agents but we'll be adding more in the future.
This tool will be useful for anyone working in Search Engine Optimisation (SEO) or digital marketing, as it gives a clear visual indication if the page is being blocked by robots.txt (many existing extensions don’t flag this). Crawl or indexation issues have a direct bearing on how well your website performs in organic results so this extension should be part of your developer toolkit for Google Chrome.
This extension is useful for:
- Faceted navigation review and optimisation (useful to see the robot control behind complex / stacked facets)
- Detecting crawl or indexation issues
- General SEO review and auditing within your browser
Found a bug or want to make a suggestion? Please email firstname.lastname@example.org
1.0.2: Fixed a bug preventing meta robots from updating after a URL update
1.0.3: Various bug fixes, including better handling of URLs with encoded characters. Robots.txt expansion feature to allow the viewing of extra long rules