Built out the Site Health Crawl feature for the SEO Auditor — a full-site crawler that checks robots.txt, sitemap, meta tags, H1s, schema, and broken links across every page.

What’s in it

• Crawl engine — walks the full site from the sitemap and scans every discovered page
• Robots.txt check — confirms a user-agent directive is present and the site isn’t accidentally blocking itself
• Sitemap detection — locates sitemap.xml, counts pages listed, and confirms it’s reachable
• Crawl map — visual list of every page discovered so you can see site structure at a glance
• Page-level audit table — per-page breakdown of HTTP status, title length, meta description length, H1 presence, and schema markup, each flagged with issue counts where something’s off
• Broken link detection — surfaces dead links site-wide instead of per-page

Notes

• Click into individual page issues to show exactly what’s wrong (title too short/long, missing meta, etc.) instead of just an issue count
• Add historical crawl comparison so users can see what changed since the last scan