Site Crawler

Crawl your website to index pages for AEO analysis, internal link mapping, and content gap discovery.

Last updated August 31, 2026

The Site Crawler indexes your website so AEO Goal can analyze content quality, internal linking, and AEO optimization across all your pages.

How the crawler works

The crawler starts from your homepage and follows internal links to discover all publicly accessible pages on your domain. It respects robots.txt and noindex tags by default. Crawled pages are stored in AEO Goal’s index and used by:

  • Site Audit (for technical SEO issues)
  • Quality Gate (for AI citability scoring)
  • Blog Generation (for internal link suggestions)
  • Ask AI (for content analysis questions)

Running a crawl

  1. Go to Audit → Crawl (or the crawler is triggered automatically when you run a Site Audit).
  2. Click Start Crawl.
  3. Enter your domain if not pre-filled.
  4. Click Crawl.

Crawl speed depends on site size. Most sites under 1,000 pages complete in under 5 minutes.

Crawl settings

Depth limit - maximum number of link hops from the homepage (default: 10).

Include/exclude patterns - use URL patterns to include or exclude specific paths. Examples:

  • Include only: ^/blog/.*
  • Exclude: ^/staging/.*, ^/wp-admin/.*

JavaScript rendering - enable if your site uses client-side rendering (React, Vue, etc.) that would otherwise show an empty page to the crawler.

Crawl results

After crawling, go to Audit → Pages to see all indexed pages:

  • Page URL
  • Title and meta description
  • Word count
  • Internal links in/out
  • Last crawled timestamp
  • AEO citability score (from the Quality Gate)

Click any page to see its full audit detail.

Recrawling

The crawler does not run automatically - you must trigger it manually, or schedule it under Audit → Schedule. We recommend crawling at least monthly, and after any major content updates.

Frequently asked questions

How does the Site Crawler discover pages?

The crawler starts from your homepage and follows internal links to discover all publicly accessible pages on your domain, and it respects robots.txt and noindex tags by default.

How long does a crawl take?

Crawl speed depends on site size, and most sites under 1,000 pages complete in under 5 minutes.

What crawl settings can I adjust?

You can set a depth limit for the maximum number of link hops from the homepage, which defaults to 10, use include and exclude URL patterns, and enable JavaScript rendering if your site uses client-side rendering such as React or Vue.

Does the crawler run automatically?

No. The crawler does not run automatically, so you must trigger it manually or schedule it under Audit -> Schedule. AEO Goal recommends crawling at least monthly and after any major content updates.