Measuring indexation rate starts with a simple distinction that teams often blur: a page can be published without being indexed, and a page can be indexed without currently being publishable. If you collapse those states into one percentage, you end up with a number that looks precise and says very little.
The useful question is not just “what percentage of pages are indexed?” It is “for a defined set of published URLs, how many entered the index, how long did it take, and what happened to the rest?” That framing turns indexation into an operational metric instead of a vanity ratio.
Published URLs versus indexed URLs
Published URLs are the pages your system has made available to crawlers and users: they return a live HTTP response, are linked from somewhere discoverable, and are intended to exist. Indexed URLs are the subset search engines have chosen to store and serve from their index. Those sets overlap, but they are not the same thing.
A published page may not be indexed yet because it is new, hard to discover, weakly linked, blocked, duplicated, canonicalized elsewhere, or simply not deemed useful enough. An indexed URL may later disappear from the index because the page changed, was removed, became noncanonical, or was excluded for some other reason.
That means indexation rate is a moving measurement over a changing population. You are not measuring a fixed asset list; you are measuring a time-based outcome on a cohort of URLs.
Use cohorts, not one sitewide percentage
A single sitewide indexation percentage is usually too blunt to act on. It blends together pages with very different expectations:
- brand-new pages
- pages published months ago
- pages in low-link subdirectories
- pages with strong internal linking
- pages marked noindex or canonicalized away
- pages excluded for quality or duplication reasons
If you mix all of those into one denominator, the result obscures the problem you actually need to diagnose. A better approach is cohort analysis: group pages by publication date, page type, directory, template, or content source, then measure indexation over time for each cohort.
For example, if pages published this week are at 12% indexation and pages published 60 days ago are at 92%, that is a very different signal from a sitewide 74% average. The first points to indexing lag or discovery issues. The second suggests the system is eventually working, even if slowly.
Cohorts also help isolate architectural differences. A programmatic template with sparse internal links may index far more slowly than editorial pages with strong navigation. Treating those as one bucket hides the underlying mechanics.
If you are also working through content architecture decisions, it is worth pairing this metric with the way content is structured and linked in the first place. A clean content model and predictable internal-link structure often make indexation outcomes easier to reason about; see how to separate SEO content data from presentation and how to choose internal links automatically.
Measure indexing lag, not just final status
Indexation is time-sensitive. A page that is not indexed on day 2 may be perfectly healthy; the same page still unindexed on day 45 may indicate a real problem.
So instead of asking only whether a page is indexed, measure the time from publication to first observed indexation. That lag distribution is usually more useful than a single endpoint percentage. Track median, upper quartile, and long-tail cohorts if you have enough volume.
A practical way to organize this is:
- Record the publish timestamp.
- Check index status repeatedly over time.
- Calculate first-indexed date where possible.
- Bucket pages by age since publication.
- Compare cohorts over the same age windows.
This gives you a curve, not a snapshot. A page set with slow but steady progress is different from one that flatlines early.
Define exclusions before you calculate anything
A lot of indexation reporting becomes misleading because the denominator includes pages that were never supposed to be indexed.
Before calculating the rate, exclude or separately segment pages that are:
- intentionally noindexed
- canonicalized to another URL
- blocked from crawling by robots rules
- redirected
- deleted or expired
- duplicates that were published for operational reasons but should not count as target indexable pages
- non-HTML assets or utility URLs
If you do not separate those cases, your indexation rate becomes an accidental measure of policy drift rather than search visibility.
This is especially important for systems that generate pages programmatically or operate across multiple templates. Some page families are meant to be indexable; others are support surfaces, filters, or transitional states. Those should not be judged together. The publishing and state logic around these pages matters as much as the content itself, which is why measurement works best when it sits alongside reliable pipeline instrumentation such as how to track SEO content from generation to indexing and how to add observability to an SEO content pipeline.
A workable formula
For a defined cohort of pages intended to be indexable, a basic indexation rate can be calculated as:
Indexed pages / eligible published pages
That is the simplest version, but it only works if “eligible published pages” has been defined carefully.
In practice, teams often need multiple versions of the metric:
- Published indexation rate: indexed / published and intended-to-index
- 30-day indexation rate: indexed within 30 days / pages published in that cohort
- Template-specific indexation rate: indexed / eligible pages for a given page type
- Discovery-adjusted rate: indexed / pages that have been published long enough to have a fair chance of being discovered
The last point matters. A page published yesterday has not had the same opportunity as a page published six weeks ago. If you do not normalize for age, you are comparing exposure windows rather than indexability.
What to investigate when the rate is low
A low indexation rate does not point to one root cause. It is a symptom, and the symptom can come from several layers of the system.
Start by checking whether the pages are actually eligible:
- Is the canonical tag self-referential or pointing elsewhere?
- Are robots directives preventing indexing?
- Are the pages linked from crawlable locations?
- Is the sitemap current?
- Are the URLs returning the correct status code?
- Are there duplicate or near-duplicate pages competing for the same cluster?
Then check operational questions:
- Are these pages published but not deployed?
- Are deploys lagging behind database state?
- Are new URLs being added to sitemaps before they are live?
- Are internal links generated after publication, or not at all?
- Are there structural exclusions in the template or CMS workflow?
Those distinctions matter because the fix is different in each case. A crawl-discovery problem is not solved by changing copy. A canonicalization problem is not solved by increasing publishing volume. A queue or deployment problem is not solved by asking for faster indexing.
If you are diagnosing why pages are not being discovered or why the system has drifted, it is often useful to step back and inspect the broader publishing architecture. Articles like how to audit an automated SEO publishing system and how to monitor thousands of SEO pages after publishing help frame the problem as a pipeline issue rather than a content-only issue.
What a good measurement setup looks like
A useful indexation dashboard usually separates four views:
- published pages by cohort
- eligible pages by cohort
- currently indexed pages by cohort
- time-to-index distribution by cohort
That arrangement gives you both state and trajectory. It also makes exclusions visible instead of hidden inside the denominator.
When teams can see those views together, they usually stop asking whether indexation rate is “good” or “bad” in the abstract. The better question becomes: which page families index quickly, which lag, which never should have been counted, and what changed in the publishing system before the curve moved?
That is the level at which indexation becomes actionable.
A single percentage rarely gets you there.