The mistake is treating “published” and “indexed” as if they are adjacent states. They are not. A page can be live in the CMS, discoverable in a sitemap, fetched by a crawler, indexed, shown in impressions, and still produce little or no traffic. Each step reflects a different system, and each one can fail independently.
If you want to track SEO pages properly, you need a pipeline model, not a single count.
The lifecycle you are actually measuring
A useful version of the lifecycle looks like this:
- Published — the URL is created and publicly accessible.
- Discovered — search systems can find the URL through internal links, sitemaps, external links, or other signals.
- Crawled — a bot requests the page and receives content.
- Indexed — the page is eligible to appear in search results.
- Impressions — the page begins to surface for queries.
- Clicks and traffic — the page earns visits from search demand.
That sequence is conceptually simple, but the timing is not. Some pages are crawled quickly and indexed slowly. Others are discovered immediately through internal links but sit for days or weeks before they are crawled again. Some pages index and still do not earn impressions because they match no meaningful demand. None of that is necessarily a problem by itself.
This is why articles like Published vs. Indexed SEO Pages: What Should You Track? and How to Measure Time to Index for Newly Published SEO Pages are best read together: one tells you which counts matter, the other tells you how to measure delay across cohorts instead of judging a single page in isolation.
Track cohorts, not just page counts
A common measurement failure is to look at a rolling total of “published pages” and a rolling total of “indexed pages” and assume the gap tells you something clean about quality. It usually does not.
A newly published page is competing for discovery, crawl attention, and index eligibility at the same time. An older page has already cleared some of those steps. If you mix them together, you lose the timing signal.
Instead, group pages by publish date and track them as cohorts:
- pages published on the same day or week
- pages from the same template or page family
- pages in the same market, directory, or product set
- pages created under the same release or workflow
For each cohort, measure how many pages move through each stage over time. That gives you the shape of the pipeline rather than a single snapshot.
A cohort view also exposes structural differences. A category page may be discovered through internal linking almost immediately, while a low-linked programmatic page may depend heavily on sitemap discovery. If the two page types are blended, the average hides the cause.
Define each stage with data you can actually collect
The data source should match the stage.
Published usually comes from your CMS or database. That should be deterministic: a URL exists, status is live, and canonical or indexation settings are known.
Discovered is harder to observe directly. In practice, you infer discovery from when a crawler first reports a URL in logs, when it appears in Search Console coverage-style reporting, or when the page starts appearing in crawl exports. If you have server logs, discovery can often be approximated by the first bot request after publication.
Crawled is easier to instrument. Server logs tell you whether a bot requested the page, when it did so, and how often it returned. For technical SEO teams, this is one of the most useful signals because it tells you whether the page is getting any crawl attention at all.
Indexed should be measured through search engine reporting and URL inspection workflows where appropriate, but only as a status signal, not a guarantee of stable visibility. Indexing is not the same as ranking.
Impressions and traffic come from Search Console and analytics. Those are downstream signals, and they depend on query demand, SERP layout, ranking position, title match, and a lot of other factors. A page can be indexed and still have zero impressions if no one searches for the topic or if the page is poorly aligned with demand.
Use the pipeline to diagnose where the system is slowing down
When pages stall, the question is not “Why isn’t this page working?” It is “Which stage is the bottleneck?”
If a page is published but not discovered, look at internal linking, sitemap inclusion, canonicalization, robots directives, and whether the page is actually reachable from the site structure. If you need a broader model for this, How SEO Teams Should Work With Developers is useful because the failure may be in deployment, rendering, or data flow rather than content itself.
If a page is discovered but not crawled, the issue may be crawl demand, site architecture, or crawl waste elsewhere on the site. On larger properties, crawl access is finite in practical terms even if people avoid using the word “budget.” The site has to earn recurring attention through structure, freshness, and link paths.
If a page is crawled but not indexed, then the question shifts to eligibility. That can include duplicate or near-duplicate content, thinness relative to the template family, canonical conflicts, or sitewide signals that make the page a poor candidate for the index. This is the stage where Why Google Is Crawling Your Pages but Not Indexing Them becomes relevant.
If a page is indexed but produces no impressions, the issue may simply be lack of demand. That is not the same as failure. A page can be correctly published, correctly crawled, and correctly indexed without being commercially useful. You still need to decide whether it should exist.
If a page gets impressions but no clicks, the problem is often query-page mismatch, title and snippet presentation, or poor result positioning. That is a different workstream from indexation. It may require reworking the page’s role in the query set rather than changing crawl behavior.
Avoid pretending the timeline is universal
One reason teams misread these metrics is that they expect one normal indexation timeline. That is usually a bad assumption.
The time from publication to crawl, crawl to index, and index to impressions will vary by:
- page type and template
- internal link depth
- site size and change frequency
- domain history and crawl reputation
- content uniqueness and demand
- how often the page family changes
- whether the URL is part of a well-linked cluster or an isolated endpoint
A homepage and a newly generated location page should not be judged on the same timing expectations. A freshly published article and a deep programmatic landing page do not move through the pipeline the same way. The right analysis is relative to page family, not sitewide averages.
That is also why performance reporting should separate “published and not yet crawled” from “crawled but not indexed.” Those are not interchangeable gaps.
Build the tracking model around transitions, not totals
The most useful dashboard is usually not the one with the most metrics. It is the one that answers a few operational questions clearly:
- How many pages were published in each cohort?
- How many were discovered within a given window?
- How many received a first crawl?
- How many were indexed after that crawl?
- How many produced impressions within a reasonable time window?
- How many turned impressions into clicks?
Once you measure transitions, you can also measure percentiles instead of averages. Median time to crawl is useful, but the 90th percentile often reveals the pages that are waiting too long and creating operational drag. That kind of distribution view is more informative than a single “indexation rate” number.
For teams operating at scale, this becomes part of the broader publishing system. How to Build a Search Console Feedback Loop for SEO Publishing is a good companion piece because it turns downstream search data into a signal that can change publishing decisions, not just reporting.
What to do with pages that never progress
Some pages will never move cleanly through every stage. That is not unusual.
The practical response is to classify failures by stage and by page family. A page family that repeatedly gets published but rarely discovered probably has an architecture problem. A family that is discovered and crawled but rarely indexed may need template or duplication work. A family that indexes but does not earn impressions may need demand validation, not technical fixes.
That is the value of modeling the lifecycle end to end: it stops teams from treating every underperforming page as the same problem. Publication is only the start of the system, and traffic is only the last visible output. The work in between is where most of the diagnosis happens.