Duplicate SEO pages are usually created long before anyone notices them in Search Console. The problem is not just duplicate text. In programmatic systems, duplication often starts earlier: two records generate the same slug, a template emits the same title pattern across too many pages, different entities receive nearly identical inputs, or a page family expands into combinations that do not add distinct search value.
If you want to catch duplicates before publishing, you need to validate at the level of identifiers, rendered output, and page intent. That means checking for slug collisions, normalized title collisions, template repetition, semantic overlap, and database rules that prevent an invalid page from ever reaching the publish step.
Start with the lowest-level collisions
The cheapest duplicate to catch is the one that shares an identifier.
A slug should be unique within the scope of the site or page family, not just “likely unique.” Normalize before comparing. Lowercase the value, strip punctuation that your router ignores, collapse whitespace, remove accents if your URL policy does that, and compare the final canonical slug rather than the raw source field. If San Francisco, san-francisco, and san francisco all resolve to the same URL shape, they are duplicates even if the source records look different.
Do the same for other routing keys: locale, category, product ID, integration ID, and any parent-child path segments. Two pages can look distinct in the CMS and still collide after URL generation.
This is where database constraints matter more than review checklists. If your slug field is not backed by a unique index at the right scope, you are relying on process to catch what the database should reject. For programmatic systems, that is backwards.
Compare normalized titles, not just raw titles
Raw title strings are noisy. One page says “Best CRM for Startups,” another says “Top CRM for Startups,” and a third says “Best CRM Software for Startups.” A human can see they are close; a system should be able to measure that before the pages publish.
Normalize titles before comparison:
- lowercase
- strip stopwords only if that fits your use case
- remove boilerplate suffixes like brand names or repeated separators
- collapse punctuation and spacing
- optionally map synonyms that your template system treats as equivalent
Once titles are normalized, look for collisions and near-collisions within the same page family. Exact matches are obvious. Near-matches matter when a template is producing pages that differ only by one token. That is often a signal that the underlying data does not justify separate pages.
The key question is not “Are the titles different?” It is “Would a searcher see these as separate intents or as the same page with a different label?”
Audit the template, because templates create duplication at scale
A template can make hundreds of pages feel unique while still producing the same page structure, the same sections, and the same claims.
That is why duplicate detection should inspect the rendered template, not just the source data. Compare:
- heading structure
- section order
- presence or absence of variable blocks
- repeated copy fragments
- paragraph-level boilerplate
- repeated schema fields
- static intro and outro text
If two page types use the same template and differ only in three nouns, they are often functionally duplicates even if the record data is technically distinct. This is especially common in programmatic SEO architecture systems where templates are optimized for speed but not for differentiation.
A useful test is to ask whether a page family changes the answer it gives. If the only difference is the entity name in the H1 and a few template variables, the page may not deserve to be indexable.
Detect semantic similarity across rendered content
Exact-match checks miss the more common failure mode: pages that are not identical, but are close enough that they compete with each other or fail to offer enough unique value.
For this, compare rendered content after the template has been populated. Focus on the parts that users actually see, not just metadata. A practical similarity check can combine:
- text fingerprints for repeated phrasing
- section-level comparison
- embedding or vector similarity for broader semantic overlap
- threshold rules for identical introduction paragraphs, feature lists, or FAQ blocks
You do not need a perfect language model score. You need a repeatable rule that blocks pages that are too close to existing pages in the same cluster.
The threshold will vary by page type. Two comparison pages may share a lot of structure and still be legitimate if the underlying comparison axis changes. Two location pages with the same service description, same testimonials block, and same generic city insertions are much more likely to be duplicates.
This is why similarity checks work best when paired with page-family rules. A candidate page should be compared against other pages in the same family, the same parent topic, and any pages that could satisfy the same query intent.
Look for data overlap, not just text overlap
Sometimes the text is different, but the data source is the same enough to create duplicate value.
Examples:
- two directory pages built from the same inventory set with different sort orders
- two city pages that pull the same service mix and the same “local” copy blocks
- two integration pages where the only variable is the order of logos or features
- two product pages that differ only by an attribute combination users would never search separately
To catch this, inspect the source records behind the page. Compare the entity sets, not just the rendered output. If two pages share most of their source data, you should ask whether one page should exist at all, whether the data should be merged, or whether one version should be canonical and the other excluded.
This is closely related to the eligibility logic discussed in how to build SEO pages from a database. The page should not be created unless the data model can support a real distinction.
Decide your canonical strategy before pages are generated
Duplicate detection is not only about blocking bad pages. It also needs a clear answer for legitimate overlaps.
Some pages will share substantial similarity because the site architecture requires it. That is where canonical strategy matters. You need to decide, before publish time, which version is primary, which versions are variants, and which versions should never be indexed.
Canonical decisions should be deterministic. A page cannot be allowed to publish first and then “figure out” the canonical later if the system has already emitted duplicate URLs, sitemap entries, and internal links.
Common patterns:
- one preferred URL per entity, with variants canonicalized to it
- one indexable page plus filtered or parameterized variants marked noindex
- parent pages canonicalized from child combinations when child pages add no new value
- consolidation into a single page when multiple candidates target the same intent
If you need a more centralized rule set, how to manage canonical URLs at scale is the right companion piece. The important point here is that duplicate detection and canonical policy should be built together. If they are separate processes, they will drift.
Put hard constraints in the database
A review workflow can catch some duplicates. A database constraint can prevent them.
At minimum, consider:
- unique constraints on slug within the correct scope
- unique constraints on normalized title within a page family when appropriate
- composite uniqueness across locale, entity, and page type
- status-aware constraints so drafts can coexist, but published records cannot collide
- foreign key and eligibility checks that prevent invalid entity combinations from entering the queue
If your publishing system supports retries or regeneration, add idempotency keys so the same generation job cannot create two publishable records by accident. This is especially useful in automated systems where the same content may be generated more than once.
Database constraints are not a substitute for editorial judgment. They are the backstop that keeps a bad state from becoming public.
Build duplicate detection into the publish gate
The right place to catch duplicates is before the content leaves draft or queued state. A publish gate should combine deterministic checks and similarity checks in one pass:
- Normalize routing keys and verify slug uniqueness.
- Normalize titles and compare against existing page families.
- Render the page and compare its structure against sibling pages.
- Compare the source data set for overlap with existing pages.
- Apply canonical or noindex rules where overlap is expected.
- Block anything that exceeds your similarity threshold or fails uniqueness constraints.
This is the same philosophy behind how to validate SEO pages before publishing and how to build automated QA gates for SEO content: validation should be deterministic where possible, and opinionated where the system can measure meaningful overlap.
The practical test: would both pages survive as separate results?
A useful way to pressure-test duplication is to imagine the two pages sitting side by side in search results. If a searcher would not understand why both exist, your system probably should not publish both.
That is not the same as demanding every page be radically different. Large sites can and should reuse structures. But reuse has to be controlled. The more your system depends on templated generation, the more you need explicit rules for uniqueness, canonicalization, and scope.
If you get those rules right, duplicate detection becomes a compile-time problem instead of a cleanup project.