Two URLs can look different in a spreadsheet and still deliver almost the same answer. On an ecommerce site, that might be a tracking variant or a product variation. On a service website, it might be an old landing page still competing with its replacement.
To find duplicate pages with Screaming Frog, crawl the site, review exact duplicates, enable near-duplicate analysis where your licence supports it, then inspect the matched pages before choosing a fix. Similarity identifies candidates; it does not decide whether they should be deleted.
In my original tutorial I demonstrate the Content filters and Duplicate Details view. This guide keeps the useful workflow and updates the interpretation: similar regional pages and low word counts are not automatically errors.
Decide what you want the crawl to answer
Start with a clear question. Are you looking for multiple indexable copies, checking a set of programmatic pages, or investigating unnecessary crawl routes? The settings and review should match that purpose.
Record the start URL, crawl limits, rendering mode and exclusions. A restricted crawl cannot establish that you found every duplicate on the whole site. For a large site, begin with a useful section while planning wider coverage.
Keep a backup of the crawl and your settings. Your recommendations should be reproducible, particularly when developers need to compare the result after a template fix.
Configure near-duplicate analysis deliberately
Screaming Frog’s duplicate-content tutorial explains the current configuration. Exact duplicate checks are available in the normal workflow; the configurable near-duplicate features require a licence. The free crawler also has a URL limit, so note whether your scope exceeds it.
Enable near duplicates in the content duplicate settings and review the similarity threshold. A lower threshold broadens the candidate set and can increase manual review. The 80% setting demonstrated in my video is an exploratory choice, not a universally correct score.
Review whether the analysis includes only indexable pages. That can suit a search-result duplication review. A crawl-efficiency investigation may also need URLs already canonicalised elsewhere, so do not exclude them without considering your question.

Separate exact matches from similar text
Once the crawl has finished, run the required Crawl Analysis for near duplicates. Then review the Content filters and select a candidate to inspect its duplicate details.
| Finding | What to inspect | Common next decision |
|---|---|---|
| Exact duplicate | Two routes delivering the same content | Consolidate unintended copies |
| Near duplicate | Similar text with different details or purpose | Keep, improve or merge after review |
| Shared navigation and footer | Boilerplate dominates the comparison | Review the configured content area |
| Product variation | Different attributes and customer demand | Preserve useful variation or consolidate deliberately |
| Regional version | Genuine audience or availability differences | Check the localisation strategy |
Open both pages. Read the title, main content, product or service details, canonical and available navigation. A percentage cannot tell you whether the difference matters to a customer.

The source screenshot shows a historical comparison in my tutorial. The regional BBC pages illustrate how similar text can appear at different addresses; the image is not evidence that those pages should be removed or that they suffer a search penalty.
Choose the fix from the page’s actual job
If an obsolete URL is genuinely replaced, a relevant permanent redirect may be appropriate. If a necessary alternate must remain accessible but the content is equivalent, review canonicalisation. If both pages meet different needs, improve their distinct answers instead of forcing a merge.
Imagine three shop URLs serving the same red lamp: a clean product address, an email-tracking variant and an obsolete category route. The clean product page may be the preferred destination. Confirm the behaviour and customer routes before implementing that decision.
By contrast, a red lamp and a blue lamp may have meaningful differences in availability, photography and demand. The decision depends on the catalogue and browsing experience, not a rule that every similar description is harmful.
Google’s canonicalisation guidance explains how redirects, canonical annotations and sitemap signals express preferences. Make those signals agree with the URLs you link to internally.

Turn the export into a developer-ready action list
An export of 800 candidates is not yet a useful implementation brief. Group related URLs and document the reason for each proposed change. Many apparent page-level problems originate in one template or routing rule.
Use these columns:
- Candidate URLs: all affected addresses in the group.
- Observed relationship: exact copy, parameter variant, overlapping article or legitimate alternate.
- Intended destination: the final address or the reason separate URLs should remain.
- Action: update, redirect, canonical review, template fix or no change.
- Dependencies: internal links, sitemap entries, navigation and localisation.
- Verification: owner, test result and review date.
For overlapping articles, use the update-versus-new-content framework before rewriting. For disconnected or redundant routes, review your website architecture.
Re-crawl the affected routes after implementation
Test old and final URLs. Confirm response codes, destination content and the intended canonical. Check that your own links point directly to the correct route, then compare the new crawl with the original evidence.
An intended alternate may no longer appear in an indexable-only duplicate report. That is useful only if the underlying settings and behaviour are correct; a disappearance from a filter is not enough by itself.
Does duplicate content automatically trigger a penalty?
No. Ordinary duplication and malicious or manipulative publishing need different analysis. Focus on reducing confusing copies and maintaining useful pages rather than assuming every similar pair is penalised.
Should every short page be expanded?
A contact page or a clear product specification can be concise. Add information when it helps the page fulfil its purpose, not to satisfy a crawler’s word-count warning.
Start with one duplicate group and write a justified action before changing it. My SEO consulting can help turn a large crawl into prioritised work.
Watch the original video
This guide adapts my original Let’s Talk SEO tutorial. Historical examples are distinguished from current documentation; no example above promises search results.

