A directory business cut index bloat from 340,000 junk URLs to 12,000 useful ones

Representative examplesEvery case study in this library is an illustrative composite of the kind of engagement we deliver — written to show our method and standards, not to name clients.

Where they started

Regional services directories live and die by their listings being findable, and this one had spent years adding facets users genuinely value — filter by suburb, service, opening hours — without anyone counting the URLs those filters created. Empty filtered views outnumbered real listings by tens to one. The platform is a bespoke build with a templating layer that can express rules per template and not much beyond that; a small team runs the product, and its developer already splits time between feature work and maintenance.

What it was costing

Faceted navigation had generated hundreds of thousands of near-empty URLs — filtered lists with zero listings. Crawl budget drowned in them, quality pages aged out of the index, and Search Console coverage was a wall of 'Crawled – currently not indexed'.

What they could see

• Search Console coverage was a wall of 'Crawled – currently not indexed' stretching across hundreds of thousands of URLs.
• New and edited listings took weeks to appear in search while the crawler waded through empty filtered pages.
• The indexed-URL count sat near 340,000 against roughly 12,000 pages anyone would want a searcher to land on.
• Quality listing pages aged out of the index and had to be resubmitted by the editorial team manually.

The constraints we worked inside

• The facets were product features users valued — the links had to keep working for people, just not for crawlers.
• The platform's templating made per-template rules the only practical lever.
• History mattered: previously indexed junk URLs needed careful treatment to avoid shedding legit equity.

What had been tried before

The platform vendor disallowed the filter parameters wholesale in robots.txt as a one-line fix.

The block also removed faceted pages that ranked and converted; organic entries fell within a month, the rule was quietly pulled, and bloat resumed.

An outside consultant ran a monthly service that manually noindexed junk URLs from crawler exports.

Hand-processing hundreds of thousands of URLs loses to a generator that mints new ones daily; the backlog aged out faster than the list shrank.

What we proposed

We proposed classifying before cutting: a full crawl would separate URL patterns by whether the facet combination ever returns listings, and each pattern would get one rule — index, canonical, or noindex — applied template by template with crawl statistics watched between waves so a mistake stayed contained. Genuinely empty archives would be retired with 410s rather than left in noindex limbo. Users keep their filters; only the crawler-facing side changes. History mattered too: previously indexed URLs got individual treatment where equity existed.

Just as important is what we ruled out, and why:

• Removing the facets from the user interface entirely — Visitors filter by suburb and hours constantly — the facets are product features; only the crawler's view needed constraining, not the shopper's.
• A blanket noindex across every faceted URL — Some facet combinations are established landing pages with rankings and links; the templating could not express exceptions, so a blanket rule would shed real equity.
• Replatforming to a CMS with native facet controls — A rebuild would take quarters and retrain the team for a problem the existing templating could express; the bloat was rules, not architecture.

How the work ran

Delivered by the growth pod — technical SEO specialist over 7 weeks, with working increments reviewed with the client every week.

The stack, and the reasoning

Crawlers

Classifying patterns required enumerating the faceted space and sampling each combination's result state at scale; logs could not say whether a filter ever returned listings.

Search Console

Coverage and crawl-stats reports were the accepted baseline and the scoreboard; the client's team already read them, which made the waves legible internally.

Log-file analysis

Crawl budget is spent by bots, not crawlers; the logs showed exactly how much Googlebot time the junk patterns consumed — the number that funded this project.

Robots/noindex governance

The templating could only express rules per template, so a documented rules file per pattern made each rollout reviewable, reversible, and teachable to the developer.

What went wrong

How we worked together

Cadence

A fortnightly 45-minute call with the product owner and the platform developer; between waves, a short written status against the rollout plan.

Client side

The product owner judged which facets carried real user value; the platform developer implemented the template rules and owned the cache behavior.

Decisions

Classification disputes were settled against the sampled data, not opinion; any template touching paid listings needed the commercial lead's explicit approval.

They provided

Full crawl and log access, Search Console, the developer's hours between feature sprints, and facet-usage records from the product's own analytics.

What changed

The headline: indexed urls, with 'crawled – not indexed' reports falling accordingly, over the 10 weeks after full rollout — 340k → 12k, read from Search Console coverage. A second check: impressions on quality listing pages, same period at +18%.

New listings surface in days now, and the editorial team stopped resubmitting pages that had never actually disappeared from anything but Google's patience. Crawl reports fit on one screen, which changed their meetings: the product owner asks what URL pattern a new feature will generate before it ships. The developer keeps the rules file open in a tab — the discipline survived the engagement, which is the part we care about most.

The result was read from Search Console coverage against the pre-engagement baseline over the stated window, with a guardrail check on impressions on quality listing pages, same period. Where platform-reported numbers and business outcomes differ, this record says which layer it is quoting.

What they own now

• The pattern classification file mapping every faceted URL pattern to its rule
• A facet-review checklist for any new navigation or filter feature
• The log-analysis query pack used to track crawl-budget recovery
• A documented 410 policy covering when archives are retired versus redirected
• A Search Console monitoring routine with the coverage views that mattered