Deindex Monitoring for Pages Google Drops

A deindexed page with the reason “URL marked 'noindex'” carries an indexing directive that tells Google to keep it out. The directive lives on your side, in the page or in the response headers, which makes this one of the clearest drops to trace.

What You Can Check

1. Run the URL through the free indexability checker. Its fourth check reads noindex or none in meta robots and in the X-Robots-Tag header.
2. Look at what changed near the drop date: a template, a plugin setting, a CMS visibility switch or a server rule.
3. Remove the directive and test the page again until the checker reports no blocking directive.

What SearchAnalysis.io Records

The recheck that found the drop is saved as a check history row with its time, the state NOT_INDEXED, the coverage state and the source monitor recheck. The health snapshot stores the indexing state, so the row shows whether the block came from the meta tag or the HTTP header. The URL turns amber Not indexed, and the verified indexed count on the Google index card goes down by one.

A cleared directive does not force Google to index the page again. It removes the reason Google reported.

“URL blocked by robots.txt” means a robots.txt rule stops Google from crawling the page. For a page that dropped, deindex monitoring shows the robots.txt state Google reported at the recheck. Google lists a separate warning, “Indexed, though blocked by robots.txt”, for pages that stay indexed while blocked, so a block and a drop do not always arrive together.

What Google Reports

• robotsTxtState: DISALLOWED
• indexingState: BLOCKED_BY_ROBOTS_TXT
• Page indexing report: URL blocked by robots.txt

What You Can Check

Test the URL in the indexability checker. It reads robots.txt with three rules: the Googlebot group wins over *, the longest matching rule decides, and Allow wins a tie. A broad Disallow added during a release is a common thing to look for.

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

In this example, Googlebot follows its own group and is allowed, even though every other crawler is blocked. Delete the Googlebot group and the * rule applies to Google too.

What SearchAnalysis.io Records

The check history row for the drop stores robotsTxtState and the indexing state with the date of the recheck. Earlier rows keep ALLOWED, so the first DISALLOWED row dates the block as SearchAnalysis.io saw it. The block may have started any time in the 7 days before that recheck, so check your deploy log for that window.

“Not found (404)” means Google requested the page and the server answered that it does not exist. Common causes to test for a deindexed page in this state are a deleted page, a changed slug and a broken route, and every one of those is on your side.

What Google Reports

• Page indexing report: Not found (404).
• URL Inspection pageFetchState: NOT_FOUND, a field the API returns.
• Neighboring reasons: Blocked due to unauthorized request (401), Blocked due to access forbidden (403) and URL blocked due to other 4xx issue.

What You Can Check

• The first check of the free indexability checker reports the HTTP status. A non-2xx status fails.
• If the page moved, redirect the old URL to the new one. Google indexes the destination, so submit the destination.
• If the page was removed on purpose, a 404 is the expected answer and the drop is the result you wanted.

What SearchAnalysis.io Records

The recheck is stored as a NOT_INDEXED row with the coverage state, the verdict and the last crawl time. The sitemap monitoring inventory adds a second record: when the URL leaves your sitemap, the inventory marks it Removed and keeps its first seen and last seen dates. Set the last seen date beside the drop date and you can tell whether the page left the sitemap before or after Google dropped it.

“Server error (5xx)” means your server answered Google's request with a 5xx status. For a deindexed page, the places to look are hosting, application errors and origin load at the time Google crawled. The URL Inspection API returns pageFetchState SERVER_ERROR for the same condition.

Two Kinds of 5xx

An ERROR row is a failed inspection, and SearchAnalysis.io does not treat it as a drop. Transient errors (429, 5xx and network failures) retry after 30 minutes. Repeated failures back off exponentially:

What You Can Check

Load the page through the free indexability checker. It fetches as SearchAnalysisBot/1.0 with a 10-second timeout, and a non-2xx status fails the first check. A pass now does not prove the server answered Google at crawl time, so compare the stored last crawl time with your server logs and uptime history for that window.

The check history keeps the row with its date, coverage state and last crawl time, which gives you the window to search in your logs.

“Page with redirect” means the URL now redirects somewhere else. Google indexes the destination, so a page that was indexed and now redirects drops out by design, and the destination takes its place if Google indexes it. Deindex monitoring shows the drop on the old URL.

What Google Reports

What You Can Check

1. Run the old URL through the indexability checker. It follows up to 5 redirects and reports where the chain ends.
2. Confirm the destination returns a 2xx status, carries no noindex and has a self-referencing canonical.
3. Submit the destination URL, since that is the page Google indexes.

What SearchAnalysis.io Records

The old URL gets a NOT_INDEXED row with the coverage state and the date of the recheck, and it turns amber. The destination is a new URL to SearchAnalysis.io. Once you submit it, the destination climbs the inspection ladder and, when Google confirms it, enters the 7-day watch. The 30-day dedup applies to URLs already tracked inside 30 days, which matters if the destination was submitted recently.

A planned migration produces a wave of these rows. The Monitor's live state filter collects them in one list, and the CSV export gives you the old URLs to check against your redirect map.

“Duplicate without user-selected canonical” means Google treats the page as a duplicate of another URL and the page declares no canonical of its own. A deindexed page in this state has lost its place to a URL Google picked. Google lists a related reason, “Alternate page with proper canonical tag”, for pages that point their canonical at another URL on purpose.

What Google Reports

• The reason in the Page indexing report.
• googleCanonical: “The URL of the page that Google selected as canonical”.
• userCanonical: the canonical the page declares, a field the API returns. For this reason, Google found none.

What You Can Check

Run the page through the indexability checker. Its fifth check warns when the canonical is missing, passes when it references the page itself, and warns when it points elsewhere. Then open the stored Google-selected canonical and compare the two pages.

What SearchAnalysis.io Records

Each recheck stores the Google-selected canonical in the health snapshot. The row that found the drop shows the URL Google chose that day, next to the coverage state and the date.

“Duplicate, Google chose different canonical than user” means your page declares a canonical and Google picked a different URL. The page drops because Google decided another URL represents it better. Deindex monitoring shows the drop and the URL Google picked; it does not know why Google chose it.

What You Can Check

• Open the URL detail page and read the stored Google-selected canonical next to the URL that dropped.
• Confirm your declared canonical with the fifth check of the free indexability checker.
• Compare the two pages side by side: content, internal links pointing at each, and which one your sitemap lists.
• Check that redirects and hreflang or parameter rules do not send Google toward the other URL.

What SearchAnalysis.io Records

Each check history row stores the Google-selected canonical as it was on that day. Earlier rows, from when the page was indexed, show the canonical Google picked then, so you can see the recheck where Google's choice moved and compare it with your own change log.

If Google's choice is a page you are glad to see in search, submit it so the Google index checker confirms it and the 7-day watch moves to that URL.

Google describes “Crawled, currently not indexed” in one line: “The page was crawled by Google but not indexed.” Google fetched the page and decided to leave it out of the index, and deindex monitoring reports that decision without knowing Google's private reasons.

What Google Reports

• The coverage state with this reason.
• A last crawl time that shows Google fetched the page.
• A related reason for pages not fetched yet: Discovered, currently not indexed, described as “The page was found by Google, but not crawled yet.”

What You Can Check

Rule out the causes you control with the free indexability checker: status, redirects, robots.txt, directives and canonical. Its sixth check warns when the page has under about 100 words of visible text. After that, the page itself is the variable: its content, its purpose and the links that point to it. The guide to Crawled, currently not indexed covers this state in depth, with the checks to run before you change the page.

What SearchAnalysis.io Records

The recheck row stores this coverage state with the date, the verdict and the last crawl time. Read the last crawl time against the drop date: a recent crawl means Google saw the current version of the page before it left the index, so changes to that version are the place to start.

“Soft 404” is a reason in the Page indexing report. Google applies it from its own reading of the page, so a deindexed page in this state carries Google's label, and deindex monitoring stores the label without guessing at the judgment behind it. The URL Inspection API returns pageFetchState SOFT_404 for the same condition.

What You Can Check

1. Run the page through the indexability checker and read the first check (HTTP status) and the sixth check (visible text under about 100 words warns as thin).
2. Look at what the page shows to a visitor: an empty listing, a removed product, a search page with no results or a placeholder.
3. If the page is truly gone, return a real 404 or redirect to the closest live page, and submit the destination.
4. If the page has real content, make that content visible in the HTML.

What SearchAnalysis.io Records

Pages on ecommerce sites with out-of-stock or retired products are worth filtering in the Monitor after each weekly cycle, since a retired product page may show little visible content.

The check history is the core record of deindex monitoring. Every inspection of a URL is stored as a row, from the first check after submission to the latest 7-day recheck, so the history of a dropped page shows when it was confirmed and when it left.

state:  INDEXED | NOT_INDEXED | ERROR
source: inspection | daily check | monitor recheck

Reading the Sources

• Inspection: a check on the ladder after submission, from about 10 minutes out to about 3 days.
• Daily check: the fallback for URLs the ladder did not confirm.
• Monitor recheck: the 7-day recheck of an Indexed URL. A drop is found by a monitor recheck.

Each row comes with Google's health snapshot: verdict, coverage state, robots.txt state, indexing state, Google-selected canonical and last crawl time. An ERROR row records a failed inspection (429, 5xx or network), retried after 30 minutes, and does not count as a drop.

You read the full history on the URL detail page. The Google index checker page explains each stored field.

Read a dropped page's history from the bottom up. You see the inspection that first confirmed it, the run of monitor rechecks that kept it green every 7 days, and the single NOT_INDEXED row that turned it amber. That last row's date is the date you work from when you look for the change that caused the drop.

Deindex monitoring tracks live index state, which is separate from the submission lifecycle. Every submitted URL gets three channel rows: Google Indexing API, Search Console and Bing (IndexNow). Only the Search Console row carries index confirmation, and only that row turns green.

Submission Lifecycle Labels

Live Index State

Amber Not indexed is the deindex state: a URL that was Indexed and a later recheck found not indexed. It differs from Dropped, which describes a URL filtered out before it was ever sent. Green means verified and nothing else.

Accepted notifications to the Google Indexing API and IndexNow show Processing and never Indexed, because a notification does not confirm indexing. A deindexed page can still show Processing on those two rows while its Search Console row is amber.

Read the two together this way: the lifecycle label tells you what happened to the submission, and the live state tells you what Google reports about the page today. A URL can finish its lifecycle as Indexed and later carry amber Not indexed after a 7-day recheck.

Each site has a Google index card, and deindex monitoring feeds its numbers. When a recheck finds a page not indexed, the verified indexed count falls and the change shows in the deltas.

What the Card Shows

• Verified indexed: pages URL Inspection confirmed.
• Change over 24 hours and over 7 days.
• A 30-day trend line of the verified indexed count.
• Not indexed: pages a check found outside the index.
• Awaiting check: the count of pages waiting on a check.
• Coverage: verified indexed divided by tracked, always printed with its denominator.
• Appearing in search: pages with impressions in Search Console Search Analytics over the last 7 days.

coverage = verified indexed ÷ tracked

Why the Denominator Is Printed

A coverage figure without its denominator hides how many pages are tracked. The card prints both numbers, so a coverage change from new submissions reads differently from one caused by drops.

Appearing in search is an independent whole-site signal. It comes from Search Analytics impressions and does not depend on URL Inspection, so read it as a second view next to the verified count. Search performance beyond appearing in search (queries, clicks, positions) is PLANNED. See the SearchAnalysis.io product overview.

Read the 24-hour and 7-day change after each round of rechecks. A negative change with no new submissions points to drops, and the Monitor's live state filter lists the URLs behind the number.

The Monitor is where deindex monitoring turns into a work list. It shows every URL with its per-channel statuses and updates by live polling.

Filters

Actions

1. Filter by site, then by live state, to list the pages that dropped.
2. Export the list to CSV for your developer, content team or client.
3. Open each URL's detail page to read the coverage state and the date of the drop.
4. Use bulk retry for URLs that show Failed, after you read the error text on each one.

Developers get the same data by API. GET /api/v1/submissions/{batchId} returns a batch with per-URL, per-channel state, and the check_url_index_status MCP tool answers for one URL. The SearchAnalysis.io developer surface lists every endpoint, and the OpenAPI 3.1 document is public at /openapi.json.

Agencies that manage client sites can filter by site and export one CSV per client. Workspaces above sites and shareable status pages for clients are PLANNED and will group sites by client when they ship.

Each CSV row comes from the same records the dashboard shows, so the list you hand off matches what you see on the URL detail pages.

Deindex alerts are PLANNED. SearchAnalysis.io will send alerts by email and Slack when deindex monitoring finds a drop, and it will send weekly digest reports. Until they ship, drops show in the dashboard, the Monitor and each URL's check history.

Shipped Today vs. PLANNED

What to Do Until Alerts Ship

• Open the Monitor once a week, filter by live state and export the amber URLs.
• Watch the 7-day change on the Google index card for each site.
• Read the check history of any dropped URL for the date and coverage state.

Waitlist members get an invite before public launch, launch pricing locked and an extended free trial. Join the SearchAnalysis.io waitlist to get access when invites go out.

The alerts will use the same check that powers the amber state today, the 7-day recheck through URL Inspection. Nothing about how a drop is found will change when they ship; they will add a way to hear about it without opening the dashboard.