Sitemap Monitoring That Submits New URLs for You

Sitemap monitoring starts with a file Google caps. A single sitemap is limited to 50MB uncompressed or 50,000 URLs. A site with more pages than that splits its URLs across several sitemap files and lists those files in a sitemap index.

Read the table from top to bottom and the funnel narrows. A scan records every URL in a full 50,000-URL sitemap in the page inventory. Confirmation moves slower: the URL Inspection API allows 2,000 inspections a day on one property, so a large new sitemap gets confirmed across several days.

The Google Indexing API submission queue holds URLs over the day's quota until the next UTC window, and each URL keeps its place in line. SearchAnalysis.io respects each quota and never tries to get around it.

For most sites the limits never bind. A blog that adds 10 posts a week fits inside every quota on the first day. The table matters when you launch a new section, migrate a site or connect a large catalog for the first time.

Sitemap monitoring follows sitemap index files the way Google reads them. A sitemap index is a sitemap whose entries are other sitemaps. Each child sitemap then lists pages. A site with more than 50,000 pages needs more than one file, and an index ties those files together.

What a Scan Does With an Index File

• Reads the index file you added for the site.
• Opens each child sitemap the index lists.
• Records every page URL from every child in one page inventory for the site.
• Keeps the first seen and last seen dates per page URL, whichever child file listed it.

What You Add per Site

You add sitemap URLs per site. Adding the index file once covers its children. You can add several sitemap URLs to the same site when your CMS publishes separate files for posts, products or categories without an index.

When a Child Sitemap Disappears

If a later scan no longer finds a page in any listed file, that page turns Removed and its last seen date stays on the row. A whole child sitemap dropping out of the index shows up the same way: every page it held turns Removed on the next scan. That pattern points to a change in how the site builds its sitemaps.

Indexed pages stay on the 7-day recheck even after they leave the sitemap, so deindex monitoring tells you if Google drops one later.

Sitemap monitoring reads the <lastmod> tag Google treats as a conditional signal. Google uses <lastmod> "if it's consistently and verifiably accurate". A date that changes on every page view, or a date stamped on every URL at each build, gives Google no reason to trust it.

The practical reading of Google's wording: set <lastmod> when the main content of a page changes, and leave it alone when nothing on the page changed. A sitemap where every date matches a real edit is a sitemap whose dates Google can verify against what it crawls.

What SearchAnalysis.io Does With lastmod Today

Scans record the URLs in each file and track first seen and last seen dates. Auto-submit sends newly discovered URLs. A URL that stays in the sitemap with a new <lastmod> is not resubmitted automatically today.

Until then, resubmit an updated page from the SearchAnalysis.io product by hand. The 30-day dedup rule skips a URL that is already actively tracked on your account inside 30 days, and reports the skip in the batch.

Sitemap monitoring treats <priority> and <changefreq> as values Google does not use. Google's sitemap documentation says it plainly: "Google ignores <priority> and <changefreq> values." Setting every product page to 1.0, or tagging the home page as hourly, changes nothing in how Google crawls.

Many sitemap generators still write both tags by default, and every byte counts toward the 50MB uncompressed cap.

Two Different Kinds of Priority

The word priority means two separate things on this page, so the table below keeps them apart.

SearchAnalysis.io priority orders URLs inside your own queue. When the Google Indexing API quota of 200 publish requests a day is close, High URLs go out before Normal and Low ones, and the rest wait for the next UTC window. The Google Indexing API submission page covers that queue in full.

To send one page ahead of the rest, submit it as a single URL with High priority.

Sitemap monitoring no longer has a Google ping to send. Google retired its sitemap ping endpoint, announced in June 2023 on the Google Search Central blog in the post titled "Sitemaps ping endpoint is going away". Scripts and plugins that still call that ping address get nothing useful from it.

If your deploy pipeline or CMS plugin pinged Google after each publish, that step now does nothing for Google. The routes that still work are the three documented ways to tell Google about a sitemap, plus the per-URL channels below.

What Replaced the Ping for New URLs

• Search Console sitemap submission, made once per sitemap.
• Google Indexing API, a URL_UPDATED notification per URL, 200 publish requests a day per Cloud project, documented for JobPosting and BroadcastEvent pages.
• IndexNow, one POST of up to 10,000 URLs, shared among Bing, Yandex, Seznam, Naver, Yep, Internet Archive and AmazonBot. Google is not on the IndexNow list.
• URL Inspection API, which confirms indexing for one URL at a time, 2,000 a day per property.

SearchAnalysis.io sends through the last three for every auto-submitted URL. Each URL gets three status rows: Google Indexing API, Search Console and Bing. Only the Search Console row turns green, because only URL Inspection confirms indexing.

Read the IndexNow submission page for the key file step, and remove any old Google ping call from your build scripts.

Sitemap monitoring in SearchAnalysis.io reads the sitemaps you add to it, and Google learns about the same sitemaps through its own routes. Google documents three of them.

1. Search Console. Open the property, go to the Sitemaps report and submit the sitemap URL.
2. Search Console API. Submit the sitemap with an API call from your own code.
3. robots.txt. Add a Sitemap: line with the full sitemap URL to the robots.txt file at your site root.

Pick one or use all three. They point Google at the same file.

Where SearchAnalysis.io Fits

Adding a sitemap URL to SearchAnalysis.io tells SearchAnalysis.io where to scan. It builds the page inventory and, with auto-submit on, sends each new URL to the Google Indexing API and IndexNow and inspects it with URL Inspection. Telling Google about the sitemap file itself stays one of the three routes above.

The robots.txt route has a side effect worth checking: a robots.txt rule that blocks a page stops Googlebot even when the sitemap lists it. Run a listed page through the free indexability checker to see which robots.txt rule applies, with the Googlebot group winning over * and the longest matching rule deciding.

Use the same Search Console property in both places. When you connect Google in SearchAnalysis.io you pick the property, a domain property (sc-domain:) or a URL-prefix property, and inspections run against it.

Sitemap monitoring keeps two facts apart: a URL is in your sitemap, and a URL is in Google. A sitemap tells Google a page exists. Google decides what it crawls and indexes, and nobody can guarantee indexing.

Search Console's Page indexing report names the reasons a listed URL stays out. These are some of the ones sitemap owners meet most.

SearchAnalysis.io checks the Google side with the URL Inspection API. Each inspection stores the verdict, coverageState, robotsTxtState, indexing state, the Google-selected canonical and the last crawl time. A URL turns Indexed when the verdict is PASS or the coverage state says indexed.

The guide to Crawled, currently not indexed covers the most common gap between a sitemap and the index. For one page at a time, the Google index checker shows every verdict field.

Discovered is the first state in the sitemap monitoring page inventory. A URL turns Discovered when a scan finds it in one of the sitemaps you added for the site, and the scan writes the first seen date on the row at the same moment.

What Happens Next

• Auto-submit on: the scan puts each newly discovered URL in a batch with the source scheduled scan, and the URL moves to Submitted.
• Auto-submit off: the URL stays Discovered until you pick it in a sitemap import and send it in a batch yourself.
• Already tracked: a URL actively tracked on your account inside 30 days is skipped and reported, and not resent.

Two Different Kinds of Discovered

SearchAnalysis.io uses the word for its own scan. Google's Page indexing report has a reason with a similar name, Discovered, currently not indexed, which Google describes as "The page was found by Google, but not crawled yet." The two are separate facts. A URL may show Discovered in the inventory before Google has seen it at all, and a URL may sit in Google's Discovered, currently not indexed group long after SearchAnalysis.io submitted it.

When a URL spends days in Google's Discovered group, the cause sits on Google's side of the line. Check the page with the free indexability checker first: a page that fails to load, redirects, carries noindex or points its canonical elsewhere gets fixed there before any resubmission.

Submitted is the sitemap monitoring state for a URL that has entered a batch. From there, every URL gets three separate status rows, one per channel.

The Google Indexing API and IndexNow rows are fire-and-forget. Google acknowledging a notification means the ping was accepted. It does not mean the page is indexed, so the row stays Processing. The Search Console row carries the confirmation.

Queue Order and Quotas

Inside the batch, each URL keeps a priority of High, Normal or Low, which sets order within your own queue. The Google Indexing API submission queue sends up to 100 URLs per batch request and stays inside 200 publish requests a day per Google Cloud project. URLs over the day's quota wait for the next UTC window and keep their place in line.

Filtered Before Sending

Some URLs never reach a channel. A URL outside the site's domain, a duplicate or an invalid format shows Dropped. An error that needs a person shows Failed, with the error text on the row, and the Monitor offers bulk retry.

The batch page lists the URLs 100 per page, with the batch name and its source.

Indexed is the sitemap monitoring state that only Google can set. SearchAnalysis.io calls the Search Console URL Inspection API for the URL, and when the verdict is PASS or the coverageState says indexed, the row turns Indexed. Green means verified and nothing else.

The Confirmation Ladder

1. First inspection about 10 minutes after submission.
2. If not yet indexed, the next check at +15 minutes.
3. Then +1 hour.
4. Then +6 hours.
5. Then every 24 hours, up to 6 checks on the ladder, a window of about 3 days.
6. After the ladder, a daily fallback keeps checking unconfirmed URLs.

Each inspection becomes a row in the URL's check history with the time, the state (INDEXED, NOT_INDEXED or ERROR), the coverage state and the source. Each one stores Google's health snapshot too: verdict, coverageState, robotsTxtState, indexing state, the Google-selected canonical and the last crawl time.

When a URL turns Indexed, SearchAnalysis.io sends a transactional email titled "Indexed: <url>". From then on the URL is inspected again every 7 days for as long as the site is connected. The Google index checker page covers every verdict field in detail.

Transient errors from Google (429, 5xx or network) retry after 30 minutes, so a brief outage does not stall the ladder.

Removed is the sitemap monitoring state for a URL a later scan no longer finds in any sitemap you added. The row keeps its last seen date, which marks the final scan that still listed it.

Removed describes your sitemap. It says nothing on its own about Google. A page you deleted, redirected or merged usually leaves the sitemap on purpose, and a page that vanished after a plugin update leaves it by accident. The inventory shows both the same way, so the dates help you tell them apart.

Questions to Ask About a Removed Row

• Did the page move? Google indexes the redirect destination, so submit the destination URL.
• Did the page return a 404? Google's Page indexing report lists that as Not found (404).
• Did a whole child sitemap drop out of the index file? Many rows turning Removed in one scan points to a sitemap setting.
• Was the page Indexed before it left? Then the 7-day recheck still covers it.

Removed and Deindexed Are Separate Events

An Indexed URL is inspected again every 7 days for as long as the site is connected, whether or not it is still in the sitemap. If a recheck finds it not indexed, the row shows amber Not indexed, a deindex event. Deindex monitoring tracks those drops across the whole site.

Alerts for deindex events by email and Slack are PLANNED and will arrive in a later release. Today the Monitor filters and the Google index card show the change.

Sitemap monitoring stamps two dates on every inventory row. First seen is the scan that found the URL for the first time. Last seen is the most recent scan that still listed it.

Reading the Gap to Indexed

The time between a first seen date and the inspection that turned the row Indexed is how long the page took to get from your sitemap into Google, as far as URL Inspection can see. The full check history on the URL detail page shows every inspection in between, each with its time and coverage state.

The dates make a site migration readable. After a URL change, old URLs show a last seen date on migration day and new URLs show a first seen date on the same day. The Monitor exports submitted URLs as CSV, and the SearchAnalysis.io product tour shows where each view lives.

Auto-submit is the sitemap monitoring switch that turns a scan into a submission. You turn it on per site. With it on, newly discovered URLs are submitted on their own as a batch with the source scheduled scan.

scheduled scan (every 6 hours) or Scan now
  new URL found              -> Discovered, first seen date set
  auto-submit on             -> batch created, source: scheduled scan
  sent to 3 channels         -> Submitted
  URL Inspection PASS        -> Indexed
  URL leaves the sitemap     -> Removed, last seen date kept

Rules the Batch Follows

• Every URL in the batch gets three status rows: Google Indexing API, Search Console and Bing.
• A URL already actively tracked on your account inside 30 days is skipped and reported.
• URLs outside the site's domain, duplicates and invalid formats are Dropped before sending.
• The Google Indexing API quota of 200 publish requests a day decides how many go out today; the rest wait for the next UTC window.
• IndexNow sends only after SearchAnalysis.io verifies the key file at https://yourdomain/{key}.txt.

Eligibility on Each URL

SearchAnalysis.io reads each page for JobPosting or BroadcastEvent structured data, the scope Google documents for the Indexing API. A blog post or product page shows Best-effort, which is normal for most content sites, and submission proceeds best-effort. URL Inspection and IndexNow submission run either way.

Each auto-submitted batch has its own page, and a confirmed URL triggers the "Indexed: <url>" email.

Resubmission on <lastmod> change is a PLANNED sitemap monitoring feature. When it ships, SearchAnalysis.io will resubmit a URL automatically when its <lastmod> value in the sitemap changes. It is not in the product today.

What Works Today

• Auto-submit sends URLs a scan finds for the first time.
• A URL that stays in the sitemap keeps its state and its dates.
• You can resubmit an updated page by hand from a single URL, a bulk paste or a REST API call for developers.
• The 30-day dedup rule skips a URL already actively tracked on your account inside 30 days.

Why lastmod Is the Trigger

Google uses <lastmod> "if it's consistently and verifiably accurate". A site that updates <lastmod> only when a page changes publishes a clean signal, and the planned feature will read that same signal to decide when a page goes out again. A sitemap that stamps every URL with the build time would trigger a resubmission for every URL at every build, so accurate dates will matter on both sides.

Other Planned Sitemap Work

A free sitemap checker tool is PLANNED and will join the free indexability checker. Drip-feed submission, paced over a number of days, is PLANNED too. Waitlist members get an invite before public launch and will see these features as they ship.