Technical SEO for AI Search · How to
Discovered, currently not indexed: why Google has not crawled it
Part of the guide Technical SEO for AI search: the 2026 checklist, in order
"Discovered, currently not indexed" means Google knows the URL exists but has not crawled it yet. Google's help page says that typically "Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl", which is why the last crawl date is empty. The fix is to make your server fast and error free, stop wasting crawls on junk URLs, and give important pages stronger internal links.
What does "discovered, currently not indexed" mean?
The page is in Google's queue but has not been fetched. Google's full definition: "The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report."
Crawling is a prerequisite for everything else in the technical SEO for AI search checklist: a page Google has not fetched cannot be indexed or quoted.
So unlike crawled, currently not indexed, Google has not judged the content at all. Its decision is about crawling: how much your server can take, and how much Google wants to crawl your site. Rewriting the page will not help until Google fetches it.
Google's crawl budget guide names this status in its audience list: the guide is "intended primarily for" very large sites, and for sites with "a large portion of their total URLs" in this status in Search Console. If that is you, the crawl budget ideas apply even if your site is small.
The two levers: crawl capacity and crawl demand
Google decides how much to crawl from two things, both defined in its crawl budget guide (updated 22 July 2026):
| Lever | Google's definition | What lowers it | What you can do |
|---|---|---|---|
| Crawl capacity limit | Limits "the total amount of time your server spends holding connections open for Google" | Slow responses, 5xx errors, 429 rate limits | Faster server, caching, no bot rate limits on Googlebot |
| Crawl demand | Varies "based on a site's size, update frequency, page quality, and relevance, compared to other sites" | Many low value or duplicate URLs, weak signals of importance | Fewer junk URLs, better internal links, distinct pages |
Google says that if a site "slows down ... or responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429), the limit goes down and Google crawls less." The "overload" in the status definition points at the first lever. A low demand site with many thin URLs hits the second.
Step 1: check what Googlebot gets from your server
Start with your logs, because they show the real responses. Count Googlebot hits by status code:
grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
Mostly 200 and 304 is healthy. Any meaningful share of 5xx or 429 explains the status by itself. Common causes on small sites: a CDN or security plugin rate limiting Googlebot, shared hosting that times out under load, and a plugin that makes every page slow to generate. Make sure the requests are really from Google: Google publishes its crawler IP ranges (the common crawlers list is at developers.google.com/static/crawling/ipranges/common-crawlers.json), and our curl test guide includes a script that checks an IP against it.
Then measure response time for an uncached request:
curl -s -o /dev/null -w "status %{http_code} first byte %{time_starttransfer}s total %{time_total}s\n" \
-A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
"https://www.example.com/blog/new-post?nocache=$RANDOM"
The query string forces most caches to miss, so you see what your server does without help. If first byte time runs into seconds, page caching or a faster host will do more for this status than anything else on this page. Google's guide supports this: "If Google can load and render your pages faster, we might be able to read more content from your site."
Step 2: stop spending crawls on URLs that do not matter
Every crawl of a junk URL is a crawl not spent on your new article. Google's crawl budget guide lists the fixes; here they are with the usual culprits:
- Consolidate duplicates. "Eliminate duplicate content to focus crawling on unique content rather than unique URLs." Typical sources:
wwwand nonwww, trailing slash variants, tracking parameters. - Block URLs you never want crawled with robots.txt, such as sorted and filtered versions of a list. Google's examples include "differently sorted versions of the same page". Do not use noindex for this: Google "will still request, but then drop the page", which wastes the crawl.
- Return 404 or 410 for removed pages. "a 404 status code is a strong signal not to crawl that URL again."
- Fix soft 404s, pages that say "not found" but return 200. They "will continue to be crawled, and waste your budget."
- Avoid long redirect chains, which Google says "have a negative effect on crawling."
- Keep sitemaps to real, canonical URLs with an accurate
<lastmod>. Google "reads your sitemap regularly". See how to submit a sitemap to Google.
Do not block a section with robots.txt hoping Google will spend the saved crawls elsewhere. Google says it "won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit."
Step 3: raise demand for the pages you care about
Google crawls what it thinks is important, and internal links are your strongest signal of importance. For each discovered URL that matters:
- Link to it from your home page, its category page or pillar, and two or three related indexed articles. A URL that is only in the sitemap and in a paginated archive looks unimportant.
- Make sure the linking pages are themselves crawled often (indexed, linked from navigation).
- Check that the page is not one of many near identical pages. Demand depends partly on page quality, so a large batch of similar location or product pages can hold the whole set back.
For a few critical URLs, "Request indexing" in URL Inspection puts the page in the queue, within a daily limit. It is a nudge, not a fix for the causes above.
Step 4: publish at a pace your site can absorb
On a young site, publishing many URLs at once can produce this status: Google can discover them faster than it chooses to crawl them. A steady pace of distinct, well linked pages is easier for Google to keep up with than a burst. Keep new articles linked from the moment they go live, and let older ones get indexed before adding the next batch.
Postbing lets you set the cadence per site (which weekdays, what time, how many days ahead to write) and links each article into a topic cluster, so a small site is not flooded. We do not promise indexing or a timeline. For Bing and other engines that accept it, IndexNow tells them about new URLs right away; Google does not take part in IndexNow.
The rest of the list (access, rendering, canonicals, schema) is in the technical SEO for AI search checklist. Plans and article counts are on the pricing page.
How do you know it is working?
Watch two things for two to four weeks. In the Page indexing report, the count for this status should fall as URLs move to indexed or to "crawled, currently not indexed". In your logs, Googlebot requests should be mostly 200 and 304 with steady response times. Google says "Indexing is never instant, even when you submit a crawl request directly", so judge trends, not single days.
FAQ
Is "discovered, currently not indexed" a problem?
For a few URLs on a new site, it is normal queueing. For a large share of your URLs, it signals that Google is limiting crawling, usually because of server load or many low value URLs.
Why is the last crawl date empty?
Because Google has not crawled the page yet. Google's help page says it rescheduled the crawl, which is why no crawl date is shown.
Will rewriting the page fix it?
Not directly. Google has not fetched the page, so it has not judged the content. Improve server speed, cut wasted URLs and add internal links so the page gets crawled first.
Should I use noindex on low value pages to save crawl budget?
No. Google says it still requests noindexed pages and then drops them, which wastes crawling. Block pages you never want crawled with robots.txt, or remove them with a 404 or 410.
Does a small site need to care about crawl budget?
Usually not. Google says sites whose pages are crawled the same day they are published do not need its crawl budget guide, but it also lists sites with many discovered but not indexed URLs as part of its audience.
Sources
- Google Search Console Help, Page indexing report: definition of discovered, currently not indexed; indexing never instant, accessed 16 November 2026
- Google, Optimize your crawl budget: audience, crawl capacity limit, crawl demand, best practices, updated 22 July 2026, accessed 16 November 2026
- Google Search Console Help, URL Inspection tool: Request indexing daily limit, accessed 16 November 2026
- Google, verifying Googlebot and other crawlers: IP range files, accessed 16 November 2026
Related reading
Last updated: November 16, 2026. Spotted an error? Email support@postbing.com and we will fix it. Editorial policy.
Try Postbing free
Postbing reads your site, plans a month of topics from what your customers search for, writes in your voice and publishes to WordPress, Shopify, Webflow or Ghost. Your first article is free.
Try Postbing freeNo card needed. No link exchanges, ever.