How Googlebot Reads Your Website: 3 Things You May Never Have Considered

Your Product Pages Haven't Been Indexed for Three Weeks—and It's Not a Content Problem

A friend who sells hydraulic fittings launched a batch of new product pages last year.

After three weeks, Google Search Console (GSC) still showed "Discovered—currently not indexed." He assumed the content was the issue, so he rewrote every page description. Two more weeks passed, and nothing changed.

I checked the rendered screenshot in GSC's URL Inspection tool. The page structure was there, but the entire section with product specifications, material details, and technical parameters was blank.

We found the cause: he had built the pages in Elementor and set the product-specifications section to appear only when a visitor scrolled to it. When Googlebot rendered the page, it did not simulate a user scrolling, so the block was never triggered. Each time Google saw the page, it saw an empty area, judged the page to have no substantive content, and declined to index it.

He rewrote the content twice. The content was never the problem.

That is the gap between being "crawled" and being understood: Googlebot may have visited your page, but that does not mean it saw what you intended it to see.

Crawling and Rendering Are Two Different Things

Most people think Googlebot visits a website the same way you open it in a browser: it arrives, the page loads, it sees everything, and that's that.

The actual process looks like this:

1. Step 1: Discover the URL
2. Step 2: Crawl (download the HTML) — Googlebot leaves at this point
3. Step 3: Render (execute JavaScript) — this step is usually completed within minutes to hours
4. Step 4: Index (store it in the database so it can participate in ranking)

Google's JavaScript SEO documentation makes this clear: crawling and rendering are two fully separate processes, each with its own processing queue.

In Step 2, Googlebot downloads your HTML file and leaves. At that moment, it has not seen any content that requires JavaScript to run before it appears—it has only the raw HTML shell.

In Step 3, the HTML is placed in a rendering queue, where Google's Web Rendering Service (WRS)—a system based on Headless Chrome—executes JavaScript and reconstructs dynamic content. This wait is currently usually measured in minutes to hours.

The key is not how long the wait is. The key is whether the content can actually be seen during rendering.

There is one type of content Googlebot really cannot see: content that loads only after a button is clicked, such as "Load More" or content fetched from the server only after someone expands a section. Googlebot does not simulate clicks, so this content never appears in its rendered result.

The "appear on scroll" animations used by Elementor and similar page builders are slightly more nuanced. If the animation only controls how content appears—for example, a fade-in or upward slide—and the content is already in the HTML, Googlebot can usually see it.

But if the animation also triggers a server request for data, problems can arise. Most Elementor scroll animations fall into the first category. What probably made my friend's hydraulic-fittings pages blank was a plugin configuration that blocked the loading path for a critical JavaScript file. A blank area in GSC's rendered screenshot is the most direct evidence.

Which Googlebot Is Visiting Your Website Right Now?

Here is a fact many people do not know.

Googlebot is not a single crawler. Google has a desktop Googlebot, a mobile Googlebot (Smartphone), an image crawler, a video crawler, and other crawler types.

So which one is mainly visiting your site now? The mobile Googlebot.

Since 2023, Google has completed its move to mobile-first indexing for all websites. The content Google's mobile Googlebot sees is what Google uses for indexing and ranking in nearly every case.

What does that mean?

For ranking, Google is not looking at the carefully designed desktop version of your product page. It is looking at the mobile version. If your mobile and desktop content differ, Google's index contains only the portion seen on mobile.

Less Content on Mobile Means Less Ranking Potential

This is the most common trap in mobile-first indexing.

Google's requirement is clear: the mobile page's content must match the desktop page (content parity). Content removed from mobile makes no contribution to rankings.

Situations that are fine:

• Mobile puts specification details in a collapsible section, but the content is present in the initial HTML—Google can index it normally.
• Responsive design presents content in a different layout—the same HTML is used, while CSS controls the display for different screens.

Situations with SEO risk:

• To keep the mobile page "clean," you remove product descriptions or specifications that appear on the desktop page.
• Mobile content appears only after someone clicks a JavaScript-driven "Load More" button.
• You run a separate m-dot site, but its content is much thinner than the main site.

That removed content does not exist in Google's index, no matter how detailed your desktop version may be.

There is a question worth thinking through here: does "mobile first" mean sacrificing the desktop experience?

No. Those two goals do not conflict—but you need to understand whom each experience serves:

For B2B export buyers, a mobile encounter may help them discover you. But when they open your site on a computer to compare specifications, review PDFs, and submit an inquiry form, that is when conversion happens. Both scenarios matter; they simply happen on different devices.

Google recommends responsive design precisely because it can serve both goals: the same HTML, with CSS controlling the layout on different screens. Googlebot gets the full content, mobile users get an adapted layout, and desktop buyers get a professional wide-screen presentation—without compromise for any of the three.

The real danger is not choosing one device over the other. It is falling into two traps: thinning mobile content (which hurts rankings) and neglecting the desktop experience (which hurts conversions). Responsive design can avoid both risks at the same time.

Crawl Budget: Should Small and Mid-Sized Export Sites Worry?

Google's own wording is:

"Crawl budget is not something most publishers have to worry about."

Sites with no more than a few thousand pages usually do not need to manage crawl budget proactively.

That is true—with one exception.

A buildup of low-value URLs.

WordPress automatically generates many URLs you may not even realize exist:

• Tag archive pages: /tag/stainless-steel/
• Paginated category pages: /category/products/page/2/
• Search results: /?s=wire+mesh
• Filtered URLs with parameters: /products/?color=silver&size=large

These pages have no search value of their own, but Googlebot still considers them on every visit. As a result, your truly important product pages may be pushed later in the crawl queue, and new pages may take longer to be discovered.

This is not a case of "using up" crawl budget. It is a case of diluting Googlebot's attention.

For small and mid-sized export sites, crawl budget does not need to be a core concern. But you should regularly clean up these low-value URLs—using robots.txt or noindex—so Googlebot can focus on the pages you actually want it to see.

How Can You Tell What Googlebot Actually "Saw"?

You do not have to guess. You can check in five minutes.

1. Open Google Search Console.
2. Open the URL Inspection tool.
3. Enter the page URL.
4. Select "Test Live URL."
5. Open the "Screenshot" tab.

You will see the actual screenshot Googlebot produced after rendering the page. If content is blank in the screenshot, or an area did not load, that is content Googlebot cannot actually see—and it is content that will not be indexed.

Check your most important product page now. If the product description or specifications are blank in the screenshot, you have found the real reason your rankings are not improving.

Key Takeaways

Sources: Google Search Central — Googlebot Documentation · JavaScript SEO Basics · Mobile-First Indexing Best Practices · Large Site Crawl Budget Management