Google Unveils New Bot for Commercial AI Users: What Site Owners Need to Know

Google has launched a new bot tailored for its commercial AI clients, along with documentation to help site owners track its activity.

This new bot, named Google-CloudVertexBot, is designed to crawl websites on behalf of customers using the Vertex AI platform. Site owners can request this bot to access their sites for data crawling.

Vertex AI Agents

Unlike other bots mentioned in the Search Central documentation, which are primarily associated with Google Search or advertising, the Google-CloudVertexBot specifically targets content for Vertex AI clients.

According to the official Google Cloud documentation, the bot outlines six types of data, including data from public websites, and describes two levels of website crawling:

• Basic Website Indexing
• Advanced Website Indexing

Documentation Summary

The documentation provides detailed insights into how website data is processed.

• Basic Website Indexing: Interestingly, it does not require site owner verification. However, a Google representative clarified that this method relies on a portion of data that has already been crawled by Google.
• Advanced Website Indexing: This method uses the Google-CloudVertexBot and does require domain verification, along with specific indexing quotas. Notably, this crawler does not automatically index public websites; it operates solely at the request of site owners, meaning it won’t visit public sites unless explicitly invited.

Changelog Highlights

The changelog for this new crawler includes key information:

• New Google Crawler: The Google-CloudVertexBot has been introduced to assist site owners in recognizing traffic stemming from this new crawler.

According to the documentation, the Google-CloudVertexBot does not index public sites. The changelog specifies that it was created to enable site owners to identify traffic from this new bot.

Should You Block the New Crawler?

You might wonder if it’s necessary to block this new bot using a robots.txt file. However, it seems that this is not required, as the Google-CloudVertexBot only crawls sites at the request of the owners.

In summary, the introduction of the Google-CloudVertexBot presents an opportunity for site owners to leverage Google’s AI capabilities while maintaining control over their content. By understanding the different indexing methods and how the bot operates, website owners can make informed decisions about how to interact with this new technology.