AI and Your Website

Our goal is to help you be discovered, both in traditional search engines and AI responses

Let’s start by breaking down how AI might access your website now and in the future.

Search Bots

Use bots or crawlers to navigate and index web pages. These tools collect data like text, images, links, and keywords.

AI Bots, Crawlers, and Agents

Extract data from websites for various purposes, including training AI models, performing market research, or gathering information for other applications.

How AI bots use data

Search bots and crawlers have been around for years. The difference now is that AI bots can utilize large language models and machine learning to extract more data, gather research from multiple websites, and navigate complex website structures and dynamic content. AI can interpret content contextually using language models, which enables them to summarize and extract key points from your site. This data may be used to train AI models like ChatGPT and Claude.

Can my website be crawled by AI bots?

Yes. We allow AI bots to access the website’s content.

We will make no attempt to block AI bots unless you specifically request that we do so.

Why does Orbit allow bots by default?

Search visibility

Allowing bots ensures that your website’s content can be indexed and ranked by search engines, making it easier for your audience to find you online.

AI discovery

When bots from tools like ChatGPT or Perplexity can access your site, your content becomes eligible to appear in AI-generated summaries and answers, expanding your digital footprint.

Brand reputation and reach

Permitting bots to access your site helps reinforce your expertise and visibility across trusted platforms, strengthening your brand’s presence and credibility.

Blocking bots can limit how your site appears in search results or AI summaries, which may reduce discoverability online.

A person stands smiling while holding a coffee mug, in front of a perforated metal wall with a red circular design. They wear a striped shirt over a dark t-shirt.

How does Orbit use AI?

We view AI as a powerful tool.

At Orbit, we utilize AI for research, data analysis, keyphrase planning, and measuring the effectiveness of our content and deliverables.

When used correctly, AI is a great resource for making more thorough, data-driven decisions. That’s why it’s important that your website remains open to AI and search bots: allowing bots to crawl your site ensures we can better analyze performance, refine your content, and improve your visibility in search engines over time. And you can use it this way, too!

How do I know what bots are visiting my website?

A web analytics table shows traffic data from various AI referrer sources, including page views, sessions, engagement rate, and session duration percentages.

Google Analytics

While GA4 typically filters out bot traffic and doesn’t label specific bot names in standard reports, some bot traffic data gets through. You can set up an Exploration in GA4 to see what bots are accessing what pages on your Website.

See Andy’s article on how to do this

Table showing web crawlers with columns for bot name, provider, type, number of requests, and robots.txt compliance status, with most bots flagged for issues.

Your Server / Cloudflare

Depending on your current website configuration, you may be able to see bot data in a server audit. The image above is from Cloudflare’s AI Audit (beta).

How do I know if my site is blocking bots?

Go to your home page and add /robots.txt to the URL. On this robots.txt page, you will see a ‘botname’ if you are blocking bots (i.e., ChatGPT-User, Googlebot, etc.).

User-agent: botname
Disallow: /

When the robots.txt file does not contain a bot name specifically, it is not used to block AI bots.

TIP! If you’re still curious about the robots.txt file on your site, take a screenshot and ask ChatGPT, “Here’s a screenshot of my site’s robots.txt file, is it blocking any bots?”

Additionally, there are other tools, such as your Server or Host, Cloudflare, and Google Search Console, that can give you insight into what bots are being blocked. Look there next if you need further validation.

A browser window displays a robots.txt file blocking the user agent "botname" from accessing all content on the domain yourdomain.com.

When should I consider blocking bots?

  • Sensitive data not meant for public distribution – This typically refers to content behind a login area, which bots cannot access.
  • Internal pages (like thank-you or admin pages) – We set these pages to no index, no follow as a best practice.
  • When a bot is malicious, site security tools can help identify such instances. This is rare, but it can happen, and they can be blocked.

What do I share with leadership?

  • AI and bots are part of the modern web.
  • Blocking bots is possible, but often unnecessary.
  • We’re here to provide transparency, control, and support.

Exceptions are handled case-by-case

If desired, you can request to block AI bots from your website.

Please notify your Project Manager if you have any concerns or require a custom policy.

In summary

By default, Orbit allows AI bots to access websites.

We want you to get all the benefits of your website redesign and since AI isn’t going away, we feel it’s in your best interest to allow them. We’ll assume this is the plan unless you tell us otherwise.

We’re your partner and will respect your decision either way.

Want to learn more about AI and measurement?

Here are some great articles to save to your bookmarks: