EverWeb has a setting called Block AI Bots, and it's on by default. It asks a list of AI tools and other automated programs not to read your website. Google and Bing aren't on the list, and neither is the bot OpenAI uses for ChatGPT search. You can turn the setting off, but that doesn't guarantee your website will appear in ChatGPT or any other AI tool's answers.
What the setting does
When Block AI Bots is ticked and you publish, EverWeb adds a list of names to a file on your website called robots.txt. It tells each program on the list that it shouldn't read your website.
The file is a request. Well-behaved programs follow it, and others can ignore it. It's there because some bots scan websites to collect training content and can use a lot of your bandwidth.
It's one setting for the whole list. You can't choose bots one at a time with this checkbox.
Which bots are on the list?
It isn't only AI tools.
| Type | Names on the list |
|---|---|
| AI companies | GPTBot, ClaudeBot, CCBot, Amazonbot, Bytespider, PhindBot |
| Other search engines | YandexBot, Baiduspider, PetalBot, Sogou web spider, SeekportBot |
| SEO and marketing tools | AhrefsBot, SemrushBot, MJ12bot, DotBot, BLEXBot, DataForSeoBot, Barkrowler, ZoominfoBot |
| Web archive and monitoring | ia_archiver, SiteUptime |
| Scanners and general-purpose programs | CensysInspect, Expanse, l9explore, zgrab, GenomeCrawlerd, Go-http-client, python-requests, Custom-AsyncHttpClient, crawl, nbot, Unknown |
Googlebot and Bingbot aren't on the list, and neither is anything that covers every program, so this setting doesn't ask Google or Bing to stay away.
The list doesn't cover every AI tool either. Nothing on it is named after Grok. AI companies can use several bots, and these names change.
For ChatGPT specifically: OpenAI says GPTBot is about training its AI models, and OAI-SearchBot is what decides whether a website appears in ChatGPT search. GPTBot is on this list and OAI-SearchBot isn't, so this setting isn't what keeps a website out of ChatGPT search.
If an AI tool's report says your robots.txt lists many bots, it's correct. If it says EverWeb stops Google from seeing your website, this setting doesn't do that. For help with Google, see Why can't people find my website on Google?
Turn it off
The checkbox appears only when Publish To is set to EverWeb.
- Open your project in EverWeb.
- Go to File > Edit Publishing Settings.
- Scroll down to Advanced Options and untick Block AI Bots.
- Go to File > Publish Entire Site.
To turn it back on, tick the box and publish your entire site again. These bots can use a lot of bandwidth, so keep an eye on yours if you leave it off.
Check that it worked
Go to yourdomain.com/robots.txt in a private browser window, using your own address.
It worked if the names above are gone. You may see a short file, or a page-not-found message. Either is fine.
If the list is still there:
- Clear EverWeb's saved publishing files. On a Mac, go to EverWeb > Settings. On Windows, go to Edit > Options. Then go to Backup > Manage Cache Files > Publish Cache, select your website and choose Move To Trash.
- Go to File > Publish Entire Site again and recheck.
- If it's still there, contact us. Publishing doesn't delete files that are already online, so the old file may need removing from the server. If you host with us, we can do that.
If you publish to another host
You won't see the checkbox, but your robots.txt may still include the list. Check it at the address above. If it does and you'd like it changed, contact us.
Will my website then appear in ChatGPT?
We can't promise that. Each AI company decides what its tool reads, keeps and shows, and EverWeb has no setting that submits your website to them. Turning this off only removes the request. If a tool still can't read a page, the reason may be on its side.
What you can control is having clear text on your pages, because that's what any tool can read. Why can't people find my website on Google? covers this.