@tjrobertson52

So Cloudflare just declared war on ChatGPT and all the large language models. If you have a website of any kind, it's gonna have huge implications. So it's really important that you understand this.

Cloudflare is a content delivery network. They also offer a bunch of security products and optimization products. It's actually where I host my domains.

In fact, 20% of websites online are currently using Cloudflare. Come on, even if you're hosting your website somewhere else, you might also be using Cloudflare. A lot of platforms are just using it by default.

And they just announced a new product that automatically blocks all AI's from crawling their websites. Why does that matter? If large language models like Chat, GPT can't call your website, you can't show up in AI responses.

So why would Cloudflare do this? If you're a website that depends on traffic to earn money, if you're making money from ads or affiliate sales, these large language models are taking your content to train on and as a result, you're getting way less content because people are just getting their answers directly from chat, GPT or Google's AI overviews. This is becoming a huge problem for publishers and that's why pretty much the entire internet supports Cloudflare.

In doing this, they're also introducing a pay-per-crawl model where AI's could pay the publishers to crawl their website if the publishers allow it. So I think this is great for publishers, but if you actually offer your own product or service, you definitely want these large language models to be crawling your content. You don't need website traffic to make money.

You just need your products and services to end up in front of your ideal customer. Large language models actually do that better than search engines and if they're not crawling your content, they're just gonna recommend your competitors instead of you. There's no benefit to you blocking them.

Now fortunately, Cloudflare is letting you choose not only if you block these AI bots, but which bots you block. So my advice is figure out if your website is on Cloudflare, log in and make sure you're not blocking any of them unless of course, you're a publisher that relies on web traffic to make money. Now if you were already on Cloudflare like I was, they should be off by default.

You shouldn't be blocking anyone. But for all new Cloudflare customers, they will be blocked by default. So just make sure you're checking that if you're starting a new website or joining Cloudflare.

I also assume this is just the first domino to fall. Cloudflare is getting a ton of great press for this and I imagine other companies are gonna follow suit. I think this is great for the internet and if you're a business selling your own product or service, this could be really good for you as a lot of your competitors will probably unknowingly be blocking the bots.