I know people will refer me to Cloudflare, but that is a non-starter for me. I have tried it before, and it caused issues with forum members being logged out after one minute and people not being able to see images on the forum. It was nothing I configured in Cloudflare. I only used it to try and limit bots. As soon as I reverted to my original nameservers instead of Cloudflare, everything resolved itself. I don't know why there were these problems, but I don't want it to happen again.
Ok, so got that out of the way. My forum seems to be hammered by Google Bot. The real one, not a fake one. Over the past 90 days, it has crawled my forum 628,0000 times. That is averaging nearly 7000 crawls every day, and as I am on shared hosting it is starting to cause issues.
Google Search Console does not have an option to rate limit Googlebot bot and I know Google ignores the robots.txt so I need to find another solution. I don't want to kill it altogether, but if I could halve the number of crawls, that would certainly help.
Does anyone know of a solution to this problem?
Ok, so got that out of the way. My forum seems to be hammered by Google Bot. The real one, not a fake one. Over the past 90 days, it has crawled my forum 628,0000 times. That is averaging nearly 7000 crawls every day, and as I am on shared hosting it is starting to cause issues.
Google Search Console does not have an option to rate limit Googlebot bot and I know Google ignores the robots.txt so I need to find another solution. I don't want to kill it altogether, but if I could halve the number of crawls, that would certainly help.
Does anyone know of a solution to this problem?
Statistics: Posted by thetgc — Thu May 01, 2025 6:47 pm