Block bad crawlers, bots and spiders to save bandwidth
The bad crawlers have been blocked in the nginx configs. The useful links are http://www.blogtips.org/web-crawlers-love-the-good-but-kill-the-bad-and-the-ugly/ and https://github.com/mariusv/nginx-badbot-blocker
Price : 5
Country : Ukraine
Tags : nginx robots.txt web-crawler bots
Published : 7 years, 9 months ago