Block bad crawlers, bots and spiders to save bandwidth

The bad crawlers have been blocked in the nginx configs. The useful links are http://www.b... and https://gith...

nginx robots.txt web-crawler bots

5$ - Ukraine - 7 years, 9 months ago