Learned or Solved Something Today?

Dump it here! Let others learn from you

Showing 2 of 2 from page 1

anonymous posted in #nginx #devops #snippets #apache #bot

List of About 500 Bad Bots and Blocking Access by User Agent in Nginx and Apache

Crafted an ultimate bad bots list to block in a gist and example nginx, apache configuration to block them by user-agent. Using robots.txt to block bots it might be good idea to start with. But ddding robots.txt might not be enough when they don't r

anonymous posted in #text #snippets #robot #bot

Example robots.txt file to block bad bots from scanning your website

This example robots.txt is to block most of the known bots that you might not want them to craw your website Bots, spiders, and other crawlers hitting your website to copy or download your contents or the entire website. They tend to send too many re