To avoid unwanted content inside the search indexes, webmasters can instruct spiders never to crawl specific files or directories with the standard robots.txt file in the basis directory in the area. In addition, a website page can be explicitly excluded from a internet search engine's databases by utilizing a meta tag specific to robots (normally