robots.txt

Simple Definition

A text file webmasters create to instruct web robots how to crawl pages on their website.