Robots Exclusion Protocol
Robots.txt follows the Robots Exclusion Protocol (REP), a standard developed in 1994 to provide website owners with a way to communicate with web crawlers. The file must be placed in the root directory and named exactly 'robots.txt'. It uses a simple syntax with directives like User-agent, Disallow, Allow, Sitemap, and Crawl-delay to control crawler behavior.