Create custom robots.txt files for your website
The robots.txt file tells search engine crawlers which pages or files they can or can't request from your site. It's placed in your website's root directory and is one of the first files crawlers check.