A robots.txt file is used to control access to your site’s content by spiders that crawl the web. Reputable robots and spiders will respect the directives in your robots.txt file, but less scrupulous operators will just ignore them. A robots.txt file doesn’t enhance your site’s security, because it’s up to a crawler to voluntarily respect the directives you provide. While it’s not a universal solution to controlling access to your site’s content, a robots.txt file is a valuable tool to manage how your content might be indexed by reputable robots and spiders.
By default, WordPress will create a virtual robots.txt and serve it to crawlers that attempt to access it. A virtual file doesn’t actually exist on a server, but it’s served as if it does exist whenever it’s requested. The default robots.txt file served by WordPress is a simple one that contains only 3 lines. For all user agents, it blocks access to the wp-admin and wp-includes folders, including folders that exist within them.
For small sites with few plugins, the default robots.txt file is all that’s required. But if your site is running plugins that create folders or generate URL’s over which you want more control, you’ll need to create a custom robots.txt file.
To do that, visit the Plugins menu of your site’s dashboard and activate the WP Robots Txt plugin. It’s available on all of our plans. Once activated, the settings for the plugin will appear on the Settings->Reading menu of your dashboard. The screenshot below shows our recommended robots.txt file, already pasted into the metabox. To create your custom robots.txt file, enter it in the metabox and press Save Changes.
You can grab our recommended robots.txt file by following the link or clicking the link to This Gist below. Then, copy the content and paste it into the metabox as shown above. When copying, be careful to copy only the content and not the line numbers, which are there only to facilitate discussion.
Our recommended robots.txt file contains some directives that are specific to certain plugins. They’re recorded in the form of a comment and therefore do nothing. You only need to uncomment these lines if you use the plugins referenced in the comment and you want the directive to be effective. Of course, you can remove the comments completely if they don’t apply, but the few bytes saved aren’t significant.