Guide on How to Add Robots.txt File on WordPress Websites

Want to increase the traffic and buzz about your website? There are powerful SEO tools that tell the search engines to show your website to the users thus increasing traffic to your website. Robots.txt file is a very powerful and useful tool. In this article, we’ll share with you steps to add robots.txt file on WordPress websites. Therefore, giving structure to your website and search engines a channel to crawl through it.

This article contains useful information on what Robots.txt file is and how you can add Robots.txt files in WordPress.

What is Robots.txt File?

Robots.txt file is a simple .txt file, which you can make in your system or by using any online tool.

It’s a document that gives information to a search engine crawler bot about the particular and important pages to crawl and index on. When you create a website, search engines send their bots and make a map of the content of your page.

They know the content and related keywords, thus showing those websites containing the keywords and increasing the number of visitors.

Why Do You Need a Robots.txt File?

If you do not have a robots.txt file, the search engines will still crawl and mark your website. In this process, you will not be able to mark and index the important pages accordingly, thus creating not a strong impact.

In this procedure, the bots can crawl over the pages on your site, which are not important. These search bots have a certain quota for every website.

At times if they do not finish their crawling, they do come back and complete their session. This takes a lot of time and slows down your indexing rate.

Thus, you can take control of this by stopping them to crawl over the unnecessary pages which save your crawl quota.

It helps the search engines to crawl more on your website and index them quickly and thus, helping your website gain more visitors quickly.

Example of a Robots.txt File

How to Add Robots.txt File on WordPress?

Now, you can edit and use the robots.txt file tool on your WordPress. There are two methods to do so.

Thus, below is a complete detailed guide on how to add robots.txt file on WordPress. You can use the best method according to you. They are as follows:

Method 1: Edit Robots.txt File Using Yoast SEO Plugin

• When you use a Yoast SEO plugin, it comes with a robots.txt file generator. Yoast is one of the widely used WordPress Plugins for SEO.
• You need to use it to create a robots.txt file from your WordPress admin area directly.
• You need to go to SEO, and select on the tools button present on the WordPress admin and then need to click on the file editor link.

• A new page would appear, where the Yoast SEO page will show your already existing robots.txt file.
• If you won’t have an existing file then Yoast SEO will automatically generate a new robots.txt file for you.

The Yoast SEO’s robots.txt file generator will automatically add with the 2 rules which are:

• User-agent:
• *Disallow: /

This is a basic format which is considered as a complete robots.txt file, also one robots file can contain multiple lines of directives and user agents. The user agent means that the file applies to all web robots which visit the website whereas disallow tells the robots to not visit any pages on the website.

• You need to delete this text or else it will block all the search engines from crawling your website.
• You now need to go and add your own robots.txt rules.
• Click on the save robots.txt file button to save your changes.

Method 2: Edit Robots.txt File Manually Using FTP

Another easiest method to add robots.txt file in wordpress, Use your FTP client and you are good to go:

• You will need an FTP client to edit a robots.txt file.
• You will thus connect your WordPress hosting account using an FTP client.
• As you will enter, you will be able to see the robots.txt file present in your website’s folder.

• If you don’t have an existing file, you can create one and add it.
• A robots.txt is a plain file you can initially download on your desktop and with the help of any plain editor, you can create one.

• Save the changes and finally, you will be able to upload it to your website’s folder.

Now, you can always test and check your robots.txt file. It is a good idea to check for avoiding any mistake or glitch. The following are the steps to test your robots.txt file. They are as follows:

• As you have created your robots.txt file, there are many testing tools present. You can therefore now open your Google Search Console account and use the file present inside.
• If you don’t have an account yet, we suggest you do it right away. Here’s a detailed guide on how to add your website to Google Search Console.
• You simply need to login to your Google Search Console account and need to switch to the older version of Google Search Console.

• Now, you will launch the robots.txt file tester tool and locate it under the crawl menu available.

The tool will automatically get your website’s robots.txt file and will thus highlight any errors or warnings.

Bottomline

That is it guys, I hope you would have learned how add robots.txt file on WordPress.

Usually, among the SEO experts, there is a common myth that blocking the WordPress category, tags and archive pages may improve the crawling rate and therefore result in fast indexing and lead to higher ranking, which is not true.

Therefore, the initial goal of robots.txt file is to stop the search engines from crawling into your website’s pages that are not publicly available. One should follow the above methods and thus use them in order to create a positive exposure. Also, you should be aware of the search engine bots and index them to crawl only on the relevant information you want to show on to your visitors.

Thus, in this way, they will be able to present the users with the exact needed and useful content accordingly.