How To Optimize Robots.txt File To improve SEO Of Site

9.August.2018| No Comments

To get the search engine optimization results for your site the very first and the foremost thing that is required is to improvise is the robots.txt file. The robots file is that file which simply instructs the search engine what content of your website to crawl.

Your website’s ranking is completely dependent on what search engine crawls out of your website. So it becomes really important to figure out “How to optimize robots.txt file? so that all the important content keeps crawling time by time.

If a site doesn’t have a robots.txt file than the search engine will stop crawling and index that website. This blunder will lead to vanishing site performance, no matter how hard someone works on that site. All SEO process starts from crawling the robots.txt file. All sites content like images, content, pages, etc all is crawl by robots.txt file.

Where is robots.txt is located ?

how to optimize robots.txt
The robots.txt file can be found in the root folder of your website. You can view and edit the robots.txt by C-panel file manager. This file is just like any other text file which can be view by a simple text editor. If you don’t have any robots.txt file in your root folder then you can easily create just like any another text file.

For creating a robots file, create a simple text file and rename it to robots.txt in the root folder of your site. The very first thing that comes inside the robots file is the user agent. The user agent is nothing but the name of the search bot like Google Bot, Yahoo bot, Bing bot, etc that you want to get your pages crawl by it.

The user agent is declare as User-Agent: *
Where * is use to give instruction all search engines. The * is use to instruct which part to crawl of the website and which to not.

Here is the example of robots.txt file :

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /serch/
Disallow: /tag/
Disallow: /members/
Disallow: /offers/
Disallow: /community/
Disallow: /helpdesk/
Disallow: /livesupport/
Disallow: /preview/
Disallow: /documentation/
Allow: /wp-admin/admin-ajax.php

You can find your site’s robots file by this:

Hava a look what Google say’s about robots file basics:
how to optimize robots.txt file
The things that you want it to crawl are written with Allow tag, and the things that you don’t want to get crawl are written with Disallow tag. Which thing to write with Allow and what with Disallow is a very simple but important aspect.

You should Allow the content upload folder of your site. If by mistake the content folder gets Disallow than search spiders will not able to crawl any content of the site. And this thing will affect the site’s ranking. Crawl-delay specifies for how much msec the crawling should be delay page by page.

But the Disallow tag is also very important for site’s performance maintenance. There are so many files and folders which are not necessary to get crawl. Files like the sites archive files, backup files, etc.

By disallowing all these files let site perform faster. As the Google Crawling budget is finite for every site, so it is very important to utilize the crawling budget efficiently.

Search Engine Crawls the site on the basis of this Robots.txt file. Apart from all visible main pages of the site, there are so many pages on your website which are not meant to be crawled.

For better crawling and best search ranking results, it’s important to choose what to allow and what to disallow in the file because the search engine has a limited “crawling budget” for every website.

If you have allowed various futile files in the robot file than the search engine will take more time than usual and it will negatively affect the site’s ranking. So, it becomes absolutely important to optimize the file carefully.

How to check robots.txt file in website
This file can be easily tested from the Google Search Console (webmasters tools). Login to your search console and go to Crawl -> robots.txt Tester and click on the Test button. If you see the Allowed message then the file is working perfectly.

You can also check errors and warnings from there and resolve them if there are any.

What affects the robots.txt file

How to optimize robots.txt file

For having a proper Search Engine optimize robots.txt file it is very important to decide what to keep with the Allow tag and what with the Disallow tag. Content folder, image folder etc. are those things that must use the Allow tag and things like Archive files, duplicate web pages, duplicate content etc are those things which can be put with the Disallow tag.


For better optimization results from the robots.txt, we must follow the webmasters guideline to stay away from any kind of penalty. According to webmasters guidelines, there is no need to hide the old and low-quality content.

As the crawlers will still crawl that content. It’s is a very good option to edit the old content of your website for better optimization results. You should keep updating old blogs on your site. It will let the crawlers to crawl the old blogs which will add an extra score to your sites SEO.



Leave a Reply

Your email address will not be published. Required fields are marked *


Subscribe Now

Get free download on your email

Send me new freebies, offers & news

Subscribe Now

Get free download on your email

Send me new freebies, offers & news