Friends, if you are also a beginner blogger then you will definitely want to know what is robot.txt file? How to use it and how to generate it.

Friends Robot.txt is a small file that stays in the root directory of your website. Robot.txt file is used for which content or pages of your website is not crawl (search) on google search engine by GoogleBot.

Know Everything How to create and use Robot.txt file in 2020 

So let's know everything about robot.txt file:

What is ROBOT.TXT File?

If you are a blogger or website developer, then it is important for you to know what this robot.txt file is in the end. Fiends robot.txt is a text file which contains very small code that tells that if you do not want to index any page of your website on google then they have to mention in robot.txt.

If robot.txt file is not there, then friends google bot starts indexing all the content, page, category, label of your website and if that page cannot be found on google then the error of page not found will be displayed.

So therefore the pages or links that you do not want to crawl google are used in robot.txt file for that.

For example. Danger sign board on road that means that is wrong place and you have not be visit that place. Other wise you may go on wrong path.

Format of ROBOT.TXT File

Friends, the code of a robot.txt file is only 2-3 lines. Which remains the same for all websites. The format of robot.txt file is very simple like

User-agent: [User Name]

Disallow : [URL or Page that you don’t want to crawl]

Friends, in a robot.txt, the file is completed on these two main commands but there are other commands like: Allow  & Search:

So friends, lets start to know about all these:

User-agent: These are search engine crawlers Bot. If you want to give the same instruction to all search engines, then you can use the * symbol after the user-agent.

Different crawler for different search engine  :

Goolge: Googlebot

Bing: Bingbot

Yahoo: Slurp

DuckDuckGO: DuckDuckBot

Google Images: Googlebot-Image

Disallow: This prevents files or pages from crawl

Allow: This allows your file or page to be crawl.

Crawl-delay: decide how many seconds the crawler will have to wait to crawl your page or content

Example  Of robot.txt file

User-agent: *

Disallow:

Example 2

User-agent: *

Disallow:/Privatefile.html

Disallow: /Category

Disallow: /Search

How to generate robot.txt file

Creating a Friends robot.txt file is very easy. If you want, you can also make it online or you can create it yourself by creating a text file.

https://www.tools.seobook.com
https://www.seoptimer.com
https:// www.smallseotools.com
https:// www.labnol.org

Where to put the robot.txt file in your website?
Friends if you have created a robot.txt file and you do not know where to put it. So friends you have to place the robot.txt file in the root folder of your website.

That means you will go to the Cpanel of your website and that root folder with http folder where all your pages are kept, you have to keep your robot.txt file as well.

For example

https://www.abc.com / robot.txt

How to Use in Blogger:

So friends, you can generate online robot.txt content and put it in your blog account.

Step 1: Generate online robot.txt file or sitemap generator file

Go to: https://www.labnol.org/blogger/sitemap

Enter your full website name: https://www.technicalbook.in and click on Generate sitemap.

Step 2: Copy the Complete code with Sitemap URL

Step 3: Go to Blogger account and open Search and Preference from Setting option.

Step 4: Open Custom.robot.txt option and paste the copied code into the box.and click on save changes.

How to check

Friends, if you have added robot.txt and you want to check whether this file was added to my website or not, friends, you can check it by going to the browser and entering the URL of your website with robot.txt.

1. Go to browser

2. Enter URL https://www.technicalbook.in/robots.txt

3. See the Result.

Friends If you get your answer about robot.txt file please share your feedback with us.

5 Things To Know Before Starting Blogging As Career

How To Create Own Successful Blog Step By Step With Easy Guide

Basic Blog Setting Of Blogger Has The Answer To Everything.

Keep 13 Things In Mind Before Applying On Google Adsense In 2020