what is the role of robot.txt in SEO?

@oliverjonesMar 23.2013

Hello Guys,
Please share your review about robot.txt, what is this and how its important for seo. I am not more familiar to seo task so please help me to know about it. Thanks

to post a comment

SEO

30 Comments(s) _↴

@dennyleonMar 23.2013 — #Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.

@iTechMar 25.2013 — #You can make use of robot.txt files if you don't want to crawl any page etc..

@creditcardsdealMar 26.2013 — #It is important part for the site . From that you can restrict the crawler to carwl that particular page you doesn't want to crawl.

@arunshoryMar 27.2013 — #Robots.txt helps, not to index website personal pages in search engines. It acts like security for your personal page.

@frankdevineApr 03.2013 — #Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

@andysmithApr 04.2013 — #way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots metatag.

@Steve_SmithApr 04.2013 — #Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

@opensourceApr 04.2013 — #It is used to restrict or allow bots to visit our site.

@SinelogixcompMay 22.2015 — #We should use robot.txt if we want to restrict crawlers to crawl any private pages. You can do it using notepad file and upload it in your server. robots.txt tells google, yahoo, bing crawlers what not to crawl on your pages.

An example of how we use it is the following.

We have a test directory on our server where we upload sites to be tested before we go public. Obviously, we do not want the crawlers to take note of this directory since it is a directory for testing only. Below is the robots.txt file contents.

User-agent: *

Disallow:/testing/

@brainminwaeMay 23.2015 — #To have Robots.txt for your websites is one oh the important factor fro SEO perspective.

It is a text file which tell serach engine about which pages of your website should be crawled and which should not...

It is simple text file which is uploaded on server.

@advent_geekMay 23.2015 — #You can make use of robot.txt files if you don't want to crawl any page etc..[/QUOTE]

Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.[/QUOTE]

Most of the Search engines won't fllow the Robots.txt rules. but the purpose of Robots.txt is to control or specify the bots what should they do.

Kind Regards

@Vikas_PatelMay 25.2015 — #Robots.txt is stop the search engines crawling your web site or web page. You can show your site online but without crawling and it is use when you are working on your site or page without showing anyone.

@kampuzzMay 25.2015 — #how to crawl and index pages on their website.

[URL="http://www.kampuzz.com"]

www.kampuzz.com[/URL]

@daviddakaraiMay 25.2015 — #In a website that the site owner is asking the search engines to "skip" (or "disallow") some URL's which are not necessary.

Syntax: User-agent:

* Disallow: /cgi-bin/

Disallow: /tmp/

Disallow: /junk/

@EllaJonesMay 27.2015 — #The robots.txt file is a simple text file. It is used to tell the search engine crawler that which pages of our website should be indexed.

@anirban09PJun 01.2015 — #Robots.txt file is used to tell the search engines which pages to be indexed by and which pages not to index. Robots.txt is file is used to block the admin pages in the site

@richardstevensJun 01.2015 — #A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

@ZelmaWiseJun 09.2015 — #Yes in a simple way it is used to restrict or allow bots to visit our site.

@avadevineJun 10.2015 — #robots.txt file should be uploaded to your website rood directory and a valid sitemap.xml reference should be detected from it. This file is useful to search engine to know what pages are allowed to be crawled and what are not. Though it looks very small file it is very valuable for SEO.

@Kevin_PeterJun 29.2015 — #To remove unwanted web pages, we use Robot.txt to tell the bots not to index the listed URLs.

@Isabella_martinJul 01.2015 — #The robots.txt is a text file which is developed by website developer to instruct robots (typically search engine robots) how to crawl and index pages on their website.

@samueljohnJul 02.2015 — #The Robots.txt is a file that tells search engine bots what pages of the website not to be crawled. It is a file website owner's restrictions that direct the search engine bots to follow the rules specified in Robots.txt

@shiny012Jul 10.2015 — #A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers.

yad

@GauravsagJul 18.2015 — #Robots.txt file to give instructions to search engine robot that which page need to be crwal and which page not be.

Syntex

User-agent: *

Disallow: /

@aapbudgie98Jul 18.2015 — #Robots.txt files inform search engine spiders how to interact with indexing your content.

By default search engines are greedy. They want to index as much high quality information as they can, & will assume that they can crawl everything unless you tell them otherwise.

If you specify data for all bots (*) and data for a specific bot (like GoogleBot) then the specific bot commands will be followed while that engine ignores the global/default bot commands.

@Bruce_AthertonJul 20.2015 — #A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be crawled.

@aapbudgie98Jul 23.2015 — #Robots.txt file is use to block site inner page so that Google crawler not reach on your site inner page.

@uditshJul 29.2015 — #Hello Guys,

Please share your review about robot.txt, what is this and how its important for seo. I am not more familiar to seo task so please help me to know about it. Thanks[/QUOTE]

The main use of robots.txt is, when you do not want some of your site's webpages or directory to be indexed/seen on search results. You can use either 'robots.txt' file or 'NoIndex meta robot tag', but never use both for the same page.

For SEO, some of your site pages/directories may have nothing to do with your service, like [I]temp[/I], [I]cgi-bin[/I] or any other out dated product pages. If they have good rank on search engines, you may loss many of visitors/customers related to your service/business.

@JenithaJul 29.2015 — #In simple words, Robots.txt is used to tell the google which pages want to crawl. It plays vital for doing SEO.

@AlanRonJul 29.2015 — #Robot.txt is the file which contains the list of pages that are hidden from search engine bots.

Also in #SEO _↴

What is the best way to traffic generate?How to Reach People on Different Social Media Platforms Serptat.com here to say hello

Success!

Help @oliverjones spread the word by sharing this article on Twitter...

Tweet This

what is the role of robot.txt in SEO?

30 Comments(s) ↴

Also in #SEO ↴

Success!

The web is an endless sea of information. Don't miss the boat... Subscribe!

Social

Version

30 Comments(s) _↴

Also in #SEO _↴