/    Sign up×
Community /Pin to ProfileBookmark

Major role of Robots.txt file in a website

Copy linkTweet thisAlerts:
Nov 13.2013

[I]Robots. text file is considered as a useful convention to prevent cooperating web robots and web crawlers from accessing all or part of a website or its content for which we don’t want to be crawled and indexed but publicly viewable. It is also employed by search engines to archive and categorize website and to generate a rule of no follow regarding some particular areas of our websites.. pls share your experience..[/I]

to post a comment
SEO

3 Comments(s) ↴

Copy linkTweet thisAlerts:
@kpkarthikNov 15.2013 — If we adding the particular and major URL in robots.txt file,Google does not crawl that particular URL of our website.That is the main role of robots.txt file.
Copy linkTweet thisAlerts:
@JessicacisNov 18.2013 — In robots.txt file we put those links which we want should not be crawled and indexed by the Google but we want to be viewable by the users.
Copy linkTweet thisAlerts:
@farhadankhiNov 18.2013 — If you want to create a website with responsive design and seo services then you can get many website design company on the net. But you have to ensure before creating your site to provide them this responsibilities. They help you to create robots.txt file and indexing your site on google.
×

Success!

Help @bhadriram spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.18,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...