/    Sign up×
Community /Pin to ProfileBookmark

Benefits of Excluding robots

Copy linkTweet thisAlerts:
Apr 22.2013

Excluding robots from certain sections of the site can benefit you :

  • 1.

    You can prevent search engine crawlers from eating up unnecessary bandwidth of your site, For some hosting bandwidth is costly and it is important bandwidth is saved by excluding crawlers from certain sections.

  • 2.

    Excluding robots from comments section if you run a blog is very good, It prevents robots from seeing or giving credit to any unnecessary links.

  • 3.

    Excluding robots also helps in preventing sensitive information of your site not getting exposed in search results

    • I just know things above, so can you list more?

    to post a comment
    SEO

    1 Comments(s) ↴

    Copy linkTweet thisAlerts:
    @vincewicksApr 22.2013 — It can be used for the files or pages which owner does not want bots to crawl.

    it helps in hiding urls which were showing in crawl errors, we can put those also in robot.txt
    ×

    Success!

    Help @Steve_Smith spread the word by sharing this article on Twitter...

    Tweet This
    Sign in
    Forgot password?
    Sign in with TwitchSign in with GithubCreate Account
    about: ({
    version: 0.1.9 — BETA 5.18,
    whats_new: community page,
    up_next: more Davinci•003 tasks,
    coming_soon: events calendar,
    social: @webDeveloperHQ
    });

    legal: ({
    terms: of use,
    privacy: policy
    });
    changelog: (
    version: 0.1.9,
    notes: added community page

    version: 0.1.8,
    notes: added Davinci•003

    version: 0.1.7,
    notes: upvote answers to bounties

    version: 0.1.6,
    notes: article editor refresh
    )...
    recent_tips: (
    tipper: @AriseFacilitySolutions09,
    tipped: article
    amount: 1000 SATS,

    tipper: @Yussuf4331,
    tipped: article
    amount: 1000 SATS,

    tipper: @darkwebsites540,
    tipped: article
    amount: 10 SATS,
    )...