/    Sign up×
Community /Pin to ProfileBookmark

Robots file error, how to correct this one?

Copy linkTweet thisAlerts:
Aug 26.2015

Hi Everyone,

I submitted the robots.txt file on the server, its shows the error [B]“A description for this result is not available because of this site’s robots.txt – learn more.”[/B] Then, i updated the new robots.txt file on last week, still google shows the same error. Link:- [URL=”https://onlinepiping.com/robots.txt”]https://onlinepiping.com/robots.txt[/URL]

What is the problem with my robots.txt file?
I don’t know why it still displaying error. Can anyone able to give the solution for this problem?

to post a comment
SEO

7 Comments(s) ↴

Copy linkTweet thisAlerts:
@richardstevensAug 28.2015 — What is a robots.txt fetch error?

Before Googlebot crawls your site, it accesses your robots.txt file to determine if your site is blocking Google from crawling any pages or URLs. If your robots.txt file exists but is unreachable (in other words, if it doesn’t return a 200 or 404 HTTP status code), we’ll postpone our crawl rather than risk crawling URLs that you do not want crawled. When this happens, Googlebot will return to your site and crawl it as soon as we can successfully access your robots.txt file. More information about the robots exclusion protocol.

How deal with robots.txt file errors

You don't always need a robots.txt file.

You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file—not even an empty one. If you don’t have a robots.txt file, your server will return a 404 when Googlebot requests it, and we will continue to crawl your site. No problem.

Make sure your robots.txt file can be accessed by Google.

It's possible that your server returned a 5xx (unreachable) error when we tried to retrieve your robots.txt file. Check that your hosting provider is not blocking Googlebot. If you have a firewall, make sure that its configuration is not blocking Google.
Copy linkTweet thisAlerts:
@Kevin2Aug 28.2015 — User-agent: *
Disallow: /mailer.aspx
Disallow: /abdul/
Allow: /

The official robots.txt syntax does not have the [b]Allow:[/b] field. See the bottom of [B]this page[/B]. Google does support it however, but only under certain circumstances. From https://support.google.com/webmasters/answer/6062596?hl=en :

[...] to give Google access to [b][COLOR="#FF0000"]a particular URL that is a child directory in a disallowed parent directory[/COLOR][/b], then you can use a third key word, Allow.[/quote]

Get rid of the [b]Allow: /[/b] line.
Copy linkTweet thisAlerts:
@Fadia_SheetalAug 31.2015 — Robots file is for stopping the search engine to crawling the website. It is not showing the error. If you find this one than may be its another problem.
Copy linkTweet thisAlerts:
@kghoshalSep 10.2015 — Robots file is for stopping the search engine to crawling the website. It is not showing the error. If you find this one than may be its another problem.
Copy linkTweet thisAlerts:
@ramansharmaJun 04.2018 — I have the same issue with my website. Can anyone help me for the same?


Google couldn't crawl your site because we were unable to access your site's robots.txt file.

Robots txt is unreachable. Sowing this error: Your server returned a 5xx (unreachable) error when we tried to retrieve your robots.txt file.

My website robots.txt file is: https://onlinetvhelp.com/robots.txt

How to resolve this?
Copy linkTweet thisAlerts:
@rootJun 04.2018 — If you have a problem, please ask a question in your own thread please and not resurrect old post from 4 years ago..! Thanks.

Copy linkTweet thisAlerts:
@rootJun 04.2018 — {"locked":true}
×

Success!

Help @JenniferJeny spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.17,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...