/    Sign up×
Community /Pin to ProfileBookmark

File does not exist:../public_html/robots.txt

Copy linkTweet thisAlerts:
Jul 31.2005

What is this file? and what can I do about it?
It keeps showing up in my error log.
I tried making a blank robots.txt file and placed it in my public_html folder, but I still get this “File does not exist:../public_html/robots.txt” error.

I’ve tried putting a meta tag in the head section:
<META NAME=”ROBOTS” CONTENT=”ALL”>

But Validator did not like that, and now getting errors indicating that something is trying to access folders that are not linked to my public pages.

Does this meta give robots permission to crawl all over pages not linked up to the public pages?

Is there a better meta tag for the robots to use? and not let them crawl into pages they shouldn’t.

thanks for any advice,
René

to post a comment
HTML

1 Comments(s) ↴

Copy linkTweet thisAlerts:
@FangJul 31.2005 — The robots.txt file is best placed in the root of your site. Not all robots look at this, so also use a meta tag. The example you gave would allow the robot to index and follow links on the page. It should validate for html, the validator has possibly picked-up on a previous error.

[URL=http://www.robotstxt.org/wc/faq.html]robot faq[/URL]

[URL=http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.4]W3C search engine recommendations[/URL]

If another site has linked to your site, then robots may index your site regardless of your efforts to keep them out.
×

Success!

Help @aRen__ spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.18,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...