/    Sign up×
Community /Pin to ProfileBookmark

Spider robots not allowed to spider page ?

Copy linkTweet thisAlerts:
Mar 15.2006

I use Meta Tag Analyzer:
[URL=http://www.widexl.com/remote/search-engines/metatag-analyzer.html]widexl.com[/URL]
to analyze the meta info of a website that I did.
I get the message:
[B]“Spider robots are not allowed to spider the web page.”[/B]
I get that for every page except the index page. On the index I have: [B]index,follow[/B] while on the rest of the pages I have [B]noindex,follow[/B]. If I reverse that to [B]follow,noindex [/B] would that make any difference, so that the robots can look at all the pages for meta info? Or, what other solution is there? Thanks.

-jms

to post a comment
SEO

8 Comments(s) ↴

Copy linkTweet thisAlerts:
@jamesx521authorMar 15.2006 — I changed [B]noindex,follow[/B] to [B]follow,noindex[/B], but there was no-change; I get the same message: [B]"Spider robots are not allowed to spider the web page."[/B]. However, I don't get that message on the [B]index[/B] page -- which is the only page listed to be [B]indexed[/B] though ?!
Copy linkTweet thisAlerts:
@johnevaMar 15.2006 — Remove the noindex or change it to index.

If that still dont work check to see if your using a robots.txt file.
Copy linkTweet thisAlerts:
@jamesx521authorMar 15.2006 — But I want only the [B]index[/B] page to be indexed, not the rest. I want [B]follow[/B] on the other pages, (but not index), so the robots follow the other pages and look for keywords -- to improve the ranking of the index page -- which is the only page I want indexed on a search engine. Have I learned it wrong?

I don't have a robots.txt file. I don't even know what it is.
Copy linkTweet thisAlerts:
@johnevaMar 15.2006 — Yep that is partly right what you have said, but if you have denied access to the page from spiders thats why it carnt check you pages. If you allow indexing of the page it will work then.

If the pages are not indexed you will lose out SEO wise anyways as one of the things search engines like is a large site.

You should only prevent spiders access to private files or files not needed to be indexed like scripts and pdf files stuff like that.

The robots.txt file is just a way of preventing spiders access to any files you wish to stop them crawling.
Copy linkTweet thisAlerts:
@jamesx521authorMar 15.2006 — So, if I use [B]index,follow[/B] on all pages what can I do to make sure that search engines index [B]www.test.com[/B] and [U]not[/U] [B]www.test.com/page2.html[/B]?
Copy linkTweet thisAlerts:
@jamesx521authorMar 15.2006 — ...they are not real pages by the way
Copy linkTweet thisAlerts:
@johnevaMar 15.2006 — A robots.txt file but it does just the same job and will stop you from being able to use the meta tag analyzer tool.

As I say it is advisabe not to stop search engine indexing all your pages anyways.

As long as your site is a good one then it is a good thing to have all your pages indexed. Why do you think there is such a thing a site maps for search engines?

They are used to aid the search engines to make sure they index every last one of your pages cos they do sometimes miss the odd one here and there, even more so with very large sites.
Copy linkTweet thisAlerts:
@KeveyMar 15.2006 — http://www.searchengineworld.com/robots/robots_tutorial.htm

A great tutorial on what you can do, and it addresses your specific questions. Keep in mind that many "bad bots" will do whatever they want on your site regardless of the robots.txt file.
×

Success!

Help @jamesx521 spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.8,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...