/    Sign up×
Community /Pin to ProfileBookmark

web crawlers and root directory

Copy linkTweet thisAlerts:
Nov 12.2010

Hi everyone,
Does google index pages that are in the root directory on the server but not linked to from any of the pages on the website? I work with another guy on a few sites and we have some test pages in the root directory (so we can both access them) that are basically duplicates of some of the other pages and I don’t want them crawled. Is this a problem or should I put them in a separate folder that is blocked by robots.txt?
Thanks!

to post a comment
SEO

2 Comments(s) ↴

Copy linkTweet thisAlerts:
@chris22Nov 12.2010 — Hi, if there are no links from other pages, there is no way for google bots to find them. It is only you who knows the URL to open them.
Copy linkTweet thisAlerts:
@noahqwDec 10.2010 — I'd just place a robot.txt in your root directory and list your pages in that, just to be on the safe side. However, I'm sure if it's only a few pages it wouldn't hurt and or affect your Google ranking.
×

Success!

Help @Webitseo spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.19,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...