/    Sign up×
Community /Pin to ProfileBookmark

Hellp with a Pattern in Robots.txt

Copy linkTweet thisAlerts:
Apr 26.2014

Hi,
I am using a CMS that is generating all sorts of urls related to same page. So I want to block all variations of the the url but sill let search engines crawl the url without parameters, you think this will work?

Disallow: /member/profile_*.html?*

As the pattern contains “?” I believe /member/profile_*.html will still be crawled

I want /member/profile_*.html to be crawled but none of the other variations with the parameters.

Please advise, thank you!

to post a comment
SEO

2 Comments(s) ↴

Copy linkTweet thisAlerts:
@Kevin2Apr 26.2014 — From my reading of info at robotstxt.org, it seems that while you can wildcard robots with a * you cannot do the same with files, nor can you match a pattern or parameters. You can disallow entire folders and specifically named files.

Specifically:
you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".[/quote]

Read more at: http://www.robotstxt.org/robotstxt.html
Copy linkTweet thisAlerts:
@fozailauthorApr 26.2014 — From my reading of info at robotstxt.org, it seems that while you can wildcard robots with a * you cannot do the same with files, nor can you match a pattern or parameters. You can disallow entire folders and specifically named files.

Specifically:


Read more at: http://www.robotstxt.org/robotstxt.html[/QUOTE]

Thanks for your reply but the above url does explain much about file paths, read this post from Google, especially the section "URL matching based on path values", it explains the use "*" and "$" to match file paths.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
×

Success!

Help @fozail spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 5.6,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...