/    Sign up×
Community /Pin to ProfileBookmark

Why are Search Engines not crawling to my site?

Copy linkTweet thisAlerts:
Apr 01.2005

I have a site map, all my links work. but for some reason the Search engines don’t seem to crawl past my first page.

Here is my search logs…

4 different robots* Hits Bandwidth Last visit
[B]Inktomi Slurp [/B] 7+2 23.68 KB 27 Mar 2005 – 12:51
[B]Unknown robot [/B] (identified by hit on ‘robots.txt’) 0+6 732 Bytes 30 Mar 2005 – 07:00
[B]Googlebot[/B] 2 5.79 KB 02 Mar 2005 – 20:16
[B]AskJeeves [/B] 1+1 3.40 KB 30 Mar 2005 – 23:59

[URL=http://clarus-solutions.com]www.clarus-solutions.com[/URL]

My questions are….
1. Is php affecting the crawl? If so, what can I do to get the robots to crawl php.
2. I have a robots.txt file, is there anything else i should have
3. Even though they have crawled my first page, I still am not showing up on any search results. Why?
4. Is there any other method I should consider to move up the ranks? (besides paid advertising)

Thanks!

Robert Banta
[url]http://clarus-solutions.com[/url]
[email][email protected][/email]

to post a comment
SEO

9 Comments(s) ↴

Copy linkTweet thisAlerts:
@FretsApr 01.2005 — php won't affect the bots

Your meta tag is way off

And it resides above the html tag

meta tags should be placed in the head

<i>
</i>&lt;META
content="web design, graphics design, IT, instruction, Microsoft, hosting, freelance, sql, html, php, asp, database, mysql, banner, flash, xml, wml, smart phone, pda, wireless, javascript, michigan, troy, sterling heights, printer, publisher, designer, 3D, animation, web page design, proofer, data storage, video storage, hard disk, mouse, graphic tablet, firewire"
name=keywords&gt;
&lt;html&gt;
&lt;head&gt;


Keywords should be limited to ten words

Those words must be inside your <body>

the proper meta syntax is
<i>
</i>&lt;meta name="keywords" content="web design, graphics design, IT, instruction, Microsoft, hosting, freelance, sql, html, php, asp, database, mysql, banner, flash, xml, wml, smart phone, pda, wireless, javascript, michigan, troy, sterling heights, printer, publisher, designer, 3D, animation, web page design, proofer, data storage, video storage, hard disk, mouse, graphic tablet, firewire" /&gt;


You should have a robots meta tag such as
&lt;meta name="robots" content="index,follow" /&gt;

Oddly googlebot likes link outs If you have a page on your site

with listings that link out to other sites It actually boosts your ratings.
Copy linkTweet thisAlerts:
@clarussolutionsauthorApr 01.2005 — Thanks, I 'll make the changes! No idea why the bot is not crawling to my site map tho?
Copy linkTweet thisAlerts:
@andyshepApr 01.2005 — <meta name="robots" content="index,follow" />[/QUOTE]

Most people do not need the meta robots tag on their web pages. Having it on a web page does not help the page get indexed in any way. In fact, the tag is mainly used as a way to keep your web pages out of the search engines. Nor does it even work well for this, since most major search engines do not support it.

It tells search engines to index the page and follow links to inside pages, so that they can be index. But search engines will do this even without the code, to the same degree that they index any web site. So don't waste your time with it.


Oddly googlebot likes link outs If you have a page on your site

with listings that link out to other sites It actually boosts your ratings.[/QUOTE]

googlebot likes links in and out of a site only if the sites linking to are relevant

to the content of your site.

i.e. if its a web design site linking to web design related pages helps ranking

andy
Copy linkTweet thisAlerts:
@solomonApr 02.2005 — You might want to look at using a [URL=http://www.searchengineworld.com/robots/robots_tutorial.htm]robots.txt[/URL] file
Copy linkTweet thisAlerts:
@_lt_Eddie_gt_Apr 03.2005 — Link check ([url=http://www.google.co.uk/search?complete=1&hl=en&q=link%3Aclarus-solutions.com&btnG=Google+Search&meta=]link:clarus-solutions.com[/url]) - 0

Site check ([url=http://www.google.co.uk/search?complete=1&hl=en&q=site%3Aclarus-solutions.com&btnG=Search&meta=]site:clarus-solutions.com[/url]) - 1 (Home Page)

Character check ([url=http://www.google.co.uk/search?complete=1&hl=en&q=%22%2Bclarus-solutions.com%22&btnG=Search&meta=]"+clarus-solutions.com"[/url]) - 8

Google has a nasty habit of only scanning the first page of a new domain it finds. Make sure you get some inbounds links and just to make sure Google finds them, run the address of the page your links reside into [url=http://www.google.com/addurl/?continue=/addurl]Googles AddURL[/url] page. Unfortunately, you have to give Google a reason to deep crawl you site. A check also found Yahoo has done the same, home page only. MSN has even found you yet. I'd work on the links.
Copy linkTweet thisAlerts:
@leocharreApr 16.2005 — A remote possibility is that your pages are banned??? Try looking that up by going to google and typing this in "yoursite.com" no http, etc..

Google does NOT only scan the first page of a domain it finds.

If that were true, google would have 0 catalogue entries.. google does not discriminate pages by domain.

"websites" are determined by linking structure, not by ip/url domain etc.
--


Are you making sure that your links make sense?

for example this is a worthy link

[B]<a href="/about/index.html" title="about me">About Me</a>[/B] Obviously index.html should have appropriate meta and titles etc.. otherwise google has nothing to record.

a crappy link is

[B]<a href="/about/inde.html">Read more</a> [/B]and then there's nothing in that page for title, etc.
Copy linkTweet thisAlerts:
@_lt_Eddie_gt_Apr 16.2005 — Google does NOT only scan the first page of a domain it finds.[/QUOTE]So the fact that only the first page of his site has been indexed doesn't point to that fact? Being banned does not mean your home page appears in Google. Anchor text and title attributes do not contribute to the decision Google makes when indexing a new site. They only assist in the ranking process but what good is that if the majority of your pages aren't listed?

Work on backlinks instead of tweaking internal links on the page as you'll get more immediate benefit. Statistical analysis will force you to amend page copy to suit your market anyway.
Copy linkTweet thisAlerts:
@clarussolutionsauthorApr 17.2005 — Statistical analysis will force you to amend page copy to suit your market anyway.[/QUOTE]


What do you mean?
Copy linkTweet thisAlerts:
@_lt_Eddie_gt_Apr 17.2005 — What do you mean?[/QUOTE]
Keyword optimisation is a market driven task and will always evolve. You need stats to inform you which keywords people are typing into which search engines so you can amend the primary keyphrases accordingly.
×

Success!

Help @clarussolutions spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 — BETA 4.24,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,

tipper: @Samric24,
tipped: article
amount: 1000 SATS,
)...