/    Sign up×
Community /Pin to ProfileBookmark

robots.txt logs: results?

I’ve discovered that msnbot has crawled my site. I wanted to know the results, I don’t understand it too well, is the “…404.. ” an error code?:

[QUOTE]

65.54.188.66 – – [25/Aug/2004:13:03:52 -0400] “GET /robots.txt HTTP/1.0” 404 640 “-” “msnbot/0.11 (+[url]http://search.msn.com/msnbot.htm[/url])”
65.54.188.66 – – [25/Aug/2004:13:03:53 -0400] “GET /javascript/inputwrite.html HTTP/1.0” 404 640 “-” “msnbot/0.11 (+[url]http://search.msn.com/msnbot.htm[/url])”

[/QUOTE]

The IP I understand, the time stamp I understand, what does “…

“GET /javascript/inputwrite.html HTTP/1.0” 404 640 “-” “msnbot/0.11

…” mean?
and obviously the (+[url]http://search.msn.com/msnbot.htm[/url]) is the crawler.
In any case, this means that a crawler is trying to index my site?
Also, do I need to have the robots.txt file?

to post a comment
SEO

3 Comments(s)

Copy linkTweet thisAlerts:
@hdtvtechnoSep 13.2004 — yes u need to have the robots.txt


http://www.hdtvtechno.com
Copy linkTweet thisAlerts:
@hdtvtechnoSep 22.2004 — it looks like the msnbot got a 404 error when reached ur website.
Copy linkTweet thisAlerts:
@Robert_WellockSep 29.2004 — Most likely a JavaScript link it couldn't follow hence 404.
×

Success!

Help @anothen spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 BETA 5.21,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,

tipper: @Yussuf4331,
tipped: article
amount: 1000 SATS,

tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,
)...