I’ve discovered that msnbot has crawled my site. I wanted to know the results, I don’t understand it too well, is the “…404.. ” an error code?:
[QUOTE]
65.54.188.66 – – [25/Aug/2004:13:03:52 -0400] “GET /robots.txt HTTP/1.0” 404 640 “-” “msnbot/0.11 (+[url]http://search.msn.com/msnbot.htm
65.54.188.66 – – [25/Aug/2004:13:03:53 -0400] “GET /javascript/inputwrite.html HTTP/1.0” 404 640 “-” “msnbot/0.11 (+[url]http://search.msn.com/msnbot.htm
The IP I understand, the time stamp I understand, what does “…
“GET /javascript/inputwrite.html HTTP/1.0” 404 640 “-” “msnbot/0.11
…” mean?
and obviously the (+[url]http://search.msn.com/msnbot.htm
In any case, this means that a crawler is trying to index my site?
Also, do I need to have the robots.txt file?