/    Sign up×
Community /Pin to ProfileBookmark

SiteMaps – How Many Levels Deep They Usually Go ?

Howdy,

How many levels deep can a site go like this to list their html files ?
I need to know this to program my crawler how many levels it should check before giving up. Do not want to be going in an endless loop and get my crawler get into a trap.

You know what ?
I really thought Sitemap Xml files will list all .html and .hml and .shtm and .shtml files. All pages of the website.
But now I see, Sitemap xml files also list other xml files. Example:
https://www.rocktherankings.com/sitemap_index.xml

Now, I got to program my web crawler to go one level deep to find the site links (html files) on that xml file.
I got to ask you: Does this happen more than one level deep ?
I mean does it do this ….
I go to a Sitemap xml file.
I see further xml files. I clickover to an xml file. Thus go one level deep.
I see more xml files listed. I clickover to an xml file. Thus go two level deep.
And so on.

REALLY: How many levels deep can a site go like this to list their html files ?

to post a comment
PHP

1 Comments(s)

Copy linkTweet thisAlerts:
@novice2022authorOct 15.2022 — @nogdog

I learnt the answer: Infinite levels.

Kindly, close this thread.
×

Success!

Help @novice2022 spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 BETA 6.18,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @nearjob,
tipped: article
amount: 1000 SATS,

tipper: @meenaratha,
tipped: article
amount: 1000 SATS,

tipper: @meenaratha,
tipped: article
amount: 1000 SATS,
)...