I want to scrape every URL on a specific website. I’ve tried Xenu’s Link Sleuth but that doesn’t give me all of them. The point of this task being, I want to scrape all of the downloadable text files from this site.
@RoopatgDec 15.2014 — #Hi, I did not get your question clearly, could you explain elaborately. Xenu's Link Sleuth is a FREE download,main focus is all about finding broken links on your site. There are other websites are there to find the broken links in your website. If you want to find errors list URL wise, you will get through Google webmaster tools. If you want to know total URLs in your website, site:yourodomain.com, also create sitemap using tool, you will get all URLs in a single file.
The custom scraper is a tool by which we can extract all the data from any url. You can extract information from all the rendered source including anything not rendered in the browser.