/    Sign up×
Community /Pin to ProfileBookmark

How To Crawl The Web Without Having My Local Station On 24/7 ?

Folks,

I can always use a php crawler to crawl sites. Crawler can be installed on paid vps or my localhost (Xampp).
Now, providing I run the php crawler on my localhost, do I have to keep the Admin Section (on the vps host) open on my browser (client side) for the crawler to keep on crawling ? If I close down my web browser and go to sleep then will the php crawler stop crawling ?
Same question goes if I was running the php crawler on a paid vps.
I do not want to keep my pc on 24/7 regardless of whether the crawler is on my end or the paid host end. Understand ?
You will ask me which php code I will use. Well, I am searching right now. I will copy from tutorials. You have a fair idea what kind of crawlers they are built with php using DOM.

To begin with, let’s say I will use the crawler found on this following tutorial:

https://potentpages.com/web-crawler-development/tutorials/php/creating-a-simple-php-website-crawler
NOTE: It only downloads the first starting page’s title, description, h1, download time. Not the crawled pages’. I can amend the script code so it extracts the same for the crawled links.

The above link was just an example I am showing you what kind of php scripts I will use found on tutorials or in open source markets. So, to run such crawlers, do I have to keep my pc on with my web browser open ?
If so then what’s the solution ? I have no experience with CRON jobs.
Any workarounds ?

to post a comment
PHP

1 Comments(s)

Copy linkTweet thisAlerts:
@daveyerwinDec 21.2021 — well, you can run a crawler on your local machine

without any need for a browser

now, anything you can do on your local machine

you can also do on the vps(), you will need to learn

how to operate the vps with ssh https://www.hostinger.com/tutorials/ssh-tutorial-how-does-ssh-work

OR

you can install a desktop on the vps and operate it with a remote desktop client

https://www.faqforge.com/windows/windows-10/access-remote-desktop-windows-10/
×

Success!

Help @developer_web spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 BETA 6.2,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @meenaratha,
tipped: article
amount: 1000 SATS,

tipper: @meenaratha,
tipped: article
amount: 1000 SATS,

tipper: @AriseFacilitySolutions09,
tipped: article
amount: 1000 SATS,
)...