/    Sign up×
Community /Pin to ProfileBookmark

General advice for dealing with long running processes

Copy linkTweet thisAlerts:
Jan 16.2014

Hi there. I’m getting ready to create a website for work that allows a user to start, monitor, and ultimately get results from a long running calculation.

The calculation will run long enough that a page would normally time out waiting for the server to respond.

The calculation code will be in .NET and most likely we will run the calculation on a Linux machine, possibly the web server itself (not sure about that yet).

What I need is some good ideas on handling things like:

  • 1.

    How to get progress updates to the user, while the calc is running. The calculation code will be able to emit progress messages (such as 25% done), but I’m not sure how to get these to the user in the browser. I could probably require the user to press F5 every so often, to get updated results from the server, but this doesn’t seem very friendly

  • 2.

    How to inform the user the calculation is done and get results to him. I guess this is related to the first item; if I can figure out how to get progress updates to the user, then getting him a “job done” message, and the results themselves, may use the same solution.

  • 3.

    What’s a good architecture to handle the i/o between the web server app and the calculation that’s running? The web server app will be Python/Pylons. In addition to starting a job, and getting updates and results, the user may cancel a job that has been started.

    • Any ideas on any of these topics would be greatly appreciated. I’m pretty sure I could create my own solutions from scratch, but if there are best-practices to follow, I’d rather do that!

    Thanks for any ideas!
    Michael

    to post a comment
    Full-stack Developer

    2 Comments(s) ↴

    Copy linkTweet thisAlerts:
    @NogDogJan 17.2014 — First thought that came to my mind (so it's not necessarily the best one)

  • 1. Make use of JavaScript AJAX calls to launch the process (so you never have to actually leave the web page), then to also periodically send requests to check on the status and get the results when it's done.


  • 2. The actual calculation process could be kicked off as a background process on the server, using whatever functions .NET provides to do so. I'm a PHP guy, so I'd probably do something like:
    [code=php]
    $args = getArrayOfArgsFromGetOrPost(); // or whatever
    $statusFile = uniqid('', true).".txt";
    exec('nohup /path/to/script.sh '.implode(' ', $args).' > '.$statusFile.' &');
    [/code]

  • 3. I'd stick $statusFile in the session data, or pass it back to the AJAX caller so that it can pass it back when making its status checks.


  • 4. Those status calls would call a server-side script that would check the contents of that file for a completion indicator and results data (or an indicator where to get the results in the database, perhaps?), or error conditions if it fails, and perhaps status messages?
    • Copy linkTweet thisAlerts:
    @John_marksJan 25.2014 — I agree ''The actual calculation process could be kicked off as a background process on the server, using whatever functions''.....?
    ×

    Success!

    Help @michaelveloz spread the word by sharing this article on Twitter...

    Tweet This
    Sign in
    Forgot password?
    Sign in with TwitchSign in with GithubCreate Account
    about: ({
    version: 0.1.9 — BETA 5.18,
    whats_new: community page,
    up_next: more Davinci•003 tasks,
    coming_soon: events calendar,
    social: @webDeveloperHQ
    });

    legal: ({
    terms: of use,
    privacy: policy
    });
    changelog: (
    version: 0.1.9,
    notes: added community page

    version: 0.1.8,
    notes: added Davinci•003

    version: 0.1.7,
    notes: upvote answers to bounties

    version: 0.1.6,
    notes: article editor refresh
    )...
    recent_tips: (
    tipper: @AriseFacilitySolutions09,
    tipped: article
    amount: 1000 SATS,

    tipper: @Yussuf4331,
    tipped: article
    amount: 1000 SATS,

    tipper: @darkwebsites540,
    tipped: article
    amount: 10 SATS,
    )...