The Dangers of Implementing Recurring Background Tasks In ASP.NET

asp.net, code 0 comments suggest edit

I like to live life on the wild side. No, I don’t base jump off of buildings or invest in speculative tranches made up of junk stock derivatives. What I do is attempt to run recurring background tasks within an ASP.NET application.

110121-M-2339L-074 Writing code is totally just like this - Photo by DVIDSHUBCC BY 2.0 

But before I do anything wild with ASP.NET, I always talk to my colleague, Levi (sadly, no blog). As a developer on the internals of ASP.NET, he knows a huge amount about it, especially the potential pitfalls. He’s also quite the security guru. As you read this sentence, he just guessed your passwords. All of them.

When he got wind of my plan, he let me know it was evil, unsupported by ASP.NET and just might kill a cat. Good thing I’m a dog person. I persisted in my foolhardiness and suggested maybe it’s not evil, just risky. If so, how can I do it as safely as possible? What are the risks?

There are three main risks, one of which I’ll focus on in this blog post.

  1. An unhandled exception in a thread not associated with a request will take down the process. This occurs even if you have a handler setup via the Application_Error method. I’ll try and explain why in a follow-up blog post, but this is easy to deal with.
  2. If you run your site in a Web Farm, you could end up with multiple instances of your app that all attempt to run the same task at the same time. A little more challenging to deal with than the first item, but still not too hard. One typical approach is to use a resource common to all the servers, such as the database, as a synchronization mechanism to coordinate tasks.
  3. The AppDomain your site runs in can go down for a number of reasons and take down your background task with it. This could corrupt data if it happens in the middle of your code execution.

It’s this last risk that is the focus of this blog post.

Bye Bye App Domain

There are several things that can cause ASP.NET to tear down your AppDomain.

  • When you modify web.config, ASP.NET will recycle the AppDomain, though the w3wp.exe process (the IIS web server process) stays alive.
  • IIS will itself recycle the entire w3wp.exe process every 29 hours. It’ll just outright put a cap in the w3wp.exe process and bring down all of the app domains with it.
  • In a shared hosting environment, many web servers are configured to tear down the application pools after some period of inactivity. For example, if there are no requests to the application within a 20 minute period, it may take down the app domain.

If any of these happen in the middle of your code execution, your application/data could be left in a pretty bad state as it’s shut down without warning.

So why isn’t this a problem for your typical per request ASP.NET code? When ASP.NET tears down the AppDomain, it will attempt to flush the existing requests and give them time to complete before it takes down the App Domain. ASP.NET and IIS are considerate to code that they know is running,such as code that runs as part of a request.

Problem is, ASP.NET doesn’t know about work done on a background thread spawned using a timer or similar mechanism. It only knows about work associated with a request.

So tell ASP.NET, “Hey, I’m working here!”

The good news is there’s an easy way to tell ASP.NET about the work you’re doing! In the System.Web.Hosting namespace, there’s an important class, HostingEnvironment. According to the MSDN docs, this class…

Provides application-management functions and application services to a managed application within its application domain

This class has an important static method, RegisterObject. The MSDN description here isn’t super helpful.

Places an object in the list of registered objects for the application.

For us, what this means is that the RegisterObject method tells ASP.NET that, “Hey! Pay attention to this code here!” Important! This method requires full trust!

This method takes in a single object that implements the IRegisteredObject interface. That interface has a single method:

public interface IRegisteredObject
{
    void Stop(bool immediate);
}

When ASP.NET tears down the AppDomain, it will first attempt to call Stop method on all registered objects.

In most cases, it’ll call this method twice, once with immediate set to false. This gives your code a bit of time to finish what it is doing. ASP.NET gives all instances of IRegisteredObject a total of 30 seconds to complete their work, not 30 seconds each. After that time span, if there are any registered objects left, it will call them again with immediate set to true. This lets you know it means business and you really need to finish up pronto! I modeled my parenting technique after this method when trying to get my kids ready for school.

When ASP.NET calls into this method, your code needs to prevent this method from returning until your work is done. Levi showed me one easy way to do this by simply using a lock. Once the work is done, the code needs to unregister the object.

For example, here’s a simple generic implementation of IRegisteredObject. In this implementation, I simply ignored the immediate flag and try to prevent the method from returning until the work is done. The intent here is I won’t pass in any work that’ll take too long. Hopefully.

public class JobHost : IRegisteredObject
{
    private readonly object _lock = new object();
    private bool _shuttingDown;

    public JobHost()
    {
        HostingEnvironment.RegisterObject(this);
    }

    public void Stop(bool immediate)
    {
        lock (_lock)
        {
            _shuttingDown = true;
        }
        HostingEnvironment.UnregisterObject(this); 
    }

    public void DoWork(Action work)
    {
        lock (_lock)
        {
            if (_shuttingDown)
            {
                return;
            }
            work();
        }
    }
}

I wanted to get the simplest thing possible working. Note, that when ASP.NET is about to shut down the AppDomain, it will attempt to call the Stop method. That method will try to acquire a lock on the _lock instance. The DoWork method also acquires that same lock. That way, when the DoWork method is doing the work you give it (passed in as a lambda) the Stop method has to wait until the work is done before it can acquire the lock. Nifty.

Later on, I plan to make this more sophisticated by taking advantage of using a Task to represent the work rather than an Action. This would allow me to take advantage of task cancellation instead of the brute force approach with locks.

With this class in place, you can create a timer on Application_Start (I generally use WebActivator to register code that runs on app start) and when it elapses, you call into the DoWork method here. Remember, the timer must be referenced or it could be garbage collected.

Here’s a small example of this:

using System;
using System.Threading;
using WebBackgrounder;

[assembly: WebActivator.PreApplicationStartMethod(
  typeof(SampleAspNetTimer), "Start")]

public static class SampleAspNetTimer
{
    private static readonly Timer _timer = new Timer(OnTimerElapsed);
    private static readonly JobHost _jobHost = new JobHost();

    public static void Start()
    {
        _timer.Change(TimeSpan.Zero, TimeSpan.FromMilliseconds(1000));
    }

    private static void OnTimerElapsed(object sender)
    {
        _jobHost.DoWork(() => { /* What is it that you do around here */ });
    }
}

Recommendation

This technique can make your background tasks within ASP.NET much more robust. There’s still a chance of problems occurring though. Sometimes, the AppDomain goes down in a more abrupt manner. For example, you might have a blue screen, someone might trip on the plug, or a hard-drive might fail. These catastrophic failures can take down your app in such a way that leaves data in a bad state. But hopefully, these situations occur much less frequently than an AppDomain shutdown.

Many of you might be scratching your head thinking it seems weird to use a web server to perform recurring background tasks. That’s not really what a web server is for. You’re absolutely right. My recommendation is to do one of the following instead:

  • Write a simple console app and schedule it using Windows task schedule.
  • Write a Windows Service to manage your recurring tasks.
  • Use an Azure worker or something similar.

Given that those are my recommendations, why am I still working on a system for scheduling recurring tasks within ASP.NET that handles web farms and AppDomain shutdowns I call WebBackgrounder (NuGet package coming later)?

I mean, besides the fact that I’m thick-headed? Well, for two reasons.

The first is to make development easier. When you get latest from our source code, I just want everything to work. I don’t want you to have to set up a scheduled task, or an Azure worker, or a Windows server on your development box. A development environment can tolerate the issues I described.

The second reason is for simplicity. If you’re ok with the limitations I mentioned, this approach has one less moving part to worry about when setting up a website. There’s no need to configure an external recurring task. It just works.

But mostly, it’s because I like to live life on the edge.

Technorati Tags: asp.net,appdomain,web farms,background,threads,tasks

Found a typo or error? Suggest an edit! If accepted, your contribution is listed automatically here.

Comments

avatar

84 responses

  1. Avatar for Marcus McConnell
    Marcus McConnell October 16th, 2011

    I just finished up a similar background scheduler but I wasn't sure how to handle app domain shutdowns. Thanks for the code. Very helpful!
    It would be great if we had a way for ASP.NET apps to register scheduled tasks outside of the web app domain but sandboxed to only access their parent application. Kind of a safe way to allow a medium trust web app to add a windows scheduled task. Just brainstorming...

  2. Avatar for David Crowell
    David Crowell October 16th, 2011

    A proper hosting environment should have a way for you to run scheduled tasks... Dedicated or virtual servers come to mind.
    Shared hosting often has a feature to call a specific url at a specified interval, therefore you can do your background task within a ASP.NET request and avoid all the silliness. :)
    I don't like my code to live on the edge. I'd rather do an unsupported 100-kilometer bicycle race on gravel roads - doing that next month.
    Having your clever ASP.NET hack cause an outage while on said bike race (and not being around to support it) would be bad. :)

  3. Avatar for Peter
    Peter October 16th, 2011

    Had to work on something similar, but ended up using "low" level threads and managing the queue manually. The main reason, from what I remember, was to be able to use Monitor.Pulse and Monitor.Wait to allow the threads to sleep and for throttling.
    Too bad RegisterObject requires full trust. A workaround is to ping yourself every so often to make sure the pool doesn't shut down.

  4. Avatar for haacked
    haacked October 16th, 2011

    This is a scenario I'd love to see better support for within ASP.NET in the future. Some of us have talked about it, but who knows where it'll end up. :)

  5. Avatar for Siler
    Siler October 16th, 2011

    Look at nopCommerce implementation of background tasks

  6. Avatar for Daniel Marbach
    Daniel Marbach October 16th, 2011

    Hy
    Why don't you use quartz.net? You could easily integrate quartz.net in your application, you then have the power to schedule task in cron like manner or even do more powerful stuff. You would also gain task persistency etc. Quartz.net is really easy to integrate into any IOC container. Happy scheduling
    Daniel

  7. Avatar for felix
    felix October 16th, 2011

    What about wcf-wf hosted in IIS?

  8. Avatar for Mike Chaliy
    Mike Chaliy October 16th, 2011

    >> Why don't you use quartz.net?
    Because it looks like guy dead few hundreds hears ago...
    That's funny, but few weeks ago I also started working on scheduler(https://github.com/chaliy/inline_scheduler) for asp.net :).

  9. Avatar for Saeed Neamati
    Saeed Neamati October 16th, 2011

    I was doing recurring background tasks in 2 applications (running another thread and executing a method every x minutes), and I had no crash till now at all.
    Though, seems that I was a lucky guy. I think a better, easier approach is to let a windows server (or another single ASP.NET application) run the recurring task. This is easier to me.
    Thanks.

  10. Avatar for kamranayub
    kamranayub October 16th, 2011

    Currently this is what I do, but without the fancy app domain handling. I do exactly what David Crowell said, my hosting provider lets me schedule a "hit a URL and report back" task and I pass in a very long token (yah yah, security by obscurity) to help prevent people from just hitting the URL directly. Unfortunately, their username/password method just won't work with all the ways I tried so it has to be open for now. It works pretty good! I get a nightly email with the output of my tasks so I can see quickly if it failed or not.
    Maybe if there was a cheap way to do start a background job (i.e. $1/month or something), I'd do that. My task runs between 20s-1min depending on the response time of the 3rd party service I use.

  11. Avatar for ms440
    ms440 October 16th, 2011

    Hi Phil,
    I’ve also been working on the very similar problem – I need a way to execute some code at pre-scheduled future time for MVC app without having an additional service running.
    I’m using cache expiration event with absolute expiration time and the fact that IIS7.5 on x64 does not wait for the first request but is firing Application_Start as soon as the thread pool is recycled or the server restarts.
    In Application_Start I’m checking the database to find out the next absolute expiration time. The task itself is executed in the cache expiration event handler inside database transaction, and it also sets up the cache expiration for the next cycle.
    This method has certain limitations (IIS7.5, x64, no farm) but it handles the major problem (web.config changed, server went down, etc.) well.
    My question: am I missing some use case that makes my approach “dangerous” also?

  12. Avatar for rekna
    rekna October 16th, 2011

    What about running Windows Workflow from an ASP.NET application ? Is it susceptible to these dangers as well ?

  13. Avatar for Code Dog
    Code Dog October 17th, 2011

    Some lower priced hosting setups do not allow you to set up scheduled tasks for windows services so this is a handy alternative.

  14. Avatar for Daniel Marbach
    Daniel Marbach October 17th, 2011

    @mike
    Actually that's not true. 2.0 beta has been released 2 weeks ago. The project is alive and under futher development...
    Daniel

  15. Avatar for anon
    anon October 17th, 2011

    What about async controllers in mvc?

  16. Avatar for Jeff Putz
    Jeff Putz October 17th, 2011

    I have several background tasks running in POP Forums for MVC. The motivation is simply that it requires no expertise or access to install them, which is fairly important for the mass of people who run in shared environments and such. Maybe it's not ideal, but it "generally" works.
    There is the issue that if no one pings the app, it may shut down. That might be true, but if no one is using the app, it won't need to reindex a thread for searching, send e-mail notifications or update the session table. :)

  17. Avatar for haacked
    haacked October 17th, 2011

    @Jeff that's exactly my feeling. :)

  18. Avatar for Wyatt Barnett
    Wyatt Barnett October 17th, 2011

    Is the 29 hour recycle hard coded? By default app pools will recycle every 1740 minutes, but that is easily reconfigurable. I know we set it to never unless we've got a reason not to.

  19. Avatar for Rob Kent
    Rob Kent October 17th, 2011

    I've used a background thread in the past and am about to do it again :) purely because I cannot install a Windows service on my ASP hosting site.
    I might just link the process-check to normal user activity, because if nobody is using the site, why should I care that the scheduler has not run?
    Thanks for the good tips for making it more resilient though.

  20. Avatar for Khalid Abuhakmeh
    Khalid Abuhakmeh October 17th, 2011

    So this might sound foolish, but why doesn't the ASP.Net team just develop a feature in web apps that allows people to access some sort of safe scheduler? Seems like everyone wants to do it, and there is good rationale which you make yourself above.
    Not sure how they would do it, but I'm sure there are a lot of smart people at Microsoft. Maybe a worker process that starts when the app pool starts and sits outside of ASP.Net and is always running unless explicitly stopped by your app or IIS Manager?
    It would be a killer feature for the next ASP.Net release. Start working your political charm Haack, I believe you can do it.

  21. Avatar for Matthew Wills
    Matthew Wills October 17th, 2011

    Would it be reasonable to assume that this problem could also impact cache usage with CacheItemUpdateCallback ( msdn.microsoft.com/...)? I would presume that the cache refresh is being done on a background thread...

  22. Avatar for Matt Wrock
    Matt Wrock October 17th, 2011

    Awesome info regarding HostingEnvironment.RegisterObject. I've been using background threads in asp.net for years and this is the first I have heard of this. This code will definitely be making its way on to RequestReduce soon!

  23. Avatar for Chris Moschini
    Chris Moschini October 19th, 2011

    Since you work at Microsoft... can you shake the appropriate trees to get RegisterObject() documentation updated and fixed? There are so many places in .Net documentation where you're cruising through things you're CERTAIN are important, and you get useless method descriptions like:
    AddSubTask: Adds a subtask.
    Or as a more legit example: msdn.microsoft.com/...
    "Gets or sets the URL pattern for the route." Great, what do those patterns look like? How about a list of supported syntax, or a link to one, like one would have in a Regex entry? Nope. How about an example of how to call this that's not from pre-beta days? Also no.

  24. Avatar for DmitryTheA
    DmitryTheA October 20th, 2011

    wow nice thing is that HostingEnvironment.RegisterObject.. but guys you're crazy =) why not just to run timer on windows service..

  25. Avatar for Matt Wrock
    Matt Wrock October 21st, 2011

    Dmitry,
    While there is certainly a place for Windows Services, they are often overkill for alot of scenarios. For example, my OSS project, RequestReducehas a background thread that merges/minifies css and js as well as sprites and optimizes background images. This has to be done in the background in a web app obviously and I want the adoption/migration story to be as painless and friction free as possible. I want adopters to drop my dll in their bin (preferably via nuget) and be done with it. Can you imagine how adoption would plummet if this installed a windows service? Not just the extra overhead but it would also eliminate a large portion of my user base on hosted or cloud solutions. Its also just one more moving part that can break down and complicate troubleshooting.

  26. Avatar for Softlion
    Softlion October 22nd, 2011

    Yo,
    thanks for your blog post, it is a question i retained since so a long time !
    Here is an optimized version of your code.

    public class WebTaskHost : IRegisteredObject
    {
    private readonly object _lock = new object();
    private bool isShuttingDown;
    public WebTaskHost()
    {
    HostingEnvironment.RegisterObject(this);
    }
    public void Stop(bool immediate)
    {
    if (isShuttingDown)
    return;
    lock (_lock)
    {
    if (isShuttingDown)
    return;
    isShuttingDown = true;
    HostingEnvironment.UnregisterObject(this);
    }
    }
    public void DoWork(Action action)
    {
    if (isShuttingDown)
    return;
    lock (_lock)
    {
    if (isShuttingDown)
    return;
    action();
    }
    }
    }

  27. Avatar for softlion
    softlion October 22nd, 2011

    [code]
    public class WebTaskHost : IRegisteredObject
    {
    private readonly object _lock = new object();
    private bool isShuttingDown, isUnregistered;
    public WebTaskHost()
    {
    HostingEnvironment.RegisterObject(this);
    }
    public void Stop(bool immediate)
    {
    if (isShuttingDown)
    return;
    lock (_lock)
    {
    if (isShuttingDown)
    return;
    isShuttingDown = true;
    if (immediate)
    {
    HostingEnvironment.UnregisterObject(this);
    isUnregistered = true;
    }
    }
    }
    /// <summary>
    ///
    /// </summary>
    /// <param name="action"></param>
    /// <returns>false if the caller should shut down the work</returns>
    public bool DoWork(Action action)
    {
    if (isShuttingDown && isUnregistered)
    return false;
    lock (_lock)
    {
    if (isShuttingDown)
    {
    if (!isUnregistered)
    {
    HostingEnvironment.UnregisterObject(this);
    isUnregistered = true;
    }
    return false;
    }
    action();
    }
    return true;
    }
    }
    [/code]

  28. Avatar for Richard
    Richard October 24th, 2011

    It is worth to take a look at FluentScheduler, I'm using this now for some time and it is really great (package is on NuGet). Can do things like
    Schedule<MyTask>().ToRunNow().AndEvery(2).Seconds();
    or
    Schedule<MyTask>().ToRunNow().AndEvery(1).Months().OnTheFirst(DayOfWeek.Monday).At(3, 0);

  29. Avatar for Leonardo Campos
    Leonardo Campos October 25th, 2011

    Should I use HostingEnvironment.IncrementBusyCount() on this case?
    "If the busy count is greater than zero, the hosting application will not time out even if there are no pending requests."

  30. Avatar for mike johnson
    mike johnson October 27th, 2011

    In Azure you could just queue this up to a web worker, but strange that there really is no batch queue service into which you can just plug in a blob of code to execute periodically.
    Is there such a thing?

  31. Avatar for Chuck
    Chuck October 29th, 2011

    Thomas Marquardt wrote about something similar a while back:
    [1] blogs.msdn.com/...
    [2] blogs.msdn.com/...

  32. Avatar for John Haigh
    John Haigh November 2nd, 2011

    I would ask the question: What are you running in the background task i.e. what is the function being performed at a specified interval?
    Is there a better way to solve the problem at hand with a solution other than background tasks? Is it possible that using a Service Bus with Workflow might better solve the problem domain?

  33. Avatar for Bruce Chapman
    Bruce Chapman November 20th, 2011

    DotNetNuke has had a scheduler built in for many versions. It can be configured to either request (piggybacks on some hapless users request) or timer modes. Timer modes sounds suspiciously like what you are talking about here. The DotNetNuke implementation has all sorts of smarts for running on only one server in a webfarm scenario, and smarts to pick up and resume dead requests, and provides a handy overridable Scheduler class to hook your work into, so you can build your own tasks. It's not perfect, but it does allow the extension builder a known scheduled service that even the most basic shared hosting implements.

    That said, it's still a path to living dangerously for the uninitiated developer for all the reasons listed in your post. But if you're attempting an ASP.NET scheduled service, you could do worse than downloading the DNN source code and watching how the scheduler works.

  34. Avatar for Konstantin Tarkus
    Konstantin Tarkus December 7th, 2011

    It would be great to have a "background tasks" functionality right within ASP.NET (MVC). So, that switching between different background tasks execution mechanisms would be easier (Azure Worker, Console App, Windows Service, Web App..)

  35. Avatar for Professordave
    Professordave December 11th, 2011

    Hello,
    Problem: we are getting a thread abort exception from a thread started from application_start using thread.start. We are running iis 7.5 with the always running set to true, asp.net mvc 3, .net 4.0
    One theory is that because of this explicitly started thread the system thinks that the startup code has hung and therefore aborts the thread.
    Is there a difference between "spawning" a thread implicitly using a timer as phil does above, and "spawning" a thread explicitly using thread.start.
    In application_start we are doing a thread.start and that thread does some work in an infinite loop.
    We are going to try to re-write it with a timer, but there is a reason we did it this way which I will mention later.
    We are getting thread abort exception on that thread. It looks like we get this after the amount of time corresponding to the app startup timeout.
    It looks like the thread is the restarted automatically, we think, we will do some more testing.
    However it seems that this rarely or never happens when run from visual studio, so debugging is not straight forward, so until I add some more logging I do not know whether application_start is being called again or the thread is restarted or what.
    The work we are doing in the infinite loop involves a blocking call, so there is no need for a periodic timer.
    We are getting messages from an Oracle advanced queue. The oracle .net routines to get messages include a blocking call. Using this blocking call ensures that we get messages when they arrive and that when there are no messages there is no wasted CPU that you get with busy polling.
    Since the amount of work is variable, sometimes many messages, sometimes no messages, a periodic timer might cause issues, either we might get a new thread before the first thread completes or not processing messages fast enough, depending on the duration of the timer.
    We could re-write the code to use a one shot timer that is called at the end of the function and have the function run once instead of running for ever.
    We changed the timeout for the oracle API call from infinite to something much less. We noticed, at least in visual studio, that the thread would never quit if it was currently waiting in the oracle API call.
    For example, if I touch the web.config file, so that application_end  gets called and then apllication_start gets called again, I can see a second thread get created, but the first thread is still running. It stays alive until a message arrives, at which point it returns from the API call but then it gets an exception, I think because the thread is not really still alive.

    Can anyone help us to understand conceptually what is happening?
    Thanks in advance.

  36. Avatar for haacked
    haacked December 11th, 2011

    @ProfessorDave probably better to ask this question in StackOverflow and then post a link to the question here. Include some bits of code so we can see exactly what you mean.

  37. Avatar for Chris M
    Chris M December 17th, 2011

    Not a big fan of the long-lived lock: The code here wraps the task, which may run for a long time, in a lock. In addition to being named vaguely, "_lock", any lock left lying around leaves room for coding mistakes that cause deadlocks.
    The good news is the lock is unnecessary, at least so long as the documentation is correct:
    msdn.microsoft.com/...
    "The object can either complete processing, call the UnregisterObject method, and then return or it can return immediately and complete processing asynchronously before calling the UnregisterObject method."
    Can you verify that documentation is correct?

  38. Avatar for haacked
    haacked December 17th, 2011

    Without the lock, the Stop method will return immediately, even though your task is still ongoing. This could cause ASP.NET to tear down the app domain prematurely in the middle of your operation. Hence the lock.
    In this particular case, if you were to deadlock, the app domain shutdown timeout will expire and it goes down anyways. So no more harm there than w/o the lock. But if your code is properly written to not deadlock (not hard in this case), then your code is more robust and given time to complete its task. Just keep them short!

  39. Avatar for Chris M
    Chris M December 18th, 2011

    So then the documentation is incorrect? The app will be torn down as soon as you return from the Stop method, rather than only after UnregisterObject() is called as the docs state?

  40. Avatar for haacked
    haacked December 18th, 2011

    @Chris the documentation is not incorrect. You're right, if the first time Stop is called, I could return immediately and let Stop continue to do its work. But now I have to deal with the second time Stop is called which does require the method to call UnregisterObject before it's done.
    My initial implementation is intended to keep it simple. In effect, I implemented Stop as if it's being called the second time rather than handling these two cases.
    A more complex implementation would have the first call to Stop signal the work that it needs to wrap up, while the second call to Stop wraps it up immediately. Of course this assumes that the nature of the work being done is even amenable to that sort of interaction. :)
    I might look into that later, but for now, given the nature of the work I do in the background, this felt sufficient.
    The part I'm a little unclear on, now that you mention it, is whether the application will try and wait at all for all objects to be unregistered unless you block on one of the Stop calls. If it tried to, that would allow you to simply go asynchronous in both calls of Stop and hope for the best. But as I understand it, this is not the case. I'll double check. :)

  41. Avatar for Chris M
    Chris M December 20th, 2011

    Thanks for looking into this. When you posted this, I had existing code that did all of the above (handled long-running tasks inside an ASP.Net app with no additional install burden like a Windows Service) - but it did not implement IRegisterObject because frankly, with that documentation, how could it?
    I've abstracted the code to a library:
    github.com/.../ASP.Net-Long-Running-Interval-Task
    Feedback welcome.

  42. Avatar for mm
    mm January 6th, 2012


    does your soloution work on asp.net web forms too?
    or it is just for mvc?

  43. Avatar for haacked
    haacked January 6th, 2012

    It works on ASP.NET 4.0 and above. So yes.

  44. Avatar for Anthony
    Anthony February 21st, 2012

    In the code sample above, is there no need for the SampleAspNetTimer class to ever detect app shutdown?
    If the JoBHost surfaced a "Stopped" property it could stop the timer at the right time, but the fact that you haven't done that suggests that it's not needed, and IIS will tear it down anyway.

  45. Avatar for Brian
    Brian April 4th, 2012

    How does this compare to ThreadPool.QueueUserWorkItem? ThreadPool seems to be such a simplier solution. Is the only advantage of using IRegisteredObject over ThreadPool for long running processes that ThreadPool doesn't handle app pool shut downs as gracefully?

  46. Avatar for haacked
    haacked April 4th, 2012

    @Brian well it depends if you have a recurring task or not. You still need some sort of timer. And you don't want to do the work on the same thread as your timer. Hence the use of the Timer class.
    The solution I wrote also makes sure app pool shuts down gracefully and doesn't corrupt data.

  47. Avatar for Niaz
    Niaz April 7th, 2012

    what about this, pros & cons of this implementations
    www.codeproject.com/...
    Regards
    -MN

  48. Avatar for rodrijp
    rodrijp May 2nd, 2012

    I send you a modified proposal.
    translate.google.com/translate
    The link was translated; the code you can get from next original link (not translated).
    rodrigopb.wordpress.com/...

  49. Avatar for Ari
    Ari July 18th, 2012

    Hi,
    I am trying to implement this. My issue is that the timer doesn't start until there is a page request. In other words, if the application is restarted, the timer won't tick off unless a user hits the site.
    Am I missing something? Is there a way around this?
    startMode="AlwaysRunning" is not an option for me as this is being run in a hosted environment.

  50. Avatar for John
    John August 7th, 2012

    I'm using this but I am getting this error
    A process serving application pool 'DefaultAppPool' exceeded time limits during shut down. The process id was '1111'.
    What am I doing wrong?

  51. Avatar for Ramasamy
    Ramasamy August 30th, 2012

    will it work with mvc 3 web application....

  52. Avatar for haacked
    haacked August 30th, 2012

    Yep!

  53. Avatar for Ramasamy
    Ramasamy August 30th, 2012

    can u tell me how to integrate the webBacgrounder in web application...

  54. Avatar for Ramasamy
    Ramasamy August 30th, 2012

    i m developing mvc3 web application which is using javascript,jquery..my request is starts from javascript(it is taking datas from HTML 5 websql database) to C# code.i sending request using jquery ajax..and this process should happen in background will it possible with WebBackgrounder..and in the response i m sending datas that should insert to HTML 5 websql DB..Please help me out from this...

  55. Avatar for Benjamin Eidelman
    Benjamin Eidelman September 21st, 2012

    It's an interesting idea.
    I have taken an opposite way and turnned into a framework you might find interesting.
    https://github.com/cinchcast/Roque
    the idea is use common C# methods or events to send a job into a Redis list (used as a queue), and then have multiple windows services pulling jobs from each queue, each job ends up in calling the same method or event using reflection.
    To get scheduling you can write a console app (called on schedule) and there call a method (or raise an event) to get the same effect.

  56. Avatar for Badcop666
    Badcop666 October 20th, 2012

    I dealt with a similar scenario in a windows service which spawned worker threads. Service stop request had to tell each thread that a stop was pending. Each thread, operating in a wait-do loop, simply checked a stopping flag prior to each loop. The unit of work was well within the service stop wait period, hence, allowing started tasks to complete was un-problematic.
    I also like the simplicity of implementing this simple mechanism in an existing platform. In this case, IIS is the overall process controller rather than windows service host. Good discussion thanks to all.

  57. Avatar for Rick Anderson [riande@MSFT]
    Rick Anderson [riande@MSFT] October 31st, 2012

    Task update that works on Azure would be great, that would save folks from paying for a worker role when they have simple/short tasks to run in the background.

  58. Avatar for John K.
    John K. April 12th, 2013

    I came across another method that requires very little code. In Application_start, add an item to the cache with the timeout you want, and add a callback function that will run your scheduled code. Then in the callback function, re-add the item back into the cache, with the same timeout and same callback function.

    Simple scheduler with almost no code.

  59. Avatar for haacked
    haacked April 12th, 2013

    ASP.NET might not guarantee that it won't shut down the AppDomain in the middle of the callback function. Might want to check with the team on that to make sure you still don't need to call RegisterObject.

    Also, that doesn't help you coordinate scheduled tasks when you have a web farm with multiple servers running the same code.

  60. Avatar for John K.
    John K. April 16th, 2013

    I agree, the job that runs is not mission critical, just a clean up job.

  61. Avatar for Rob Church
    Rob Church May 9th, 2013

    Instead of locking just to set the flag, could you not use the volatile keyword on the bool? It would prevent any code hanging around for locks and you could even check it more easily through-out your work() function.

  62. Avatar for Christopher Deutsch
    Christopher Deutsch June 29th, 2013

    I know this post is super old, but I've been working on a hosted service to move these types of tasks out of ASP.NET and into something easy to manage. I'd love feedback from some other .NET devs:
    http://sharphooks.net

  63. Avatar for Markus Buehler
    Markus Buehler July 29th, 2013

    You made my day: "...I modeled my parenting technique after this method when trying to get my kids ready for school."

  64. Avatar for Simon
    Simon August 1st, 2013

    I ended up using a different approach (which I'm sure has just as many problems) - A windows Service that loads the assemblies of the website into an AppDomain. It uses reflection to look for classes which implement ITask and executes them when scheduled (as defined by attributes).

    The good news is that this allows me to control the lifecycle, have my own Stop(Force) mechanism, etc.

    I detect when a deploy/update occurs to the site (via a WCF call) and start shutting down the old tasks. When I'm done, I kill the old AppDomain and load new task definitions into a new one.

    This works very smoothly in most circumstances. The only problem is if a publish has to perform database updates, I need some logic in the website to co-ordinate with the service (via WCF) so that it doesn't attempt to pull the rug out from beneath it and either waits for the jobs to complete (with that webserver being unavailable while we wait) or by telling the service to abort running jobs after a short shutdown time so updates can be applied rapidly.

    I don't see this as any worse than any other solution (ie it's a process issue, not an implementation one).

    Of course, this does require that the service can see the assemblies of the website...

  65. Avatar for Miguel Angel Goicochea
    Miguel Angel Goicochea January 3rd, 2014

    Hello,

    Did you write a blog entry about how to handle situation #1? If not then what's the best approach?

  66. Avatar for haacked
    haacked January 4th, 2014

    I did not. I simply make sure background tasks don't throw exceptions by wrapping them in a try/catch and terminating the thread gracefully in the catch.

  67. Avatar for Roger Martin
    Roger Martin January 15th, 2014

    Do I still need to implement IRegisterObject if I am starting tasks using Task.Factory.StartNew() from the request thread?

  68. Avatar for odinserj
    odinserj April 13th, 2014

    Yes, you still need to implement IRegisteredObject, but this solution has different problems. If you want to use fire-and-forget tasks in a reliable way, take a look at http://hangfire.io.

  69. Avatar for Christopher Thomas
    Christopher Thomas May 22nd, 2014

    Nothing constructive to add here, i'd just like to say thanks for this article. Very well written and instantly understandable, a nice break from the monotony of internet explinations ironically aimed at those who already have a firm grasp of the subject matter.

  70. Avatar for odinserj
    odinserj June 8th, 2014

    Last week I checked the last statement of this awesome article – that writing code is like skydiving, so I can claim myself as an expert :D

    If you want to run any kind of background tasks: fire-and-forget, delayed, or recurring, inside ASP.NET application in a reliable way with single line of code, try http://hangfire.io It is backed by persistent storage – Redis, SQL Server, SQL Azure or MSMQ (RabbitMQ support is coming), and works anywhere – on shared hosting, dedicated server or in cloud.

  71. Avatar for V...
    V... June 10th, 2014

    Great article. Need to solve similar stuff. Windows service is overkill, if I take in account that most of my task are most to 1 min. length.

    Although there's still a problem not solved: assume you have to execute task in T1, T2 and T3 time. If IIS goes down after T1, task T2, T3 won't be executed. Even if you manage to bring up IIS after T2, T2 task still won't be executed.

    So the long running task problem (or hosting in IIS) must be separated into smaller parts:
    1) keeping the IIS on - with a windows task scheduler, trying to open a simple HTML page in every f.e. 5 minutes this is accomplished
    2) planning and persisting info about task - this can be done with a timer, inside IIS. Quartz is a good solution, you can use Quartz's Cron(Expression or parser class) to get next schedule time. Then you create a task, with a name, which is scheduler (not startable before) given time, has state: scheduled
    3) another timer which handles the given task and related job in a transaction - either both succeed, or neither. the task manager simply checks every 5 secs. if there's a task to start, if yes, he starts it...

    This solution is neither ideal, some tasks may be executed a bit later than planned (since IIS ping is every 5 sec), but it allows us even to implement retrying, etc.

    There are also several issues with this design:
    if IIS goes up, and between last planned task and DateTime.Now the task should have been started X times, should we create a task for each occurrence? or should we create a task only for the next occurrence? or should we create a task for the nearest previous occurrence and start it immediately. actually with pinging IIS every 5 minutes, having time between most recurring tasks more than 5 minutes this is not an issue, but maybe others have another domain, where these problems must be handled...

  72. Avatar for Micah
    Micah December 24th, 2014

    I've used Fluent Scheduler (https://github.com/jgeurts/... which is pretty nice. I mainly use it to consume RSS feeds, or updated the database for date related items (expiration, etc)

  73. Avatar for kns98
    kns98 January 2nd, 2015

    nice solution and good tip to reference the timer :) - to avoid garbage collection. I think you can also use HostingEnvironment.QueueBackgroundWorkItem and avoid needing the iregisterobject.

  74. Avatar for Максим
    Максим January 19th, 2015

    Thank you, very very cool and helped me.

  75. Avatar for ricibald
    ricibald January 25th, 2015

    Great article! Thanks

  76. Avatar for sally
    sally March 6th, 2015

    Good post, some good advice there.

    I have an extra snippet of info for package authors: if you are authoring a package which is likely to be installed as a development dependency in almost all cases, starting with NuGet 2.8 you can add <developmentdependency>true</developmentdependency> to your nuspec metadata. This will inform NuGet to add developmentDependency="true" in packages.config when developers install your package.

    Of course, a developer can still remove the attribute if they need to.

  77. Avatar for jason
    jason March 8th, 2015

    A scenario I never see is a validator that hits the database to check the name is unique, say if the user is registering and you require a unique username. Anyone know of any good examples for MVC 1 or 2?

  78. Avatar for Daniel Vieira Costa
    Daniel Vieira Costa September 30th, 2015

    Hi haacked! How could I use this in the "Fire and Forget" situation? I'm facing something like:

    public async Task<partialviewresult> SaveBudget(PostOrcamentoServicoProposta proposta)
    {
    // persist the budget
    SaveData(proposta);

    // get all companies...
    var companies = getCompanies(proposta);

    foreach (var company in companies)
    await EmailFactory.SendBudget(proposta, company).SendAsync();

    return PartialView(proposta);
    }

    In this situation, the await condition will cause a temporary lock in the return of the View, so, few guys told me to use Task.Factory.StartNew, however, we know about the tears down, and bla bla bla... but, Could I mix your solution with Task?

  79. Avatar for haacked
    haacked September 30th, 2015

    How does await create a lock? If SendAsync is truly async, what should happen is that the current thread is returned to the threadpool while SendAsync runs on an async thread (or IO Completion port depending on what it does). When it returns, a thread from the threadpool is revived and completes the method.

  80. Avatar for Daniel Vieira Costa
    Daniel Vieira Costa October 1st, 2015

    It create a lock unfortunately, asynchronous controller actions does not help in this scenario, because they do not yield response to the user while waiting for the asynchronous operation to complete. They only solve internal issues related to thread pooling and application capacity.

    How could we use your example with Async?

  81. Avatar for Lucas Teixeira
    Lucas Teixeira January 29th, 2016

    I have 1 site (one AppDomain) running under 1 ApplicationPool.
    When a request is made to an especific endpoint, it creates a new AppDomian (System.AppDomain.CreateDomain) and put its code to run under a Task.Run thread. The request awaits this Task to return after the new AppDomain get closed.

    When I press the button STOP in IIS. Will this pool thread running my new AppDomain get killed?

    When IIS Recycle or Stops it actually calls Thread.Abort()? (not good)

    Suppose I've got 2 sites (two AppDomains) running under the same ApplicationPool. Then I press STOP in one of them. Will it kill the threads in under control of the TPL ?

  82. Avatar for Zignd
    Zignd February 12th, 2016

    I have found a better solution, it's a library called Hangfire, it's available on Nugget and will help you easily write those tasks and even schedule them to run periodically. http://hangfire.io/

  83. Avatar for WannabeLearner
    WannabeLearner September 2nd, 2016

    "who really made it possible to get Quartz.NET working on .NET Core. Daniel Marbach also contributed a lot with ideas and code to async side." i see you are neutral about it :p

  84. Avatar for Josh
    Josh September 5th, 2016

    Should not _shutdown be volatile?