A Subtle Gotcha with Azure Deployment Slots and ASP.NET Core

When I deploy software, I’m lazy. Very lazy. This is why I lean heavily on Continous Deployment (CD) to automatically test and deploy software when it’s merged into my main branch. I don’t have time to deploy code by hand. So gauche!

Of course this requires a lot of trust in my automation and testing before merging code into main. But the overall time savings and agility gained through CD are tremendous!

Deploying code

It’s possible to set up a continuous blue-green deployment process on Azure using deployment slots.

I did this recently, but ran into a subtle gotcha when I tried to set up a custom warm-up with my ASP.NET Core site. A custom warm-up is a URL that Azure will hit after code is deployed to the offline deployment slot or slots. This gives the app a chance to run custom warm-up code before the slot is swapped into production.

The documentation mentions two app settings you can use to set up a custom warm-up:

WEBSITE_SWAP_WARMUP_PING_PATH: The path to ping to warm up your site. Add this app setting by specifying a custom path that begins with a slash as the value. An example is /statuscheck. The default value is /.

WEBSITE_SWAP_WARMUP_PING_STATUSES: Valid HTTP response codes for the warm-up operation. Add this app setting with a comma-separated list of HTTP codes. An example is 200,202. If the returned status code isn’t in the list, the warmup and swap operations are stopped. By default, all response codes are valid.

Seems straightforward. So I set something up like this:

WEBSITE_SWAP_WARMUP_PING_PATH: /statuscheck
WEBSITE_SWAP_WARMUP_PING_STATUSES: 200

Afterwards, all my deployments failed. I used curl to confirm my /statuscheck page returned a 200 code.

curl -i https://mysite.example.com/statuscheck

I tried running the swap manually from the Azure Portal and it gave a very unhelpful error message. First of all, the error message contains the following string:

Cannot swap slots for site '{0}' because the application initialization in '{1}' slot either took too long or failed.

It looks like someone forgot to pass parameters to string.Format.

Second, it mentioned a Bad Request: 400 status code.

I then looked in my logs and didn’t see any 400 status code requests. Instead, I saw some 307 Temporary Redirects. I looked more closely and noticed requests for:

http://mysite.example.com/statuscheck

Notice the problem? The Azure warmup ping request is making HTTP requests, not HTTPS requests. And you can’t change this. The setting WEBSITE_SWAP_WARMUP_PING_PATH does not allow a fully qualified URL. It will always make an HTTP request.

The problem for me is that I have the following code in my app startup (Startup.cs).

public void Configure(IApplicationBuilder app) {
    // Bunch of stuff
    app.UseHttpsRedirection();
    // More stuff.
}

You might recognize that UseHttpsRedirection method call from the default ASP.NET Core project templates. That adds middleware to redirect HTTP requests to HTTPS.

On top of that, I usually configure Azure Web Applications to be HTTPS only via the TLS/SSL settings. This is why the Azure swap warmup request fails. It expects a 200 but is getting a 307 and doesn’t follow the redirect.

In a useful troubleshooting blog post, the author mentions using rewrite rules to solve this. Not sure that’s going to work for my site since I don’t even have a web.config file and I’m not using rewrite rules to enforce HTTPS in the first place.

Fortunately the solution is straightforward. First, I went to the TLS/SSL settings for my site in the Azure Portal and turned HTTPS Only to Off.

Then I branched my request pipeline using the UseWhen extension method.

public void Configure(IApplicationBuilder app) {
    // ...
    app.UseWhen(context =>
    {
        var path = context.Request.Path.Value ?? string.Empty;
        return !path.Equals("/statuscheck", StringComparison.OrdinalIgnoreCase);
    },
    mainApp =>
    {
        mainApp.UseHttpsRedirection();
    });
    // ...
}

In my actual code, I wrote an extension method like so:

public static bool IsStatusCheckRequest(this HttpRequest request) {
    var path = request.Path.Value ?? string.Empty;
    return path.Equals("/statuscheck", StringComparison.OrdinalIgnoreCase);
}

That way I could simplify the code in my startup class to:

public void Configure(IApplicationBuilder app) {
    // ...
    app.UseWhen(context => !context.Request.IsStatusCheckRequest(),
      mainApp =>
      {
        mainApp.UseHttpsRedirection();
      });
    // ...
}

UseWhen takes two parameters. The first is the condition. In this case, when the request is NOT a status check request. The second is the code to call when that condition is true, in this case it registers the HTTPS redirect middleware.

In effect, this code forks my app pipeline into two paths. For requests to /statuscheck HTTP requests are allowed. For all other requests, we continue to redirect HTTP to HTTPS.

Problem solved!

What’s particularly subtle about all this is if you set up WEBSITE_SWAP_WARMUP_PING_PATH but do not set up WEBSITE_SWAP_WARMUP_PING_STATUSES, the warm-up request by default accepts all status codes, including 307. So your deploys will pass, but it may not necessarily be done with the warm-up as it’s unclear if Azure follows the redirect in this case.

This was a frustrating experience, but I got it working in the end. My hope is that if you have a similar setup to me, this will save you time and headaches.

UPDATE: My deploys still fail. If I specify WEBSITE_SWAP_WARMUP_PING_STATUSES = 200 my deploys still fail even though I can use curl to see the status check URLs return 200. I’m at a loss here and I’ve opened up a support ticket with Microsoft. I’ll post back here with what I learn.

UPDATE: Resolved the issue. I tried to be clever and pass a query string parameter to my warmup URL. For example, by setting WEBSITE_SWAP_WARMUP_PING_PATH to /statuscheck?azure-warmup=true. This way I could quickly distinguish my own testing of this URL from the request made by Azure in my logs. Upon reflection, of course this doesn’t work because Azure URL Encodes WEBSITE_SWAP_WARMUP_PING_PATH before requesting it. So the request path is /statuscheck%3Fazure-warmup=true and not /statuscheck as I expected.

Conclusion

In the end, I figured out what I was doing wrong and it’s easy to chalk it up to PEBCAK (Problem Exists Between Chair And Keyboard). While that may be true, it’s also true that the feature could improve its usability to reduce PEBCAK occurrences.

Sebastian Ros on Twitter suggested a couple of improvements:

It should allow for https or accept a redirect to the https endpoint
It should not encode the URL you passed

I definitely agree with the first suggestion. Otherwise everyone deploying a standard ASP.NET Core app will run into this and my not even realize it!

My preference would be to follow the redirect so IT JUST WORKS^TM. I understand this is not trivial and requires a bit of thought. For example, it can’t blindly follow the redirect. The initial request is made internal to Azure, bypassing (as I understand it) DNS and the load balances etc. So it would need to be smart about following the redirect. If the redirect is to the same endpoint, but only the scheme is changed, then re-make the original request with HTTPS.

Might be easier to add a radio button group to select between HTTP and HTTPS. Either way, the current behavior is a hidden gotcha.

As for the second suggestion, I also agree. It would require validation of the user input. For example, if I put something that isn’t allowed for the end of the URL, it should present an error message. Alternatively, have two fields, one for the path and one for the query string. Either way, the current behavior is also confusing and a hidden gotcha.

Comments

8 responses

Rasmus • September 28th, 2020
Another way is to check for the user agent string “IIS Application Initialization Warmup”.

Handy when having multiple warmup urls
Rando • September 29th, 2020
Couldn’t you just skip all of that and simply hit that endpoint in your pipeline to warm it up?
haacked • September 29th, 2020
@Rasmus thanks for the tip!

@Rando that won’t work for me. I’m trying to set up an auto-swap. As part of the auto-swap, Azure will do the following steps:
1. Apply production settings to the staging deployment slot
2. Send the ping request to ensure it’s warmed up and that it returns a 200 status code (per my settings).
3. If the ping request returns 200, it’ll swap otherwise it won’t.
So I could have my pipeline make the warmup request after deployment, but that wouldn’t guarantee that it’s done before Azure does the auto-swap. I need Azure to make that request and wait for a 200 status code to be sure.
Haacked • September 29th, 2020
@Rasmus I just looked in my logs and the actual User Agent is “SiteWarmup”.

Erik A. Brandstadmoen • September 29th, 2020

We use client certificates, and would like to skip that for the site warmup and ping from alwaysOn. After a lot of trial and error, we’ve actually had to use all of this to allow for both site warmups, the always on request, etc. Look at the list of user-agents. And it might also use an http header. It’s a bit messy. But this is all the variants we’ve had to allow to make it work. It has taken some time to find them all :)

   private bool ShouldSkipCertificateValidation(HttpContext context)
        {
            return UserAgentIsRecognized(context) &&
                   (IsFromLocalHostAddress(context) ||
                   (IsFromInternalAddress(context) && HasWellKnownMicrosoftAuthHeader(context)));
        }

        private static bool HasWellKnownMicrosoftAuthHeader(HttpContext context) =>
            context.Request.Headers["x-ms-auth-allow-http"].Equals("true");

        private static bool IsFromInternalAddress(HttpContext context) =>
            GetClientIpAddress(context).GetAddressBytes().First() == 10;

        private static bool IsFromLocalHostAddress(HttpContext context) =>
            IPAddress.IsLoopback(GetClientIpAddress(context));
        

        private bool UserAgentIsRecognized(HttpContext context) => GetUserAgent(context).In(RecognizedUserAgents);
        
        private static IEnumerable<string> RecognizedUserAgents => new[]
        {
            "AlwaysOn",
            "HostnameSyncPinger",
            "SiteWarmup",
            "IIS Application Initialization Warmup",
            "IIS Application Initialization Preload"
        };


        private static IPAddress GetClientIpAddress(HttpContext context) => context.Connection.RemoteIpAddress;
        private static string GetUserAgent(HttpContext context) => context.Request.Headers["User-Agent"];

drdamour • October 1st, 2020
was turning https only off really required? i’d think the warmup call would come to the backend instance and not through the load balancer (and i assumed the https only was at the LB)
Mark Bowler • October 20th, 2020
I only hit this issue if the redirect rule was specified in the web.config, which is mentioned in the troubleshooting guide https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots#troubleshoot-swaps

Not seen the issue when HTTPS only is configured in the TLS/SSL settings.
Mohammad • October 20th, 2020
Wasn’t it easier to set environment to “staging”, then just check if IsStaging() in Startup.Configure() (or anywhere else)?