Handling Errors and Poison Blobs in Azure Functions With Azure Blob Storage Triggers

(This article applies to Azure Functions V2)

An Azure Function can be triggered by new blobs being written (or updated). If an unhandled exception occurs in the function, by default Azure Functions will retry the blob 5 times. This means the function will be triggered again for the same blob up to 5 times. If the same blob causes errors 5 times, no further attempts will be made and the processing of the blob will be “lost”.

Understanding Blob Processing Errors in Azure Functions

When a new (or updated) blob triggers a function, the Azure Functions runtime makes sure that the same blob is not processed twice (if no error occurs in the function execution). To do this the runtime makes use of “blob receipts”. These are stored in the Azure storage account associated with the function app (as defined in the AzureWebJobsStorage Function App settings).

As an example, suppose a new blob (called “followupletterrequest.data”) triggered the following function:

class FollowupLetterRequest
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

public static class PoisonBlobExampleFunctions
{
    [FunctionName("PoisonBlobExampleFunctions")]
    public static void Run(
        [BlobTrigger("followup-letters/{blobname}.data")]string blobData, 
        string blobname,
        [Blob("followup-letters/{blobname}.txt")] out string letter,
        ILogger log)
    {
        var settings = new JsonSerializerSettings
        {
            MissingMemberHandling = MissingMemberHandling.Error
        };

        // This code assumes blob JSON is valid, if not an exception will be thrown
        var request = JsonConvert.DeserializeObject<FollowupLetterRequest>(blobData, settings);

        string firstName = request.FirstName;
        string lastName = request.LastName;

        letter = RenderFollowUpLetterText(firstName, lastName);
    }
    
    private static string RenderFollowUpLetterText(string firstName, string lastName)
    {
        string simulateLetterText = WaffleEngine.Text(paragraphs: 3, includeHeading: false);

        return $"Dear {firstName} {lastName}\r\n \r\n{simulateLetterText}";
    }
}

After the function runs, in the storage account under a path like “azure-webjobs-hosts/blobreceipts” the blob receipt can be seen. On a development machine using the local storage emulator the full path would be something like: “blobreceipts/desktop/DontCodeTiredDemosV2.PoisonBlobExampleFunctions.Run/"0x8D69224161F4590"/followup-letters/followupletterrequest.data”.

This full path to the blob receipt blob represents:

  • Function Id that the blob triggered (DontCodeTiredDemosV2.PoisonBlobExampleFunctions.Run)
  • Blob Container Name (followup-letters)
  • Name of triggering blob (followupletterrequest.data)
  • Triggering blob version ETag (“0x8D69224161F4590”)

 

If we now added another new blob called “followupletterrequest_bad.data” that contains bad data (e.g. a missing JSON property), so that an exception is thrown, a second blob receipt will be generated: “blobreceipts/desktop/DontCodeTiredDemosV2.PoisonBlobExampleFunctions.Run/"0x8D692245985E910"/followup-letters/followupletterrequest_bad.data”.

Because this blob generated an error, after the default number of retries (5) there will be no more attempts to process it.

Manually Forcing a Blob to Be Reprocessed

The documentation states that if the blob receipt is manually deleted, this will force the blob to reprocessed. This may be suitable to force reprocessing of a set of blobs that failed processing due to some transient error such as a database or network being temporarily offline. You should obviously take care that reprocessing blobs wont cause problems such as duplicate orders, emails, etc. or other errors in the system. You  may also need to consider what would happen if blobs are retried in a different order and/or interleaved with new blobs being added. Also blobs may not be reprocessed immediately. Using the local function runtime development environment, once the blob receipt has been deleted, it seems that the function app needs restarting to cause the blob to be reprocessed (either that or I didn’t wait long enough…). Once deployed to Azure there can be a delay between when the blob receipt is deleted and the blob being retried, the following timeline shows the delay between the blob receipt being deleted and the retry attempt 1.

2019-02-14 03:40:24.374 <attempt 1 - failure>
2019-02-14 03:40:24.763 <attempt 2 - failure>
2019-02-14 03:40:24.891 <attempt 3 - failure>
2019-02-14 03:40:25.007 <attempt 4 - failure>
2019-02-14 03:40:25.117 <attempt 5 - failure>
<blob receipt deleted>
2019-02-14 04:24:24.327 <retry attempt 1 - failure>
2019-02-14 04:24:25.155 <retry attempt 2 - failure>
2019-02-14 04:24:25.288 <retry attempt 3 - failure>
2019-02-14 04:24:25.455 <retry attempt 4 - failure>
2019-02-14 04:24:25.592 <retry attempt 5 - failure>

Automatically Responding to Blob Failures in Azure Functions

When a blob fails for the last time, information about the failure will written as a message to a Storage queue called “webjobs-blobtrigger-poison”. The message contains a JSON payload describing the triggering blob that didn’t complete processing successfully, for example:

{
  "Type": "BlobTrigger",
  "FunctionId": "DontCodeTiredDemosV2.PoisonBlobExampleFunctions.Run",
  "BlobType": "BlockBlob",
  "ContainerName": "followup-letters",
  "BlobName": "followupletterrequest_bad.data",
  "ETag": "\"0x8D692245985E910\""
}

The information contained in the JSON can be used to alert support people about the error and take appropriate action as required such as writing to a support ticket database or sending an email. You could also implement logic to automatically delete the blob receipt to force reprocessing but there would probably want to be some retry count otherwise bad data could cause an infinite processing loop. Exactly how you handle failed blob processing will depend on the business scenario.

As an example, the following function monitors the “webjobs-blobtrigger-poison” queue and grabs the information about the failed blob:

[FunctionName("PoisonBlobQueueProcessor")]
public static void PoisonBlobQueueProcessor(
    [QueueTrigger("webjobs-blobtrigger-poison")] string message,
    ILogger log)
{
    var poisonBobDetails = JsonConvert.DeserializeObject<dynamic>(message);

    log.LogInformation($"Found an unprocessed blob {poisonBobDetails.ContainerName}/{poisonBobDetails.BlobName}\r\n");
    
    // Send an email, log a ticket in a fault system, log a CRM issue, etc.            
}

SHARE:

Getting Blob Metadata When Using Azure Functions Blob Storage Triggers

(This article refers to Azure Functions V2)

Basic Blob Metadata

There are a few basic pieces of metadata that are often useful.

The following code show a simple example of a blob-triggered Azure Function:

[FunctionName("BlobMetadataExample")]
public static void Run(
    [BlobTrigger("decline-letters/{name}")]Stream myBlob, 
    string name, 
    ILogger log)
{
    log.LogInformation($"Name: {name} Size: {myBlob.Length} Bytes");
}

With the preceding code, if we add a blob called “declineletterrequest.data” to the “decline-letters” container, the function will be triggered with the output: “Name: declineletterrequest.data Size: 50 Bytes”.

Notice that the string name parameter has been automatically populated with the full name of the blob that triggered the function execution.

If you want to get the blob name and blob extension separately you could write the following:

[FunctionName("BlobMetadataExample")]
public static void Run(
    [BlobTrigger("decline-letters/{blobname}.{blobextension}")]Stream myBlob,
    string blobName,
    string blobExtension,
    ILogger log)
{
    log.LogInformation($"Name: {blobName} Extension: {blobExtension} Size: {myBlob.Length} Bytes");
}

If the preceding function executes we get the output: “Name: declineletterrequest Extension: data Size: 50 Bytes”.

In addition to being able to use this simple blob metadata in code, you can also use the elements of the triggering blob name in other bindings:

[FunctionName("BlobMetadataExample")]
public static void Run(
        [BlobTrigger("decline-letters/{blobname}.{blobextension}")]Stream myBlob,
        string blobName,
        string blobExtension,
        [Queue("output-queue-{blobextension}")] out string message,
        ILogger log)
{
    log.LogInformation($"Name: {blobName} Extension: {blobExtension} Size: {myBlob.Length} Bytes");

    message = "Hello world";
}

In the preceding code, the output queue that is written to is dependent on the extension of the triggering blob. If the triggering blob name was “declineletterrequest.bankofmars” then a message will be written to the queue “output-queue-bankofmars” or if the input blob was called “output-queue-bankofvenus” then a message would be written to the “output-queue-bankofvenus”.

You can also do a similar thing by binding an input blob binding to the contents of a triggering queue message.

Advanced Metadata

There are a number of additional metadata items that you can get by simply adding the correct method arguments with the correct names:

[FunctionName("BlobMetadataExample")]
public static void Run(
        [BlobTrigger("decline-letters/{blobname}.{blobextension}")]Stream myBlob,
        string blobName,
        string blobExtension,
        string blobTrigger, // full path to triggering blob
        Uri uri, // blob primary location
        IDictionary<string, string> metaData, // user-defined blob metadata
        BlobProperties properties, // blob system properties, e.g. LastModified
        ILogger log)
{
    log.LogInformation($@"
blobName      {blobName}
blobExtension {blobExtension}
blobTrigger   {blobTrigger}
uri           {uri}
metaData      {metaData.Count}
properties    {properties.Created}");
}

Executing the preceding code will give the following output:

blobName      declineletterrequest
blobExtension data
blobTrigger   decline-letters/declineletterrequest.data
uri           http://127.0.0.1:10000/devstoreaccount1/decline-letters/declineletterrequest.data
metaData      0
properties    12/02/2019 2:15:53 AM +00:00

The BlobProperties give you access to a host of information such as ETag, DeletedTime, ContentEncoding, etc.

You can use this additional metadata in further binding expressions, the following example shows how to bind a blob output name to the ETag of the original triggering blob:

[FunctionName("BlobMetadataExample")]
public static void Run(
[BlobTrigger("decline-letters/{blobname}.{blobextension}")]Stream myBlob,
string blobName,
BlobProperties properties,
[Blob("decline-letters/{properties.ETag}")] out string message,
ILogger log)
{
    message = "Hello world";
}

The preceding code would create an output blob with a name such as “0x8D6909193F68C10”.

SHARE:

Dealing With Unprocessed Storage Queue Poison Messages in Azure Functions

If an Azure Function that is triggered by a message on a Storage Queue throws an exception, the message will automatically be returned to the queue and retried again in the future.

In addition to specifying how soon the message will be retried, you can also configure how many times the message will be retried by editing the host.json file. By default a message will be retried 5 times before finally failing. The following host.json specifies that a message should be retried 10 times before finally failing:

{
  "version": "2.0",
  "extensions": {
    "queues": {      
      "maxDequeueCount": 10
    } 
  }  
}

Handling Poison Messages in Azure Functions

Once a message has been retried the maximum number of times, it will be considered a poisonous message, essentially that if we keep it on the queue it will “poison” the application/function and cause harm. Poison messages will be removed from the queue and placed onto a poison queue.

For example, if the queue that triggers the function is called “input-queue”, poison messages will be moved to a queue called “input-queue-poison”.

Because we know the name of the poison queue, we can process these poison messages somehow. Exactly how you choose to process these messages will depend on the application you are building.

One thing to think about is why the message may have failed:

  • Is the message content itself corrupted?
  • Is the function code itself defective/have a bug?
  • Were the exceptions caused by a transient error in a service the function uses?
  • Etc.

You could have some automated process (function) attempt to resolve the poison messages or forward them to a human to resolve (for example writing the message to database that a human can query).

Triggering an Azure Function From a Poison Message Queue

As an example, the following function retrieves messages from the “input-queue-poison” queue and writes out to table storage for a human to manually correct somehow:

using System;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage.Queue;

namespace DontCodeTiredDemosV2
{
    public class PoisonMessageDetails
    {        
        public string RowKey { get; set; }
        public string PartitionKey { get; set; }
        public string OriginalMessageContent { get; set; }
        public string OriginalMessageId { get; set; }
    }

    public static class HandlePoisonMessages
    {
        [FunctionName("HandlePoisonMessages")]
        [return: Table("HumanInterventionRequired")]
        public static PoisonMessageDetails Run(
            [QueueTrigger("input-queue-poison")]CloudQueueMessage poisonMessage,            
            ILogger log)
        {
            log.LogInformation($"Processing poison message {poisonMessage.Id}");            

            return new PoisonMessageDetails
            {
                RowKey = Guid.NewGuid().ToString(),
                PartitionKey = "input-queue",
                OriginalMessageContent = poisonMessage.AsString,
                OriginalMessageId = poisonMessage.Id
            };                       
        }
    }
}

Once a poison message is processed (for example with the content of “Amrit”) a row will be added to the table as the following screenshot shows:

Azure Table Storage to process poison messages

SHARE:

Specifying How Soon a Storage Queue Message Will Be Retried in an Azure Function

By default, if an exception occurs in an Azure Function that uses a Storage Queue trigger, the message will be returned to the queue and automatically retried again in the future (up to a maximum number of times).

By default, there is no delay in how soon the message can be retried.

public static class MakeUppercase
{
    [FunctionName("MakeUppercase")]
    public static void Run(
        [QueueTrigger("input-queue")]CloudQueueMessage inputQueueItem,
        ILogger log)
    {
        log.LogInformation($"Message Dequeued : {inputQueueItem.DequeueCount} time(s)");
        log.LogInformation($"Message Next Visible : {inputQueueItem.NextVisibleTime}");

        throw new Exception("Forced exception for demonstration purposes.");
    }
}

With the preceding function, when a single message is added to the queue, the following (abbreviated) output will be seen:

[15/01/2019 11:55:49 PM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=44f95504-7a99-4f23-81f2-096f0bd434a2)
[15/01/2019 11:55:49 PM] Message Dequeued : 1 time(s)
[15/01/2019 11:55:49 PM] Message Next Visible : 16/01/2019 12:05:49 AM +00:00
[15/01/2019 11:55:50 PM] Executed 'MakeUppercase' (Failed, Id=44f95504-7a99-4f23-81f2-096f0bd434a2)
[15/01/2019 11:55:50 PM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=bbede56f-e22a-461f-945c-3f3b47114de3)
[15/01/2019 11:55:50 PM] Message Dequeued : 2 time(s)
[15/01/2019 11:55:50 PM] Message Next Visible : 16/01/2019 12:05:50 AM +00:00
[15/01/2019 11:55:50 PM] Executed 'MakeUppercase' (Failed, Id=bbede56f-e22a-461f-945c-3f3b47114de3)
[15/01/2019 11:55:50 PM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=10b7495f-9cfb-4d75-bc31-db68581b8055)
[15/01/2019 11:55:50 PM] Message Dequeued : 3 time(s)
[15/01/2019 11:55:50 PM] Message Next Visible : 16/01/2019 12:05:50 AM +00:00
[15/01/2019 11:55:50 PM] Executed 'MakeUppercase' (Failed, Id=10b7495f-9cfb-4d75-bc31-db68581b8055)
[15/01/2019 11:55:50 PM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=385beb36-80b7-47a5-ba65-b0d04f956cc6)
[15/01/2019 11:55:50 PM] Message Dequeued : 4 time(s)
[15/01/2019 11:55:50 PM] Message Next Visible : 16/01/2019 12:05:50 AM +00:00
[15/01/2019 11:55:51 PM] Executed 'MakeUppercase' (Failed, Id=385beb36-80b7-47a5-ba65-b0d04f956cc6)
[15/01/2019 11:55:51 PM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=f6acfb11-41fb-416e-88b1-e113aa4424f5)
[15/01/2019 11:55:51 PM] Message Dequeued : 5 time(s)
[15/01/2019 11:55:51 PM] Message Next Visible : 16/01/2019 12:05:51 AM +00:00
[15/01/2019 11:55:51 PM] Executed 'MakeUppercase' (Failed, Id=f6acfb11-41fb-416e-88b1-e113aa4424f5)
[15/01/2019 11:55:51 PM] Message has reached MaxDequeueCount of 5. Moving message to queue 'input-queue-poison'.

Notice in the preceding output, the next visible times don’t include a delay in when the message can potentially be retried.

The next visible time controls when the message will become visible to be consumed. This default value in Azure Functions is 0. You may want to change this default if you want to add some delay between message retries (for example to help prevent message loss* for transient failures).

* Eventually, failed messages will be moved to a poison message queue.

The next visible time can be configured in the host.json file (we are using Azure Functions V2 in this article):

{
  "version": "2.0",
  "extensions": {
    "queues": {
      "visibilityTimeout": "00:00:30" 
    } 
  }  
}

The visibilityTimeout value represents a timespan (HH:MM:SS) to wait before a message becomes visible next time, in the preceding configuration, 30 seconds. Running again with this new configuration, the following output can be seen:

[16/01/2019 12:13:01 AM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=1f4f7177-4de6-4f4c-98c1-48d318892112)
[16/01/2019 12:13:01 AM] Message Dequeued : 1 time(s)
[16/01/2019 12:13:01 AM] Message Next Visible : 16/01/2019 12:23:00 AM +00:00
[16/01/2019 12:13:01 AM] Executed 'MakeUppercase' (Failed, Id=1f4f7177-4de6-4f4c-98c1-48d318892112)
[16/01/2019 12:13:32 AM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=93e9a90c-dd83-410f-9435-003712f64513)
[16/01/2019 12:13:32 AM] Message Dequeued : 2 time(s)
[16/01/2019 12:13:32 AM] Message Next Visible : 16/01/2019 12:23:32 AM +00:00
[16/01/2019 12:13:33 AM] Executed 'MakeUppercase' (Failed, Id=93e9a90c-dd83-410f-9435-003712f64513)
[16/01/2019 12:14:04 AM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=7ffe157b-7186-4f84-b8eb-02c43a260352)
[16/01/2019 12:14:04 AM] Message Dequeued : 3 time(s)
[16/01/2019 12:14:04 AM] Message Next Visible : 16/01/2019 12:24:04 AM +00:00
[16/01/2019 12:14:04 AM] Executed 'MakeUppercase' (Failed, Id=7ffe157b-7186-4f84-b8eb-02c43a260352)
[16/01/2019 12:14:36 AM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=ba3c186b-8b33-4b4c-b896-724a95fa2b25)
[16/01/2019 12:14:36 AM] Message Dequeued : 4 time(s)
[16/01/2019 12:14:36 AM] Message Next Visible : 16/01/2019 12:24:36 AM +00:00
[16/01/2019 12:14:36 AM] Executed 'MakeUppercase' (Failed, Id=ba3c186b-8b33-4b4c-b896-724a95fa2b25)
[16/01/2019 12:15:07 AM] Executing 'MakeUppercase' (Reason='New queue message detected on 'input-queue'.', Id=3de85cf0-89ad-430f-8219-ebd7b1701d4d)
[16/01/2019 12:15:07 AM] Message Dequeued : 5 time(s)
[16/01/2019 12:15:07 AM] Message Next Visible : 16/01/2019 12:25:07 AM +00:00
[16/01/2019 12:15:07 AM] Executed 'MakeUppercase' (Failed, Id=3de85cf0-89ad-430f-8219-ebd7b1701d4d)
[16/01/2019 12:15:07 AM] Message has reached MaxDequeueCount of 5. Moving message to queue 'input-queue-poison'.

Notice now that the message won’t be retried for 30 seconds between each attempt (look at the “Message Next Visible” lines).

Setting a visibility other than zero will not prevent other messages that come into the queue from being processed while waiting for retried messages to become visible again.

SHARE:

Getting Message Metadata When Using Azure Functions Storage Queue Triggers

When creating an Azure Function that is triggered by incoming messages on a Storage Queue, the type specified for the message parameter can be a simple string as follows:

public static class MakeUppercase
{
    [FunctionName("MakeUppercase")]
    public static void Run(
        [QueueTrigger("%in%")]string inputQueueItem,
        [Queue("%out%")] out string outputQueueItem,
        [Blob("%blobout%")] out string outputBlobItem,
        ILogger log)
    {
        inputQueueItem = inputQueueItem.ToUpperInvariant();
        outputQueueItem = inputQueueItem;
        outputBlobItem = inputQueueItem;
    }
}

In the preceding code, the inputQueueItem represents the content of the message that triggered the function.

If you want additional information about the queue message item itself, rather than use a string you can use CloudQueueMessage. Doing this gives you access to the metadata about the queue message including the following:

  • Message ID
  • Time message was inserted into queue
  • Time the message expires
  • How many times the message has been dequeued (i.e. read off the queue )*
  • Message next visible time
  • Message pop receipt

* A message can be returned to the queue if an exception occurs during execution of the function, this will increment the dequeue count.

In addition to the message metadata, you can still get the message content either as a string or byte array using the AsString or AsBytes properties respectively:

public static class MakeUppercase
{
    [FunctionName("MakeUppercase")]
    public static void Run(
        [QueueTrigger("%in%")]CloudQueueMessage inputQueueItem,
        [Queue("%out%")] out string outputQueueItem,
        [Blob("%blobout%")] out string outputBlobItem,
        ILogger log)
    {
        log.LogInformation($"Message Id: {inputQueueItem.Id}");
        log.LogInformation($"Message Inserted : {inputQueueItem.InsertionTime}");
        log.LogInformation($"Message Expires : {inputQueueItem.ExpirationTime}");
        log.LogInformation($"Message Dequeued : {inputQueueItem.DequeueCount} time(s)");
        log.LogInformation($"Message Next Visible : {inputQueueItem.NextVisibleTime}");
        log.LogInformation($"Message Pop Receipt : {inputQueueItem.PopReceipt}");

        log.LogInformation($"Message content (bytes) : {BitConverter.ToString(inputQueueItem.AsBytes)}");
        log.LogInformation($"Message content (string) : {inputQueueItem.AsString}");

        
        var inputQueueItemContent = inputQueueItem.AsString;
        inputQueueItemContent = inputQueueItemContent.ToUpperInvariant();
        outputQueueItem = inputQueueItemContent;
        outputBlobItem = inputQueueItemContent;
    }
}

SHARE:

Configuring Queue Names and Blob Path Bindings in Azure Functions Configuration

When working with Azure Functions in C# (specifically Azure Functions V2 in this article) you can specify bindings with hard-coded literal values.

For example, the following function has a queue trigger that is reading messages from a queue called “input-queue”, an output queue binding writing messages to “output-queue”, and a blob storage binding to write blobs to “audit/{rand-guid}”:

public static class MakeUppercase
{
    [FunctionName("MakeUppercase")]
    public static void Run(
        [QueueTrigger("input-queue")]string inputQueueItem,
        [Queue("output-queue")] out string outputQueueItem,
        [Blob("audit/{rand-guid}")] out string outputBlobItem,
        ILogger log)
    {
        inputQueueItem = inputQueueItem.ToUpperInvariant();
        outputQueueItem = inputQueueItem;
        outputBlobItem = inputQueueItem;
    }
}

All these binding values in the preceding code are hard coded, if they need to be changed once the Function App is deployed, a new release will be required.

Specifying Azure Function Bindings in Configuration

As an alternative, the %% syntax can be used inside the binding string:

public static class MakeUppercase
{
    [FunctionName("MakeUppercase")]
    public static void Run(
        [QueueTrigger("%in%")]string inputQueueItem,
        [Queue("%out%")] out string outputQueueItem,
        [Blob("%blobout%")] out string outputBlobItem,
        ILogger log)
    {
        inputQueueItem = inputQueueItem.ToUpperInvariant();
        outputQueueItem = inputQueueItem;
        outputBlobItem = inputQueueItem;
    }
}

Notice in the preceding code, parts of the binding configuration strings are specified between %%: "%in%", "%out%", and "%blobout%".

At runtime, these values will be read from configuration instead of being hard coded.

Configuring Bindings at Development Time

When running locally, the configuration values will be read from the local.settings.json file, for example:

{
    "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "in": "input-queue",
    "out": "output-queue",
    "blobout" :  "audit/{rand-guid}"
  }
}

Notice the “in”, “out”, and “blobout” configuration elements that map to  "%in%", "%out%", and "%blobout%”.

Configuring Bindings in Azure

Once deployed and running in Azure, these settings will need to be present in the Function App Application Settings as the following screenshot demonstrates:

Specifying Azure Function Bindings in application settings

Now if you want to modify the queue names or blob path you can simply change the values in configuration. It should be noted that you may have to restart the Function App for the changes to take effect. You will also need to manage the switch to new queues, blobs, etc.such as what to do if after the change there are still some messages in the original input queue, etc, etc.

SHARE:

Remote Debugging Azure Functions V2 "The breakpoint will not currently be hit. No symbols have been loaded for this document"

Sometimes it can be tricky to attach the Visual Studio debugger to a deployed Azure Functions app. For example if you use Cloud Explorer you can right click on the deployed Azure Function and choose Attach Debugger as the following screenshot shows:

Using Cloud Explorer to debug an Azure Function

While this may seem to work at first, you may experience a problem with your breakpoints actually being hit when function app code is executing with the message “The breakpoint will not currently be hit. No symbols have been loaded for this document”.

The breakpoint will not currently be hit. No symbols have been loaded for this document

As an alternative to attaching via Cloud Explorer, you can try the following approach:

1 Log in to Azure Portal and navigate to your deployed function app.

2 Download the publish profile

Downloading publish profile for Azure Function app

3 Open the downloaded file

Make a note of the userName, userPWD and destinationAppUrl values.

4 Attach Visual Studio Debugger to Azure Function App

  1. Make sure your function app project is open in Visual Studio
  2. Make sure that you have deployed a debug version of the function app to Azure
  3. On the Debug menu choose Attach to Process..
  4. For the Attach to value, click the Select.. button and un-tick everything except Managed (CoreCLR) code
  5. In the Connection target enter the  destinationAppUrl (without the preceding http) followed by :4022 – for example: investfunctionappdemotest.azurewebsites.net:4022 – and hit Enter
  6. You should now see an Enter Your Credentials popup box, use the userName and userPWD from step 3 and click Ok
  7. Wait a few seconds for Visual Studio to do its thing
  8. Click the w3wp.exe process and the click the Attach button (see screenshot below) Be patient after clicking as it may take quite a while for all the debug symbols to load.

Attaching to the Azure Functions w3wp.exe process

5 Set your breakpoints as desired

Breakpoint set in function run method

6 Invoke your function code and you should see your breakpoint hit

Once again this may take a while so be patient, you may also see “a remote operation is taking longer than expected”.

Azure Function breakpoint being hit in Visual Studio

SHARE:

Azure Functions Dependency Injection with Autofac

This post refers specifically to Azure Function V2.

If you want to write automated tests for Azure Functions methods and want to be able to control dependencies (e.g. to inject mock versions of things) you can set up dependency injection.

One way to do this is to install the AzureFunctions.Autofac NuGet package into your functions project.

Once installed, this package allows you to inject dependencies into your function methods at runtime.

Step 1: Create DI Mappings

The first step (after package installation) is to create a class that configures the dependencies. As an example suppose there was a function method that needed to make use of an implementation of an IInvestementAllocator. The following class can be added to the functions project:

using Autofac;
using AzureFunctions.Autofac.Configuration;

namespace InvestFunctionApp
{
    public class DIConfig
    {
        public DIConfig(string functionName)
        {
            DependencyInjection.Initialize(builder =>
            {
                builder.RegisterType<NaiveInvestementAllocator>().As<IInvestementAllocator>(); // Naive

            }, functionName);
        }
    }
}

In the preceding code, a constructor is defined that receives the name of the function that’s being injected into. Inside the constructor, types can be registered for dependency injection. In the preceding code the IInvestementAllocator interface is being mapped to the concrete class NaiveInvestementAllocator.

Step 2: Decorate Function Method Parameters

Now the DI registrations have been configured, the registered types can be injected in function methods. To do this the [Inject] attribute is applied to one or more parameters as the following code demonstrates:

[FunctionName("CalculatePortfolioAllocation")]
public static void Run(
    [QueueTrigger("deposit-requests")]DepositRequest depositRequest,
    [Inject] IInvestementAllocator investementAllocator,
    ILogger log)
    {
        log.LogInformation($"C# Queue trigger function processed: {depositRequest}");

        InvestementAllocation r = investementAllocator.Calculate(depositRequest.Amount, depositRequest.Investor);
    }

Notice in the preceding code the [Inject] attribute is applied to the IInvestementAllocator investementAllocator parameter. This IInvestementAllocator is the same interface that was registered earlier in the DIConfig class.

Step 3: Select DI Configuration

The final step to make all this work is to add an attribute to the class that contains the function method (that uses [Inject]). The attribute used is the DependencyInjectionConfig attribute that takes the type containing the DI configuration as a parameter, for example: [DependencyInjectionConfig(typeof(DIConfig))]

The full function code is as follows:

using AzureFunctions.Autofac;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;

namespace InvestFunctionApp
{
    [DependencyInjectionConfig(typeof(DIConfig))]
    public static class CalculatePortfolioAllocation
    {
        [FunctionName("CalculatePortfolioAllocation")]
        public static void Run(
            [QueueTrigger("deposit-requests")]DepositRequest depositRequest,
            [Inject] IInvestementAllocator investementAllocator,
            ILogger log)
        {
            log.LogInformation($"C# Queue trigger function processed: {depositRequest}");

            InvestementAllocation r = investementAllocator.Calculate(depositRequest.Amount, depositRequest.Investor);
        }
    }
}

At runtime, when the CalculatePortfolioAllocation runs, an instance of an NaiveInvestementAllocator will be supplied to the function.

The library also supports features such as named dependencies and multiple DI configurations, to read more check out GitHub.

SHARE:

Testing Precompiled Azure Functions Overview

Just because serverless allows us to quickly deploy value, it doesn’t mean that testing is now obsolete. (click to Tweet)

If we’re using Azure Functions as our serverless platform we can write our code (for example C#) and test it before deploying to Azure. In this case we’re talking about precompiled Azure Functions as opposed to earlier incarnations of Azure Functions that used .csx script files.

Working with precompiled functions means the code can be developed and tested on a local development machine. The code we write is familiar C# with some additional attributes to integrate the code with the Azure Functions runtime.

Because the code is just regular C#, we can use familiar testing tools such as MSTest, xUnit.net, or NUnit. Using these familiar testing frameworks it’s possible to write tests that operate at different levels of granularity.

One way to categorize these tests are into:

  • Unit tests to check core business logic/value
  • Integration tests to check function run methods are operating correctly
  • End-to-end workflow tests that check multiple functions working together

To enable effective automated testing it may be necessary to write functions in such a way as to make them testable, for example by allowing function run method dependencies to be automatically injected at runtime, whereas at test time mock versions can be supplied for example using a framework such as AzureFunctions.Autofac.

There are other tools that allow us to more easily test functions locally such as the local functions runtime and the Azure storage emulator.

To learn more about using these tools and techniques to test Azure Functions, check out my Pluralsight course Testing Precompiled Azure Functions: Deep Dive.

SHARE:

Automatic Input Blob Binding in Azure Functions from Queue Trigger Message Data

Reading additional blob content when an Azure Function is triggered can be accomplished by using an input blob binding by defining a parameter in the function run method and decorating it with the [Blob] attribute.

For example, suppose you have a number of blobs that need converting in some way. You could initiate a process whereby the list of blob files that need processing are added to a storage queue. Each queue message contains the name of the blob that needs processing. This would allow the conversion function to scale out to convert multiple blobs in parallel.

The following code demonstrates one approach to do this. The code is triggered from a queue message that contains text representing the input bob filename that needs reading, converting, and then outputting to an output blob container.

using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;

namespace FunctionApp1
{
    public static class ConvertNameCase
    {
        [FunctionName("ConvertNameCase")]
        public static void Run([QueueTrigger("capitalize-names")]string inputBlobPath)
        {
            string originalName = ReadInputName(inputBlobPath);

            var capitalizedName = originalName.ToUpperInvariant();

            WriteOutputName(inputBlobPath, capitalizedName);
        }
        
        private static string ReadInputName(string blobPath)
        {
            CloudStorageAccount account = CloudStorageAccount.DevelopmentStorageAccount;
            CloudBlobClient blobClient = account.CreateCloudBlobClient();
            CloudBlobContainer container = blobClient.GetContainerReference("names-in");

            var blobReference = container.GetBlockBlobReference(blobPath);

            string originalName = blobReference.DownloadText();

            return originalName;
        }

        private static void WriteOutputName(string blobPath, string capitalizedName)
        {
            CloudStorageAccount account = CloudStorageAccount.DevelopmentStorageAccount;
            CloudBlobClient blobClient = account.CreateCloudBlobClient();
            CloudBlobContainer container = blobClient.GetContainerReference("names-out");

            CloudBlockBlob cloudBlockBlob = container.GetBlockBlobReference(blobPath);
            cloudBlockBlob.UploadText(capitalizedName);            
        }

    }
}

In the preceding code, there is a lot of blob access code (which could be refactored). This function could however be greatly simplified by the use of one of the built-in binding expression tokens. Binding expression tokens can be used in binding expressions and are specified inside a pair of curly braces {…}. The {queueTrigger} binding token will extract the content of the incoming queue message that triggered a function.

For example, the code could be refactored as follows:

using System.IO;
using Microsoft.Azure.WebJobs;

namespace FunctionApp1
{
    public static class ConvertNameCase
    {
        [FunctionName("ConvertNameCase")]
        public static void Run(
        [QueueTrigger("capitalize-names")]string inputBlobPath,
        [Blob("names-in/{queueTrigger}", FileAccess.Read)] string originalName,
        [Blob("names-out/{queueTrigger}")] out string capitalizedName)
        {
                capitalizedName = originalName.ToUpperInvariant();         
        }
}

In the preceding code, the two [Blob] binding paths make use of the {queueTrigger} token. When the function is triggered, the queue message contains the name of the file to be processed. In the two [Blob] binding expressions, the {queueTrigger} token part will automatically be replaced with the text contents of the incoming message. For example if the message contained the text “File1.txt” then the two blob bindings would be set to names-in/File1.txt and names-out/File1.txt respectively. This means the input blob nameBlob string will automatically be read when the function is triggered,

To learn more about creating precompiled Azure Functions in Visual Studio, check out my Writing and Testing Precompiled Azure Functions in Visual Studio 2017 Pluralsight course.

SHARE: