Push Notifications and Buttons with Microsoft Flow: Part 3

In part 1 we crated a Flow to enable/disable the sending of push notifications and in part 2 we created an Azure Function to generate random phrases of positivity.

In this third and final part of this series we’ll go and create the second Flow that sends the push notifications to the phone.

This Flow will be automatically triggered every 15 minutes by using a Recurrence action followed by an action to get the blob content containing whether or not we should send a notification as the following screenshot shows:

Running a Microsoft Flow every 15 minutes

Now that we have the blob content, we can examine it in an If condition. If the content of the blob is “enabled” we can continue the Flow:

If condition in Microsoft Flow

If the condition is satisfied (“IF YES”) two actions are  performed: the first an HTTP GET to the Azure function, the second a push notification action that as content uses what was returned in the body of the HTTP GET. Notice in the HTTP GET, the URI includes the function authorization key as a query string parameter.

flow4

After saving the new Flow, we can head back to the Flow app and hit the button to enable “positivity pushes”:

Microsoft Flow iOS app button

Then every 15 minutes (until we turn them off by hitting the button again) we’ll get a positivity notification on the phone:

iOS push notifications from Microsoft Flow

Push Notifications and Buttons with Microsoft Flow: Part 2

In part 1 we created a Flow to toggle the sending of push notifications on and off by storing the configuration in Azure blob storage.

Now that we have a way of enabling/disabling notifications we can start to build the second Flow.

Before jumping into the Flow designer, we need to consider how to generate random positivity phrases and how to integrate this into the second Flow. One option to do this is to create a simple Azure Function with an HTTP trigger. The Flow can then use an HTTP action to issue a GET to the server that will return the string content to be sent via push notifications.

In the Azure Portal function editor a new function can be created with a HTTP trigger configured to GET only as the following screenshot shows:

Creating an Azure Function with a HTTP trigger

Notice in the preceding screenshot the authorization level has bee set to “function”. This means the key needs to be provided when the function is called.

We can now write some code in the function code editor window as follows:

using System.Net;

public static HttpResponseMessage Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("C# HTTP trigger function processed a request.");

    string phrase = GeneratePhrase();

    return req.CreateResponse(HttpStatusCode.OK, phrase);
}

public static string  GeneratePhrase()
{
    var phrases = new string[]
    {
        "Don't worry, be happy :)",
        "All is well",
        "Will it matter in 100 years?",
        "Change what you can, don't worry about what you can't"
    };

    var rnd = new Random();
    
    return phrases[rnd.Next(phrases.Length)];
}

Azure Function code editor window

Clicking the Run button will test the function and we can see the random phrases being returned with a 200 status as shown in the following screenshot:

Azure function HTTP test output

In the final part of  this series we’ll go and create the second Flow that uses this function and the configuration value created in the previous article to actually send random positivity push notifications on a 15 minute schedule.

Push Notifications and Buttons with Microsoft Flow: Part 1

Microsoft Flow allows the creation of serverless cloud workflows. It is similar to services such as If This Then That and has more of a business focus. It allows custom Flows to integrate with Azure services such as blob storage, the calling of arbitrary HTTP services, in addition to a whole host of services such as Facebook, Dropbox, OneDrive, etc.

In addition to executing in the cloud, Flows can create push notification to the Flow app on iOS and Android.

Once installed, the Flow app can be used to design/edit Flows, view Flow activity/recent executions, and initiate the execution of flows via “buttons” as shown in the following screenshot.

Microsoft Flow iOS app

Tapping this software button will trigger the Flow in the cloud.

Example Scenario

To see buttons and push notification in action, imagine a scenario where sometimes you want cheering up with regular messages of positivity.

In this scenario, when enabled, you’ll get a push notification on your phone every 15 minutes with a random positive phase such as “Don't worry, be happy :)”.

To accomplish this two separate (but related) Flows can be created. The first Flow uses a button in the phone Flow app to toggle wether the “positivity pushes” will be sent. The second Flow is triggered automatically every 15 minutes and if enabled, sends a push notification.

Creating the Toggle Positivity Push Flow

This Flow will enable/disable the push notifications. To do this, a manual button trigger will be added to the Flow that will be pushed on the phone. To hold the enabled/disabled state, we can use the content of a blob in Azure blob storage. When triggered, the Flow will retrieve the content of the blob which can be the string “enabled” or “disabled”.

Microsoft Flow reading content from blob storage

Once the blob content has been retrieved, its content can be examined in a If condition. If the content of the blob is currently “enabled” it will be updated to “disabled” and vice versa. Finally we’ll send a push notification to confirm the state.

Microsoft Flow examining blob content

Pressing the button in the app a couple of times results in the expected push notifications:

Microsoft Flow sending push notifications to iOS

The blob content also gets toggled as expected:

Blob content being toggled from Microsoft Flow

In part 2 of this series we’ll start the process of creating another Flow to actually send the random positivity phrases to the phone.

Custom Session Logging in Marten

Marten is a .NET document database library that uses an underlying PostgreSQL database to store objects as JSON. The library has a variety of features that allow the logging of SQL statements issued to the underlying PostgreSQL including previewing LINQ query SQL statements.  One of the other logging features available allows custom logging to be created for individual session operations such as successfully issued database SQL commands, failed commands, and changes that were saved. There are also numerous other logging/extension points that can be utilized such as logging schema change SQL and automatically using a logger for all sessions.

The following code shows a console application that writes two customers and then retrieves them:

using System;
using static System.Console;
using Marten;

namespace MartenFKDemo
{
    class Customer
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public override string ToString() => $"{Id} {Name}";
    }

    class Program
    {
        static void Main(string[] args)
        {
            const string connectionString = "host = localhost; database = OrderDb; password = g7qo84nck22i; username = postgres";

            var store = DocumentStore.For(connectionString);

            WriteLine("Creating new customer");
            using (var session = store.OpenSession())
            {
                session.Store(new Customer {Name = "Amrit"}, new Customer {Name = "Sarah"});

                session.SaveChanges();
            }


            WriteLine("All customers:");
            using (var session = store.QuerySession())
            {
                foreach (Customer customer in session.Query<Customer>())
                {
                    WriteLine(customer);
                }
            }
            
            ReadLine();
        }
    }
}

Running the application results in the following output:

Creating new customer
All customers:
9001 Amrit
9002 Sarah

To create a custom session logger, the IMartenSessionLogger interface can be implemented, a simple version that logs to the console is shown as follows:

class ColorConsoleLogger : IMartenSessionLogger
{
    public void LogSuccess(Npgsql.NpgsqlCommand command)
    {
        ForegroundColor = ConsoleColor.Green;

        WriteLine(command.CommandText); // additional properties (e.g. SQL parameters) are available
    }

    public void LogFailure(Npgsql.NpgsqlCommand command, Exception ex)
    {
        ForegroundColor = ConsoleColor.Red;

        WriteLine(command.CommandText); // additional properties (e.g. SQL parameters) are available
        WriteLine(ex);
    }

    public void RecordSavedChanges(IDocumentSession session, IChangeSet commit)
    {
        ForegroundColor = ConsoleColor.Gray;

        foreach (object insertedItem in commit.Inserted) //  updated/deleted are also available
        {
            WriteLine(insertedItem);
        }
    }
}

To configure individual sessions to use this logger, the sessions Logger property can be set as the following modified code demonstrates:

ResetColor();
WriteLine("Creating new customer");
using (var session = store.OpenSession())
{
    // Set logger for this session
    session.Logger = new ColorConsoleLogger();

    session.Store(new Customer {Name = "Amrit"}, new Customer {Name = "Sarah"});

    session.SaveChanges();
}

ResetColor();
WriteLine("All customers:");
using (var session = store.QuerySession())
{
    // Set logger for this session
    session.Logger = new ColorConsoleLogger();

    foreach (Customer customer in session.Query<Customer>())
    {
        ResetColor();
        WriteLine(customer);
    }
}

This produces the following output:

Creating new customer
select public.mt_upsert_customer(doc := :p0, docDotNetType := :p1, docId := :p2, docVersion := :p3);select public.mt_upsert_customer(doc := :p4, docDotNetType := :p5, docId := :p6, docVersion := :p7);
13001 Amrit
13002 Sarah
All customers:
select d.data, d.id, d.mt_version from public.mt_doc_customer as d
13001 Amrit
13002 Sarah

screenshot of Marten custom logger

To learn more about the document database features of Marten check out my Pluralsight courses: Getting Started with .NET Document Databases Using Marten and Working with Data and Schemas in Marten or the documentation.

What Would Easy Be Like?

As another (Gregorian) year edges ever closer, it’s a period that offers a natural reflection time on what happened in the past year and the forging, often weakly, of “New Years Resolutions”.

One interesting question to ask for your work (or personal) life is: “what would easy be like?”.

For example, looking back to the past year where was the pain, sadness, frustration, long hours, constant overtime, feelings of falling behind in new technologies, failed deployments, the same bugs re-occurring over and over again, unsupportive management, mean co-workers, feelings of inadequacy, disappointed customers, etc.

you should feel like you’re doing your best work

Take some time. Make a list. Write them all down.

Now imagine what it would feel like if all of the things on the list did not exist in the upcoming year – what would easy be like?

Fixing some of the things may mean radical decisions such as quitting your job and moving employer.

Some things may not be quite as radical: better automated testing to catch bugs earlier.

You probably won’t be able to fix everything on your list. Starting from the point of view that things should be easy, that you should feel like you’re doing your best work, can be a powerful way to frame where you’re currently at and where you want to be by this time next year.

Previewing the Generated PostgreSQL SQL for a Query in Marten

Marten is a .NET document database library that uses an underlying PostgreSQL database to store objects as JSON. The library has a variety of features that allow the logging of SQL statements issued to the underlying PostgreSQL database in addition to being able to do things such as get the PostgreSQL query plan for a given LINQ query.

One simple way to get the generated SQL for a Marten LINQ query is to use the ToCommand() extension method.

As an example, suppose we are developing some query code as follows (this code uses the Include method to include the related documents in a single database round-trip):

Customer customer = null;

List<Order> orders = session.Query<Order>()
                            .Include<Customer>(joinOnOrder => joinOnOrder.CustomerId, includedCustomer => customer = includedCustomer)
                            .Where(x => x.CustomerId == 4001).ToList();

If we want to get an idea of what SQL Marten will generate for this LINQ query, we can change the code as shown in the following:

Customer customer = null;

IQueryable<Order> orders = session.Query<Order>()
                                  .Include<Customer>(joinOnOrder => joinOnOrder.CustomerId, includedCustomer => customer = includedCustomer)
                                  .Where(x => x.CustomerId == 4001);

// Get the SQL command that will be issued when the query executes
NpgsqlCommand cmd = orders.ToCommand();

// Output some selected command info
Console.WriteLine(cmd.CommandText);

foreach (NpgsqlParameter parameter in cmd.Parameters)
{
    Console.WriteLine($"Parameter {parameter.ParameterName} = {parameter.Value}");
}

// Ensure included customer variable is populated
List<Order> orderResults = orders.ToList();

Console.WriteLine(customer.Name);
foreach (Order order in orderResults)
{
    Console.WriteLine($" Order {order.Id} for {order.Quantity} items");
}

Running this preceding code  results in the following console output:

select d.data, d.id, d.mt_version, customer_id.data, customer_id.id, customer_id.mt_version from public.mt_doc_order as d INNER JOIN public.mt_doc_customer as customer_id ON d.customer_id = customer_id.id where d.customer_id = :arg0
Parameter arg0 = 4001
Sarah
 Order 3001 for 42 items
 Order 4001 for 477 items
 Order 5001 for 9 items

To learn more about the document database features of Marten check out my Pluralsight courses: Getting Started with .NET Document Databases Using Marten and Working with Data and Schemas in Marten.

Retrieving Raw JSON Data in Web API with Marten

Marten is a .NET document database library that uses an underlying PostgreSQL database to store objects as JSON.

Ordinarily, Marten takes care of retrieving the JSON from the database and deserializing it into an object. We can however instruct Marten to perform a query to retrieve document(s) and not perform the deserialization, instead giving us the JSON as it appears in the underlying PostgreSQL record. If we are exposing documents via a Web API, this feature can be taken advantage of to reduce some processing overhead on the server.

As an example, we could have the following Customer document defined:

public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; }

    // etc.
}

And in a CustomersController we could start with a method as follows to add Customers to the database:

public void Post(Customer customer)
{
    // no validation

    // DocumentStore would normally be created only once in app, e.g. via IOC singleton 
    using (var store = DocumentStore.For(ConnectionString))
    {
        using (var session = store.LightweightSession())
        {
            session.Store(customer);
            session.SaveChanges();
        }
    }
}

To get all Customers, the following method could be written:

// GET api/customers
public IEnumerable<Customer> Get()
{
    // DocumentStore would normally be created only once in app, e.g. via IOC singleton 
    using (var store = DocumentStore.For(ConnectionString))
    {
        using (var session = store.QuerySession())
        {
            return session.Query<Customer>();
        }
    }
}

The preceding method however incurs the additional overhead of Marten deserializing the database JSON into Customer objects, only to be serializing it again back into JSON on the way out of the API.

When creating the LINQ query, the Marten ToJsonArray() method can be added to instruct Marten to simply return the JSON directly from the database.

We can then modify the Get method as follows:

public HttpResponseMessage Get()
{
    // DocumentStore would normally be created only once in app, e.g. via IOC singleton 
    using (var store = DocumentStore.For(ConnectionString))
    {
        using (var session = store.QuerySession())
        {
            string rawJsonFromDb = session.Query<Customer>().ToJsonArray();

            var response = Request.CreateResponse(HttpStatusCode.OK);
            response.Content = new StringContent(rawJsonFromDb, Encoding.UTF8, "application/json");
            return response;
        }
    }
}

We could also write the parameterized Get method and use Marten’s AsJson() method to get the JSON string for the individual Customer document as  in the following code:

public HttpResponseMessage Get(int id)
{
    // DocumentStore would normally be created only once in app, e.g. via IOC singleton 
    using (var store = DocumentStore.For(ConnectionString))
    {
        using (var session = store.LightweightSession())
        {
            var rawJsonFromDb = session.Query<Customer>().Where(x => x.Id == id).AsJson().FirstOrDefault();

            if (string.IsNullOrEmpty(rawJsonFromDb))
            {
                throw new HttpResponseException(HttpStatusCode.NotFound);
            }

            var response = Request.CreateResponse(HttpStatusCode.OK);
            response.Content = new StringContent(rawJsonFromDb, Encoding.UTF8, "application/json");
            return response;
        }
    }
}

To learn more about the document database features of Marten check out my Pluralsight courses: Getting Started with .NET Document Databases Using Marten and Working with Data and Schemas in Marten.

New Pluralsight Course: Working with Data and Schemas in Marten

Marten is a .NET document database library to allows objects to be stored, retrieved, and queried as documents stored as JSON in an underlying PostgreSQL database. This new course is a follow-on from the previous Getting Started with .NET Document Databases Using Marten course, if you’re new to Marten I’d recommend checking out the previous course first before continuing with this new one.

Among other topics, this new course covers how to log/diagnose the SQL that is being issued to PostgreSQL; how to enable offline optimistic concurrency; bulk document inserts; a number of ways to improve query performance; and the customization of database schema objects.

You can check out the new course on the Pluralsight site.

Screen Scraping As A Service with Azure Functions in 5 Mins

If you have some data in a web page but there is no API to get the same data, it’s possible to use (the often brittle and error prone) technique of screen scraping to read the values out of the HTML.

By leveraging Azure Functions, it’s trivial (depending on how horrendous the HTML is) to create a HTTP Azure Function that loads a web page, parses the HTML, and returns the data as JSON.

The fist step is to create an HTTP-triggered Azure Function:

Creating a new Azure Function with a HTTP Trigger

To help with the parsing, we can use the HTML Agility Pack NuGet package. To add this to the function, create a project.json file and add the NuGet reference:

Using NuGet Packages in Azure Functions

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "HtmlAgilityPack": "1.4.9.5"
      }
    }
   }
}

Once this file is saved, the NuGet package will be installed and a using directive can be added: using HtmlAgilityPack;

Now we can download the required HTML page as a string, in the example below the archive page from Don’t Code Tired, use some LINQ to get the post titles, and return this as the HTTP response. Correctly selecting/parsing the required data from the HTML page is likely to be the most time-consuming part of the function creation.

The full function source code is as follows. Notice that there’s no error checking code to simplify the demo code. Screen scraping should usually be a last resort because of its very nature it can often break if the UI changes or unexpected data exists in the HTML. It also requires the whole page of HTML be downloaded which may raise performance concerns.

using System.Net;
using HtmlAgilityPack;

public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    HttpClient client = new HttpClient();

    string html = await client.GetStringAsync("http://dontcodetired.com/blog/archive");    

    HtmlDocument doc = new HtmlDocument();
    doc.LoadHtml(html); 
    
    var postTitles = doc.DocumentNode
                        .Descendants("td")
                        .Where(x => x.Attributes.Contains("class") && x.Attributes["class"].Value.Contains("title"))
                        .Select(x => x.InnerText);

    return req.CreateResponse(HttpStatusCode.OK, postTitles);
}

We can now call the function via HTTP and get back a list of all Don’t Code Tired article titles as the following Postman screenshot shows:

Using Postman to call Azure Functions

Azure HTTP Function Authorization with Function Keys

When creating an Azure Function triggered via HTTP, one way to authorize use of the function is to configure the HTTP function trigger to require the caller to provide a function key.

Azure Function HTTP Trigger Authorization Modes

With the authorization set to Anonymous, as expected anyone can call it.

When set to Function Authorization, the caller needs to provide the function key either as a URL query string parameter or in a header.

The function key can be found by navigating to Manage tab as the following screenshot shows:

Finding the Azure Function Key

Once Function Authorization is enabled, if the client does not provide it correctly the function will return a 401 Unauthorized.

To supply the function key in the URL, the “code” query string parameter can be used, e.g. “https://myazurecloudfunctions.azurewebsites.net/api/SayHi?code=udXhf3pviSICFMtViW/pqmV/1Q5vLH5aMcRWXfD/q6NXk2VVxRlfYw==”.

Alternatively an “x-functions-key” header can be added containing the key as the following Postman screenshot shows:

Calling Azure Function with Postman and Function Key