Showing posts with label GitHub. Show all posts
Showing posts with label GitHub. Show all posts

Wednesday, April 04, 2018

Azure Functions and Azure Redis Cache - Making integration Simpler

Introduction

Azure Functions provides integration out of the box with several Azure components such as Azure storage - Queues, Table and BLOBs, Event Hubs and has support for CRON timers. It also has an extensibility SDK, which allows many more providers of first and third-party service such as Cosmos DB, Twillio, SendGrid to integrate with Azure Functions as either triggers or bindings.

Having used Azure Redis Cache with Azure Functions in couple of projects, I missed the ease and code reduction and cleanliness that binding provides. So, I decided to use the extensibility SDK for Azure Functions v2 and build one myself and share it with community.

Caveat

At the time of writing this blog entry - April 2018, the Azure Functions v2 SDK is currently in beta and may encounter breaking changes. Not recommended for production yet.


Visual Studio Tooling 

To be able to use the code successfully in Visual Studio, you must have Azure Functions and WebJobs Extension version 15.0.40405.0 or higher.

Code Repository

The code for this WebJobs Extension is available on GitHub  https://github.com/codepossible/WebJobs.Extensions.AzureRedis

Usage

Azure Redis Cache as Async Collector

The Async Collector is the most common use for most target bindings. In the case of Azure Redis Cache, that would be updating the cache - adding or updating existing keys.

The extensibility SDK provides a simple way to achieve this using interface. - IAsyncCollector, which requires implementation of two methods - AddAsync and FlushAsync.

The most common efficient of writing items to a store and cache being no exception is to write it in batches (or buffered writes).

The implementation of the Async collector can include an internal collection to support buffered writes, where AddAsync method would write to a internal collection and performed a buffered/batch update, once the item count reaches a certain limit (for example 100 items) or other event such as a timer. User can also perform a forced write as FlushAsync method.

In most cases for cache, values to be updated immediately.

So to simply things, I chose to create two constructs - interface called -IRedisCacheItem and a class called RedisCacheItemsBatch which is just a wrapper generic list of IRedisCacheItem instances. (The beta version of SDK support does not support nested collection for Async Collectors).

    public interface IRedisCacheItem
    {
        RedisKey Key { get; set; }
        RedisValue Value { get; set; }
    }

    public class RedisCacheItemsBatch
    {
        private List _items = new List();
        public List Items
        {
            get { return _items; }
        }
    }

It is the RedisCacheItemsBatch class which is bound to Collector. An example signature of the Azure Function would be.

public static async Task Run(
  [BlobTrigger("%SourceBlob%", Connection =  "BlobStorageConnection")] Stream myBlob,
  [AzureRedis] IAsyncCollector redisCache,
  TraceWriter log
){ ...

In the sample code, the Azure Function is triggered by update to a CSV file stored in a BLOB storage, which is assumed to contain a comma-delimited key-value pair.

Azure Redis Cache as IDatabase

To use more richer native functionality to the Redis Cache, the extension also allows binding to an instance - StackExchange.Redis.IDatabase interface as the Redis Cache client.

Though not required, IRedisCacheItem can help simplify the code. The sample Azure Function code uses the IDatabase Binding to accomplish the same updates.

An example signature of the Azure Function would be.

public static async Task Run(
  [BlobTrigger("%SourceBlob%", Connection =  "BlobStorageConnection")] Stream myBlob,
  [AzureRedis] IDatabase redisCache,
  TraceWriter log
){ ...


Azure Redis Cache Configuration

The Azure Redis Cache binding requires a connection string to the Azure Redis Cache instance to work. This can be specified inline in the function or enclosing the name of AppSettings key within the "%" signs.

If not specified, the code looks for a value for the Redis Cache connection string in AppSettings under the name of - "AzureWebJobsAzureRedisConnectionString"

Feedback and Sharing

If you find the code useful, please share it with your fellow developers, Twitter followers, Facebook groups, Google+ Circle and other social media networks.

Since this is built on a beta release of the SDK, updates and breaking changes are expected. If that happens, please raise an issue in the GitHub project - https://github.com/codepossible/WebJobs.Extensions.AzureRedis

Noteworthy

Another colleague of mine at Microsoft - Francisco Beltrao in Switzerland, has also written WebJobs Extensions which include one for Redis Cache. Do check out Francisco's work at https://github.com/fbeltrao/AzureFunctionExtensions 

Saturday, March 10, 2018

Combining Analytics and Serverless - Azure Functions and Azure Analysis Services

The "Serverless" computing is one of the term catching lots of attention. Linked with the advent of microservices and cloud computing, serverless probably represent the smallest unit of pay-for-use consumption model, scale on demand and code to meet a well scoped and limited purpose.

AWS, Azure and Google all have their implementation of serverless computing with support for various languages and runtime.

As a practical application of the serverless computing paradigm, I had the opportunity to automate operations for an Analytics application built on Azure Analysis Services.

As part of building a quick turnaround reporting and analytics platform for multi-national beverage company, the tables in Azure Analysis Services (AAS) tabular model had to be processed as soon as the ingestion step was completed.

The limitation we ran into was that the orchestration engine used in this case was running on Linux and could not launch Microsoft Windows native executables.

So in comes, Azure Functions with HTTP triggers to the rescue, which allow processing of the tabular model - tables and partitions on demand. Additionally automate creating and merging of partitions as the business requirement demanded.

There are other examples of a similar approach being taken, but they use Timer triggers to perform the operations. (Orsolya Gal's Azure Blog, SqlDusty) or have HTTP Triggers with specific example (Ust Oldfield's Blog)

I believe, this Azure Function App code is more comprehensive and flexible combining the approaches above and has been production tested.

It would be quite useful for developers who find themselves in this situation, so I decided to open source the foundational code which can used to build more complex operations.


The project is titled - Azure Function App for Azure Analysis Services Tabular Model Operations.


Find the source code on GitHub.

Feel free to download, fork and comment. Looking forward to hear back from the community.