Feedbacks and Enhancements to Azure Functions for Azure Analysis Services

I am happy to report that after I released the Azure Functions to perform operations on Azure Analysis Services early March 2018 (as my first GitHub projects), it has been tried out and used in several places and I have received lots of feedback.

Based on that I have been able to make several fixes and enhancements to solution including integration with Azure Data Factory.

Today, I added a more significant feature which the client can specify multiple tables to be processed in a single request in synchronous or asynchronous manner.

Thanks to suggestion to one of the users, it made the change quite easy.

Endpoints For Multiple Table Process

The new signature for processing table is:

GET /ProcessTabularModel/{Database}/tables/{tableList}

GET /ProcessTabularModel/{Database}/tables/{tableList}/async

Example:

https://azf-processtabularmodel.azurewebsites.net/ProcessTabularModel/AdventureWorks/tables/DimAccount, DimProduct?code=vxdmfmm45esfau3ddjffd

https://azf-processtabularmodel.azurewebsites.net/ProcessTabularModel/AdventureWorks/tables/DimAccount, DimProduct/async?code=vxdmfmm45esfau3ddjffd

Looking forward to more users and feedback. The Github repo is available at - https://github.com/codepossible/AzFunctions-AASOperations

Azure Functions and Azure Redis Cache - Making integration Simpler

Introduction

Azure Functions provides integration out of the box with several Azure components such as Azure storage - Queues, Table and BLOBs, Event Hubs and has support for CRON timers. It also has an extensibility SDK, which allows many more providers of first and third-party service such as Cosmos DB, Twillio, SendGrid to integrate with Azure Functions as either triggers or bindings.

Having used Azure Redis Cache with Azure Functions in couple of projects, I missed the ease and code reduction and cleanliness that binding provides. So, I decided to use the extensibility SDK for Azure Functions v2 and build one myself and share it with community.

Caveat

At the time of writing this blog entry - April 2018, the Azure Functions v2 SDK is currently in beta and may encounter breaking changes. Not recommended for production yet.

Visual Studio Tooling 

To be able to use the code successfully in Visual Studio, you must have Azure Functions and WebJobs Extension version 15.0.40405.0 or higher.

Code Repository

The code for this WebJobs Extension is available on GitHub  https://github.com/codepossible/WebJobs.Extensions.AzureRedis

Usage

Azure Redis Cache as Async Collector

The Async Collector is the most common use for most target bindings. In the case of Azure Redis Cache, that would be updating the cache - adding or updating existing keys.

The extensibility SDK provides a simple way to achieve this using interface. - IAsyncCollector, which requires implementation of two methods - AddAsync and FlushAsync.

The most common efficient of writing items to a store and cache being no exception is to write it in batches (or buffered writes).

The implementation of the Async collector can include an internal collection to support buffered writes, where AddAsync method would write to a internal collection and performed a buffered/batch update, once the item count reaches a certain limit (for example 100 items) or other event such as a timer. User can also perform a forced write as FlushAsync method.

In most cases for cache, values to be updated immediately.

So to simply things, I chose to create two constructs - interface called -IRedisCacheItem and a class called RedisCacheItemsBatch which is just a wrapper generic list of IRedisCacheItem instances. (The beta version of SDK support does not support nested collection for Async Collectors).

    public interface IRedisCacheItem
    {
        RedisKey Key { get; set; }
        RedisValue Value { get; set; }
    }

    public class RedisCacheItemsBatch
    {
        private List _items = new List();
        public List Items
        {
            get { return _items; }
        }
    }

It is the RedisCacheItemsBatch class which is bound to Collector. An example signature of the Azure Function would be.

public static async Task Run(
  [BlobTrigger("%SourceBlob%", Connection =  "BlobStorageConnection")] Stream myBlob,
  [AzureRedis] IAsyncCollector redisCache,
  TraceWriter log
){ ...

In the sample code, the Azure Function is triggered by update to a CSV file stored in a BLOB storage, which is assumed to contain a comma-delimited key-value pair.

Azure Redis Cache as IDatabase

To use more richer native functionality to the Redis Cache, the extension also allows binding to an instance - StackExchange.Redis.IDatabase interface as the Redis Cache client.

Though not required, IRedisCacheItem can help simplify the code. The sample Azure Function code uses the IDatabase Binding to accomplish the same updates.

An example signature of the Azure Function would be.

public static async Task Run(
  [BlobTrigger("%SourceBlob%", Connection =  "BlobStorageConnection")] Stream myBlob,
  [AzureRedis] IDatabase redisCache,
  TraceWriter log
){ ...

Azure Redis Cache Configuration

The Azure Redis Cache binding requires a connection string to the Azure Redis Cache instance to work. This can be specified inline in the function or enclosing the name of AppSettings key within the "%" signs.

If not specified, the code looks for a value for the Redis Cache connection string in AppSettings under the name of - "AzureWebJobsAzureRedisConnectionString"

Feedback and Sharing

If you find the code useful, please share it with your fellow developers, Twitter followers, Facebook groups, Google+ Circle and other social media networks.

Since this is built on a beta release of the SDK, updates and breaking changes are expected. If that happens, please raise an issue in the GitHub project - https://github.com/codepossible/WebJobs.Extensions.AzureRedis

Noteworthy

Another colleague of mine at Microsoft - Francisco Beltrao in Switzerland, has also written WebJobs Extensions which include one for Redis Cache. Do check out Francisco's work at https://github.com/fbeltrao/AzureFunctionExtensions 

Azure Functions: Updating Azure Redis Cache from BLOB storage

Background

Couple of weeks back, one of colleagues specializing in Data Platform was working with a customer, who uses Azure to support many of their customer facing applications. They use Azure cache for one of application and the scenario the challenge they were facing was to keep the cache updated with part of the information coming from a legacy mainframe application.

The data was being exported as CSV file. The challenge was speed up the initial upload of the large file containing ~120 million records into cache. This process was taking nearly 10 hours based on the a console application built from a sample code found on the Internet which was running on their local network..

Then, there were also subsequent updates (smaller files), which were expected to run multiple times a day.

Upon evaluating code, we determined that the program was only writing one key at a time and had an over-engineered the use of threading. They also had a slow outbound network to Azure to contend with. 

So we decided to address these limitations in a single solution using Azure Functions. 

Solution 

Using Azure function and BLOB trigger, the cache update process is started as soon as new extracted file is available on BLOB storage. 

When the Azure Function detects the change, the BLOB is provided as a file stream, which the function:

• Reads line by line 
• Converts the line into key-value pair,
• Batches up the key-value pair into a configurable set.
• Writes the batch to Azure Cache. 

Result

The result was quite encouraging. With a batch size of 2000 items, 120 million items was processed under 20 minutes. 

Code

Considering this a common scenario for applications using Azure Redis cache and solution that others may find useful, I have made the code available in GitHub.

Combining Analytics and Serverless - Azure Functions and Azure Analysis Services

The "Serverless" computing is one of the term catching lots of attention. Linked with the advent of microservices and cloud computing, serverless probably represent the smallest unit of pay-for-use consumption model, scale on demand and code to meet a well scoped and limited purpose.

AWS, Azure and Google all have their implementation of serverless computing with support for various languages and runtime.

As a practical application of the serverless computing paradigm, I had the opportunity to automate operations for an Analytics application built on Azure Analysis Services.

As part of building a quick turnaround reporting and analytics platform for multi-national beverage company, the tables in Azure Analysis Services (AAS) tabular model had to be processed as soon as the ingestion step was completed.

The limitation we ran into was that the orchestration engine used in this case was running on Linux and could not launch Microsoft Windows native executables.

So in comes, Azure Functions with HTTP triggers to the rescue, which allow processing of the tabular model - tables and partitions on demand. Additionally automate creating and merging of partitions as the business requirement demanded.

There are other examples of a similar approach being taken, but they use Timer triggers to perform the operations. (Orsolya Gal's Azure Blog, SqlDusty) or have HTTP Triggers with specific example (Ust Oldfield's Blog)

I believe, this Azure Function App code is more comprehensive and flexible combining the approaches above and has been production tested.

It would be quite useful for developers who find themselves in this situation, so I decided to open source the foundational code which can used to build more complex operations.


The project is titled - Azure Function App for Azure Analysis Services Tabular Model Operations.

Find the source code on GitHub.

Feel free to download, fork and comment. Looking forward to hear back from the community.

BlogSpot - 10 years later

Wow, it is amazing (and sad) to realize that I have not blogged in this spot for almost 10 years.

My last post being in October 2008. Just before I was made to realize another fragility of human life. It was in November 2008, that I was made aware that my father - Dr. Luxmi Kant Banerjea, the tough man of my life, was suffering from an aggressive form of lymphoma. For a person, who had never been in hospital bed other than when he was born, it was big change. Even bigger change, as he rapidly bed ridden by this debilitating disease.

As things played out for good, bad and ugly, he passed away, less than three months later in February 2009. I was not there by his side, I was back in US trying to see if I get second opinions from doctor's here. Maybe balance my time throughout the year with my sister who lived in the UK and me take turns helping out in his treatment. But it was not to be. Now with after nine years, I still miss him. But I also feel his presence never left me. I do wonder, if I can half as good parent has he was, my own kids would turn out just fine.

What was amazing to see was his legacy. After his passing, it seems people came out of the woodwork to help our family. They claimed my father had helped them when their chips were down and sometimes gone above and beyond to make sure they were taken care of, while staying within the bounds of honesty and the law. My father being a quite, reserved man never shared these stories of kindness or empathy with us, but it did seem to have rubbed off a little bit on me.

I also noticed an unfinished post from May 2009, I started writing about my father's life. But I guess that will be for another time.

Now that that I am well past grieving and life has moved on. I had my second daughter in December 2009 which completed the circle of life, so as to speak. I went full time on my own venture and then joined my dream company - Microsoft in July 2011.

Last seven years at working at Microsoft has been has been an interesting experience. I remember sending out my resume by post (aka snail mail) in 1999 as soon as I graduated from my masters to Microsoft in Redmond, WA. Now I did receive a post card back from HR saying they would consider me for future position. Boy, it took them - 12 years but I was there, finally. Well, I would hope that is how it played out... :)

Microsoft as awesome as it is, was different from what I imagined it to be. As big corporations are it has it fallacies. Probably back in 2011, too complacent in past years success, at one time considered an evil empire, dark side (for Star Wars fan), but you could feel the winds of change. Humbled by the tenuous and smart competition - Google, Amazon, Apple, who had customer mindshare, we were the big white drunk mad elephant. But as we know from Louis V.Gerstner Jr.'s book - "Who say's elephants can't dance? (Harper Collins, November 2002)", we know the mad elephants still has some tricks up its trunk".... :)

Fast forward into 2018, under leadership of Satya Nadella (since February 2014), Microsoft is forging new grounds into Cloud Computing, Artificial Intelligence, Mixed Reality and Quantum Computing. Well, Satya even wrote a book about it - "Hit Refresh (Harper Collins, September 2017)", if you have not had a chance to read it.

The biggest change at Microsoft - emphasis on openness (not just open source, that work started before the CEO change, but has new vigor now), leveraging work of others - within and outside Microsoft and most of all empathy - toward customers, partners and colleagues. I cannot say for sure if the change has come to every part of the organization but the new attitude is contagious.

One word to catches my attention - empathy and to that I go back to my late father, as evidence by people who came to help us, after he passed way, he had lots of it.

I think he applied a great deal of empathy and compassion to his work everyday. As a person responsible for lives of hundreds of people who reported to him and those his reports effected, he went through his daily rigors without much worry about how is legacy would be perceived, just how best it would make the lives he touched better.

All I can hope is that some of it rubbed on me as well.