Thursday, March 22, 2018

Azure Functions: Updating Azure Redis Cache from BLOB storage

Background

Couple of weeks back, one of colleagues specializing in Data Platform was working with a customer, who uses Azure to support many of their customer facing applications. They use Azure cache for one of application and the scenario the challenge they were facing was to keep the cache updated with part of the information coming from a legacy mainframe application.

The data was being exported as CSV file. The challenge was speed up the initial upload of the large file containing ~120 million records into cache. This process was taking nearly 10 hours based on the a console application built from a sample code found on the Internet which was running on their local network..

Then, there were also subsequent updates (smaller files), which were expected to run multiple times a day.

Upon evaluating code, we determined that the program was only writing one key at a time and had an over-engineered the use of threading. They also had a slow outbound network to Azure to contend with. 

So we decided to address these limitations in a single solution using Azure Functions. 

Solution 

Using Azure function and BLOB trigger, the cache update process is started as soon as new extracted file is available on BLOB storage. 

When the Azure Function detects the change, the BLOB is provided as a file stream, which the function:
  • Reads line by line 
  • Converts the line into key-value pair,
  • Batches up the key-value pair into a configurable set.
  • Writes the batch to Azure Cache. 

Result

The result was quite encouraging. With a batch size of 2000 items, 120 million items was processed under 20 minutes. 

Code

Considering this a common scenario for applications using Azure Redis cache and solution that others may find useful, I have made the code available in GitHub.


Saturday, March 10, 2018

Combining Analytics and Serverless - Azure Functions and Azure Analysis Services

The "Serverless" computing is one of the term catching lots of attention. Linked with the advent of microservices and cloud computing, serverless probably represent the smallest unit of pay-for-use consumption model, scale on demand and code to meet a well scoped and limited purpose.

AWS, Azure and Google all have their implementation of serverless computing with support for various languages and runtime.

As a practical application of the serverless computing paradigm, I had the opportunity to automate operations for an Analytics application built on Azure Analysis Services.

As part of building a quick turnaround reporting and analytics platform for multi-national beverage company, the tables in Azure Analysis Services (AAS) tabular model had to be processed as soon as the ingestion step was completed.

The limitation we ran into was that the orchestration engine used in this case was running on Linux and could not launch Microsoft Windows native executables.

So in comes, Azure Functions with HTTP triggers to the rescue, which allow processing of the tabular model - tables and partitions on demand. Additionally automate creating and merging of partitions as the business requirement demanded.

There are other examples of a similar approach being taken, but they use Timer triggers to perform the operations. (Orsolya Gal's Azure Blog, SqlDusty) or have HTTP Triggers with specific example (Ust Oldfield's Blog)

I believe, this Azure Function App code is more comprehensive and flexible combining the approaches above and has been production tested.

It would be quite useful for developers who find themselves in this situation, so I decided to open source the foundational code which can used to build more complex operations.


The project is titled - Azure Function App for Azure Analysis Services Tabular Model Operations.


Find the source code on GitHub.

Feel free to download, fork and comment. Looking forward to hear back from the community.


BlogSpot - 10 years later

Wow, it is amazing (and sad) to realize that I have not blogged in this spot for almost 10 years.

My last post being in October 2008. Just before I was made to realize another fragility of human life. It was in November 2008, that I was made aware that my father - Dr. Luxmi Kant Banerjea, the tough man of my life, was suffering from an aggressive form of lymphoma. For a person, who had never been in hospital bed other than when he was born, it was big change. Even bigger change, as he rapidly bed ridden by this debilitating disease.

As things played out for good, bad and ugly, he passed away, less than three months later in February 2009. I was not there by his side, I was back in US trying to see if I get second opinions from doctor's here. Maybe balance my time throughout the year with my sister who lived in the UK and me take turns helping out in his treatment. But it was not to be. Now with after nine years, I still miss him. But I also feel his presence never left me. I do wonder, if I can half as good parent has he was, my own kids would turn out just fine.

What was amazing to see was his legacy. After his passing, it seems people came out of the woodwork to help our family. They claimed my father had helped them when their chips were down and sometimes gone above and beyond to make sure they were taken care of, while staying within the bounds of honesty and the law. My father being a quite, reserved man never shared these stories of kindness or empathy with us, but it did seem to have rubbed off a little bit on me.

I also noticed an unfinished post from May 2009, I started writing about my father's life. But I guess that will be for another time.

Now that that I am well past grieving and life has moved on. I had my second daughter in December 2009 which completed the circle of life, so as to speak. I went full time on my own venture and then joined my dream company - Microsoft in July 2011.

Last seven years at working at Microsoft has been has been an interesting experience. I remember sending out my resume by post (aka snail mail) in 1999 as soon as I graduated from my masters to Microsoft in Redmond, WA. Now I did receive a post card back from HR saying they would consider me for future position. Boy, it took them - 12 years but I was there, finally. Well, I would hope that is how it played out... :)

Microsoft as awesome as it is, was different from what I imagined it to be. As big corporations are it has it fallacies. Probably back in 2011, too complacent in past years success, at one time considered an evil empire, dark side (for Star Wars fan), but you could feel the winds of change. Humbled by the tenuous and smart competition - Google, Amazon, Apple, who had customer mindshare, we were the big white drunk mad elephant. But as we know from Louis V.Gerstner Jr.'s book - "Who say's elephants can't dance? (Harper Collins, November 2002)", we know the mad elephants still has some tricks up its trunk".... :)

Fast forward into 2018, under leadership of Satya Nadella (since February 2014), Microsoft is forging new grounds into Cloud Computing, Artificial Intelligence, Mixed Reality and Quantum Computing. Well, Satya even wrote a book about it - "Hit Refresh (Harper Collins, September 2017)", if you have not had a chance to read it.

The biggest change at Microsoft - emphasis on openness (not just open source, that work started before the CEO change, but has new vigor now), leveraging work of others - within and outside Microsoft and most of all empathy - toward customers, partners and colleagues. I cannot say for sure if the change has come to every part of the organization but the new attitude is contagious.

One word to catches my attention - empathy and to that I go back to my late father, as evidence by people who came to help us, after he passed way, he had lots of it.

I think he applied a great deal of empathy and compassion to his work everyday. As a person responsible for lives of hundreds of people who reported to him and those his reports effected, he went through his daily rigors without much worry about how is legacy would be perceived, just how best it would make the lives he touched better.

All I can hope is that some of it rubbed on me as well.