Combining Analytics and Serverless - Azure Functions and Azure Analysis Services

The "Serverless" computing is one of the term catching lots of attention. Linked with the advent of microservices and cloud computing, serverless probably represent the smallest unit of pay-for-use consumption model, scale on demand and code to meet a well scoped and limited purpose.

AWS, Azure and Google all have their implementation of serverless computing with support for various languages and runtime.

As a practical application of the serverless computing paradigm, I had the opportunity to automate operations for an Analytics application built on Azure Analysis Services.

As part of building a quick turnaround reporting and analytics platform for multi-national beverage company, the tables in Azure Analysis Services (AAS) tabular model had to be processed as soon as the ingestion step was completed.

The limitation we ran into was that the orchestration engine used in this case was running on Linux and could not launch Microsoft Windows native executables.

So in comes, Azure Functions with HTTP triggers to the rescue, which allow processing of the tabular model - tables and partitions on demand. Additionally automate creating and merging of partitions as the business requirement demanded.

There are other examples of a similar approach being taken, but they use Timer triggers to perform the operations. (Orsolya Gal's Azure Blog, SqlDusty) or have HTTP Triggers with specific example (Ust Oldfield's Blog)

I believe, this Azure Function App code is more comprehensive and flexible combining the approaches above and has been production tested.

It would be quite useful for developers who find themselves in this situation, so I decided to open source the foundational code which can used to build more complex operations.


The project is titled - Azure Function App for Azure Analysis Services Tabular Model Operations.

Find the source code on GitHub.

Feel free to download, fork and comment. Looking forward to hear back from the community.

BlogSpot - 10 years later

Wow, it is amazing (and sad) to realize that I have not blogged in this spot for almost 10 years.

My last post being in October 2008. Just before I was made to realize another fragility of human life. It was in November 2008, that I was made aware that my father - Dr. Luxmi Kant Banerjea, the tough man of my life, was suffering from an aggressive form of lymphoma. For a person, who had never been in hospital bed other than when he was born, it was big change. Even bigger change, as he rapidly bed ridden by this debilitating disease.

As things played out for good, bad and ugly, he passed away, less than three months later in February 2009. I was not there by his side, I was back in US trying to see if I get second opinions from doctor's here. Maybe balance my time throughout the year with my sister who lived in the UK and me take turns helping out in his treatment. But it was not to be. Now with after nine years, I still miss him. But I also feel his presence never left me. I do wonder, if I can half as good parent has he was, my own kids would turn out just fine.

What was amazing to see was his legacy. After his passing, it seems people came out of the woodwork to help our family. They claimed my father had helped them when their chips were down and sometimes gone above and beyond to make sure they were taken care of, while staying within the bounds of honesty and the law. My father being a quite, reserved man never shared these stories of kindness or empathy with us, but it did seem to have rubbed off a little bit on me.

I also noticed an unfinished post from May 2009, I started writing about my father's life. But I guess that will be for another time.

Now that that I am well past grieving and life has moved on. I had my second daughter in December 2009 which completed the circle of life, so as to speak. I went full time on my own venture and then joined my dream company - Microsoft in July 2011.

Last seven years at working at Microsoft has been has been an interesting experience. I remember sending out my resume by post (aka snail mail) in 1999 as soon as I graduated from my masters to Microsoft in Redmond, WA. Now I did receive a post card back from HR saying they would consider me for future position. Boy, it took them - 12 years but I was there, finally. Well, I would hope that is how it played out... :)

Microsoft as awesome as it is, was different from what I imagined it to be. As big corporations are it has it fallacies. Probably back in 2011, too complacent in past years success, at one time considered an evil empire, dark side (for Star Wars fan), but you could feel the winds of change. Humbled by the tenuous and smart competition - Google, Amazon, Apple, who had customer mindshare, we were the big white drunk mad elephant. But as we know from Louis V.Gerstner Jr.'s book - "Who say's elephants can't dance? (Harper Collins, November 2002)", we know the mad elephants still has some tricks up its trunk".... :)

Fast forward into 2018, under leadership of Satya Nadella (since February 2014), Microsoft is forging new grounds into Cloud Computing, Artificial Intelligence, Mixed Reality and Quantum Computing. Well, Satya even wrote a book about it - "Hit Refresh (Harper Collins, September 2017)", if you have not had a chance to read it.

The biggest change at Microsoft - emphasis on openness (not just open source, that work started before the CEO change, but has new vigor now), leveraging work of others - within and outside Microsoft and most of all empathy - toward customers, partners and colleagues. I cannot say for sure if the change has come to every part of the organization but the new attitude is contagious.

One word to catches my attention - empathy and to that I go back to my late father, as evidence by people who came to help us, after he passed way, he had lots of it.

I think he applied a great deal of empathy and compassion to his work everyday. As a person responsible for lives of hundreds of people who reported to him and those his reports effected, he went through his daily rigors without much worry about how is legacy would be perceived, just how best it would make the lives he touched better.

All I can hope is that some of it rubbed on me as well.

Adventures in Virtualization and Sun's Virtual Box

Virtualization has been around for a while and of recent has gathered lot of interest due to "GO GREEN" initiatives and reduction in total cost of ownership. I had written in note in 2003 of a War of Third Kind about - on demand computing and later posted on this blog in 2007. (See- http://codingunknowingly.blogspot.com/2007/04/war-of-third-kind-reflection-4-years.html)

Now, it looks like my prediction on grid computing on top of virtualization is manifiesting itself as what we know as - CLOUD Computing. Sure, not in the most accurate description, but to achieve services through the cloud, an organization will need to maximize the processing power of its physical machines powering the cloud and then in comes the virtualization. There are already specialized tools and dedicated applicances to enable cloud powered services. But what is needed is a mindset shift for IT and software development community.

My own experience is virtualization started with VMWare Workstations and then went on to the ESX Server. More recently, I have been playing around with Windows 2008 and Hyper-V, as well a free open source option from Sun Microsystems called - Virtual Box. Check out - http://www.virtualbox.org/ for details.

Sun does have its more commerically tragetted offerings to build and manager virtual server infrastructures but virtual box is a good playing ground for enthuisants like me. Given a reasonably powerful desktop/laptop (really what I mean is non server class) machine, it provide a decent simulation for servers and virtual machines. So far, I installed Windows XP, Windows 2003 and Ubuntu Linux Server on Windows Vista host and had good success.

Couple of caveats on this free version of virtualization software, you cannot duplicate machines by simply copying the virtual hard drive image. (That would be strict no-no under Microsoft Licensing anyway.) and you need to plan for drive space on the virtual machine. You cannot increase the maximum size for the virtual hard drive. (It can be increased dynamically, in the sense, it only uses the space being used by the Guest OS. There are obvious downsides to that in terms of performance.)

My experience with Hyper-V has not been that pleasent. Though, it looks to be more polished and enterprise ready than Virtual Box. Using it on machine with a single NIC (network interface card) has been quite a challange. Inspite of reading multiple blogs and visit Microsoft support site, I have not been able to get the virtual machine and host to be on the Internet and local network on the same time. I am still researching the problem.