How to Create an RSS Feed from Windows Azure Table Storage Data

May 3rd, 2012

Today I am going to talk about how you can create an RSS feed from Azure Table Storage data. Realistically, you could very easily change this code to create a feed from SQL Azure or any store for that matter (if you get stuck email me at Liam AT Cotega dot com). The code that I am going to use is based on a sample created by DeveloperZen. I am also going to use the trick I blogged about to return the top X rows from Table Storage in reverse chronological order.

The reason I did this is because I needed to be able to provide my customers with a RSS / XML feed of the most recent items logged by the Cotega service. This service logs database statistical information on a users database such as query performance, blocked queries, etc. By providing this data as an RSS feed it is easy for my customers to then embed the feed into their own monitoring systems if they do not wish to use the Cotega dashboard. I am also hoping to use this feed (or another I am creating in JSON) to allow people to build mobile apps so that they can monitor their database from a smartphone. By the way, if you are interested in helping with this, let me know.

How it Works
At the bottom of this post, I have included an MVC3 project that shows how this works. If you open the solution in Visual Studio, you will want to focus on the HomeController.cs file which is located in the /Controllers folder.

For this to work, I will use the ServiceModel Syndication namespace for RSS and the Windows Azure namespaces to connect to my storage. As such, I have added the following to the controller.

using System.Xml;
using System.ServiceModel.Syndication;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.Diagnostics;

Within this controller there is one key functions and two classes. The first class defines the structure of the data that will be queries from Azure table storage and the second defines an ActionResult for the RSS feed. The Feed function is what gets called when you enter http://x.x.x.x/home/feed in your browser. You may want to set a breakpoint on this when you run the code. Some of the things that this function does is:

  • Receive an agentname and username.  These parameters are used to filter the results that come back from Azure storage
  • Create a connection string to my Windows Azure Table store called storageConnectionString . Notice you will need to replace [ACCOUNTNAME] and [ACCOUNTKEY] with your own credentials.
  • Create an EndDt that is set to todays date.  I use this as a trick to sort the results in reverse chronological order.  For more details on why I do this see this page.
  • Take the results of this query and fill up the RSS feed with the 50 most recent items from Table Storage and returns this to the browser
[HttpGet]
public ActionResult Feed(string agentName, string userName)
{
string storageConnectionString = "DefaultEndpointsProtocol=http;AccountName=[ACCOUNTNAME];AccountKey=[ACCOUNTKEY]";

//Select most recent 50 items in desceneding order of date
DateTime endDt = DateTime.Now.ToUniversalTime();

//create syndication feed
SyndicationFeed feed = new SyndicationFeed("Monitoring Agent Feed", "This is an RSS feed for monitoring agent logged data.",
new Uri("http://127.0.0.1:81/alert/feed"), "075211", DateTime.Now);

try
{
// Get the loggged data
CloudStorageAccount Account;
Account = CloudStorageAccount.Parse(storageConnectionString);

// Connect to Azure Tables
CloudTableClient TableClient = new CloudTableClient(Account.TableEndpoint.ToString(), Account.Credentials);
TableServiceContext tableServiceContext = TableClient.GetDataServiceContext();

// Use rowEndKeyToUse so that I can sort the data in reverse chronological order
string rowEndKeyToUse = string.Format("{0:D19}", DateTime.MaxValue.Ticks - endDt.Ticks);

var results = (from g in tableServiceContext.CreateQuery("logevent")
where g.PartitionKey == userName
&& g.RowKey.CompareTo(rowEndKeyToUse) > 0
&& g.EventType.CompareTo(agentName) == 0
select g).Take(50);

//add feed items
List syndicationItems = new List();

foreach (LogEvent logEvent in results)
{
var syndicationItem = new SyndicationItem(String.Format("{0}", logEvent.EventType),
String.Format("Watching: {0}
Value: {1}", logEvent.Watching, logEvent.EventValue),
new Uri("http://cotega.com"),
// You may want to replace this guid with the rowkey and partitionkey if not concerned about publishing this in the feed
System.Guid.NewGuid().ToString(),
logEvent.Timestamp);

syndicationItems.Add(syndicationItem);
}

feed.Items = syndicationItems;
}
catch (Exception ex)
{
//Log error
}

//return feed
return new RssActionResult() { Feed = feed };
}
}

public class LogEvent : TableServiceEntity
{
public string UserName { get; set; }
public string EventType { get; set; }
public int EventValue { get; set; }
public string Watching { get; set; }
public DateTime EventTime { get; set; }
}

///
 /// RssActionResult class. /// 

public class RssActionResult : ActionResult { public SyndicationFeed Feed { get; set; } public override void ExecuteResult(ControllerContext context) { context.HttpContext.Response.ContentType = "application/rss+xml"; Rss20FeedFormatter rssFormatter = new Rss20FeedFormatter(Feed); using (XmlWriter writer = XmlWriter.Create(context.HttpContext.Response.Output)) { rssFormatter.WriteTo(writer); } } }

That’s it. If you have any questions, let me know in the comments.
Download Sample – Azure RSS Feed

What I Learned Building a Startup on Microsoft Cloud Services: Part 12 – Your Customers are my Customers

April 27th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

Over the years I have constantly been amazed at how customers end up using products I work on.  No matter, how much thought and planning has been put into a product it seems you can never fully appreciate how customers will use your products until they actually start using them.  I suppose that is a key reason for building a “Minimum Viable Product” and to get your product out the door as soon as possible so that you can get this feedback an iterate quickly.

One example of this became very clear early on in the Cotega beta.  Initially, I thought that charting of logged data would be really useful for DBA’s to be able to visualize the trends that were happening over time within their database.  This was a key reason for adding this feature.  After talking with various people, I started to realize that although this was in fact useful to DBA’s, what many of them really felt would be useful would be to take these charts and embed them in their own web site so that their customers could see the health of the system.  In a way, their customers would be my customers.   This was really a surprise to me and something I had never thought of.  As it turned out, it was very easy to implement because it was just a matter of creating a new MVC page that accepted the name of the chart and the user name to be used to builds the chart.  Then I could use either <OBJECT> or <IFRAME> tags to take this page and embed it.  Here is an example that shows a historical look at how long it takes to connect to my SQL Azure database. The numbers to the right indicate how many milliseconds it took to complete the connection.


This chart will be loaded dynamically each time you load this page. Here is the code that I used to embed this chart. Notice how, I used iFrame, which is because WordPress does not work well with the Object tag.

<iframe src=”http://cotega.com/home/charts?agentName=Monitor Customer Database&amp;userName=liam” frameborder=”0″ marginwidth=”0″ marginheight=”0″ scrolling=”no” width=”500″ height=”400″></iframe>

Cloud Competitive Advantage

As it turned out, this ended up being a real competitive advantage for me over traditional on-premises monitoring solutions.  Since Cotega is hosted completely in the Azure environment, it is very easy for me to make these charts available to be embedded in customer’s web sites.   If Cotega was an on-premises system, this would have been much more difficult due to firewall and other issues.  Also, if the charting was only used by DBA’s, this capability to embed charts and data would not be nearly as critical.

Protovis Charting

While I am on the subject, you might be interested to learn how I chose to implement the charting.  In the early stages of the beta, I used a charting control from Infragistics.  This control was great and very easy to use (although certainly not cheap).  In the end, I decided to use a charting control from Protovis because it had the ability to copy and paste charts directly from the Dashboard so that they could be used in something like Excel or Word.    If you are using Internet Explorer, give this a try by right clicking on the above chart and choosing paste into a Word document. Very cool, right? The other nice part about Protovis is that it is easy and free.

What I Learned Building a Startup on Microsoft Cloud Services: Part 11 – Vacation Time, Profitability and Listening to the Customers

April 17th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

Vacation Time

It has been a few weeks since my last post and I apologize for that.  One of the big reasons why I have not posted more recently is because I took a vacation with my family.  One of the downsides of starting a business by yourself is that you are never truly on vacation.  It is important that Cotega is always running and although it is pretty self sufficient and (luckily) I have not had any major issues with the service other than that one Azure outage, I am always concerned that something may happen or a customer has a problem.  With most jobs there is always someone that knows how to get a hold of you if there is a major issue and even if you have a co-founder there is always someone there who can contact you if there is a problem.  I think that is really one of the other big advantages of bringing on a co-founder.

Profitability

Since I last posted, I started charging for the service and I am  happy to say that Cotega is now officially profitable.  Yeah!…   Well, let me be more clear about that.  By profitable, I mean I am officially making enough money to cover the costs of operating the service (including those costs that will come when my BizSpark program ends).  I am still a long way from taking any major salary.  But for me this is a big step because it was one of the main goals of starting Cotega.  If you are  reading this and are thinking of doing a startup, I have to tell you that one of the most exciting things to see are those first transactions coming in to your account.  Even the small transactions are incredibly exciting.  I think it has to do with the realization that there are in fact  customers out there that are interested in what you are doing and are willing to pay for it.

New Features and Customer Suggestions

The other big reason why I have not posted is because I have been focusing on some new features for the service.  Yesterday I deployed an update that allows for monitoring of blocked and poor performing queries.    Each of these new features have come from existing customers I have been working with.  I keep the suggestions in the Github issue repository where I can track the features that are most commonly requested and start working on those first.    The other great source of ideas has been from people I have contacted that are not customers at all.  For example, Microsoft has some amazing MVP’s who work closely with different Microsoft technologies and are absolute experts in these products.  Every MVP I have ever worked with has gone way out of their way to help me and some of the best ideas (that are still in the works) have come from these people.

If any of you have not yet tried the service but are interested in seeing the service, I created this code that you can use to try any of the plans for 30 days free: 30dayfree

Liam

What I Learned Building a Startup on Microsoft Cloud Services: Part 10 – Choosing a Billing System

March 21st, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

This past week I was able to move Cotega out of its beta stage and on to the production servers.  Since my beta customers were not being charged to use the service, one final work item I needed to complete before moving out of beta was to implement a billing system.  Since a number of people have asked me about my choices in this area, I thought it would be good if I talked about it today.

Without a doubt one of the most common choices for doing online credit card processing is Paypal.  I really seriously considered using Paypal since I have trusted them so much in the past for my own purchases and I have read that their API’s are quite extensive.  However, since I am building a startup, I felt I should see if there were any startups in the area of credit card processing.  Two of the most interesting ones that I narrowed my choices down to were Stripe and Dwolla.

Stripe is a really interesting company.  They are one of the many companies that have come out of the Y Combinator startup program.  Their prices seemed to be very similar to what you would get from PayPal which is 2.9% + $0.30 / charge.  But what I really liked about Stripe was their support.  Although they have the typical support where you can email questions, they also have a Campfire based chatroom where you can go to ask questions in real-time.  I have logged on a few times and each time there were always a large number of people online to answer my questions.  But what was really surprising was that most of the questions were not in fact answered by Stripe customer support but rather by Stripe customers.  Imagine having the support of customers who would take the time to help out other customers?

The other great part about Stripe were the API’s.  Not only did Stripe offer their own API Library to integrate with languages like Python, Ruby, PHP and Java, but they also had a really extensive set of 3rd party libraries for even more languages (including two for .NET that both work really well).

Dwolla, was also really interesting.  I still don’t completely understand how they keep pricing so  outrageously low.  Their pricing is also pretty simple.  If you charge your customers less than $10 you do not pay anything for that transaction.  If you charge anything more than $10, you pay a flat $0.25 / transaction.  For this unique business model, they were ranked by “Fast Company” as one of the 50 most innovative companies.

Just as an aside.  When I signed up for a business credit card from my bank they kept calling me to see if I wanted to use their credit card processing system.  If I remember correctly, they wanted to charge me something like $100 to setup, plus $30 / month usage fee, plus 2.9% + $0.30 / transaction.  When I told them about Dwolla’s pricing they stopped calling me.  :-)

In the end, I chose to use Stripe.  I had just read so many awesome things about them and their support was just so great that it tipped the scales for me over Dwolla’s amazing pricing.

After spending a lot of time with Stripe and getting my first paid customers, I am still very happy that I chose to use them.  Their administrative dashboard is very extensive and really easy to use.  I also love how easy it is to switch from “live” and “test” mode which allowed me to fully test Cotega with simulated credit cards and subscription plans before going live.  I am still not sure if customers will demand to be able to use PayPal, but Stripe has seemed to give me everything I need for now.

What I Learned Building a Startup on Microsoft Cloud Services: Part 9 – Sending Email Notifications from Windows Azure

March 14th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

A key feature of the Cotega monitoring service is the ability to send email notifications when there are issues with a user’s database. The Windows Azure worker role executes a specified set of jobs against a users database and if there are issues such as connection drops or query performance degradation, then the service will send a notification to the administrator.

Currently there are no SMTP servers available within Windows Azure to allow me to send email notifications. Luckily, there are a huge number of third party email services that work really well with Windows Azure and are extremely cheap.

Sending Email Using Free Email Services

To get started, I first built a prototype that would send notifications from my worker roles. To do this I started with Hotmail as the SMTP server. The code to do this is pretty simple and there are a number of good examples on how to do this, such as here and here. These services worked pretty well.  Ultimately I decided not to move forward with them given the sending restrictions that free email services like Hotmail, Office365, GMail, Yahoo mail have. For example, many of them limit the number of senders you can send to within an hour or limit the total number of emails you could send over a specified period of time. I suspected it would be a long time before the Cotega service reached these limits but I preferred to avoid them if possible.

3rd Party Paid Email Services

I really did not want to host my own SMTP service in Windows Azure.  In fact, I have heard (but not confirmed) that sending emails using your own SMTP servers can have issues where emails will frequently be bounced back or will be tagged inaccurately as spam.  Many third party email services have techniques to minimize this which was very attractive to me.  So for these reasons, I started researching other third party email services. Some of the most promising ones that I found were Elastic EmailSendGrid and Amazon SES (Simple Email Service). Each of these are paid services.  I ended up using Amazon SES primarily because I wanted to get an opportunity to learn more about Amazon’s services and their cost was attractive at $0.10 per thousand email messages. The other interesting thing about Amazon SES is that they start you out at a limited number of outbound emails. This limit is quite high at 10,000 over 24 hours and increases as Amazon learns to trust that you are not sending spam. Plus you can request an increase if needed. The dashboard for monitoring your email traffic is pretty nice and allows you to visually see the number of delivered, rejected, bounced and complaints emails.

After setting up my Amazon SES account, I needed to install the Amazon SDK and include the Amazon.SimpleEmail.Model namespace which would be deployed a a reference to the worker role. Here is a snippet of code that I used which is based on a sample Amazon provides. If you use it, remember to update the [CODE] sections with your SES keys:

public static Boolean SendEmailSES(String From, String To, String Subject, String Text = null, String HTML = null, String emailReplyTo = null, String returnPath = null)
{
    if (Text != null && HTML != null)
    {
        String from = From;

        List to
            = To
            .Replace(", ", ",")
            .Split(',')
            .ToList();

        Destination destination = new Destination();
        destination.WithToAddresses(to);
        //destination.WithCcAddresses(cc);
        //destination.WithBccAddresses(bcc);

        Content subject = new Content();
        subject.WithCharset("UTF-8");
        subject.WithData(Subject);

        Content html = new Content();
        html.WithCharset("UTF-8");
        html.WithData(HTML);

        Content text = new Content();
        text.WithCharset("UTF-8");
        text.WithData(Text);

        Body body = new Body();
        body.WithHtml(html);
        body.WithText(text);

        Message message = new Message();
        message.WithBody(body);
        message.WithSubject(subject);

        AmazonSimpleEmailService ses = AWSClientFactory.CreateAmazonSimpleEmailServiceClient("[CODE]", "[CODE]");

        SendEmailRequest request = new SendEmailRequest();
        request.WithDestination(destination);
        request.WithMessage(message);
        request.WithSource(from);

        if (emailReplyTo != null)
        {
            List replyto
                = emailReplyTo
                .Replace(", ", ",")
                .Split(',')
                .ToList();

            request.WithReplyToAddresses(replyto);
        }

        if (returnPath != null)
        {
            request.WithReturnPath(returnPath);
        }

        try
        {
            SendEmailResponse response = ses.SendEmail(request);
            SendEmailResult result = response.SendEmailResult;

            Console.WriteLine("Email sent.");
            Console.WriteLine(String.Format("Message ID: {0}",
                result.MessageId));

            return true;
        }
        catch (Exception ex)
        {
            Helper.LogException("Worker - SendEmailSES", ex.Message.ToString());
            return false;
        }
    }

    Console.WriteLine("Specify Text and/or HTML for the email body!");

    return false;
}

What I Learned Building a Startup on Microsoft Cloud Services: Part 8 – I failed to plan for failure

March 7th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

On Feb 29th, I woke up to learn that Windows Azure (the hosting site of the Cotega monitoring service) was experiencing a service outage.  It surprised me that it wasn’t an email from Windows Azure or from an angry customer that notified me of this issue, but rather I learned it from reading Hacker News.  When I first saw the post I immediately checked the Cotega worker role and sure enough it had stopped monitoring user’s databases.  This was more than a little embarrassing for me since I had tried to follow the golden rule when creating services of “Planning for failure”.  I thought I had done everything possible to deal with failures.  I had multiple worker roles to handle the case where one machine fails.  As I discussed earlier, I even moved my SQL Azure queueing system to Windows Azure queues because of issues I would have monitoring a user’s database if SQL Azure failed.  What I did not plan for, was the case when Windows Azure itself became unavailable. When I first learned of the outage, my first thought was that maybe it was a bad idea to rely so heavily on the cloud for my service.  But soon after I realized that this was really just a failure on my part, not on the cloud.  All services will have issues at some point in time.  This is true regardless of whether they are in the cloud or on premises.  I had not considered all critical failure scenarios.  Not only did I not have a solution to handle failures like this, I did not even know about the failure until I read about it in the news.  In the back of my mind I had always thought either Windows Azure or a customer would notify me if there was an issue like this, but even if they had, this would not have helped me automatically handle the failure.  What I took away from this is that I needed to automate the failover of the monitoring service.  To do this I would need to set up a separate application to watch my worker roles.  Since pinging worker roles is disabled by default and I prefer to avoid opening up this port, the next best thing seems to be to check the logging table from a machine outside of the Windows Azure data centers.  If there were no entries in the log table in last 5 minutes, then there must be an issue and it would then:

  1. Send me an email and SMS notification of the failure
  2. Start running the monitoring from a separate location

Then once Windows Azure was back up and running, I could reset this tool back to the state of pinging the worker roles and disable monitoring from this separate location.  Luckily, the monitoring service is built to handle multiple instances of the monitoring service running at the same time, so when Windows Azure comes back up, the separate monitoring service will just work in parallel to it until I shut it down. The only problem I still need to handle is the case where Windows Azure queues becomes unavailable as this is a central service that is needed by my service.  I’ll have to think about how to handle this failure a little more. In any case, this was a really good test for me to learn from a major service failure and I think this will ultimately make the final service more more solid.

What I Learned Building a Startup on Microsoft Cloud Services: Part 7 – Cotega Updates and New Web Site

March 4th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

Today, I wanted to update you on the status of Cotega and let you know what I have been working on over the past few weeks.

First of all, thanks to all of you who have taken the time to use and provide feedback on the service.  I have greatly appreciated your support.  Over the past few weeks I have tried to consolidate your feedback and based on these suggestions, I have made a significant number of updates to the web site.  Below, is a screen shot of the new dashboard page that now allows you to quickly visualize the state of your databases.   In fact, from IE9 or higher, you can now copy and paste the charts.

There are a number of other updates that are meant to simplify the user experience and focus on the key tasks you have told me are important.  In addition, “notifications” are now referred to as “monitoring agents”.

Over the next few weeks Cotega will transition from the current beta servers to the production servers.  Although the service will still be in “beta” for a few additional weeks, it will allow the service to be hosted in its final location and allow me to add some final pieces such as the billing system.  During this transition time, your notifications will need to be re-created and all archived data will be lost.  If you wish to keep this data, it is recommended that you download this data using the “Logs” option.  I will give you more notice once this transition gets closer.

Once again, thank-you for your support and I hope to continue to hear from you with your feedback, questions and suggestions.  Please take a look at the updated service and let me know if you have any issues.

What I Learned Building a Startup on Microsoft Cloud Services: Part 6 – Securing the service and building trust

February 28th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

I knew my next step was to get an SSL certificate for my domain so that I could secure communications through https and encrypt the user’s connection information that is stored in the Cotega system database. Before I did this I needed to create a domain name. Unfortunately Windows Azure does not provide domain name registration or SSL certificates so I decided to look into using GoDaddy which is also where this blog is hosted. I found a domain name I liked through the GoDaddy auctions for about $10 called Cotega. I was also pretty lucky because there was a deal on the GoDaddy SSL Certificate for $12 / year. If I were to do this again and there was not a deal on the GoDaddy SSL certificate, I would probably take a closer look at FreeSSL’s free SSL certificate as I have read about other startups who chose this option.

Integrating SSL Certificates and CNames

Since I am hosting my web site on DiscountASP.NET I needed to load the certificate there. To configure the domain name, you need to set up a CName. A CName is basically a redirection that happens when someone enters www.cotega.com into a browser. In the background the CName links www.cotega.com to my DiscountASP.NET hosted IP address. This also works with the SSL certificate that is hosted with my web site so that https and http both leverage this IP address.

If I had of known at the start that I was going to host my web site on DiscountASP.net, I would have also used them to provide my domain name. Luckily I was able to use my previously created GoDaddy SSL Certificate and domain name with only minor configuration changes on DiscountASP.net.

Self Signed Certificates

The other thing that I wanted to use a certificate for is the encryption of users connection strings that are stored in the Cotega system database. For this I wanted to have a really high level of encryption so I chose to create an unsigned 2,048 bit MD5 X.509 certificate using the makecert.exe utility. You would not want to use a makecert generated certificate for your SSL traffic as browsers will reject an unsigned certificate. If you want to learn more about this, there is a great overview of the issues here. However, for encrypting the connection information within my MVC controller, this was a perfect solution because I control the entire encryption process which means the “man in the middle attack” does not apply but I could implement a really high level of encryption for my users.

Working with Customers to Building Trust

One of the interesting differences I learned in building services at Microsoft vs. Cotega is in the area of trust. At both Microsoft and Cotega, I spend a lot of time making sure a user’s information is secured. However, at Microsoft I did not have to spend as much time explaining this to customers. I think this comes from their past experiences with Microsoft where they have learned of Microsoft privacy and security policies and have grown over time to trust them. However, at Cotega, the service and company is new and has not built the level of trust that Microsoft has. As a result, I found that it was important for me to work closely with these customers to ensure that I can build that level trust which has primarily come from one-on-one discussions. A side benefit of this close relationship with customers is the early feedback I get on the service. But I will talk about that more later…

What I Learned Building a Startup on Microsoft Cloud Services: Part 5 – Migrating to Windows Azure Queues

February 20th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

In my previous post I talked about how I built the Cotega queuing system on SQL Azure. I was pleased at how well it worked. Unfortunately, it had one fundamental problem. The whole point of the Cotega service is to tell users if there are issues with their SQL Azure database such as database connectivity issues. If there was a problem with SQL Azure that was blocking users from connecting to their database, then this problem would very likely also affect the Cotega queuing system since it is also based on SQL Azure. If I am not able to connect to SQL Azure then I also cannot retrieve user’s jobs meaning I cannot tell the user there are problems.

Giving up on SQL Azure Queues

This was a big enough problem that I decided to investigate the use of Windows Azure queues. This turned out to be very easy to implement and as it turned out simplified a lot of things I had manually implemented in the SQL Azure queuing system. For example, the handling of orphaned jobs which occurred when a Worker Roles failed or crashed was automatically handled with queues.

Hiding Jobs in Windows Azure Queues

The only issue I had a little work to figure out is how to set a job to be hidden for a specific period of time. For example, if I did not want to run a job for another 5 minutes, I would need to add the job to the queue but make it invisible for 5 minutes. As it turned out, there is an overload for CloudQueue.AddMessage call which can set this value. For example, this code would set the message as invisible to 5 minutes, but set the time to live to be 10 hours:

CloudQueue queue = QueueClient.GetQueueReference(queueName);
TimeSpan runTime = new TimeSpan(10, 0, 0);
TimeSpan frequncy = new TimeSpan(0 5, 0);
queue.AddMessage(message, runTime, frequency);

Windows Azure Queue Pricing

Windows Azure Queues also turned out to be a lot cheaper than I expected. For example, let’s say I have a single thread that checks the queue for a new job every 2 seconds. Even thoug this is 1,314,000 storage transactions / month ( 30 transactions / minute * 60 min * 24 hours * [365/12] days), it still only costs $1.31 / month (1.314 million transactions / 10,000 * $0.01). Even if I have multiple worker roles with multiple threads, this is extremely low pricing.

In the end I was really happy with Windows Azure queues and wish I had of made more of a time investment in learning this technology from the start.

What I Learned Building a Startup on Microsoft Cloud Services: Part 4 – To Build or Buy a Queuing System

February 15th, 2012

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

One core piece of the Cotega services is the queuing system. This system resides within the Windows Azure Worker Roles that are constantly checking for new jobs. A job is simply a task that the Cotega service needs to execute. For example, the service may check if a user’s database is up and running, or it may check the user count in a user’s database. For each job, the data is logged and if issues are found email notifications are sent.

Building a Queuing System in SQL Azure

As you might imagine, this is a perfect job for a queue. Most of you would probably start by using Windows Azure queues, but since I am a long time database guy, my preference is to always do as much as I can within the database, simply because that is where I am most comfortable and my queuing needs are pretty simple. I also like to keep things simple and since I was already using SQL Azure for my system database it seemed simpler to use that also for queiing rather than add in another service.

I have to tell you, building a simple queuing system in SQL Azure was way easier than I expected and it worked really well. My queue was in a table called Notifications. When a user adds a new notification in the admin web site, I insert a row into this table with a timestamp of ‘1900/01/01’. Then the worker role which is constantly checking for rows that are less than the current time, picks up these notifications, marks a “Processing Time” field for this row with the current timestamp and starts executing the associated job. When the job is done, the jobs status is marked as complete and the timestamp is updated to some interval past the current time, so that it can be picked up again later on.

Handling Multiple Threads

One of the biggest problems I had to handle was the fact that I have multiple worker roles and then multiple threads within each of those worker roles. With all of these processes it is very possible for a single job to be picked up by more than one thread. To handle this problem, I needed to create a stored procedure that would use a row locking system to ensure that a job could not be picked up by more than one thread. The secret to getting this to work is in the use of WITH (UPDLOCK, READPAST) code. This tells the SQL Azure engine to skip any rows that are locked. The locking ensures no other threads will pick it up. This is what my stored procedure looked like:

create procedure get_next_job as begin

DECLARE @NextId INTEGER
BEGIN TRANSACTION

– Find next available item available where the status is enabled
SELECT TOP 1 @NextId = [NotificationId]
FROM [Notifications] WITH (UPDLOCK, READPAST) WHERE [NextRunTime] <= getdate() and [Status] = 1 ORDER BY [NextRunTime] ASC

– If found, flag it to prevent being picked up again
IF (@NextId IS NOT NULL)
BEGIN
UPDATE [Notifications]
SET [ProcessingTime] = getdate(),NextRunTime] = dateadd(mi,frequency,[NextRunTime])
WHERE [NotificationId] = @NextId
END

COMMIT TRANSACTION

– Now return the queue item, if we have one
IF (@NextId IS NOT NULL)
SELECT [NotificationId], [TableName], [EmailAddress], [ChangeColumn],[ChangeType], [MessageTextColumn], [LastSuccessfulRunValue],[DatabaseConnString], [DatabaseType]

FROM [Notifications], [UserDatabases]
WHERE [NotificationId] = @NextId and [Notifications]
.[DatabaseID] = [UserDatabases].[UserDatabaseID]
end
go

Handling Orphaned Jobs

The other issue I had to consider is when the machine crashed or the job did not complete. Although this had not happened yet, I am sure at some point it will, so I need to make sure a users job does not get lost forever. This is where the ‘“Processing Time” column comes into play. In the above stored procedure you can see I set this to the current date time. I also know that is should take no more than a minute to complete any job. Therefore, if a job is more than one minute old, I know there was a problem and I can handle it.

Throwing it all away

All of this worked incredibly well and the performance was terrific. Another advantage I found was that regardless of the number of queries I executed against my SQL Azure database, I was guaranteed a fixed cost per month. Best of all that cost was free since I already had a SQL Azure database for my system data. With Windows Azure queues the cost is not fixed, although the cost for this is very low. Ultimately, I decided to throw this all away and move to Windows Azure queues. But I will talk about that more next week.