Recently I’ve been solving some issues with queue starvation in CruiseControl.NET. Since I haven’t found much information about this problem I’ve decided to round up my findings in this blog post.

So, what is queue starvation? It’s a problem that can occur when scheduling tasks in priority queues if there are too many high priority tasks so that lower priority tasks never get scheduled and are just waiting indefinitely. For example, let’s say we have three queues: high, medium and low priority. Tasks in the high priority one always get processed first. Once this queue is empty the tasks from the medium priority queue get processed. Of course the tasks in the low priority queue get processed only if the other two queues are empty. If there are too many high and medium priority tasks incoming all the time so that the queues never get empty, then the low priority tasks are not processed at all.

And how is this related to CruiseControl.NET? As you might be aware, projects can be assigned to different queues. Only projects in the same queue (as the name implies) are built sequentially while projects in different queues are built in parallel to one another. Queues themselves therefore cannot cause starvation because different queues don’t depend on one another and projects are always put at the end of its queue. This means that sooner or later each project will be built.

Now, queue priorities enter the stage. Each project can be given a priority defining where the project will be put into the queue: before all of the lower priority projects already in the queue instead of at the end. Effectively this partitions a single sequential queue into multiple queues with different priorities as described at the beginning of the post. We have all the necessary prerequisites for queue starvation. If too many projects with higher priorities have to be built (typically caused by checking in code or triggering a forced build), the projects with lower priorities will never get processed.

What I have observed is that this can happen even if higher priority projects don’t have to be built at all. The reason for that is in the way CruiseControl.NET is processing the projects to determine whether there were any source code changes. This is taken care of with an interval trigger which puts a project in the queue, but it only gets built if a modification exists. Nevertheless, even checking if such a modification does exists, takes some time. In extreme cases this can be enough to cause queue starvation and this is also what happened in my case. Lower priority projects never got built, not even during the night and no matter how often a build was forced for them.

Fortunately there is a solution for this problem: the seconds attribute which specifies how often the project is put into the queue for checking if a modification exists. By default it is set to 60 seconds. Increasing the value can solve the queue starvation issue. Instead of simply defining the trigger:

<intervalTrigger />

A longer period can be specified:

<intervalTrigger seconds="300" />

Of course this has its downside: the latency before the project gets built after a modification increases. Therefore you should use this solution with caution. Only increase the value as much as you have to. And don’t do it at all if you can apply any of the other solutions:

• Increase the performance of the build server to speed up the processing and building of the projects.
• Parallelize the build process by using more queues. Put the projects in the same queue only if you have to.
• Don’t use queue priorities if it’s not necessary. In most cases you can get by without them.

A few months ago I worked on a small spare time project which included some manipulation of binary Excel (.xls) files. This seemingly simple task soon turned out to be quite a challenge if you want to handle it right. The post you are reading is a short summary of my experiences. They should make your choices easier if you are about to tackle a similar problem.

The most obvious choice for handling .xls files is Excel automation using the Excel object model. As long as your application is always going to be used interactively, you should be fine. It’s probably the best method to use in spite of a few downsides:

• The object model is COM based which means you’ll have to do interop if you are developing a .NET application. Fortunately there are some nice improvements in .NET Framework which make coping with COM in C# much easier.
• Your application will have to be compiled in 32-bit in order for it to work on 64-bit Windows. This usually isn’t a real limitation just don’t forget to switch the target platform from Any CPU to x86 if you’re doing development on a 32-bit OS to avoid the problem. (You’re going to notice it yourself when developing on a 64-bit OS.)
• Don’t try unit testing the Excel automation code outside your IDE (e.g. on your continuous integration server). As described below this scenario is not supported.

As soon as you want your code to be run non-interactively, you’re out of luck with Excel automation. Since all Office applications assume to be running on the interactive desktop, there are several reasons why such usage is not supported and is even strongly discouraged by Microsoft. If you want to run your code on a web server, as a Windows service, a scheduled task or just automatically test it on your build server, you’ll have to find a different approach. And in this case there are no obvious choices.

Your best bet is to use Open XML file format (.xlsx, .xlsm) instead of the binary one (.xls) if this is an option for you. Since this is a well documented XML based format you can manipulate it directly without running Excel at all. You can even use Open XML SDK for Microsoft Office which includes strongly typed classes that simplify many common tasks.

This is not the case with the binary format. Microsoft doesn’t provide or support any SDK or API for manipulating the files directly. You are only provided with detailed documentation of the format. Based on it some third party solutions have been developed. Aspose and SoftArtisans have their own commercial offerings which I haven’t evaluated because as such they weren’t suitable for me.

On the other hand the only free library that I have found is MyXLS and this one leaves much to be desired. It also seems to be pretty much abandoned with the latest release almost a year ago and only a single commit to the repository in this year so far. That being said, it still might prove useful if you only want to create the files, mostly focusing on the appearance with only minimal requirements regarding formulas. According to the samples this seems to work fine. Reading existing files is another story. You are more or less on your own as soon as you need to read cells with formulas. This made the library useless for me therefore I used another approach based on the fact that the OLE DB Provider for Jet can be used to read and write data in Excel worksheets.

At first I haven’t even considered this possibility because I was convinced that this method can only be used on worksheets designed as database tables (having columns of data with or without header columns). This article proved me wrong and in spite of many issues in the demo project it showed off techniques which turned out really useful for me. The most important one was the possibility to address regions, not only complete worksheets – this can be used to retrieve and set individual cell values as well as for accessing database table-like blocks of data in a part of the worksheet. Let’s take a look at a sample:

SELECT F1 FROM [Sheet1$C2:C2];

This query retrieves a value of a single cell. The same syntax can be used to access any region: after the worksheet name with a trailing $ character the region is defined just like in Excel formulas. If the region doesn’t include column headers (specified by including HDR=No in the Extended Properties of the connection string) the columns are named F# where # is the sequential number of the column in the defined region. Similarly the following query can be used to set a value of an individual cell:

UPDATE [Sheet1$C2:C2] SET F1 = 'Value';

A few more things worth mentioning:

• Be aware of the IMEX option in the connection string. It doesn’t seem to be documented very well, probably its best description is here.
• Make sure you keep the connection to the file open if you plan to do more data access later. Opening the connection takes quite some time therefore the performance will be terrible if you keep closing and reopening it.
• The technique doesn’t seem to work if the worksheet name begins with a space. At least I couldn’t make it work, no matter how I set the quotes.
• I encountered a file which couldn’t be opened in this way but the problem was resolved after I opened the file in Excel and saved it again.

Subversion and CruiseControl.NET can be invaluable tools in your .NET development process. There are many resources available to help you get started which I’ll try to gather in this post along with some of my personal experiences.

Let me start with the list of recommended software:

• VisualSVN Server is the ultimate Windows version of Subversion including a simple setup and powerful management tools. If you are planning to install a Subversion server on Windows it should be your first choice.
• AnkhSVN is a Subversion Source Control Provider (SCC) for Visual Studio. As long as you’re not using Express editions of Visual Studio, this is the suggested way of working with SVN directly from Visual Studio IDE.
• TortoiseSVN is a Windows shell extension for working with Subversion from within Windows Explorer. When you're not working with Visual Studio solutions this is the best choice for using SVN.
• CruiseControl.NET is a continuous integration server including a web dashboard and CCTray - a system tray client application for monitoring and controlling builds.

If you’re not already familiar with the above mentioned products, you should consult their documentation or search for tutorials. I will rather focus on setting up your development and release process. If you haven’t done so already I suggest you first read the following articles by Ariejan de Vroom:

• How to structure your repository
• How to release software properly
• How to fix bugs properly

I mostly based my configuration on the ideas in these articles. I have projects configured in CC.NET to build all copies of the project: trunk (ProjectName-Trunk), all branches (ProjectName-REL-#.#) and all tags (ProjectName-v#.#.#). To identify individual builds I am using CC.NET’s Assembly Version Labeller together with AssemblyInfo MsBuild Community Task.

Assembly Version Labeller is really simple to configure. You only need to add a short snippet to each project:

<labeller type="assemblyVersionLabeller">
    <major>1</major>
    <minor>0</minor>
    <build>0</build>
</labeller>

I’m using the following versioning policy:

• I start each project with version 1.0.0.
• Once it’s ready for release I make a copy of the trunk in the branches directory, named REL-#.# containing the major and the minor version number. Immediately afterwards I bump the version of the trunk (only minor or major and minor, depending on the nature of the new features planned).
• In the release branch I make the necessary changes before release (e.g. I change the AssemblyProduct name to distinguish between development and release quality builds) and make another copy in the tags directory, named v#.#.# containing the major, minor and build version numbers. Immediately afterwards I increase the build version number in the release branch.
• I make no changes to the copies in the tags directory. All bug fixes go to the release branch. Once I’m ready for a new release I repeat the previous step.

Since I don’t specify the revision number directly, the SVN Revision number gets used automatically. This makes it possible to match each build to the revision of the code in SVN.

To put the generated assembly version in the build I am using the AssemblyInfo MsBuild task. There are two steps involved in doing this.

First you need to move the AssemblyProduct, AssemblyInfo and AssemblyFileVersion attributes from the auto generated AssemblyInfo.cs file into a new file. In my case the AssemblyVersion.cs has the following contents:

using System.Reflection;

[assembly: AssemblyProduct("ProjectName DEV")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

Next you have to modify your project file (*.csproj) by importing the community tasks and adding a call to the AssemblyInfo MsBuild task:

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />
<Target Name="BeforeBuild">
    <AssemblyInfo Condition="'$(CCNetLabel)' != ''"
        CodeLanguage="CS" 
        OutputFile="Properties\AssemblyVersion.cs" 
        AssemblyProduct="ProjectName TRUNK" 
        AssemblyVersion="$(CCNetLabel)" 
        AssemblyFileVersion="$(CCNetLabel)" />
</Target>

If you have never edited a project file before, you might want to read these first:

• How to: Edit Project Files
• How to: Extend the Visual Studio Build Process

One more thing to note which might not be all that obvious. The Condition in the AssemblyInfo task is met only when building from CC.NET. For builds in Visual Studio the task doesn’t regenerate the AssemblyVersion.cs file therefore the revision number is always 0 and the AssemblyProduct has a DEV suffix as defined in the original file. Also I remove the TRUNK suffix from the AssemblyProduct attribute of the AssemblyInfo task when moving code from trunk to release branches to separate between the two.

Query parameters are a very useful Excel feature allowing parameterization of database queries used to import data in Excel. They are really simple to use as well. On the Definition tab of the Connection Properties dialog there is a Parameters… button at the bottom. It gets enabled as soon as there is a parameter defined in the Command text – you define it by typing in a question mark (?) instead of a value in the WHERE clause of the query, as seen in the image below.

Unfortunately there is a limitation for using this functionality which turns out pretty unintuitive in Excel 2007. It is only supported for Microsoft Query based queries. For all other types of queries available in Excel you get the error “No value given for one or more parameters.” when you add a parameter to the Command text. Since the error doesn’t even hint at the real cause of the problem it took me some time before I figured it out. The thing to remember is: If you want to use query parameters, you must select Microsoft Query as the external data source when importing the data for the first time. This can’t be changed at a later time.

Microsoft Exchange supports Send As and Send On Behalf Of permissions to be granted to users for individual e-mail addresses. Sending e-mail from Outlook for these users is very simple – they just enter the desired address in the From field of a new message (toggled with the Show From command on the Options ribbon) and if they have the required permission it will be sent accordingly – either as if it was actually sent from that address or as sent by the user on behalf of the address in the From field.

If you want to achieve this from code there is a little more work involved. First of all the user must be authenticated on the server using one of the methods below:

SmtpClient smtp = new SmtpClient("smtp.domain.com");

// use user’s existing credentials
smtp.UseDefaultCredentials = true;

// pass username and password
smtp.Credentials = new NetworkCredentials("username", "password");

The next step is to set up the correct headers in the message otherwise the server will return error code 5.7.1 describing the permission the user does not have.

To send the e-mail as only the From property has to contain the desired address:

MailMessage mail = new MailMessage();
mail.From = new MailAddress("send.as@domain.com");

To send the e-mail on behalf of another user the Sender property must additionally contain the user’s e-mail address:

MailMessage mail = new MailMessage();
mail.From = new MailAddress("send.as@domain.com");
mail.Sender = new MailAddress("user.address@domain.com");

On a related note, the required permissions can be granted using PowerShell.

To grant the Send As permission:

Add-ADPermission –Identity "user1" –User "user2" –ExtendedRights Send-As

To grant the Send On Behalf Of permission:

Set-Mailbox "user1" -GrantSendOnBehalfTo "user2"

In both cases the user1 specifies the mailbox to grant the permission for and the user2 specifies the user to grant the permission to.