Trace listeners are a great mechanism for troubleshooting and monitoring applications in production environment. After you have decorated your code with the necessary Trace and Debug calls, you only have to add the desired listeners either in code or in the configuration file and voila: the application starts emitting information to the configured destinations. Usually there is no need to have trace listeners attached all the time, you can only add them to the configuration file when you need to troubleshoot a problem.

What I didn’t know until recently, is that by adding a trace listener to your application you can cause it to crash. That’s definitely not something I wanted or expected to happen! After investigating the issue further the offender turned out to be EventLogTraceListener which (obviously) logs information to the event log. To do that the event log source is required and therefore it needs to be specified when adding (initializing) the EventLogTraceListener. And here lies the root of the problem.

To register a new event log source, administrative privileges are required. If the event log source is not yet registered and the user running the application is not an administrator, the call to Trace will raise a security exception. If this exception is not handled properly, the application will crash. To be on the safe side all Trace and Debug calls in your application should be in a try/catch block. And I have often seen this not being the case, in particular when the calls are already placed in a catch block. The other option is of course to ensure that the account has administrative privileges or that the event log source is already registered. But is it really possible to be 100% sure of that in production environment?

What I find surprising is that there doesn’t seem to be any consistency between different trace listeners provided in the framework. TextWriterTraceListener has a similar security related problem when the user doesn’t have write permissions for the specified log file location. But this situation is handled in the trace listener itself which simply doesn’t write the information to the log file if it can’t. It doesn’t raise any exceptions which (at least for me) is the expected behavior. Why is this not the case with EventLogTraceListener?

Since version 1.5 CruiseControl.NET includes an FTP Task which can be used for uploading build results to a remote FTP server. Its most typical use case is for publishing websites to a remote server. Configuring this correctly requires some thought and at the moment there are very few helpful resources available. Below are some suggestions for the most simple case when the files to be uploaded only have to be retrieved from the source control and there is no additional build process involved.

The core of the configuration is of course the FTP task. Its configuration is pretty straightforward. You only need to supply it with login details and the path to the local and the remote folder. This also means that there are almost no configuration options available – you can only choose between uploading the files recursively or not.

The problem with the lack of options is that most source control systems put additional files in the local folder which you don’t want to upload. In the case of Subversion this is a .svn folder in every local folder. Because the FTP task can’t be configured to skip or ignore certain files, these files have to be deleted before the FTP task is executed. The best way to do that is by executing a batch file:

FOR /F "tokens=*" %%G IN ('DIR /B /AD /S *.svn*') DO RMDIR /S /Q "%%G"

If you do this directly in the working directory where the files are retrieved from Subversion, then the next retrieval from source control will fail unless you set the cleanCopy configuration element to true and retrieve all of the files from Subversion every time. Doing this has two downsides:

• If the project is large, this will significantly decrease performance. Downloading lots of files from Subversion takes time.
• The file timestamps will be reset every time. This information could potentially be useful for the FTP task (but isn’t yet as I explain at the end of the post).

To avoid this you should first copy the retrieved files to a different folder. The best way to do this is by using the build publisher. Just don’t forget to set the alwaysPublish configuration element to true. Otherwise it won’t copy anything because you are using it in the tasks section where the build is not yet successful.

To sum it all up, your tasks section should something look like this (I’ve used preprocessor constants for most of the values to increase readability):

<tasks>
    <buildpublisher>
        <sourceDir>$(WorkingDir)</sourceDir>
        <publishDir>$(PublishDir)</publishDir>
        <cleanPublishDirPriorToCopy>true</cleanPublishDirPriorToCopy>
        <useLabelSubDirectory>false</useLabelSubDirectory>
        <alwaysPublish>true</alwaysPublish>
    </buildpublisher>
    <exec>
        <executable>$(ClearSvnFilesBatch)</executable>
        <baseDirectory>$(PublishDir)</baseDirectory>
    </exec>
    <ftp>
        <serverName>$(FtpServerName)</serverName>
        <userName>$(FtpUserName)</userName>
        <password>$(FtpPassword)</password>
        <action>UploadFolder</action>
        <ftpFolderName>$(FtpRemoteFolder)</ftpFolderName>
        <localFolderName>$(PublishDir)</localFolderName>
        <recursiveCopy>true</recursiveCopy>
    </ftp>
</tasks>

I’ll conclude this post with a bad news. Unfortunately there is no support in the FTP task to only upload the files that have changed since the previous build. For a small website this might not be an issue but for larger ones this is quite a problem. You really don’t want to upload tens or even hundreds of megabytes every time. Let’s just hope that the issue will be fixed soon.

Recently I’ve been solving some issues with queue starvation in CruiseControl.NET. Since I haven’t found much information about this problem I’ve decided to round up my findings in this blog post.

So, what is queue starvation? It’s a problem that can occur when scheduling tasks in priority queues if there are too many high priority tasks so that lower priority tasks never get scheduled and are just waiting indefinitely. For example, let’s say we have three queues: high, medium and low priority. Tasks in the high priority one always get processed first. Once this queue is empty the tasks from the medium priority queue get processed. Of course the tasks in the low priority queue get processed only if the other two queues are empty. If there are too many high and medium priority tasks incoming all the time so that the queues never get empty, then the low priority tasks are not processed at all.

And how is this related to CruiseControl.NET? As you might be aware, projects can be assigned to different queues. Only projects in the same queue (as the name implies) are built sequentially while projects in different queues are built in parallel to one another. Queues themselves therefore cannot cause starvation because different queues don’t depend on one another and projects are always put at the end of its queue. This means that sooner or later each project will be built.

Now, queue priorities enter the stage. Each project can be given a priority defining where the project will be put into the queue: before all of the lower priority projects already in the queue instead of at the end. Effectively this partitions a single sequential queue into multiple queues with different priorities as described at the beginning of the post. We have all the necessary prerequisites for queue starvation. If too many projects with higher priorities have to be built (typically caused by checking in code or triggering a forced build), the projects with lower priorities will never get processed.

What I have observed is that this can happen even if higher priority projects don’t have to be built at all. The reason for that is in the way CruiseControl.NET is processing the projects to determine whether there were any source code changes. This is taken care of with an interval trigger which puts a project in the queue, but it only gets built if a modification exists. Nevertheless, even checking if such a modification does exists, takes some time. In extreme cases this can be enough to cause queue starvation and this is also what happened in my case. Lower priority projects never got built, not even during the night and no matter how often a build was forced for them.

Fortunately there is a solution for this problem: the seconds attribute which specifies how often the project is put into the queue for checking if a modification exists. By default it is set to 60 seconds. Increasing the value can solve the queue starvation issue. Instead of simply defining the trigger:

<intervalTrigger />

A longer period can be specified:

<intervalTrigger seconds="300" />

Of course this has its downside: the latency before the project gets built after a modification increases. Therefore you should use this solution with caution. Only increase the value as much as you have to. And don’t do it at all if you can apply any of the other solutions:

• Increase the performance of the build server to speed up the processing and building of the projects.
• Parallelize the build process by using more queues. Put the projects in the same queue only if you have to.
• Don’t use queue priorities if it’s not necessary. In most cases you can get by without them.

A few months ago I worked on a small spare time project which included some manipulation of binary Excel (.xls) files. This seemingly simple task soon turned out to be quite a challenge if you want to handle it right. The post you are reading is a short summary of my experiences. They should make your choices easier if you are about to tackle a similar problem.

The most obvious choice for handling .xls files is Excel automation using the Excel object model. As long as your application is always going to be used interactively, you should be fine. It’s probably the best method to use in spite of a few downsides:

• The object model is COM based which means you’ll have to do interop if you are developing a .NET application. Fortunately there are some nice improvements in .NET Framework which make coping with COM in C# much easier.
• Your application will have to be compiled in 32-bit in order for it to work on 64-bit Windows. This usually isn’t a real limitation just don’t forget to switch the target platform from Any CPU to x86 if you’re doing development on a 32-bit OS to avoid the problem. (You’re going to notice it yourself when developing on a 64-bit OS.)
• Don’t try unit testing the Excel automation code outside your IDE (e.g. on your continuous integration server). As described below this scenario is not supported.

As soon as you want your code to be run non-interactively, you’re out of luck with Excel automation. Since all Office applications assume to be running on the interactive desktop, there are several reasons why such usage is not supported and is even strongly discouraged by Microsoft. If you want to run your code on a web server, as a Windows service, a scheduled task or just automatically test it on your build server, you’ll have to find a different approach. And in this case there are no obvious choices.

Your best bet is to use Open XML file format (.xlsx, .xlsm) instead of the binary one (.xls) if this is an option for you. Since this is a well documented XML based format you can manipulate it directly without running Excel at all. You can even use Open XML SDK for Microsoft Office which includes strongly typed classes that simplify many common tasks.

This is not the case with the binary format. Microsoft doesn’t provide or support any SDK or API for manipulating the files directly. You are only provided with detailed documentation of the format. Based on it some third party solutions have been developed. Aspose and SoftArtisans have their own commercial offerings which I haven’t evaluated because as such they weren’t suitable for me.

On the other hand the only free library that I have found is MyXLS and this one leaves much to be desired. It also seems to be pretty much abandoned with the latest release almost a year ago and only a single commit to the repository in this year so far. That being said, it still might prove useful if you only want to create the files, mostly focusing on the appearance with only minimal requirements regarding formulas. According to the samples this seems to work fine. Reading existing files is another story. You are more or less on your own as soon as you need to read cells with formulas. This made the library useless for me therefore I used another approach based on the fact that the OLE DB Provider for Jet can be used to read and write data in Excel worksheets.

At first I haven’t even considered this possibility because I was convinced that this method can only be used on worksheets designed as database tables (having columns of data with or without header columns). This article proved me wrong and in spite of many issues in the demo project it showed off techniques which turned out really useful for me. The most important one was the possibility to address regions, not only complete worksheets – this can be used to retrieve and set individual cell values as well as for accessing database table-like blocks of data in a part of the worksheet. Let’s take a look at a sample:

SELECT F1 FROM [Sheet1$C2:C2];

This query retrieves a value of a single cell. The same syntax can be used to access any region: after the worksheet name with a trailing $ character the region is defined just like in Excel formulas. If the region doesn’t include column headers (specified by including HDR=No in the Extended Properties of the connection string) the columns are named F# where # is the sequential number of the column in the defined region. Similarly the following query can be used to set a value of an individual cell:

UPDATE [Sheet1$C2:C2] SET F1 = 'Value';

A few more things worth mentioning:

• Be aware of the IMEX option in the connection string. It doesn’t seem to be documented very well, probably its best description is here.
• Make sure you keep the connection to the file open if you plan to do more data access later. Opening the connection takes quite some time therefore the performance will be terrible if you keep closing and reopening it.
• The technique doesn’t seem to work if the worksheet name begins with a space. At least I couldn’t make it work, no matter how I set the quotes.
• I encountered a file which couldn’t be opened in this way but the problem was resolved after I opened the file in Excel and saved it again.

Subversion and CruiseControl.NET can be invaluable tools in your .NET development process. There are many resources available to help you get started which I’ll try to gather in this post along with some of my personal experiences.

Let me start with the list of recommended software:

• VisualSVN Server is the ultimate Windows version of Subversion including a simple setup and powerful management tools. If you are planning to install a Subversion server on Windows it should be your first choice.
• AnkhSVN is a Subversion Source Control Provider (SCC) for Visual Studio. As long as you’re not using Express editions of Visual Studio, this is the suggested way of working with SVN directly from Visual Studio IDE.
• TortoiseSVN is a Windows shell extension for working with Subversion from within Windows Explorer. When you're not working with Visual Studio solutions this is the best choice for using SVN.
• CruiseControl.NET is a continuous integration server including a web dashboard and CCTray - a system tray client application for monitoring and controlling builds.

If you’re not already familiar with the above mentioned products, you should consult their documentation or search for tutorials. I will rather focus on setting up your development and release process. If you haven’t done so already I suggest you first read the following articles by Ariejan de Vroom:

• How to structure your repository
• How to release software properly
• How to fix bugs properly

I mostly based my configuration on the ideas in these articles. I have projects configured in CC.NET to build all copies of the project: trunk (ProjectName-Trunk), all branches (ProjectName-REL-#.#) and all tags (ProjectName-v#.#.#). To identify individual builds I am using CC.NET’s Assembly Version Labeller together with AssemblyInfo MsBuild Community Task.

Assembly Version Labeller is really simple to configure. You only need to add a short snippet to each project:

<labeller type="assemblyVersionLabeller">
    <major>1</major>
    <minor>0</minor>
    <build>0</build>
</labeller>

I’m using the following versioning policy:

• I start each project with version 1.0.0.
• Once it’s ready for release I make a copy of the trunk in the branches directory, named REL-#.# containing the major and the minor version number. Immediately afterwards I bump the version of the trunk (only minor or major and minor, depending on the nature of the new features planned).
• In the release branch I make the necessary changes before release (e.g. I change the AssemblyProduct name to distinguish between development and release quality builds) and make another copy in the tags directory, named v#.#.# containing the major, minor and build version numbers. Immediately afterwards I increase the build version number in the release branch.
• I make no changes to the copies in the tags directory. All bug fixes go to the release branch. Once I’m ready for a new release I repeat the previous step.

Since I don’t specify the revision number directly, the SVN Revision number gets used automatically. This makes it possible to match each build to the revision of the code in SVN.

To put the generated assembly version in the build I am using the AssemblyInfo MsBuild task. There are two steps involved in doing this.

First you need to move the AssemblyProduct, AssemblyInfo and AssemblyFileVersion attributes from the auto generated AssemblyInfo.cs file into a new file. In my case the AssemblyVersion.cs has the following contents:

using System.Reflection;

[assembly: AssemblyProduct("ProjectName DEV")]
[assembly: AssemblyVersion("1.0.0.0")]
[assembly: AssemblyFileVersion("1.0.0.0")]

Next you have to modify your project file (*.csproj) by importing the community tasks and adding a call to the AssemblyInfo MsBuild task:

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />
<Target Name="BeforeBuild">
    <AssemblyInfo Condition="'$(CCNetLabel)' != ''"
        CodeLanguage="CS" 
        OutputFile="Properties\AssemblyVersion.cs" 
        AssemblyProduct="ProjectName TRUNK" 
        AssemblyVersion="$(CCNetLabel)" 
        AssemblyFileVersion="$(CCNetLabel)" />
</Target>

If you have never edited a project file before, you might want to read these first:

• How to: Edit Project Files
• How to: Extend the Visual Studio Build Process

One more thing to note which might not be all that obvious. The Condition in the AssemblyInfo task is met only when building from CC.NET. For builds in Visual Studio the task doesn’t regenerate the AssemblyVersion.cs file therefore the revision number is always 0 and the AssemblyProduct has a DEV suffix as defined in the original file. Also I remove the TRUNK suffix from the AssemblyProduct attribute of the AssemblyInfo task when moving code from trunk to release branches to separate between the two.