A .NET application in the company I work for recently started crashing under Windows Vista when trying to open a window implemented in an ActiveX DLL. Investigations showed that they were caused by an AccessViolationException "Attempted to read or write protected memory. This is often an indication that other memory is corrupt.". The cause was one of the ActiveX controls in the window. When instantited directly in a .NET Form the exception above was contained in a TargetInvocationException "Unable to get the window handle for the '<control name>' control. Windowless ActiveX controls are not supported.". Knowing that the control didn't suddenly turn windowless we dug deeper.

It turned out that the problem appeared with .NET Framework 2.0 Service Pack 1. Apparently it caused the C# compiler in Visual Studio 2005 and later to set the NXCOMPAT bit for all build targets without an option to turn this new behavior off. For those who don't know, this means that DEP (data execution prevention) will kick in unless it is turned off completely in the operating system. This wouldn't be a big deal unless ATL before Visual Studio 2005 didn't have a bug which caused the heap allocated memory not to be flagged as executable, which under the new circumstances results as the already mentioned exception. Windows XP has DEP turned off by default therefore everything still works but in Windows Vista it is turned on and prevents such application from functioning properly.

The best solution would of course be to recompile the ActiveX controls in Visual Studio 2005 or later but this might not be possible if they are supplied by a third party. In this case the most obvious approach to dealing with the situation is to disable DEP in Vista. There is a Data Execution Prevention tab in the Performance Options which open when you click the Settings button in the Performance frame of the Advanced tab in the System Properties dialog but it only allows switching between DEP for all processes with defined exceptions and DEP for essential Windows programs and services, i.e. executables flagged with the NXCOMPAT bit. The only way to turn DEP completely off is executing the following command with administrative privileges:

bcdedit.exe /set {current} nx AlwaysOff

After restart DEP will be turned off and problematic binaries will work once again. To restore the previous (default) state replace AlwaysOff with OptIn. The AlwaysOn option enables DEP for all processes and might cause additional problems (e.g. Google Calendar Sync in its current version 0.9.3.2 doesn't work in this mode).

Unfortunately, this solution would require all your Vista using customers to disable DEP as well which really isn't an option for commercial software. Fortunately, there is another solution. Although there is no compiler option to turn keep the NXCOMPAT bit unset, you can still do this after compilation using the editbin.exe which comes with C++ compiler for Visual Studio 2005 and later:

editbin.exe /NXCOMPAT:NO <filename.exe>

This command removes the NXCOMPAT bit and restores the behavior before .NET 2.0 SP1. If your assembly was signed it also invalidates the signature so you'll have to resign it:

sn.exe -R <filename.exe> <keyfile.snk>

You can automate this by putting the two commands in the Post-build event command line for the project:

editbin.exe /NXCOMPAT:NO "$(TargetPath)"
sn.exe -R "$(TargetPath)" "$(ProjectDir)<keyfile.snk>"

You might need to call vcvars32.bat before that to put the required executables in path and enable editbin.exe dependencies to be resolved. Note that in a completely automated build scenario using MSBuild you'll have to specify full path for the vcvars32.bat because $(DevEnvDir) resolves to *Undefined* outside Visual Studio 2005. Also your strong name key should be in a snk file instead of a password protected pfx file because sn.exedoesn't allow the password to be read from a redirected standard input.

By switching from C# with P/Invoke calls to Managed C++ when implementing a managed wrapper for the ANSI C style library I stumbled upon, I wanted to avoid the tedious and error-prone task of writing the P/Invoke signatures for function calls and user-defined types for the structures they used. As a side result I also managed to avoid most of the advanced marshalling issues with complex data structures.

With simple value types not needing any explicit marshalling only strings need special attention since char* and System::String can’t be implicitly converted between. The Marshal class implements the methods necessary for doing this as demonstrated in the following snippet:

String^ MCPP::Together(String^ first, String^ second)
{
   // marshal managed strings to unamanged memory
   IntPtr firstPtr = Marshal::StringToHGlobalAnsi(first);
   IntPtr secondPtr = Marshal::StringToHGlobalAnsi(second);

   // cast unmanaged buffer to a characer array
   char* firstNative = static_cast<char*>(firstPtr.ToPointer());
   char* secondNative = static_cast<char*>(secondPtr.ToPointer());

   // perform some unmanaged calls
   int bufferSize = strlen(firstNative) + strlen(secondNative) + 4;
   char* resultBuffer = new char[bufferSize];
   sprintf_s(resultBuffer, bufferSize, "%s + %s", firstNative, secondNative);
  
   // marshal unmanaged character array to managed string
   String^ result = Marshal::PtrToStringAnsi(static_cast<IntPtr>(resultBuffer));
  
   // free all unmanaged buffers
   delete[] resultBuffer;
   Marshal::FreeHGlobal(firstPtr);
   Marshal::FreeHGlobal(secondPtr);
  
   // return managed string
   return result;
}

It should be noted that it wouldn’t make any sense switching to unmanaged code to do some string operations only as shown above. This is for demonstration purposes only and you would usually call some unmanaged libraries or the like instead of it. But it is a nice demonstration how cumbersome string operations were in C. If you were doing this once, the sprintf_s function call might have caught your attention. It’s a safe implementation of sprintf which prevents buffer overflows.

Another point worth mentioning is that you have to take care of allocated memory for the conversion since your character array is now allocated on the unmanaged heap. Make sure you use the FreeHGlobal method to do it, not the free (used for freeing memory allocated by *alloc calls – ANSI C style) or delete (used for freeing memory allocated by new calls – C++ style) functions.

To reduce the code overhead of string conversions between managed and unmanaged world you might consider wrapping it into a helper class. This should also help preventing unwanted memory leaks during the conversions.

Less than a month ago I was certain I’d never have to write managed code in C++. With such a set of mind I didn’t really care about the new features being added and improvements being made to managed C++ since the first Visual Studio .NET. Luckily there were people thinking otherwise and as a result this – often overlooked – member of the .NET family got better with every release and is actually quite nice in the 2005 version. Don’t get me wrong – it still is C++ and you probably wouldn’t want to use it if you could get away with C# or VB. But if platform invoke is giving you too many headaches you might want to take a look at it. It could be the easiest way to solve your problem.

To make this a little bit easier for those of you who haven’t used C++ for some time and didn’t bother to check managed C++ at all, I’ll provide a small sample to help out with the beginner’s problems I had when starting out.

In a situation like mine you’ll usually want to write a class library to wrap the unmanaged calls for undisturbed use within you managed project. In my case I’ll be implementing an interface but writing a standalone class is not much different.

Let’s start with the header file. You might remember that C++ requires the separation of class declaration from method definition, the former being done in a header (*.h) file and the latter in a code (*.cpp) file. I admit being a little bit spoiled by not having to do that any more in C#. Let’s name our header MCPP.h (fell free to guess what it stands for):

using namespace System;

namespace DamirsCorner
{
   namespace Samples
   {
      namespace LateBinding
      {
         public ref class MCPP : IBind
         {
         public:
            virtual String^ Together(String^ first, String^ second);
         };
      }
   }
}

It looks like a hybrid between C++ and C# and one could argue that’s what it actually is. I’d like to point out the following:

• The using directive has an additional keyword namespace. Also a double colon (static separator in C++) would be used to separate nested namespaces instead of a dot. You’ll see this in the code file which follows.
• To put your own code into a nested namespace, you’ll have to actually nest the namespace declarations. Separating several of them with double colons won’t work.
• To declare a managed class, use the additional keyword ref.
• Don’t forget the different use of the public keyword for the members in C++ compared to C#.
• The ^ character denotes a reference to a managed object (stored on the managed heap), separating it from unmanaged pointers (* character).
• Notice the interface IBind being implemented by the class. A standalone class wouldn’t need this of course. The virtual keyword in the method declaration would also be skipped in this case.

This is it for the header file. Let’s look at the code file now:

#include "MCPP.h"

using namespace System;
using namespace DamirsCorner::Samples::LateBinding;

String^ MCPP::Together(String^ first, String^ second)
{
   return String::Empty;
}

It’s pretty obvious the method doesn’t really do anything but it’s the skeleton we’re focusing on at the moment. You should turn your attention to the following:

• The code file must include the header file it implements (i.e. defines its methods). To refresh your C++ knowledge: it’s a good habit to use quotes for internal project header files and pointy brackets for external library header files. The compiler looks in the project directory first for the former and in the include directories first for the latter. It checks the other location if it doesn’t find the file but it will work faster if you use the right one (not to mention that by doing it you’re avoiding the problem of locating the wrong include with the same name).
• You can notice the double colons as namespace separators as I mentioned before.
• The methods you’re implementing are not within the class declaration any more. You already declared the class in the header file therefore you only set the method class membership here by prefixing its name with the class name and a colon, just like you were calling a static method.

Completely unexpectedly I stumbled across Platform Invocation Services today when I was introduced to the IBM Tivoli Storage Manager API. It should have been a simple case of getting to know the API and writing a sample application with it, but it turned out that there’s no managed wrapper available for it (in spite of first being told otherwise). Combine that fact with an abundance of low level calls and large structs as their parameters and you can imagine that after one day there’s still a long way to that sample application, not to mention the final solution.

For anybody else like me out there who hasn’t done more than an occasional DllImport call or two the following resources should help getting to grips with the PInvoke basics:

• Platform Invoke Tutorial – a must read for anyone starting with PInvoke. As usual, the follow-up links provide lots of additional useful information.
• PINVOKE.NET – an indispensable source of information on WinAPI PInvoke calls. Not exactly what I needed but still the abundance of example calls turned out helpful.
• The P/Invoke Wizard – a simple tool for converting C/C++ include files to PInvoke declarations. Not perfect and a bit pricy for what it does but still worth it due to the time it can save you when dealing with many large include files.