A .NET application in the company I work for recently started crashing under Windows Vista when trying to open a window implemented in an ActiveX DLL. Investigations showed that they were caused by an AccessViolationException "Attempted to read or write protected memory. This is often an indication that other memory is corrupt.". The cause was one of the ActiveX controls in the window. When instantited directly in a .NET Form the exception above was contained in a TargetInvocationException "Unable to get the window handle for the '<control name>' control. Windowless ActiveX controls are not supported.". Knowing that the control didn't suddenly turn windowless we dug deeper.
It turned out that the problem appeared with .NET Framework 2.0 Service Pack 1. Apparently it caused the C# compiler in Visual Studio 2005 and later to set the NXCOMPAT bit for all build targets without an option to turn this new behavior off. For those who don't know, this means that DEP (data execution prevention) will kick in unless it is turned off completely in the operating system. This wouldn't be a big deal unless ATL before Visual Studio 2005 didn't have a bug which caused the heap allocated memory not to be flagged as executable, which under the new circumstances results as the already mentioned exception. Windows XP has DEP turned off by default therefore everything still works but in Windows Vista it is turned on and prevents such application from functioning properly.
The best solution would of course be to recompile the ActiveX controls in Visual Studio 2005 or later but this might not be possible if they are supplied by a third party. In this case the most obvious approach to dealing with the situation is to disable DEP in Vista. There is a Data Execution Prevention tab in the Performance Options which open when you click the Settings button in the Performance frame of the Advanced tab in the System Properties dialog but it only allows switching between DEP for all processes with defined exceptions and DEP for essential Windows programs and services, i.e. executables flagged with the NXCOMPAT bit. The only way to turn DEP completely off is executing the following command with administrative privileges:
bcdedit.exe /set {current} nx AlwaysOff
After restart DEP will be turned off and problematic binaries will work once again. To restore the previous (default) state replace AlwaysOff with OptIn. The AlwaysOn option enables DEP for all processes and might cause additional problems (e.g. Google Calendar Sync in its current version 0.9.3.2 doesn't work in this mode).
Unfortunately, this solution would require all your Vista using customers to disable DEP as well which really isn't an option for commercial software. Fortunately, there is another solution. Although there is no compiler option to turn keep the NXCOMPAT bit unset, you can still do this after compilation using the editbin.exe which comes with C++ compiler for Visual Studio 2005 and later:
editbin.exe /NXCOMPAT:NO <filename.exe>
This command removes the NXCOMPAT bit and restores the behavior before .NET 2.0 SP1. If your assembly was signed it also invalidates the signature so you'll have to resign it:
sn.exe -R <filename.exe> <keyfile.snk>
You can automate this by putting the two commands in the Post-build event command line for the project:
editbin.exe /NXCOMPAT:NO "$(TargetPath)"
sn.exe -R "$(TargetPath)" "$(ProjectDir)<keyfile.snk>"
You might need to call vcvars32.bat before that to put the required executables in path and enable editbin.exe dependencies to be resolved. Note that in a completely automated build scenario using MSBuild you'll have to specify full path for the vcvars32.bat because $(DevEnvDir) resolves to *Undefined* outside Visual Studio 2005. Also your strong name key should be in a snk file instead of a password protected pfx file because sn.exe doesn't allow the password to be read from a redirected standard input.