Damir's Corner.

Parallel builds for .NET projects

I had the pleasure of troubleshooting an issue that occurred on our build server after we tried to improve performance by parallelizing the build. We need to build the same project in several configurations, so we started running the dotnet publish command for each configuration in parallel, of course with a different output folder for each one:

dotnet publish MyProject/MyProject.csproj --configuration MyConfig1 --output MyConfig1

Once we did that, occasionally one of the configurations failed with the following error:

The file .../obj/project.assets.json already exists.

Although by default each configuration uses a separate subfolder inside obj for intermediate files, the project.assets.json is put directly inside obj and is thus shared across all builds. Which, obviously, can cause the error above when the timing is just right.

The first instinct for solving the issues was to specify custom intermediate path using the BaseIntermediateOutputPath MSBuild parameter:

dotnet publish MyProject/MyProject.csproj --configuration MyConfig --output MyConfig /p:BaseIntermediateOutputPath=MyConfig-obj/

While this makes sure that the project.assets.json files from different builds end up in different folders and therefore can't interfere with each other, it introduces a new set of problems resulting in several errors because of duplicate assembly-level attributes:

Duplicate global::System.Runtime.Versioning.TargetFrameworkAttribute attribute

Duplicate System.Reflection.AssemblyCompanyAttribute attribute

Duplicate System.Reflection.AssemblyConfigurationAttribute attribute

Duplicate System.Reflection.AssemblyFileVersionAttribute attribute

Duplicate System.Reflection.AssemblyInformationalVersionAttribute attribute

Duplicate System.Reflection.AssemblyProductAttribute attribute

Duplicate System.Reflection.AssemblyTitleAttribute attribute

Duplicate System.Reflection.AssemblyVersionAttribute attribute

They occur because by default as part of the build a C# source file with assembly-level attributes is generated in the intermediate path, named MyProject.AssemblyInfo.cs. And while the custom intermediate path currently in use is excluded from the project, that's not true for custom intermediate paths of other builds running in parallel. This means that the file generated in MyConfig1-obj will be included in the build which uses MyConfig2-obj.

Of course, there are ways to work around that, but fortunately there is a simpler way to resolve the original issue. Namely, the project.assets.json file is generated during the restore phase of the build which happens implicitly as part of the dotnet publish command. This can be avoided with the --no-restore option.

dotnet publish MyProject/MyProject.csproj --no-restore --configuration MyConfig --output MyConfig

Now the file won't be generated by each of the builds running in parallel, so there's no chance for conflict. Just make sure to explicitly restore the packages with the dotnet restore command before doing any builds which will also generate the project.assets.json file:

dotnet restore

Also make sure to run the dotnet restore command with the same runtime identifier which you're going to use for your build. For example, if you're going to publish your project for linux-x64 and you're building on Windows, use the --runtime linux-x64 option both for the initial dotnet restore command and for the dotnet publish command after that:

dotnet restore --runtime linux-x64
dotnet publish MyProject/MyProject.csproj --no-restore --configuration MyConfig --output MyConfig --runtime linux-x64 --self-contained false /p:PublishReadyToRun=true

Otherwise, the dotnet publish command will complain that the restore hasn't run:

Assets file MyProject\obj\project.assets.json doesn't have a target for net8.0/linux-x64. Ensure that restore has run and that you have included net8.0 in the TargetFrameworks for your project. You may also need to include linux-x64 in your project's RuntimeIdentifiers.

I was surprised that there isn't much documentation about parallel builds for .NET. This made it more difficult to find a proper solution than it should have been. At least, in the end it can all be explained: the project.assets.json file is not configuration specific and is therefore placed in the root of the intermediate path instead of the configuration specific subfolder. Doing the restore beforehand instead of as part of each build also makes sense, both when running the builds in parallel or sequentially.