Multiple Hangfire instances in one database
I recently got involved in maintaining a project that's using Hangfire for scheduling jobs.
One of the issues with it was unreliable execution of jobs. Often, they failed several times before finally succeeding. In the logs, the cause was always a FileNotFoundException because the assembly with the code to execute couldn't be loaded:
System.IO.FileNotFoundException
Could not resolve assembly 'App1'.
System.IO.FileNotFoundException: Could not resolve assembly 'App1'.
at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark)
at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName)
at Hangfire.Storage.InvocationData.DeserializeJob()
At first, this didn't make much sense. Of course, the deployed application included the required assembly. And this issue shouldn't resolve itself without a new deployment of the application.
While searching for more information in the Hangfire dashboard, I finally noticed that there were two servers running although there should only be a single instance of the application deployed.
It turned out that there was a second application sharing the same database that was also using Hangfire for job scheduling. Of course, this application ran its own set of jobs and didn't include all the assemblies from the first one.
This was the reason for jobs running unreliably. If the wrong application tried to execute a job, it failed because it didn't have the corresponding assembly. And when the job was finally picked up by the right application, it succeeded.
To fix the issue, I changed the Hangfire configuration to use a different database schema for each application. The feature doesn't seem to be (well) documented as I could only find it mentioned in a GitHub issue.
Still, it was easy to setup. The default recommended configuration only needs to be extended with a SchemaName option:
services.AddHangfire(configuration => configuration
.SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseSqlServerStorage(Configuration.GetConnectionString("HangfireConnection"), new SqlServerStorageOptions
{
CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
QueuePollInterval = TimeSpan.Zero,
UseRecommendedIsolationLevel = true,
DisableGlobalLocks = true,
SchemaName = "HangFireApp1",
}));
This will cause the corresponding schema to be generated on application startup:
As a result, the Hangfire instance in each application will be completely independent of the other one. Each will have its own set of jobs and there will only be a single server for each instance. Hence, all jobs will succeed on first try without any assembly loading issues.
A sample with two applications configured like that is available in my GitHub repository. In a previous commit, separate database schemas for the applications are not yet configured and therefore the applications manifest the issue with jobs occasionally failing.
By default, Hangfire state is persisted in the configured database in a separate schema named Hangfire. This means that if multiple instances of Hangfire are configured with the same database, they will all share their state and run the same jobs. This might be your intention. But if it's not and you need to run multiple independent Hangfire instances, you can still use only a single database if you set a different database schema name for each instance.
