How do I configure multiple audience

I have the following configuration and I have played around with it but I still get the audience mismatch exception.

var samlConfiguration = new SAMLConfiguration
{
LocalServiceProviderConfiguration = new LocalServiceProviderConfiguration
{
Name = local.Name,
AssertionConsumerServiceUrl = local.AssertionConsumerServiceUrl,
LocalCertificateFile = local.LocalCertificateFile
}
};

foreach (var serviceProvider in samlConfig.ServiceProvider)
samlConfiguration.AddPartnerServiceProvider(new PartnerServiceProviderConfiguration() {
Name = serviceProvider.Name,
AssertionConsumerServiceUrl = serviceProvider.AssertionConsumerServiceUrl,
LocalCertificateFile = serviceProvider.LocalCertificateFile
});

samlConfiguration.AddPartnerIdentityProvider(
new PartnerIdentityProviderConfiguration()
{
Name = samlConfig.IdentityProvider.Name,
SignAuthnRequest = samlConfig.IdentityProvider.SignAuthnRequest,
WantSAMLResponseSigned = samlConfig.IdentityProvider.WantSAMLResponseSigned,
WantAssertionSigned = samlConfig.IdentityProvider.WantAssertionSigned,
WantAssertionEncrypted = samlConfig.IdentityProvider.WantAssertionEncrypted,
SingleSignOnServiceUrl = samlConfig.IdentityProvider.SingleSignOnServiceUrl,
SingleLogoutServiceUrl = samlConfig.IdentityProvider.SingleLogoutServiceUrl,
PartnerCertificateFile = samlConfig.IdentityProvider.PartnerCertificateFile,
DisableAudienceRestrictionCheck = samlConfig.IdentityProvider.DisableAudienceRestrictionCheck
});

Is the only way to support multiple audience via multiple configurations with each audience configure using LocalServiceProviderConfiguration?

As per the SAML specification, the audience restriction in the SAML assertion should match the local service provider name.
If it doesn’t you get the exception you’re seeing unless you set DisableAudienceRestrictionCheck to true.
I wouldn’t recommend setting up multiple configurations.
Instead, you should check with the IdP why they’re setting the audience restriction to an unexpected value.
The worst case scenario is that you have to set DisableAudienceRestrictionCheck to true.