Keyset does not exist error on Azure

Hi ComponentSpace,

The following error has recently started showing up randomly for us on our Azure environment after running the below code in our .NET Core 2.2 application

var result = await SamlServiceProvider.ReceiveSsoAsync();

Receiving an SSO response from a partner identity provider has failed.
Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Keyset does not exist
at ComponentSpace.Saml2.SamlServiceProvider.DecryptSamlAssertionAsync(AssertionListItem assertionListItem)
at ComponentSpace.Saml2.SamlServiceProvider.GetSamlAssertionAsync(SamlResponse samlResponse)
at ComponentSpace.Saml2.SamlServiceProvider.ProcessSamlResponseAsync(XmlElement samlResponseElement)
at ComponentSpace.Saml2.SamlServiceProvider.ReceiveSsoAsync()


I’ve been told that it happens randomly. I’ve looked around on the forums and I see topics mentioning that it may be related to permissions
https://componentspace.com/Forums/8629/CryptographicException-Keyset-does-not-exist
https://www.componentspace.com/Forums/29/Troubleshooting-Loading-X.509-Certificates

We store our certificates in the database and load them through a custom ICertificateLoader implementation.

return new X509Certificate2(file, password, X509KeyStorageFlags.Exportable | X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet);

Our ComponentSpace.Saml2.dll is version 2.0.6, and we recently updated our application to .NET Core 2.2.

On azure, we run our application in an App Service.

Do you have any suggestions? I believe it has only started happening recently.



Unfortunately the Windows crypto API, and consequently the .NET Core crypto API, don’t provide very helpful error messages.
From the stack trace, I see that the SAML assertion is being decrypted. This is done using the private key associated with the X.509 certificate.
The most likely causes are either the private key is missing or the calling application doesn’t have read access to the private key.
Make sure that the permissions are set on the associated private key container.
The following link refers to the private key container and setting permissions on it.
https://www.componentspace.com/Forums/29/Troubleshooting-Loading-X.509-Certificates
Is the X509Certificate2 constructor code you supplied how you’re loading the certificate?
If the certificates are in a database, are you copying them to the file system?

[quote]
ComponentSpace - 6/3/2019
Unfortunately the Windows crypto API, and consequently the .NET Core crypto API, don't provide very helpful error messages.
From the stack trace, I see that the SAML assertion is being decrypted. This is done using the private key associated with the X.509 certificate.
The most likely causes are either the private key is missing or the calling application doesn't have read access to the private key.
Make sure that the permissions are set on the associated private key container.
The following link refers to the private key container and setting permissions on it.
https://www.componentspace.com/Forums/29/Troubleshooting-Loading-X.509-Certificates
Is the X509Certificate2 constructor code you supplied how you're loading the certificate?
If the certificates are in a database, are you copying them to the file system?

[/quote]

Yes, the constructor code is how we load the certificate.

In our custom ICertificateLoader implementation, we override LoadCertificateFromFileAsync


Task ICertificateLoader.LoadCertificateFromFileAsync(string certificateFile, string certificatePassword)
{
if (certificateFile.Equals("idp.cer", StringComparison.InvariantCultureIgnoreCase))
{
var samlKey = SamlKeyService.GetSamlKey();

var certificate = LoadCertificate(samlKey.PublicKey, null);

return Task.FromResult(certificate);
}

if (certificateFile.Equals("sp.pfx", StringComparison.InvariantCultureIgnoreCase))
{
var samlConfiguration = SamlConfigurationService.GetSamlConfiguration();

if (samlConfiguration?.PrivateKey != null &&
!string.IsNullOrEmpty(samlConfiguration?.PrivateKeyPassphrase))
{
var privateKeyPassphrase = DecryptPassphrase(samlConfiguration.PrivateKeyPassphrase);

var certificate = LoadCertificate(samlConfiguration.PrivateKey, privateKeyPassphrase);

return Task.FromResult(certificate);
}
}

throw new ArgumentException($"Could not load unexpected certificate file {certificateFile}");
}

private X509Certificate2 LoadCertificate(byte[] file, string password)
{
return new X509Certificate2(file, password, X509KeyStorageFlags.Exportable | X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet);
}


We also have an AbstractSamlConfigurationResolver implementation too, where we override GetLocalServiceProviderConfigurationAsync, GetPartnerIdentityProviderConfigurationAsync and GetPartnerIdentityProviderNamesAsync, because we get our certificates from the database.

Since we are just deploying on Azure App Services, are we able to set permissions on the private key container?

Sorry, the “file” parameter threw me. I thought you were somehow loading a file rather than a byte[].
I’ll do some experimenting with deployment as an app service to see if I can reproduce the issue.

I haven’t been able to reproduce the issue. Perhaps you could create a small test app that simply loads the certificate and uses the private key in an attempt to reproduce the issue more consistently. Let me know what you find.

Were you able to figure out the solution? I’m running into a similar issue.
Thanks.

We never heard back and it’s not something we’ve been able to reproduce.

What SAML for ASP.NET Core version are you using?

Is this an intermittent issue? How easy is it to reproduce?

To help with the debugging, could you create a small test app that simply loads the certificate and uses the private key in an attempt to reproduce the issue more consistently?

[quote]
ComponentSpace - 1/6/2021
We never heard back and it's not something we've been able to reproduce.

What SAML for ASP.NET Core version are you using?

Is this an intermittent issue? How easy is it to reproduce?

To help with the debugging, could you create a small test app that simply loads the certificate and uses the private key in an attempt to reproduce the issue more consistently?
[/quote]

Thanks for your reply.
Our app targets "netstandard2.0" and is deployed on Azure App Service.
Cert is created like this:
new X509Certificate2(
Convert.FromBase64String(certContentSecret.Value),
string.Empty);

Certificate is cached for 60minutes using C# ObjectCache.

This cert is then used to acquireToken like this:
await context.AcquireTokenAsync(audience, certCred, sendX5c: true);

The issue is: acquireTokenAsync throws the exception "Keyset does not exist".

This issue happens every few months and is not easy to replicate. Sometimes it would go away by itself and sometimes we'd need to change the app service plan and that will fix it.

Any pointers/suggestions would be extremely helpful!

Are you using SAML for ASP.NET Core? If so, which version?

[quote]
ComponentSpace - 1/6/2021
Are you using SAML for ASP.NET Core? If so, which version?
[/quote]

We are not using SAML. We are using OAuth.

My suspicion is this is related to the private key.

Have you tried something like the following?


new X509Certificate2(
Convert.FromBase64String(certContentSecret.Value),
string.Empty,
X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.Exportable | X509KeyStorageFlags.EphemeralKeySet);



The EphemeralKeySet flag specifies that the private key should not persisted to disk.

[quote]
ComponentSpace - 1/6/2021
My suspicion is this is related to the private key.

Have you tried something like the following?


new X509Certificate2(
Convert.FromBase64String(certContentSecret.Value),
string.Empty,
X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.Exportable | X509KeyStorageFlags.EphemeralKeySet);



The EphemeralKeySet flag specifies that the private key should not persisted to disk.
[/quote]

We've tried:

new X509Certificate2(
Convert.FromBase64String(certContentSecret.Value),
string.Empty,
X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable);


Shouldn't the PersistKeySet option have fixed the issue as then the key files will not be deleted upon certificate disposal?
Also we are targetting netstandard 2.0 and the option for "EphemeralKeySet" was introduced in .netstandard 2.1. We have a lot of dependencies on this project so not sure if we could just upgrade to 2.1

We haven’t been able to reproduce this error so it’s not something we’ve been able to investigate thoroughly.

Have you asked Microsoft support or on stackoverflow?

If you find a solution please add a comment here for anyone else who might run into the issue.