Hi there,
We use the ‘ImportUrlAsync’ method to pull down the federation metadata for both UK Federation and OpenAthens on a daily basis (as per the recommendation of the UK Federation). We’re implementing the AbstractSamlConfigurationResolver interface so that we can store the metadata in our database.
We noticed that this process consumes a lot of memory and that the memory usage does not drop after the metadata import is complete. After some debugging it doesn’t seem that the memory issues are related to our own code. I have a suspicion that it may be related to some kind of internal caching behaviour of the Component Space library which is holding all of the IDP metadata in memory and not allow it to be garbage collected.
Any advice or idea how we can alleviate these problems would be much appreciated!
We don’t cache the metadata etc so I’m not sure what could be causing the high memory usage.
Please see if you can reproduce the issue using the ImportMetadata project. You might want to change this code to loop around several times. This will help to narrow down the issue.
I’ll see if I can replicate it here as well.
I modified the ImportMetadata project to loop around a few hundred times importing the 71 MB UK federation metadata.
I saw no increase in memory usage over time and, looking at the heap at various times, I saw nothing unusual.
Just as an aside, if you aren’t already you might want to use an ETag and check for the 304 return status to save downloading the metadata unnecessarily.
The CheckForMetadataUpdates example project we include under the Examples\Metadata folder demonstrates this and other mechanisms to minimize the processing required.
I saw no increase in memory usage over time and, looking at the heap at various times, I saw nothing unusual.
Just as an aside, if you aren't already you might want to use an ETag and check for the 304 return status to save downloading the metadata unnecessarily.
The CheckForMetadataUpdates example project we include under the Examples\Metadata folder demonstrates this and other mechanisms to minimize the processing required.
Thank you for the suggestion, I will take a look at that
You’re welcome.
I did some more tests. Indeed the ImportMetadata project does not exhibit any kind of memory issues.
I tried to setup a minimal reproduction of this issue by creating a new web project and adding an endpoint which triggers the import of the UK federation metadata file via the url. This project does exhibit the problems that I described earlier. Each time the endpoint is called, the memory usage of the app goes up and does not come back down. I was able to reach several GB of memory usage in this manner.
Could you please take a look and see whether you can determine why this might happen? I have attached the project where I was able to reproduce the issue as a zip file.
Thanks for the test app. I’ve run it fifty or more times and don’t believe there’s a memory leak although memory usage is high.
If I keep hitting the endpoint, process memory increases until it plateaus at around 1.5GB. Looking at the heap, there’s about 24K objects consuming 133MB. I suspect these numbers will vary depending on how much system memory is available but that’s what I see on my machine. By comparison, when the app starts and prior to hitting the endpoint, process memory usage is 68MB and the heap has 20K objects consuming 2MB.
As an experiment, I added another endpoint which simply called GC.Collect().
I tried various combinations of importing the metadata and calling the garbage collector by browsing to the respective endpoints. For example, sometimes I called the garbage collector after each import. Other times I did ten or twenty imports and then ran the garbage collector a few times. In all cases, after garbage collection completed the process memory usage dropped to 640MB and the heap had 24K objects consuming 2MB.
I’m not suggesting you call the garbage collection in a production application. I merely did this as an experiment.
I suggest doing something similar to confirm the results I saw.
Also, looking at the heap, there are just a couple ComponentSpace objects consuming a few dozen bytes. This makes sense as we’re not keeping references to any of the objects associated with the metadata import.
Given the large size of the metadata document being downloaded and imported, this generates a large number of objects and consumes significant memory. However, the .NET garbage collector will eventually clean things up.
As I mentioned earlier, I suggest employing some of the techniques used by the CheckForMetadataUpdates project to minimize the number of downloads.
Let me know if you still have any concerns.
Thank you for taking the time to investigate
You’re welcome. Let me know if you have any concerns.