Mid-Life Crisis and Polly
07/17/2019
Problem
I have been working on an application sending out millions of HTTP Post requests. The application has an issue, it has a high memory usage. It is using 2 GB over the expected of a few hundred megabytes.
An investigation revealed that a percentage of (random) requests are failing because of invalid request content. When a request fails, the application has a built in resiliency policy to wait with an exponential backoff and retry the failed requests. Unfortunately, that failed request is determined to fail again, hence triggering the above policy 3 times, before giving up for all.
One could figure, that the retry policy is referencing the failing requests for considerably longer period in memory. Let's not trust our gusts but do some measurements instead.