Perfomance testing within Unit tests bad idea
11/10/2018
2 minutes
I have doing performance tests in regular Console apps and BenchmarkDotNet for a while. Recently I have been seeing people doing the same from unit tests. At first it seems to have many advantages to run a unit test to measure how long a certain peace of code runs.
Certain things I noted immediately:
The time to run a unit tests != the time your peace of code runs
It is a lot more difficult to measure only a certain portion of an algorithm, you will likely to include measuring test initializers
Running along with a test framework brings a lot of dependencies, which you would not have in a regular empty console application
Console out: some test frameworks makes it difficult to write out to the console, for example in xUnit you can use a special dependency: ITestOutputHelper
Memory profiling is more difficult too: capturing GC heaps for comparisons brings a lot of objects into the view, that are allocated by the test framework
Test might be run in Debug
CI/CD pipelines with multiple build agents might measure different results because of different load/underlying hardware
In conclusion, I would still suggest to use regular Release built C# console apps with a well-known benchmark framework for performance testing.