Perfomance testing within Unit tests bad idea

I have doing performance tests in regular Console apps and BenchmarkDotNet for a while. Recently I have been seeing people doing the same from unit tests. At first it seems to have many advantages to run a unit test to measure how long a certain peace of code runs.

Certain things I noted immediately:

  • The time to run a unit tests != the time your peace of code runs

  • It is a lot more difficult to measure only a certain portion of an algorithm, you will likely to include measuring test initializers

  • Running along with a test framework brings a lot of dependencies, which you would not have in a regular empty console application

  • Console out: some test frameworks makes it difficult to write out to the console, for example in xUnit you can use a special dependency: ITestOutputHelper

  • Memory profiling is more difficult too: capturing GC heaps for comparisons brings a lot of objects into the view, that are allocated by the test framework

  • Test might be run in Debug

  • CI/CD pipelines with multiple build agents might measure different results because of different load/underlying hardware

In conclusion, I would still suggest to use regular Release built C# console apps with a well-known benchmark framework for performance testing.