Preview .NET with BenchmarkDotnet
09/08/2024
6 minutes
In a previous post I explored how someone can setup a custom .NET SDK and Runtime for development. It consists of a script downloads the .NET runtime to a custom folder, another script that sets up environment variables to use the given .NET version with Visual Studio and a global.json / nuget.config for build time dependencies with Visual Studio and the .NET CLI.
In this post I evolve the previous setup:
Install and use an alpha release of .NET 10 Runtime with .NET 9 SDK.
Setup BenchmarkDotNet performance measurements to use .NET 10
Update the runtime to .NET 10
First, let's install .NET 10 runtime with .NET 9 SDK. In the install script of the previous post, update the sdkVersion and runtimeVersion variables then run the script for installation:
$sdkVersion = '9.0.100-preview.7.24407.12' $runtimeVersion = '10.0.0-alpha.1.24425.2'
The script will also update the global.json file. Extend the NuGet.config file with a new source for the .NET 10 runtime dependencies:
<add key="dotnet10" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10/nuget/v3/index.json" />
At this point starting Visual Studio with startvs.cmd sets up the PATH pointing to the downloaded SDK and runtime. To switch to the .NET 10 at runtime, the build properties of the .csproj should define a rollforward option to the latest major version (as opposed to the more common latest minor version): <RollForward>latestmajor</RollForward>. As a developer, I find this as a rare case, normally I am using the same major versions for the runtime and the SDK.
To check the application is running on .NET 10, add the following line to Program.cs:
Console.WriteLine(RuntimeInformation.FrameworkDescription);
Running the application prints:
.NET 10.0.0-alpha.1.24425.2
Setup BenchmarkDotNet
BenchmarkDotNet is a popular microbenchmarking library for .NET. It helps to reliably measure and compare the performance of .NET methods. It reveals statistical metrics for execution time, memory allocations, it can give insights of the executed assembly code. It has many other features, but these are the most common ones.
BenchmarkDotNet is built to be able to execute benchmarks targeting different runtimes or frameworks. The built-in mechanism can be used to target the benchmarks against .NET 10 Runtime. However, the setup requires more than adding [SimpleJob] attribute on the benchmarked class.
In this case a custom configuration will be required that configures where the .NET runtime is located. This config is then passed to the benchmark runner:
var config = DefaultConfig.Instance.AddJob(Job.Default...);
BenchmarkRunner.Run<Benchmarks>(config);
In this case the benchmark is compiled with .NET 9 SDK .NET 9.0.0-preview.7.24405.7 that is my default installation on the machine. Visual Studio and the solution is opened without setting up the .NET the PATH environment variable pointing to the custom .NET installation. Instead, these are configured in the CsProjCoreToolchain for the given benchmark job:
Job.Default.WithToolchain(CsProjCoreToolchain.From(
new NetCoreAppSettings(
targetFrameworkMoniker: "net9.0",
runtimeFrameworkVersion: "10.0.0-alpha.1.24425.2",
name: "latest runtime",
customDotNetCliPath: "..\\..\\..\\..\\.dotnet\\dotnet.exe")));
BenchmarkDotNet has a Host process that builds and executes the test assembly for the given jobs. In the case above (and results below) it is using the installed .NET 9 SDK (9.0.100-preview.7.24407.12) with the corresponding host runtime (9.0.24.40507). There is a single job for the benchmark, that is executed on .NET 10.0.0 (10.0.24.42502) runtime.
BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4037/23H2/2023Update/SunValley3) 12th Gen Intel Core i7-1255U, 1 CPU, 12 logical and 10 physical cores .NET SDK 9.0.100-preview.7.24407.12 [Host] : .NET 9.0.0 (9.0.24.40507), X64 RyuJIT AVX2 Job-DUTWQL : .NET 10.0.0 (10.0.24.42502), X64 RyuJIT AVX2 Toolchain=latest runtime | Method | Mean | Error | StdDev | |----------- |----------:|----------:|----------:| | TestMethod | 0.0133 ns | 0.0170 ns | 0.0159 ns |
When starting the application with the PATH pointing to the local .NET SDK and installation, the Host process will also execute on .NET 10 (given the roll forward is enabled in the csproj).
In this case it prints the results as:
.NET SDK 9.0.100-preview.7.24407.12 [Host] : .NET 10.0.0 (10.0.24.42502), X64 RyuJIT AVX2 Job-APRHKM : .NET 10.0.0 (10.0.24.42502), X64 RyuJIT AVX2 Toolchain=latest runtime | Method | Mean | Error | StdDev | Median | |----------- |----------:|----------:|----------:|-------:| | TestMethod | 0.0015 ns | 0.0029 ns | 0.0027 ns | 0.0 ns |
Note, that the SDK and the TFm is still pointing to .NET 9.
Benchmarking Multiple .NET Runtimes
Runtime 9.0.0-preview.7.24405.7 is installed locally as part of the 9.0.100-preview.7.24407.12 SDK. In the next case two jobs are added to the configuration, so that the benchmarks are executed using the .NET 9 and .NET 10 runtimes. The TFM and the SDK used is still .NET 9:
var config = DefaultConfig.Instance.AddJob(Job.Default.WithToolchain(CsProjCoreToolchain.From( new NetCoreAppSettings( targetFrameworkMoniker: "net9.0", runtimeFrameworkVersion: "10.0.0-alpha.1.24425.2", name: ".NET 10 Runtime", customDotNetCliPath: "..\\..\\..\\..\\.dotnet\\dotnet.exe")))); config = config.AddJob(Job.Default.WithToolchain(CsProjCoreToolchain.From( new NetCoreAppSettings( targetFrameworkMoniker: "net9.0", runtimeFrameworkVersion: "9.0.0-preview.7.24405.7", name: ".NET 9 Runtime", customDotNetCliPath: "..\\..\\..\\..\\.dotnet\\dotnet.exe"))));
Both jobs sets the TFM to .NET 9.0, both points to a local dotnet.exe in the folder of the custom SDK and Runtime installation. The runtimeFrameworkVersion parameter specifies .NET 10 for the first job and .NET 9 for the second job's execution.
Running the above configuration prints:
BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4037/23H2/2023Update/SunValley3) 12th Gen Intel Core i7-1255U, 1 CPU, 12 logical and 10 physical cores .NET SDK 9.0.100-preview.7.24407.12 [Host] : .NET 10.0.0 (10.0.24.42502), X64 RyuJIT AVX2 Job-FTSHCB : .NET 10.0.0 (10.0.24.42502), X64 RyuJIT AVX2 Job-XVQYZI : .NET 9.0.0 (9.0.24.40507), X64 RyuJIT AVX2 | Method | Job | Toolchain | Mean | Error | StdDev | Median | |----------- |----------- |---------------- |----------:|----------:|----------:|----------:| | TestMethod | Job-FTSHCB | .NET 10 Runtime | 0.0000 ns | 0.0001 ns | 0.0001 ns | 0.0000 ns | | TestMethod | Job-XVQYZI | .NET 9 Runtime | 0.0081 ns | 0.0087 ns | 0.0082 ns | 0.0097 ns |
Extending Benchmarks on .NET 8
Extending the current benchmark with another .NET runtime, in this case .NET 8 follows the same pattern. Add a new job to the configuration with runtimeFrameworkVersion pointing to the runtime installed on the machine (or in a custom folder). In this case .NET 8 is installed on my machine, hence customDotNetCliPath is not set. To check the runtimes installed on a machine use the dotnet --info CLI command.
config = config.AddJob(Job.Default.WithToolchain(CsProjCoreToolchain.From(
new NetCoreAppSettings(
targetFrameworkMoniker: "net8.0",
runtimeFrameworkVersion: "8.0.8",
name: ".NET 8 Runtime"))));
Notice, that the targetFrameworkMoniker is also downgraded to .NET 8, otherwise the .NET 8 runtime would not be able to execute the application targeting .NET 9.Along with this change, the csproj also updated to set both .NET 8 and .NET 9 as a TFM: <TargetFrameworks>net9.0;net8.0</TargetFrameworks>.
When executing this benchmark it prints the following results on the output:
.NET SDK 9.0.100-preview.7.24407.12 [Host] : .NET 9.0.0 (9.0.24.40507), X64 RyuJIT AVX2 Job-WJPVJW : .NET 10.0.0 (10.0.24.42502), X64 RyuJIT AVX2 Job-DEFFSY : .NET 8.0.8 (8.0.824.36612), X64 RyuJIT AVX2 Job-FBFAIS : .NET 9.0.0 (9.0.24.40507), X64 RyuJIT AVX2
There are 3 jobs, the first and the last are configured as shown previously. The Host is .NET 9. The benchmark is build against target framework monikers 8 and 9 using the corresponding .NET SDK installed on the machine. The benchmarks are executed on .NET Runtime 8, 9 and 10.