# Kestrel HTTP/2 Internals In the previous post I explored some of the inner workings of Kestrel. Kestrel is one of the servers built into [ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/introduction-to-aspnet-core). First, I investigated how [HTTP socket connections are dispatched](https://blog.ladeak.net/posts/kestrel-serving-requests) to a request processor, such as the `Http2Connection`. In a different [post](https://blog.ladeak.net/posts/http2headerframes) I looked into how HEADER frames work with HTTP/2 responses, and what are the main building blocks of writing headers to the response stream in Kestrel. Lastly, I explored how HEADER and CONTINUATION frames are used in ASP.NET Core to [serve response headers larger to the frame size](https://blog.ladeak.net/posts/http2headerframes2). In this post I will explore some of the interactions of the `Http2Stream`, `Http2Connection`, `Http2OutputWriter` and `Http2FrameWriter` types. The HTTP/2 protocol definition can be found in [RFC7540](https://www.rfc-editor.org/rfc/inline-errata/rfc7540.html). I write this post in the .NET8 - .NET9 prerelease timeframe. References to type names, method names and code will reflect the [current](https://github.com/dotnet/aspnetcore/tree/d301328737c4912b95f236c4b349cf46ef7a473a) state of the [ASP.NET Core](https://github.com/dotnet/aspnetcore) repository. ## From Request Frames to Connection An incoming request is handled in `Http2Connection`. A method called [ProcessRequestsAsync](https://github.com/dotnet/aspnetcore/blob/d301328737c4912b95f236c4b349cf46ef7a473a/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs#L249) handles the incoming connection stream. In its main loop it tries to read the next frame received and it branches processing based on the frame's type. For example, HEADER, DATA or WINDOW_UPDATE frames are dispatched to different methods for further validation and processing. `Http2Connection` represents the HTTP/2 connection on the server side. It makes sure that the protocol aspects of an HTTP/2 connection are consistent and correct. Aspects such as: - incoming frames reference existing (or initiate new) streams - padding is correct - the referenced stream is in open state - server limits are not breached (such as max stream count) - it tracks connection level flow control - creates and closes streams - closes the connection on fatal errors - manages timeouts with the help of `TimeoutControl` - handles keep-alive events ## From Connection to Stream When a HEADER *frame* (not to mistaken it with an HTTP request header), is received it creates a new stream. This new stream can be a completely new object or can be rented from a stream pool. This stream is represented by `Http2Stream` type, which also [implements](https://github.com/dotnet/aspnetcore/blob/80460917e1663c2c5e97f215c360b8aee335cc0f/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Stream.cs#L19) `IThreadPoolWorkItem` interface, making it itself schedulable on the `ThreadPool`. That is exactly something that `Http2Connection` leverages: when a HEADER frame is received for a new request, then a new stream is initialized to handle the request, and this stream object is scheduled on the thread pool. ## From Stream to User Response An `Http2Stream` represents a request-response pair between the client and the server. A stream deserializes the HTTP request headers and body, then it hands off the rest of the processing to the *application*, which will invoke the middleware pipeline and route the request to the method that handles the corresponding endpoint. Once a request is processed by the user code, the response is written into a pipe, that is captured by a `Http2OutputProducer` along with the other response properties. The `Http2OutputProducer` producer signals the `Http2FrameWriter` when response data becomes available from the user, that can be written into response frames. It does this signaling by passing itself on a `Channel` (ref: [Channel](https://learn.microsoft.com/en-us/dotnet/api/system.threading.channels.channel-1?view=net-8.0)). Multiple `Http2OutputProducer`s are pushed onto the same channel, that is consumed by a single reader per connection. ## From User Response to HTTP/2 Response Frames All response streams handled by a `Http2Connection` must be serialized into the same TCP connection. This requires multiplexing the responses data of multiple streams' `Http2OutputProducer`s, while maintaining a mutual exclusion across the frames. This responsibility handled by [Http2FrameWriter](https://github.com/dotnet/aspnetcore/blob/d301328737c4912b95f236c4b349cf46ef7a473a/src/Servers/Kestrel/Core/src/Internal/Http2/Http2FrameWriter.cs). A `Http2Connection` owns a single `Http2FrameWriter`. The `Http2FrameWriter` reads a scheduled instance of `Http2OutputProducer` from the `Channel` mentioned above. This isn't the actual data to be written, the *output producer* contains references to the *HTTP response headers* and trailers as well as the response data stream. The `Http2FrameWriter` creates frames for the available data and flushes it to the TCP connection. If a `Http2OutputProducer` is not yet fully flushed, then it is rescheduled for future writes. The output producer makes sure that only a single stream's frame is written to the output at a single time. It achieves this by mutual exclusion, by locking the critical segments of the write process. Moreover, the HTTP/2 protocol is stateful, so not only the actual data write is guarded, but also operations that might mutate the shared state, such as: - static and dynamic header table - flow control - timeouts/keep-alive - etc. ## Flow Control Overview One interesting aspect is the flow-control. There are both connection and stream level flow-control windows. The server can write as much data as available in these windows. A client can update both the connection and the stream level windows by sending SETTINGS frames or WINDOW_UPDATE frames. A connection level frame update applies to all current and future frames. Typically, a WINDOW_UPDATE increases the current window, so the client acknowledges that data has been received and the server may send more data. However, a connection level update may result in the window of a stream to become negative. In such a case the stream is not scheduled for writes until its window is increased by future WINDOW_UPDATEs. Note, that window updates happen asynchronously, hence it can end up with a temporary negative window. ## Conclusion In previous posts I have explored many different parts of Kestrel handling HTTP requests in ASP.NET Core. This post focuses on the interaction between `Http2Connection`, `Http2Stream`, `Http2FrameWriter` and `Http2OutputProducer` types. It describes how a single request-response is being processed as a stream on a shared connection.