Kestrel HTTP/2 Internals
01/05/2025
6 minutes
In the previous post I explored some of the inner workings of Kestrel. Kestrel is one of the servers built into ASP.NET Core.
First, I investigated how HTTP socket connections are dispatched to a request processor, such as the Http2Connection.
In a different post I looked into how HEADER frames work with HTTP/2 responses, and what are the main building blocks of writing headers to the response stream in Kestrel.
Lastly, I explored how HEADER and CONTINUATION frames are used in ASP.NET Core to serve response headers larger to the frame size.
In this post I will explore some of the interactions of the Http2Stream, Http2Connection, Http2OutputWriter and Http2FrameWriter types. The HTTP/2 protocol definition can be found in RFC7540.
I write this post in the .NET8 - .NET9 prerelease timeframe. References to type names, method names and code will reflect the current state of the ASP.NET Core repository.
From Request Frames to Connection
An incoming request is handled in Http2Connection. A method called ProcessRequestsAsync handles the incoming connection stream. In its main loop it tries to read the next frame received and it branches processing based on the frame's type. For example, HEADER, DATA or WINDOW_UPDATE frames are dispatched to different methods for further validation and processing.
Http2Connection represents the HTTP/2 connection on the server side. It makes sure that the protocol aspects of an HTTP/2 connection are consistent and correct. Aspects such as:
incoming frames reference existing (or initiate new) streams
padding is correct
the referenced stream is in open state
server limits are not breached (such as max stream count)
it tracks connection level flow control
creates and closes streams
closes the connection on fatal errors
manages timeouts with the help of
TimeoutControlhandles keep-alive events
From Connection to Stream
When a HEADER frame (not to mistaken it with an HTTP request header), is received it creates a new stream. This new stream can be a completely new object or can be rented from a stream pool. This stream is represented by Http2Stream type, which also implements IThreadPoolWorkItem interface, making it itself schedulable on the ThreadPool. That is exactly something that Http2Connection leverages: when a HEADER frame is received for a new request, then a new stream is initialized to handle the request, and this stream object is scheduled on the thread pool.
From Stream to User Response
An Http2Stream represents a request-response pair between the client and the server. A stream deserializes the HTTP request headers and body, then it hands off the rest of the processing to the application, which will invoke the middleware pipeline and route the request to the method that handles the corresponding endpoint. Once a request is processed by the user code, the response is written into a pipe, that is captured by a Http2OutputProducer along with the other response properties. The Http2OutputProducer producer signals the Http2FrameWriter when response data becomes available from the user, that can be written into response frames. It does this signaling by passing itself on a Channel<Http2OutputProducer> (ref: Channel). Multiple Http2OutputProducers are pushed onto the same channel, that is consumed by a single reader per connection.
From User Response to HTTP/2 Response Frames
All response streams handled by a Http2Connection must be serialized into the same TCP connection. This requires multiplexing the responses data of multiple streams' Http2OutputProducers, while maintaining a mutual exclusion across the frames. This responsibility handled by Http2FrameWriter. A Http2Connection owns a single Http2FrameWriter. The Http2FrameWriter reads a scheduled instance of Http2OutputProducer from the Channel<Http2OutputProducer> mentioned above. This isn't the actual data to be written, the output producer contains references to the HTTP response headers and trailers as well as the response data stream.
The Http2FrameWriter creates frames for the available data and flushes it to the TCP connection. If a Http2OutputProducer is not yet fully flushed, then it is rescheduled for future writes.
The output producer makes sure that only a single stream's frame is written to the output at a single time. It achieves this by mutual exclusion, by locking the critical segments of the write process. Moreover, the HTTP/2 protocol is stateful, so not only the actual data write is guarded, but also operations that might mutate the shared state, such as:
static and dynamic header table
flow control
timeouts/keep-alive
etc.
Flow Control Overview
One interesting aspect is the flow-control. There are both connection and stream level flow-control windows. The server can write as much data as available in these windows. A client can update both the connection and the stream level windows by sending SETTINGS frames or WINDOW_UPDATE frames. A connection level frame update applies to all current and future frames. Typically, a WINDOW_UPDATE increases the current window, so the client acknowledges that data has been received and the server may send more data. However, a connection level update may result in the window of a stream to become negative. In such a case the stream is not scheduled for writes until its window is increased by future WINDOW_UPDATEs. Note, that window updates happen asynchronously, hence it can end up with a temporary negative window.
Conclusion
In previous posts I have explored many different parts of Kestrel handling HTTP requests in ASP.NET Core. This post focuses on the interaction between Http2Connection, Http2Stream, Http2FrameWriter and Http2OutputProducer types. It describes how a single request-response is being processed as a stream on a shared connection.