Tornado: Event loop blocking and excess memory use from chunked bodies with many small chunks

Overview

Summary

When Tornado reads an HTTP/1.1 body that uses chunked transfer encoding, it processes all chunks that have already arrived without yielding to the event loop. A client can send a body made of millions of 1-byte chunks, which blocks the event loop (and therefore every other connection served by that process) for the entire time it takes to parse the body. A body near the default max_body_size of 100 MB can block the process for close to a minute, and the attacker can repeat or parallelize these requests.

Bodies made of tiny chunks also consumed far more memory than their size: a handler that does not use @stream_request_body stored each chunk as a separate object, using roughly 20 times as much memory as the bytes received.

Impact

All Tornado HTTP servers are affected when they receive a request body for a handler that reads it, which includes any handler that does not reject the request before the body is read. No authentication is required. Lowering max_body_size reduces the maximum blocking time and memory use for each request but does not remove the issue.

SimpleAsyncHTTPClient is affected in the same way when it reads a response from a malicious server.

Patches

Fixed in Tornado 6.6. Tornado now yields to the event loop periodically while reading any body, combines small chunks before passing them to data_received, and stores non-streaming bodies more compactly.

Applications using @stream_request_body should note that the pieces passed to data_received no longer correspond to HTTP chunk boundaries.

Workarounds

• Run Tornado behind a reverse proxy that buffers request bodies and forwards them with a Content-Length header instead of chunked encoding.
• Lower max_body_size (for the whole server or for individual handlers) to limit the impact of each request.

Credit

Reported by uttej-badwane.

Affected packages

• tornado < 6.6

Remediation

Upgrade to a patched release:

• tornado 6.6