Standards Corner: IETF Revisits HTTP/1.1
By Independentid-Oracle on Jun 18, 2014
HTTP has been one of the most successful IETF specifications aside from the Internet itself. When it was created in 1999, the authors of HTTP had no idea how big and how widely used it would be. For many years the focus was on the evolving world-wide-web and HTML. The web itself went through many transformations with the introduction of Ajax and then HTML5 by the W3C. Meanwhile, non-browser use of HTTP has been steadily growing especially with the exploding popularity of smart devices, the Internet of Things, and in particular RESTful APIs.
In 2007, a working group was formed to define HTTPbis, the next generation of HTTP. This group decided to break the work into 2 steps: redefine and clarify HTTP/1.1 based on what has been learned over the years; and, set the stage for HTTP/2 by breaking the HTTP specification into key component specifications. The new specifications set the stage for HTTP/2, so that it will only need to update specific aspects of HTTP/1.1 such as messaging.
The new HTTP/1.1 specification effort was led by Roy Fielding and Julian Reschke and is as follows:
- RFC7230 - Message Syntax and Routing - low-level message parsing and connection management
- RFC7231 - Semantics and Content - methods, status codes and headers
- RFC7232 - Conditional Requests - such as If-Modified-Since
- RFC7233 - Range Requests - getting partial content
- RFC7234 - Caching - browser and intermediary caches
- RFC7235 - Authentication - a framework for HTTP authentication.
Though the HTTP protocol version has not changed, there has been a significant number of clarifications and restrictions added. In putting together this document set, the working group discussed 550 issues and produced 26 draft versions. For a complete list of changes, consult the appendix of each of the new specifications (or see summary below).
From my perspective in Identity standards, there have been a number of key improvements to security that reflect a broad maturing of the specifications:
- Userinfo (username and password) is now disallowed in request URIs due to security issues related to transmission on the wire for both HTTP and HTTPS. In general I have always believed that passing personally identifiable information (PII) or security information in URLs is both a bad security and privacy practice.
- Special characters, escaping, whitespace, and international character handling has been greatly improved. This is a reflection of ongoing work and new best security practices around comparison of strings that impact protocol and authentication. For more information check out the IETF Precis Framework.
- While Basic Auth still exists, it is now considered a legacy authentication scheme. Past readers of my blog may recall my personal call for Basic Auth to die.
The IETF kept the same HTTP version to reflect that from a core protocol inter-operability perspective, nothing has changed. It is important however to realize that if you were working on the "fringes" of HTTP, it may be that some of these clarifications will impact your application -- most likely for much improved security and inter-operability.
Some of the finer details:
- Userinfo (username and password) are disallowed in URIs because of security issues related to transmission on the wire for both HTTP and HTTPS. This is not surprising, I’ve been advocating that having filter terms with PII (personally identifiable information) is also a privacy issue as well as being a security issue in many cases.
- New rules around whitespace, NUL octets, and backslash-escaping, and other special characters
- Header content fields that span multiple lines (aka line folding) are deprecated
- NUL octet is no longer allowed in comment and quoted-string text, backslash-escaping has been clarified
- Gateways do not need to generate “Via” header fields anymore.
- Bogus Content-Length header fields are now required to be handled as errors by recipients
- The algorithms for determining message body and chunk length has been clarified
- The term “Effective Request URI” has been introduced.
- The limit of two connections per server has been removed. Requirements to retry idempotent requests due to premature connection closures have been removed.
- The expectation to support HTTP/0.9 requests has been removed.
From Semantics and Content:
- Semantics embedded in a URI be disabled when those semantics are inconsistent with the request method as this is a common cause of interop failure
- An algorithm for determining if payload is associated with a specific header
- The default charset of ISO-8859-1 has been removed. The default is whatever the media type says.
- The definition of Content-Location changed to no longer affect base URI for resolving relative URI references
- The definition of GET has been relaxed so that requests can have a body, even though the body has no meaning for GET.
- The CONNECT method has been moved from RFC2817 to RFC7231.
- The OPTIONS and TRACE request methods are now defined as being “safe”.
- The Expect header field’s extension mechanism has been removed due to widely-deployed broken implementations.
- “about:blank” URI has been suggested as a value for the Referrer header field when no referring URI is applicable. This distinguishes from the case where the referrer field was not sent or has been removed.
- Status codes 204,404,405,414,501 are now cacheable by default.
- The HTTP Status 201 (Created) description was changed to allow for the possibility of more than one resource has been created.
- The set of request methods that are safe to automatically redirect is no longer closed; user agents are able to make that determination based on method semantics. 301, 302, and 307 no longer have normative requirements on response payloads and user interactions.
From Conditional Requests:
- Weak entity-tags are now allowed in all requests except range requests
- The ETag header field ABNF has been changed to not use quoted-string to avoid escaping issues.
- The ETag is defined to provide an entity tag for the selected representation, thereby clarifying what it applies to in various situations (such as a PUT response).
- The precedence for evaluation of conditional requests has been defined.
From Range Requests:
- Servers are given more leeway in how they respond to a range request in order to mitigate abuse by malicious (or greedy) clients.
- A weak validator cannot be used in a 206 response
- The Content-Range header field only has meaning when the status code explicitly defines its use.
- The specification defines a new “Range Unit” registry.
- The conditions under which an authenticated response can be cached have been clarified.
- Caches are now allowed to calculate heuristic freshness for URIs with query components.
- The algorithm for calculating age is less conservative.
- Requirements regarding denial-of-service attack avoidance when performing invalidation have been clarified
- Cache invalidation only occurs when a successful response is received.
- Cache directives are now case-insensitive.
- The “no-store” request directive does not apply to responses
- The “no-cache” response directive’s meaning has been clarified.
- The one-year limit on Expires header field values has been removed.
- The Pragma header is deprecated.
- Requirements regarding production and processing of Warning header fields have been relaxed as it was not widely implemented.
- Introduces new Cache Directive and Warn Code IANA Registries.
- RFC7235 takes over definition of authentication framework from RFC2617
- The “realm” parameter is no longer required on challenges. ABNF ow allows challenges without any auth parameters
- The “token68” alternative to auth-param lists has been added for consistency with legacy authentication schemes such as “Basic”.
- Introduces the Authentication Schema Registry along with considerations for new authentication schemes.
Author: Phil Hunt
Phil Hunt is an active member of multiple industry standards groups and committees and has spearheaded discussions, creation and ratifications of industry standards including the Kantara Identity Governance Framework, among others. Being an active voice in the industry standards development world, we have invited him to share his discussions, thoughts, news & updates, and discuss use cases, implementation success stories (and even failures) around industry standards on this monthly column.