World Library  
Flag as Inappropriate
Email this Article

HTTP compression

Article Id: WHEBN0004954757
Reproduction Date:

Title: HTTP compression  
Author: World Heritage Encyclopedia
Language: English
Publisher: World Heritage Encyclopedia

HTTP compression

HTTP compression is a capability that can be built into web servers and web clients to improve transfer speed and bandwidth utilization.[1]

HTTP data is compressed before it is sent from the server: compliant browsers will announce what methods are supported to the server before downloading the correct format; browsers that do not support compliant compression method will download uncompressed data. The most common compression schemes include gzip and Deflate, however a full list of available schemes is maintained by the IANA.[2] Additionally, third parties develop new methods and include them in their products, for example the Google Shared Dictionary Compression for HTTP (SDCH) scheme implemented in the Google Chrome browser and used on Google servers.

There are two different ways compression can be done in HTTP. At a lower level, a Transfer-Encoding header field may indicate the payload of a HTTP message is compressed. At a higher level, a Content-Encoding header field may indicate that a resource being transferred, cached, or otherwise referenced is compressed. Compression using Content-Encoding is more widely supported than Transfer-Encoding, and some browsers do not advertise for Transfer-Encoding compression to avoid triggering bugs in servers.[3]


  • Compression scheme negotiation 1
  • Content-Encoding tokens 2
  • Servers that support HTTP compression 3
  • Problems preventing the use of HTTP compression 4
  • Security implications 5
  • References 6
  • External links 7

Compression scheme negotiation

In most cases, excluding the SDCH, the negotiation is done in two steps, described in RFC 2616:

1. The web client advertises which compression schemes it supports by including a list of tokens in the HTTP request. For Content-Encoding, the list in a field called Accept-Encoding; for Transfer-Encoding, the field is called TE.

GET /encrypted-area HTTP/1.1
Accept-Encoding: gzip, deflate

2. If the server supports one or more compression schemes, the outgoing data may be compressed by one or more methods supported by both parties. If this is the case, the server will add a Content-Encoding or Transfer-Encoding field in the HTTP response with the used schemes, separated by commas.

HTTP/1.1 200 OK
Date: mon, 23 Jun 2016 22:38:34 GMT
Server: Apache/ (Unix)  (Red-Hat/Linux)
Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
Accept-Ranges: bytes
Content-Length: 438
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip

The web server is by no means obliged to use any compression method – this depends on the internal settings of the web server and also may depend on the internal architecture of the website in question.

In case of SDCH a dictionary negotiation is also required, which may involve additional steps, like downloading a proper dictionary from the external server.

Content-Encoding tokens

The official list of tokens available to servers and client is maintained by IANA,[4] and it includes:

  • compress – UNIX "compress" program method (historic; deprecated in most applications and replaced by gzip or deflate)
  • deflate – compression based on the deflate algorithm (described in RFC 1951), wrapped inside the zlib data format (RFC 1950);
  • exi – W3C Efficient XML Interchange
  • gzip – GNU zip format (described in RFC 1952). This method is the most broadly supported as of March 2011.[5]
  • identity – No transformation is used. This is the default value for content coding.
  • pack200-gzip – Network Transfer Format for Java Archives[6]

In addition to these, a number of unofficial or non-standardized tokens are used in the wild by either servers or clients:

  • brotli – A new open-sourced compression algorithm specifically designed for HTTP content encoding, supported by Firefox and with intent-to-implement from Chromium.
  • bzip2 – compression based on the free bzip2 format, supported by lighttpd[7]
  • lzma – compression based on (raw) LZMA is available in Opera 20, and in elinks via a compile-time option[8]
  • peerdist[9]Microsoft Peer Content Caching and Retrieval
  • sdch[10][11]Google Shared Dictionary Compression for HTTP, based on VCDIFF (RFC 3284); supported natively in recent versions of Google Chrome, Chromium and Android, as well as on Google websites.
  • xpress - Microsoft compression protocol used by Windows 8 and later for Windows Store application updates. LZ77-based compression optionally using a Huffman encoding.[12]
  • xz - LZMA2-based content compression, supported by a non-official Firefox patch;[13] and fully implemented in mget since 2013-12-31.[14]

Servers that support HTTP compression

The compression in HTTP can also be achieved by using the functionality of server-side scripting languages like PHP, or programming languages like Java.

Problems preventing the use of HTTP compression

A 2009 article by Google engineers Arvind Jain and Jason Glasgow states that more than 99 person-years are wasted[17] daily due to increase in page load time when users do not receive compressed content. This occurs when anti-virus software interferes with connections to force them to be uncompressed, where proxies are used (with overcautious web browsers), where servers are misconfigured, and where browser bugs stop compression being used. Internet Explorer 6, which drops to HTTP 1.0 (without features like compression or pipelining) when behind a proxy – a common configuration in corporate environments – was the mainstream browser most prone to failing back to uncompressed HTTP.[17]

Another problem found while deploying HTTP compression on large scale is due to the deflate encoding definition: while HTTP 1.1 defines the deflate encoding as data compressed with deflate (RFC 1951) inside a zlib formatted stream (RFC 1950), Microsoft server and client products historically implemented it as a "raw" deflated stream,[18] making its deployment unreliable.[19][20] For this reason, some software, including the Apache HTTP Server, only implement gzip encoding.

Security implications

In 2012, a general attack against the use of data compression, called CRIME, was announced. While the CRIME attack could work effectively against a large number of protocols, including but not limited to TLS, and application-layer protocols such as SPDY or HTTP, only exploits against TLS and SPDY were demonstrated and largely mitigated in browsers and servers. The CRIME exploit against HTTP compression has not been mitigated at all, even though the authors of CRIME have warned that this vulnerability might be even more widespread than SPDY and TLS compression combined.

In 2013, a new instance of the CRIME attack against HTTP compression, dubbed BREACH, was published. A BREACH attack can extract login tokens, email addresses or other sensitive information from TLS encrypted web traffic in as little as 30 seconds (depending on the number of bytes to be extracted), provided the attacker tricks the victim into visiting a malicious web link.[21] All versions of TLS and SSL are at risk from BREACH regardless of the encryption algorithm or cipher used.[22] Unlike previous instances of CRIME, which can be successfully defended against by turning off TLS compression or SPDY header compression, BREACH exploits HTTP compression which cannot realistically be turned off, as virtually all web servers rely upon it to improve data transmission speeds for users.[21]


  1. ^ "Using HTTP Compression (IIS 6.0)". Microsoft Corporation. Retrieved 9 February 2010. 
  2. ^ RFC 2616, Section 3.5: "The Internet Assigned Numbers Authority (IANA) acts as a registry for content-coding value tokens."
  3. ^ 'RFC2616 "Transfer-Encoding: gzip, chunked" not handled properly', Chromium Issue 94730
  4. ^ "Hypertext Transfer Protocol Parameters - HTTP Content Coding Registry". IANA. Retrieved 18 April 2014. 
  5. ^ "Compression Tests: Results". Verve Studios, Co. Retrieved 19 July 2012. 
  6. ^ "JSR 200: Network Transfer Format for Java Archives". The Java Community Process Program. 
  7. ^ "ModCompress - Lighttpd". lighty labs. Retrieved 18 April 2014. 
  8. ^ elinks LZMA decompression
  9. ^ "[MS-PCCRTP]: Peer Content Caching and Retrieval: Hypertext Transfer Protocol (HTTP) Extensions". Microsoft. Retrieved 19 April 2014. 
  10. ^ Butler, Jon; Wei-Hsin Lee; McQuade, Bryan; Mixter, Kenneth. "A Proposal for Shared Dictionary Compression Over HTTP" (PDF). Google. 
  11. ^ "SDCH Mailing List". Google Groups. 
  12. ^ "[MS-XCA]: Xpress Compression Algorithm". Retrieved 29 August 2015. 
  13. ^ "LZMA2 Compression - MozillaWiki". Retrieved 18 April 2014. 
  14. ^ "mget GitHub project page". Retrieved May 2014. 
  15. ^ "HOWTO: Use Apache mod_deflate To Compress Web Content (Accept-Encoding: gzip)". Mark S. Kolich. Retrieved 23 March 2011. 
  16. ^ "Extra part of Hiawatha webserver's manual". 
  17. ^ a b "Use compression to make the web faster". Google Developers. Retrieved 22 May 2013. 
  18. ^ "deflate - Why are major web sites using gzip?". Stack Overflow. Retrieved 18 April 2014. 
  19. ^ "Compression Tests: About". Verve Studios. Retrieved 18 April 2014. 
  20. ^ "Lose the wait: HTTP Compression". Zoompf Web Performance. Retrieved 18 April 2014. 
  21. ^ a b Goodin, Dan (1 August 2013). "Gone in 30 seconds: New attack plucks secrets from HTTPS-protected pages". Ars Technica. Condé Nast. Retrieved 2 August 2013. 
  22. ^ Leyden, John (2 August 2013). "Step into the BREACH: New attack developed to read encrypted web data". The Register. Retrieved 2 August 2013. 

External links

  • RFC 2616: Hypertext Transfer Protocol – HTTP/1.1
  • HTTP Content-Coding Values by Internet Assigned Numbers Authority
  • Compression with lighttpd
  • Coding Horror: HTTP Compression on IIS 6.0
  • 15 Seconds: Web Site Compression at the Wayback Machine (archived July 16, 2011)
  • HTTP Compression: resource page by the founder of VIGOS AG, Constantin Rack
  • Using HTTP Compression by Martin Brown of Server Watch
  • Using HTTP Compression in PHP
  • Dynamic and static HTTP compression with Apache httpd
  • Browser HTTP Compression Test
  • Check HTTP compression
This article was sourced from Creative Commons Attribution-ShareAlike License; additional terms may apply. World Heritage Encyclopedia content is assembled from numerous content providers, Open Access Publishing, and in compliance with The Fair Access to Science and Technology Research Act (FASTR), Wikimedia Foundation, Inc., Public Library of Science, The Encyclopedia of Life, Open Book Publishers (OBP), PubMed, U.S. National Library of Medicine, National Center for Biotechnology Information, U.S. National Library of Medicine, National Institutes of Health (NIH), U.S. Department of Health & Human Services, and, which sources content from all federal, state, local, tribal, and territorial government publication portals (.gov, .mil, .edu). Funding for and content contributors is made possible from the U.S. Congress, E-Government Act of 2002.
Crowd sourced content that is contributed to World Heritage Encyclopedia is peer reviewed and edited by our editorial staff to ensure quality scholarly research articles.
By using this site, you agree to the Terms of Use and Privacy Policy. World Heritage Encyclopedia™ is a registered trademark of the World Public Library Association, a non-profit organization.

Copyright © World Library Foundation. All rights reserved. eBooks from Project Gutenberg are sponsored by the World Library Foundation,
a 501c(4) Member's Support Non-Profit Organization, and is NOT affiliated with any governmental agency or department.