Content Encoding Support for libcurl

* About content encodings:

HTTP/1.1 [RFC 2616] specifies that a client may request that a server encode

its response. This is usually used to compress a response using one of a set

of commonly available compression techniques. These schemes are `deflate' (the

zlib algorithm), `gzip' and `compress' [sec 3.5, RFC 2616]. A client requests

that the sever perform an encoding by including an Accept-Encoding header in

the request document. The value of the header should be one of the recognized

tokens `deflate', ... (there's a way to register new schemes/tokens, see sec

3.5 of the spec). A server MAY honor the client's encoding request. When a

response is encoded, the server includes a Content-Encoding header in the

response. The value of the Content-Encoding header indicates which scheme was

used to encode the data.

A client may tell a server that it can understand several different encoding

schemes. In this case the server may choose any one of those and use it to

encode the response (indicating which one using the Content-Encoding header).

It's also possible for a client to attach priorities to different schemes so

that the server knows which it prefers. See sec 14.3 of RFC 2616 for more

information on the Accept-Encoding header.

* Current support for content encoding:

Support for the 'deflate' and 'gzip' content encoding are supported by

libcurl. Both regular and chunked transfers should work fine.  The library

zlib is required for this feature. 'deflate' support was added by James

Gallagher, and support for the 'gzip' encoding was added by Dan Fandrich.

* The libcurl interface:

To cause libcurl to request a content encoding use:

    curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, <string>)

where <string> is the intended value of the Accept-Encoding header.

Currently, libcurl only understands how to process responses that use the

"deflate" or "gzip" Content-Encoding, so the only values for

CURLOPT_ACCEPT_ENCODING that will work (besides "identity," which does

nothing) are "deflate" and "gzip" If a response is encoded using the

"compress" or methods, libcurl will return an error indicating that the

response could not be decoded.  If <string> is NULL no Accept-Encoding header

is generated.  If <string> is a zero-length string, then an Accept-Encoding

header containing all supported encodings will be generated.

The CURLOPT_ACCEPT_ENCODING must be set to any non-NULL value for content to

be automatically decoded.  If it is not set and the server still sends encoded

content (despite not having been asked), the data is returned in its raw form

and the Content-Encoding type is not checked.

* The curl interface:

Use the --compressed option with curl to cause it to ask servers to compress

responses using any format supported by curl.

James Gallagher <jgallagher@gso.uri.edu>

Dan Fandrich <dan@coneharvesters.com>