请求同一个网址， requests 和 urllib2 返回的 headers 内容不同？

Date: XXX
Server: Apache
Last-Modified: XXX
Accept-Ranges: bytes
Content-Length: 12345678
Vary: Accept-Encoding
Connection: close
Content-Type: text/plain

requests:

Content-Encoding: gzip
Accept-Ranges: bytes
Vary: Accept-Encoding
Keep-alive: timeout=5, max=128
Last-Modified: XXX
Connection: Keep-Alive
ETag: xxxxxxxxx
Content-Type: text/plain

为何 requests 少了 content-length ？其它发送请求的设置是完全一样的。。 requests 和 Chrome 开发者工具查看到的一致。但是这里我又需要 content-length 的值（为了断点续传）

Supplement 1 · Jan 14, 2017

import urllib2 
import requests 

url = 'exmaple.com' 
headers = { 
"Authorization": "Basic xxxx", 
"Range": "bytes=0-" 
} 
req = urllib2.Request(url, headers=headers) 
resp = urllib2.urlopen(req) 
print resp.info() 
r = requests.get(url, headers=headers) 
print r.headers 
assert resp.info()['ETag'] == r.headers['ETag']

Date: Sat, 14 Jan 2017 09:39:50 GMT
Server: Apache
Last-Modified: Sat, 14 Jan 2017 09:39:49 GMT
ETag: "e91103-10e04f7-5460abb4743a3"
Accept-Ranges: bytes
Content-Length: 17695991
Vary: Accept-Encoding
Content-Range: bytes 0-17695990/17695991
Connection: close
Content-Type: text/plain

{'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'Accept-Ranges': 'bytes', 'Vary': 'Accept-Encoding', 'Keep-Alive': 'timeout=5, max=128', 'Server': 'Apache', 'Last-Modified': 'Sat, 14 Jan 2017 09:39:49 GMT', 'Connection': 'Keep-Alive', 'ETag': '"e91103-10e04f7-5460abb4743a3"', 'Date': 'Sat, 14 Jan 2017 09:39:50 GMT', 'Content-Type': 'text/plain'}

我也知道肯定是两次发送的请求header不一样。。。现在总算解决了。。

Supplement 2 · Jan 14, 2017

The response is different because requests indicates that it supports gzip-encoded bodies, by sending an Accept-Encoding: gzip, deflate header field. urllib2 does not. You'll find if you added that header to your urllib2 request that you get the new behaviour.

Clearly, in this case, the server is dynamically gzipping the responses. This means it doesn't know how long the response will be, so it is sending using chunked transfer encoding.

If you really must get the Content-Length header, then you should add the following headers to your Requests request: {'Accept-Encoding': 'identity'}.

15 replies • 2017-01-14 23:11:08 +08:00

redhatping

Jan 14, 2017

看官方文档

binux

Jan 14, 2017 via Android

Content-Length 不应该手动设置

dofine

Jan 14, 2017

@binux 我描述不清～～上边给出的结果是响应的 header ，我的意思是需要知道当前 content-length 的值。。但是 requests 的返回里面没有。。 urllib2 就有。。
@redhatping 文档已经看了许多遍了。。怀疑是服务器的问题？

hahastudio

Jan 14, 2017

你这是结果，肯定还是因为你发送的请求不一样

Lonely

Jan 14, 2017 via iPhone

你把代码也贴出来啊……

dofine

Jan 14, 2017

{'Range': 'bytes=0-', 'Authorization': 'Basic XXX'}

手动加了这个 header ， urllib2 和 requests 返回的 ETag 都是一样的啊。。为什么会发送请求不一样呢。。 @hahastudio

dofine

Jan 14, 2017

```
import os
import urllib2
import requests

url = 'exmaple.com'
headers = {
"Authorization": "Basic xxxx",
"Range": "bytes=0-"
}
req = urllib2.Request(url, headers=headers)
resp = urllib2.urlopen(req)
print resp.info()

r = requests.get(url, headers=headers)
print r.headers
assert resp.info()['ETag'] == r.headers['ETag']
```

dofine

Jan 14, 2017

@Lonely 贴在楼上了。。我发现我都不在回复里贴代码了。。