Python grequests模块使用场景及代码实例

陆景行

发布时间：2026-10-14 | 5人浏览过

原创

本文介绍Python中grequests模块的使用方法与适用场景。grequests基于requests和gevent实现异步并发请求，支持GET、POST等多种HTTP方法，常用于爬虫IP代理池验证及压力测试。文章提供基础用法、自定义返回结果修改源码示例以及事件钩子配置代码，帮助开发者高效处理批量网络请求。

grequests适用场景

适用场景：

1) 在爬虫开发中配置IP代理池时，用于验证IP地址是否有效

2）在进行系统压测时，执行批量并发请求等场景

grequests基础用法

grequests 是对 requests 和 gevent 库的简单封装，使用方式十分便捷。

grequests.map(requests, stream=False, size=None, exception_handler=None, gtimeout=None)

此外，由于 grequests 底层依赖 requests，因此它支持

GET，OPTIONS， HEAD， POST， PUT， DELETE 等各种http method

因此以下的任务请求都是支持的

grequests.post(url, json={“name”:“zhangsan”}) grequests.delete(url)

代码如下:


import grequests

urls = [
  'http://www.baidu.com',
  'http://www.qq.com',
  'http://www.163.com',
  'http://www.zhihu.com',
  'http://www.toutiao.com',
  'http://www.douban.com'
]
rs = (grequests.get(u) for u in urls)
print(grequests.map(rs))  # [<Response [200]>, None, <Response [200]>, None, None, <Response [418]>]
def exception_handler(request, exception):
  print("Request failed")
reqs = [
  grequests.get('http://httpbin.org/delay/1', timeout=0.001),
  grequests.get('http://fakedomain/'),
  grequests.get('http://httpbin.org/status/500')
]
print(grequests.map(reqs, exception_handler=exception_handler))

自定义返回结果

实际操作中，也可以自定义返回的结果

修改grequests源码文件:

例如:

新增extract_item() 函数合修改map()函数


def extract_item(request):
  """
  提取request的内容
  :param request:
  :return:
  """
  item = dict()
  item["url"] = request.url
  item["text"] = request.response.text or ""
  item["status_code"] = request.response.status_code or 0
  return item

def map(requests, stream=False, size=None, exception_handler=None, gtimeout=None):
  """Concurrently converts a list of Requests to Responses.

  :param requests: a collection of Request objects.
  :param stream: If True, the content will not be downloaded immediately.
  :param size: Specifies the number of requests to make at a time. If None, no throttling occurs.
  :param exception_handler: Callback function, called when exception occured. Params: Request, Exception
  :param gtimeout: Gevent joinall timeout in seconds. (Note: unrelated to requests timeout)
  """
  requests = list(requests)
  pool = Pool(size) if size else None
  jobs = [send(r, pool, stream=stream) for r in requests]
  gevent.joinall(jobs, timeout=gtimeout)
  ret = []
  for request in requests:

    if request.response is not None:
      ret.append(extract_item(request))
    elif exception_handler and hasattr(request, 'exception'):
      ret.append(exception_handler(request, request.exception))
    else:
      ret.append(None)

  yield ret

可以直接调用:


import grequests
urls = [
  'http://www.baidu.com',
  'http://www.qq.com',
  'http://www.163.com',
  'http://www.zhihu.com',
  'http://www.toutiao.com',
  'http://www.douban.com'
]
rs = (grequests.get(u) for u in urls)
response_list = grequests.map(rs, gtimeout=10)
for response in next(response_list):
  print(response)

事件钩子配置

支持事件钩子


def print_url(r, *args, **kwargs):
print(r.url)

url = “http://www.baidu.com”
res = requests.get(url, hooks={“response”: print_url})
tasks = []
req = grequests.get(url, callback=print_url)
tasks.append(req)
ress = grequests.map(tasks)
print(ress)