在Scrapy中集成PhantomJS实现异步AJAX爬取

江屿

发布时间：2026-10-13 | 5人浏览过

原创

本文介绍如何在Scrapy中集成PhantomJS以实现异步AJ数据爬取。通过重写Downloader并引入Semaphore控制并发，解决传统同步集成的性能瓶颈。代码展示了如何配置PhantomJSDownloadHandler，利用线程池和队列管理浏览器实例，从而在保持Scrapy高效并发能力的同时，处理需要JavaScript渲染的网页内容。

利用Selenium可以便捷地获取网页中的AJAX动态内容，并模拟用户点击、输入文本等多种交互行为。在Scrapy框架进行网页爬取时，这种能力具有极高的实用价值。

尽管网络上关于将Selenium集成到Scrapy的教程众多，但鲜有方案能真正实现异步爬取。下文提供的代码通过重写Scrapy的Downloader组件，成功实现了Selenium的集成与异步处理功能。

在实际使用时，需将PhantomJSDownloadHandler配置项添加至配置文件的DOWNLOADER设置中。

# encoding: utf-8
from __future__ import unicode_literals
 
from scrapy import signals
from scrapy.signalmanager import SignalManager
from scrapy.responsetypes import responsetypes
from scrapy.xlib.pydispatch import dispatcher
from selenium import webdriver
from six.moves import queue
from twisted.internet import defer, threads
from twisted.python.failure import Failure
 
 
class PhantomJSDownloadHandler(object):
 
 def __init__(self, settings):
  self.options = settings.get('PHANTOMJS_OPTIONS', {})
 
  max_run = settings.get('PHANTOMJS_MAXRUN', 10)
  self.sem = defer.DeferredSemaphore(max_run)
  self.queue = queue.LifoQueue(max_run)
 
  SignalManager(dispatcher.Any).connect(self._close, signal=signals.spider_closed)
 
 def download_request(self, request, spider):
  """use semaphore to guard a phantomjs pool"""
  return self.sem.run(self._wait_request, request, spider)
 
 def _wait_request(self, request, spider):
  try:
   driver = self.queue.get_nowait()
  except queue.Empty:
   driver = webdriver.PhantomJS(**self.options)
 
  driver.get(request.url)
  # ghostdriver won't response when switch window until page is loaded
  dfd = threads.deferToThread(lambda: driver.switch_to.window(driver.current_window_handle))
  dfd.addCallback(self._response, driver, spider)
  return dfd
 
 def _response(self, _, driver, spider):
  body = driver.execute_script("return document.documentElement.innerHTML")
  if body.startswith("<head></head>"): # cannot access response header in Selenium
   body = driver.execute_script("return document.documentElement.textContent")
  url = driver.current_url
  respcls = responsetypes.from_args(url=url, body=body[:100].encode('utf8'))
  resp = respcls(url=url, body=body, encoding="utf-8")
 
  response_failed = getattr(spider, "response_failed", None)
  if response_failed and callable(response_failed) and response_failed(resp, driver):
   driver.close()
   return defer.fail(Failure())
  else:
   self.queue.put(driver)
   return defer.succeed(resp)
 
 def _close(self):
  while not self.queue.empty():
   driver = self.queue.get_nowait()
   driver.close()

以上即为在Scrapy中使用PhantomJS实现异步爬取的全部方法分享，希望能为读者提供参考，并欢迎大家对本站给予更多支持。