While working on a scraping project, I decided to use scraperAPI.com for IP rotation. In my attempt to incorporate their new post request method, I encountered an error stating 'HtmlResponse' object has no attribute 'dont_filter'. Below is the custom start_requests function code snippet:
def start_requests(self):
S_API_KEY = {'key':'eifgvaiejfvbailefvbaiefvbialefgilabfva5465461654685312165465134654311'
}
url = "XXXXXXXXXXXXXX.com"
payload={}
headers = {
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'x-requested-with': 'XMLHttpRequest',
'Access-Control-Allow-Origin': '*',
'accept': 'application/json, text/javascript, */*; q=0.01',
'referer': 'XXXXXXXXXXX.com'
}
client = ScraperAPIClient(S_API_KEY['key'])
resp = client.post(url = url, body = payload, headers = headers)
yield HtmlResponse(resp.url, body = resp.text,encoding = 'utf-8')
Interestingly, when I run this script step by step in scrapy shell, it functions properly and returns the expected data. Any help or insights regarding this issue would be incredibly helpful as I have been troubleshooting for 4 hours now.
Notes:
- The client.post method returns a response object
- The API key mentioned above is not real, just a placeholder
- The client.post method does not include a body option