My goal is to extract the membership years data from the IMDB Users page.
Link
On this page, there are multiple badges and one badge that is common for all users is the last one.
This is the code I am using:
def getYear(review_url):
response = requests.get(review_url, headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'
})
soup = BeautifulSoup(response.text, 'html.parser')
year = soup.find_all('div', attrs={'class': 'value'})
return year[-1].get_text()
I have researched various sources and found that adding a user-agent in the headers solves similar issues, but even after doing so, it is not working in my case.
To call the function:
getYear('https://www.imdb.com/user/ur102180396')
An error message is returned:
IndexError Traceback (most recent call last)
<ipython-input-24-dc3ce3a7e637> in <module>()
----> 1 getYear('https://www.imdb.com/user/ur102180396')
<ipython-input-23-5871162c538d> in getYear(review_url)
6 year = soup.find_all('div', attrs={'class': 'value'})
7
----> 8 return year[-1].get_text()
IndexError: list index out of range
This error occurs because the method soup.find_all() is returning an empty list. This behavior is confusing to me as sometimes the function works perfectly fine and provides the output, but when applied to all the data (2136 user links), this error emerges.
Executing the function on all users:
years = [getYear(url) for url in user_links]
The list user_links contains URLs for 2136 users.