Several months ago, I created a web scraper to track job listings for a football club. Everything was running smoothly until about a week ago when the program started experiencing multiple issues.
Despite my efforts to troubleshoot and make changes to the code (primarily moving away from CSS selectors that were becoming unreliable over time), I now find myself facing a major problem with no clear solution in sight.
The intended operation of the scraper is straightforward: it should navigate to the website, locate job postings, open them in separate tabs, extract the relevant data, apply some formatting, and then move on to the next listing until every job is documented.
However, the current dilemma arises during the scraping process. The program successfully retrieves the initial link and extracts information (as indicated by output in the terminal), but it abruptly halts at line 61 where it fails to find the "linkTitle" attribute.
I have resorted to revisiting an older version of the code that had previously functioned flawlessly until recently. The sudden shift in behavior prompted me to transition from CSS selectors to Xpath selections, along with adjusting the find() method which had ceased to perform its intended task.
Why is this error occurring within the code? What is causing the breakdown on line 61?
Modifications have been made to address these issues.
You can view the revised code here:
line 61, in <module>
link = jobitem.find("a", class="linkTitle")
^^^^^^^^^^^^^
AttributeError: 'WebElement' object has no attribute 'find'
The specific reason behind the failure at line 61 remains unclear to me. While observing the program in action, it appears to successfully identify and access links before encountering a halt in operation just prior to returning to the main tab containing job listings.