I'm currently working on a web-scraping project to extract data from a platform known as "Startup India," which facilitates connections with various startups. I have set up filters to select specific criteria and then click on each startup to access detailed information. However, I am encountering an issue where the URLs I need for scraping are not displaying in the console.
Below is the code snippet:
import scrapy
from selenium import webdriver
import os
import logging
class ProductSpider(scrapy.Spider):
name = "product_spider"
allowed_domains = ['https://www.startupindia.gov.in/']
start_urls = ['https://www.startupindia.gov.in/content/sih/en/search.html?industries=sih:industry/advertising&states=sih:location/india/andhra-pradesh&stages=Prototype&roles=Startup&page=0']
def __init__(self):
cwd = os.getcwd()
self.driver = webdriver.Chrome("C:/Users/RAJ/PycharmProjects/WebCrawler/WebCrawler/WebCrawler/spiders/chromedriver.exe")
def parse(self, response):
self.driver.get(response.url)
next = self.driver.find_elements_by_css_selector('div#persona-results a')
logging.info(next)
for i in next:
try:
logging.info(i.click())
logging.info(response.url)
# Extract and store the required data using scrapy items
except:
print("Yolo")