The dynamic dropdown on https://www.nseindia.com/ does not display auto-suggestions when Selenium and Python are used to pass values

driver = webdriver.Chrome('C:/Workspace/Development/chromedriver.exe')
driver.get('https://www.nseindia.com/companies-listing/corporate-filings-actions')
inputbox = driver.find_element_by_xpath('/html/body/div[7]/div[1]/div/section/div/div/div/div/div/div[1]/div[1]/div[1]/div/span/input[2]')
inputbox.send_keys("Reliance")

I am attempting to extract data from the table on this website that appears after entering the company name in the text field above it. The code provided works well with drop-downs on other websites like Google search and Wolfram, but when used on this particular website, it only fills in the text field and results in a 'No Records Found' message in the dropdown. Manually entering the same information works correctly.

Answer №1

I tested out your code with some modifications and here are the results of my test:

  • Code Snippet:

    from selenium import webdriver        
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    
    options = webdriver.ChromeOptions() 
    options.add_argument("start-maximized")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    driver.get('https://www.nseindia.com/companies-listing/corporate-filings-actions')
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@id='Corporate_Actions_equity']//input[@placeholder='Company Name or Symbol']"))).send_keys("Reliance")
    
  • Findings: Just like your findings, I encountered the same issue with no visible outcomes:

https://i.stack.imgur.com/GASuw.png


In-Depth Analysis

The click() action on the element labeled as Get Data seemed to be executed. However, upon examining the DOM Tree of the webpage, it was noticed that some <script> tags referenced JavaScripts containing the keyword akam. For instance:

  • <script type="text/javascript" src="https://www.nseindia.com/akam/11/3b383b75" defer=""></script>
  • <noscript><img src="https://www.nseindia.com/akam/11/pixel_3b383b75?a=dD02ZDMxODU2ODk2YTYwODA4M2JlOTlmOGNkZTY3Njg4ZWRmZjE4YmMwJmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript>

This strongly suggests that the website is safeguarded by Bot Manager, an advanced bot detection service from Akamai, resulting in a blocked response.


Bot Manager Insights

According to the article Bot Manager - Foundations:


Final Thoughts

Evidently, the data request appears to be identified as originating from a Selenium-driven WebDriver instance, leading to a blocked response.


Additional Resources

Here are a few relevant documents for further reading:

  • Bot Manager
  • Bot Manager : Foundations

tl; dr

Check out these related discussions:

  • Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
  • Unable to use Selenium to automate Chase site login

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The issue of time inconsistency in model object save when using datetime.now() in Django

This table shows my admin interface with records ordered by their id in descending order (latest record at top). Below is the code snippet used for creating model objects and saving them: notification = Notification(from_user=from_user, to_user=to_user, ...

Retrieve a button using its name attribute in Selenium with Python

https://i.stack.imgur.com/MbJgQ.png In the HTML code below, there is a button that changes its x-path dynamically while the text "Show Actions" remains constant. I am uncertain whether "Show Actions" is the name, title, or id of the button. Is there a wa ...

Retrieving POST data from AngularJS AJAX in Python FlaskExplanation: In this

Utilizing Angular's AJAX call, I am sending data to my Flask backend for natural language processing. Here is the AJAX code snippet: $scope.processText = function(){ $http({ method: "POST", url: "http://127.0.0.1:5000/processTex ...

Initiate the Selenium server automatically to facilitate end-to-end testing

After reading a helpful post on Stack Overflow, I was able to configure my Gruntfile for testing. Initially, I manually downloaded Selenium standalone and specified its location in the file, which allowed my tests to run successfully. However, seeking auto ...

I am attempting to scrape the text using BeautifulSoup in python for my data collection project

Can someone guide me on how to efficiently extract specific data using Beautiful Soup? I am working on gathering text information that reads "RoSharon1977". Here is the code snippet I have been using: I'm currently attempting to use: <div id="twi ...

What is the method for extracting a string from its encapsulation to use it in another string

How can I extract a string within a string using Python? Consider the following scenario: >>> original_string "u'abcde'" I want to retrieve this string: >>> extracted_string 'abcde' ...

Encountering error java.lang.ClassNotFoundException: org.openqa.selenium.MutableCapabilities while attempting to initiate browser using Selenium

I encountered the error mentioned above when attempting to launch the Chrome browser in IntelliJ. This issue occurred while executing the code on my Ubuntu 16.04 system, and I've already attempted updating the browser as well as using the appropriate ...

Is the username you want available?

I am facing a challenge in my registration form validation process where I need to verify the username input using javascript and flask in python, but the implementation is unclear to me. The requirement is to utilize $.get in the javascript segment along ...

Obtaining the text from an element in Selenium using JavaScript

I've been trying to figure this out, and it seems like it should be a simple task, but how can I extract the text from a table displayed in a browser? When I use the "inspect" tool, the code snippet looks something like this: <tr role="row&qu ...

Python Numpy: Assigning dual outcomes when np.where condition is met

Explanation of Input: The DataFrame df is provided Category Location Min Space AA Y 0.25 0.25 BB Y 0.25 1 CC 0.5 0.625 Situation: The task at hand is to calculate Reduced_Space ...

Utilizing Robot Framework to select CSS attributes

The Selenium documentation provides an example of how to use Get Element Attribute: ${id}= Get Element Attribute css:h1 id However, I encountered a problem with this selector: ${VISIBILITY}= Get Element Attribute css:visibility mySidebar I ...

Exploring the Power of Pythons, Selenium, and XPATH for Handling New Windows

Is it possible to retrieve an element through XPath after using the .click() method when that element is located within a section of JavaScript known as BODY_BLOCK_JQUERY_REFLOW and appears only after the click action? I am attempting to access this speci ...

Switching XML to JSON using Python while confirming it against a schema

I am looking for a solution to convert XMLs into JSON format. Although I have found various packages like xmltodict in Python that can accomplish this, I'm facing an issue where single elements within a node are being converted into dictionaries. Howe ...

Extracting the Instagram followers of a specific account

Is it possible to scrape followers from an Instagram account using Selenium? I am attempting to do so with the following URL: I am encountering a problem where I am unable to scroll down a popup page, with the only option being to navigate back. Below is ...

Tips for using selenium to input a username and password when the IDs are concealed on a webpage

I am attempting to use Python code to automate the process of filling in a username and password on a webpage. However, while the code is able to successfully open the page, it is not populating the username and password fields as expected. Below is the co ...

The Selenium WebDriverWait encountered the issue of "Element is not clickable" despite being implemented

I encountered an issue with the code below (br represents the webdriver and all necessary imports are in place). Initially, the first three lines perform as expected. However, when attempting to execute link1.click(), I encounter an error: link = WebDriv ...

Need help fixing a corrupted JSON file that contains escaped single and double quotes?

Challenge I am faced with a sizable JSON file (~700,000 lines, 1.2GB in size) that contains Twitter data which needs preprocessing for both data and network analysis purposes. While collecting the data, an error occurred: instead of using " as a separator ...

Ways to verify if a string in one row is present in the string of another row within the same column of a pandas dataframe

Here is a dataframe I am working with: The "docid" is the exploded column of "DocID". I need to check if a string in the "Term" column appears in another row within the same column. For instance, rows 3 and 4 have "in the treatment" and "in the treatment ...

What is the reason behind numpy.angle() not being a universal function (ufunc)?

What makes numpy.angle() different from other numpy universal functions (ufuncs)? Although it seems to meet the criteria outlined in the numpy documentation, why is it not officially categorized as a ufunc? I initially speculated that its conversion of c ...

What enables ClusterRole to execute the list_pod_for_all_namespaces() function while restricted from accessing read_namespaced_pod_status()?

Currently utilizing Python to check the status of a pod: kubernetes.client.CoreV1Api().read_namespaced_pod_status(name=name, namespace='default') However, this results in a forbidden, 403 response. The following code works without any issues: ...