Exploring websites with Python's mechanize library by utilizing __doPostBack functions for

Is it possible to navigate a table on a web page using mechanize if the table uses __doPostBack functions?

Here is my code snippet:

import mechanize
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
br.open("http://www.gfsc.gg/The-Commission/Pages/Regulated-Entities.aspx?auto_click=1")

page_num = 2
for link in br.links(): 
    if link.text == str(page_num):
        br.open(link) #I suspect this is not correct
        break

for link in br.links():
    print link.text, link.url

While searching for all controls in the table like drop-down menus may not show the page buttons, searching for all links in the table does. The page button does not have a URL like typical links, resulting in a TypeError: expected string or buffer error.

It seems like mechanize should be able to handle this type of navigation.

Thank you for taking the time to read this.

pythonpython-2.7mechanizedopostback

Answer №1

When dealing with tables that utilize __doPostBack, Mechanize comes in handy. BeautifulSoup was my tool of choice for parsing the HTML to extract necessary parameters. I also sought valuable insights on regex manipulation. My implementation is outlined below.

import mechanize
import re # craft a regex pattern to acquire __doPostBack parameters
from bs4 import BeautifulSoup
from time import sleep

br = mechanize.Browser()
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
response = br.open("http://www.gfsc.gg/The-Commission/Pages/Regulated-Entities.aspx?auto_click=1")
# Triggering __doPostBack function for page navigation 
for pg in range(2,5):
    br.select_form(nr=0) # sole form on the webpage
    br.set_all_readonly(False) # configuring __doPostBack params

    # Leveraging BeautifulSoup for parsing
    soup = BeautifulSoup(response, 'lxml')
    table = soup.find('table', {'class': 'RegulatedEntities'})
    records = table.find_all('tr', {'style': ["background-color:#E4E3E3;border-style:None;", "border-style:None;"]})

    for rec in records[:1]:
        print 'Company name:', rec.a.string

    # Disabling 'Search' and 'Clear filters'
    for control in br.form.controls[:]:
        if control.type in ['submit', 'image', 'checkbox']:
            control.disabled = True

    # Capturing parameters for the __doPostBack function
    for link in soup("a"):
        if link.string == str(page):
            next = re.search("""<a href="javascript:__doPostBack\('(.*?)','(.*?)'\)">""", str(link))
            br["__EVENTTARGET"] = next.group(1)
            br["__EVENTARGUMENT"] = next.group(2)
    sleep(1)    
    response = br.submit()

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Attempting to decipher the solution for preserving information post program termination

I'm currently working on a project that involves saving data upon terminating a program, and I stumbled upon this code snippet online : #input variable saved_data = {"one" : 1, "two" : 2} file = open("Python.txt", " ...

python

Combining Javascript and Django for a powerful web development solution

Having trouble setting up JS on my Django web app, despite reading through the documentation and previous queries. Using the Django dev server with the following file structure: mysite/ __init__.py MySiteDB manage.py settings.py ...

javascriptpythoncssdjango

Rearrange the positions of the latitude and longitude values in a JSON file without any additional tags

I have a JSON file with the following code snippet. I am looking to swap the latitude and longitude values while keeping the output the same. "path": [ [ -0.662301763628716, 51.48792441079866 ], [ ...

pythonnode.jsjson

Having trouble finding an element with Python Selenium after switching to a frame in Firefox browser?

I'm facing difficulty locating the element within the frame even after switching to the correct frame. Below is my code, error message, and HTML source. When I right-click on the frame and choose This Frame -> Show Only This Frame, I can find the e ...

pythonseleniumxpathiframecss-selectors

Setting the starting sequence number for a TCP socket

I'm currently involved in testing and I require the ability to set the initial sequence number (ISN) of a TCP connection to a specific value. The ISN is typically a random value chosen by the OS/Network Stack, but I need to have control over it. Is t ...

pythoncsocketstcp

Creating a Stealthy Presence for Headless Chrome in Python

I'm currently exploring methods to make Chrome headless undetectable, following a specific guide I found. The challenge lies in the fact that the guide relies heavily on Javascript and executing scripts via the command prompt. My goal is to develop a ...

pythonpython-3.xseleniumselenium-webdriver

Issue with saving GIF files when using ImageIO/PIL

My current project involves using ImageIo and PIL to save a series of images as a GIF file. While the images are indeed saved in .gif format, they do not playback as a seamless "video" GIF. gif_images[0].save('path/test.gif', save_all=True, appen ...

pythonpython-imaging-librarygifpython-imageio

The Pandas DataFrame is displaying cells as strings, but encountered an error when attempting to split the cells

I am encountering an issue with a Pandas DataFrame df. There is a column df['auc_all'] that contains tuples with two values (e.g. (0.54, 0.044)) Initially, when I check the type using: type(df['auc_all'][0]) >>> str However, ...

pythonpandasdataframesplittuples

Extracting YouTube Videos Using Python and Selenium

My friend asked me to scrape all the videos from 'TVFilthyFrank'. I have access to all the links for each video. I want to determine the size of each video in MB and then proceed with downloading them. However, using driver.get(VIDEO_URL) and ext ...

pythonseleniumweb-scraping

reducing my code by using parentheses for both 'and' and 'or' outcomes

What is the best way to simplify this code snippet? if (a+b+c == 1000 and a**2 + b**2 == c**2) or (a+b+c == 1000 and a**2 + c**2 == b**2) or (a+b+c == 1000 and b**2 + c**2 == a**2) into: if a+b+c == 1000 and (a**2 + b**2 == c**2 or a**2 + c**2 == b**2 o ...

python

The scipy module encountered an error with an invalid index while trying to convert the data to sparse format

I am currently utilizing a library called UnbalancedDataset for oversampling purposes. The dimensions of my X_train_features.shape are (30962, 15637) and y_train.shape is (30962,) type(X_train_features) is showing as scipy.sparse.csr.csr_matrix An index ...

pythonnumpyscipyscikit-learn

Python Selenium for Web Scraping

I am currently working on a project to extract data from the Sunshine List website () using the BeautifulSoup library along with the Selenium package. My main challenge is figuring out how and where to instruct the driver to wait for elements to load befor ...

pythonpython-3.xselenium

Converting JSON formats with Python module: A step-by-step guide

Is there a way to use a Python module to convert one JSON format to another? I have a JSON object and need to extract the keys and values. How can this be accomplished? Thank you in advance. Input json: { "A": { "sensitive": false, "type": " ...

pythonjson

Creating Python API documentation in PyCharm automatically

My Python project is in PyCharm and I am looking to automate the generation of API documentation (in HTML format) from my Python code and docstrings. On a resource page, it lists out several tools that can be used to generate Python API documentation: a ...

pythonpycharmpython-sphinxdocumentation-generationdocutils

Troubleshooting Selenium Dropdown Visibility Issues

Hello there, I'm a beginner in using Selenium with Python and I am currently facing an issue while trying to automate a simple form. The error I keep encountering is related to the drop-down menu, where it says that the element is not visible. I have ...

pythonselenium-webdriver

Converting a flattened column of type series or list to a dataframe

I am looking to extract specific data from my initial dataset which is structured as shown below: time info bis as go 01:20 {'direction': 'north', abc {'a':12, ...

pythonpandaslistdataframe

Creating a dataframe with fixed intervals - Python

Within my dataset, I have a dataframe that includes the following columns (only showing a portion): START END FREQ VARIABLE '2017-03-26 16:55:00' '2017-10-28 16:55:00' 1234567 x &ap ...

pythonpandasperformancenumpydataframe

Entering a date in a disabled datepicker using Selenium

I am encountering an issue while trying to input my own chosen date as the datepicker is disabled. Whenever I click on the datepicker, it prompts me to select a specific date and even for changing the month, multiple clicks are required. This has left me f ...

pythonpython-3.xseleniumselenium-webdriverselenium-chromedriver

"GeoDjango Crashes Unexpectedly with Segmentation Fault Error

I recently encountered a problem with my Django App after installing GeoDjango. I am using MacOS Sierra 10.12.2 with Python 2.7.13 (installed via mac ports) and Django 1.10.4. Following GeoDjango's tutorial (homebrew), I installed the necessary packag ...

pythondjangogeodjangospatialite

Adding quotation marks to the string form of a two-dimensional array in Python

My code takes user input from a text box and needs to convert it into a 2D list of strings. The user input comes from a Jupyter Dash text input I am working on creating a design where the user does not have to quote the elements themselves and can simply ...

pythonstringcastinguser-inputplotly-dash