Tips for dynamically extracting the activated tags from a webpage with Python and Selenium

I have been working on a website where I have implemented the Google Analytics code through Google Tag Manager. The site consists of numerous pages, and I am interested in ensuring that the Google Analytics code fires correctly on all pages. One method is to manually open each URL, access the GA debugger, and verify if the pageviews are firing in the console. However, due to the large number of URLs that need to be checked, I am looking for an automated solution, preferably using Python.

Thus far, I have attempted to fetch the source code of the pages and use regex to identify specific code snippets related to GA and GTM. You can see the code snippet below. Unfortunately, this approach only captures static code, excluding any dynamic pixels or codes that fire after the page finishes loading.

from selenium import webdriver

driver = webdriver.Chrome(executable_path=r"C:\chromedriver.exe")
driver.get("url")

html1 = driver.page_source

html2 = print(driver.execute_script("return 
document.documentElement.innerHTML;"))

I have also tried utilizing BS4 and requests libraries, but have not found them useful in this context.

Answer №1

Currently, I have integrated BrowserMob Proxy with a selenium driver to intercept and analyze all HTTP requests and responses during test execution. In my process, I specifically target requests containing 'google-analytics' in the URL and proceed to validate the event values within those requests against my expected criteria.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Selenium Python can be used to save a web page as a

Trying to print a webpage as a PDF with headers and footers to mimic the output from Google Chrome. Any suggestions? Experimented with PhantomJS but couldn't get it to include headers and footers. from selenium import webdriver import selenium def ...

Searching through nested arrays within JSON structures

I have a JSON response that I need to parse in order to find the ID of a specific folder. The folder is located at a known level of depth within the path structure, and I have the complete path leading to it. For example, if the path is /Core/UI/Folder1/S ...

Tips for retrieving the Email/Text input from a field without the presence of a "value" attribute or text in the code snippet using Java Selenium

Can anyone help me retrieve the email id entered "[email protected]" using java code? I have tried various methods, but unfortunately, I always get a null value in return. My ultimate goal is to validate whether the email field contains a valid entr ...

Experiencing difficulties with executing a right click on an element in Selenium with the contextClick

After repeatedly going through this process, I am struggling to identify where my mistake lies. The code below involves right-clicking an element and then navigating down twice in the popup menu to click on the second option: System.out.println("HELLO ...

Tips for changing to a frame and retrieving an element within it

Looking to access the element containing the text Deprecated within the website "" using selenium in the Chrome browser. WebDriverWait wait = new WebDriverWait(driver, 10); wait.until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(driver.findElement(B ...

Java Selenium TestNG - Excessive Number of Parameters

I am currently working on implementing a Selenium Data Driven framework using TestNG in Java. In my framework, I have a pageObject called Login_Page.java where I store all the locators available on the page. Additionally, I have an appModules class named ...

Is there a way in Python to modify a character within an element that has been located by xpath using Selenium?

When using Selenium in Python, I am trying to retrieve the text of an element from a specific xpath. Once I have retrieved the text, I want to check if it contains a special character "-", and replace it with "0". However, every time I attempt this, I enco ...

What is the best way to retrieve the current DOM using Selenium Java 2.8?

I am currently utilizing the most recent version of Selenium along with the chromedriver to conduct tests on a ZK application. During my testing process, I am interested in extracting the DOM, or at least a portion of it, in order to help locate the neces ...

Adding an assertion to a conditional if..else statement: a step-by-step guide

I am attempting to use an if-else condition within an assert statement. My current code is not functioning as expected. I need to compare two strings one by one. String comment = driver.findElement(By.xpath("//*[@id='historyRow']/tbody/tr[1]/td ...

The sendKeys function in WebDriver is failing to register the '@' character

I am currently dealing with the following issue. Is there anyone who can offer a solution? The problem I am encountering is that the WebDriver sendKeys method is failing to send the "@" character to a text box. When I try sending "Test@123", it only input ...

Ensuring that Selenium WebDriver effectively waits for intricate webpages with extensive JavaScript to fully load

Working with Selenium to test a web application that utilizes a significant amount of JavaScript during page load can present challenges. Unfortunately, in the event of poorly written JavaScript code that cannot be modified, waiting for an element using th ...

Issue with Chrome 59 freezing when using the --headless and --proxy-server flags

With the latest release of Chrome 59 featuring headless support and an updated driver (Chromedriver 2.30), I decided to transition my code over to it. The --headless flag functions fine on its own. The --proxy-server flag also works independently, but ...

Unable to retrieve the value from an "input" element

Currently, I am utilizing Selenium with Java to create a test. My task involves retrieving the text from inside an input element: <table class="boundaryFormAdd"> <tbody> <tr> <td> <input id="id ...

The function of Selenium moveToElement() isn't functioning as expected

Having successfully logged in to , my next step is to test the logout functionality. https://i.stack.imgur.com/X1EyJ.png To log out, I need to hover over a specific div element in order for the sign out button to appear. I initially attempted to achieve ...

Using Python to scrape Shopee.sg with Selenium and BeautifulSoup for web data extraction

I am facing difficulties when trying to scrape data from shopee.sg using selenium and BeautifulSoup. The issue is that I can only extract information for the first 15 out of 50 products on a search results page, with the remaining ones returning null value ...

Having trouble loading cookies using selenium and python?

I'm working on a project that involves accessing a webpage, saving the cookie to a csv file, and then using that cookie later using selenium and python. I've made progress in successfully saving cookies, but when trying to use them later, I encou ...

Python Tips: Automatically detecting and interacting with newly visible iframes

Currently, I am facing an issue when trying to input data into an iframe. The problem is that the iframe takes anywhere between 15-30 seconds to load. I have attempted using time.sleep(30), which works if the iframe consistently loads at around 25-30 seco ...

Locating elements effortlessly using Selenium and Capybara: Search for selectors within any element

Imagine we have a <div class='something-else'> and buried deep within it lies an element <div class='inside-something-else'> My goal is to find a way to locate that specific inside-something-else-div using the methods of Ca ...

Executing the keyboard commands "Ctrl + A," "Ctrl + C," and "Ctrl + V" on a text field using Selenium with C#

Is there a way to easily input a value in a text box, select the entire text inside the text box by pressing "Ctrl+a", copy it by pressing "Ctrl+c", and then paste it back in the same text box with "Ctrl+v" using Selenium and C#? ...

Tips for finding and selecting the "speed test" link on Netflix with the help of Selenium in Python

I'm new to using selenium and may have a basic question (it happens):) I am trying to locate the Speed test link on and then click on it. I have attempted different search methods like text searching, but nothing seems to be working for me, and I&ap ...