Guide to scraping a website using node.js, ASP, and AJAX

I am currently facing an issue where I need to perform web scraping on this specific webpage form.

This webpage is dedicated to vehicle technical reviews, and you can try inputting the car license CDSR70 for testing purposes.

As mentioned earlier, I am utilizing node.js with the following package.json configuration:

{
  "name": "test",
  "dependencies": {
  "express": "^3.4.8",
  "express.io": "^1.1.13",
  "swig": "^1.3.2",
  "connect-redis": "^1.4.7",
  "request": "^2.34.0",
  "cheerio": "^0.13.1",
  "urllib": "^0.5.8"
  }
}

In addition, I am using Firebug to analyze the database parameters being sent; however, it seems that this form utilizes AJAX which makes Firebug less helpful in this context.

Below is the code snippet I am attempting to execute:

var urllib = require('urllib');
var cheerio = require('cheerio');
urllib.request('http://www.prt.cl/Paginas/RevisionTecnica.aspx', {
method: 'POST',
data: {ppu: 'CDSR70'}
}, function(err, data, res) {
            if(!err && res.statusCode == 200){
                var $ = cheerio.load(data);
        $('#resultPanel').each(function() {
            console.log($(this).text().trim()); 
        });
    }
    else
        //TODO
        throw err;
    });

The HTML containing the table results is as follows:

<div id="resultPanel" style="display: block;">

My goal is to extract the entire table result, including Vehicle information (Información del vehículo) and details of each inspection conducted on the vehicle (Información de Revisión Técnica). Currently, I am only able to retrieve a limited amount of text:

Pinche para ver información de Revisión Técnica





                    Pinche para ver información de Planta de Revisión Técnica













                    Mapa de Ubicación de PRT

It appears that the trim() function is also not functioning properly. Any assistance or suggestions would be greatly appreciated. Thank you

EDIT: Switching from the POST method to the GET method yields the same results.

javascriptasp.netajaxnode.jsweb-scraping

Answer №1

I recommend checking out PhantomJS (http://phantomjs.org/) and CasperJS (). They are both powered by node.js and have the capability to run javascript code - you can use them to extract data from that particular website.

Answer №2

At long last, I received a response from a forum. It turns out the URL was incorrect all along. The correct URL should actually be:

urllib.request('http://www.prt.cl/infovehiculomttwsNew.asmx/infoVehiculoMTT', {
    method: 'POST',
    data: {ppu: 'CDSR70'} 
}, function(err, data, res) {
    if(!err && res.statusCode == 200){
        var $ = cheerio.load(data);
        $('*').each(function() {
            console.log($(this).text());                
        });
    }
    else
        //TODO 
        throw err;
});

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The intricate scripting nestled within the jQuery function

When using jQuery, I am looking to include more codes within the .html() function. However, I also want to incorporate PHP codes, which can make the writing style quite complex and difficult to read. Is it possible to load an external PHP/HTML script? fu ...

javascriptjqueryhtmlajax

Installing and running Node.js within a tomcat server

After creating a web application using Angular, Node/Express, and MySQL, I faced an issue with deployment. My Angular app is running on a tomcat server connected to multiple PCs, but now I need to also deploy my backend (Node.js/Express.js) on the same s ...

node.jsangularexpressangular-fullstack

In Internet Explorer 10, it is not possible to access the document.links using the link's id; it can only be accessed using the

Why does the following code work in FireFox 23.0.1 and Chrome 29, but not in IE 10? <!DOCTYPE html> <html> <head> <title></title> <script type="text/javascript"> function loadFile1(){ a ...

javascripthtml

Unusual behavior observed while looping through an HTMLCollection returned by the getElementsByClassName method

After spending some time debugging, I discovered an issue with my function that changes the class of elements to modify their properties. Surprisingly, only specific elements were being affected by this change. It took me a while to troubleshoot and resolv ...

javascripthtmlgetelementsbyclassnamehtmlcollection

How to make an AJAX request in jQuery / JavaScript after a specific time interval has passed

Here is the code snippet I'm working with: $('input.myinput').each(function(){ var id = $(this).val(); var myajax = function(){ $.ajax({ url: ajax_url, type: "GET", data: ({ code: id ...

javascriptjqueryajaxforms

Flexible array for varying results

I'm attempting to display a random selection from two arrays by alternating between them, with no concern for storage capacity. Unfortunately, my code is not functioning correctly and I am unsure why. var male = ["have a shot", "item 2", "item 3"]; ...

javascriptarraysvariables

Is there a way for me to streamline the process of logging in using either Google or Facebook?

Is there a way for me to automate the scenario if I'm unable to locate the element using Appium Uiautomator? https://i.stack.imgur.com/Rjji4.png ...

javascriptandroidappium

Is there a way to achieve a transparent background while animating the second text?

I am seeking to create a unique typography animation that involves animating the second text, which is colored and consists of multiple text elements to animate. The animation should showcase each text element appearing and disappearing one after the other ...

javascripthtmljquerycss

Are you experiencing an issue with UTF8 in PHP, Ajax, or jQuery

When I insert text into my SQL INSERT statements and it is stored in the database, the displayed text does not match what was expected. For example: This is the actual text in the database: lažljivo. However, the expected text should be lažljivo (in ...

phpjquerymysqlajaxutf-8

Error: Unable to simplify version range

Attempting to follow the Express beginner's guide. I created an app and executed npm install as per the provided instructions. > npx express-generator --view=pug myapp > npm install npm ERR! semver.simplifyRange is not a function npm ERR! A com ...

javascriptnode.jsexpressnpmexpress-generator

"Utilizing jQuery to apply a class based on the attributes of CSS

Is there a way in jQuery (or plain JS) to create a condition based on whether a div has a specific CSS attribute? For instance, I need jQuery to apply position:fixed to an element's CSS when another element has display:none, but switch back to positi ...

javascriptjqueryconditional-statementsjquery-events

What is the process for implementing a new requirement for 'confirmed' users when they log in with passport.js?

Looking to enhance the functionality of passport.js by implementing a conditional check to verify if the user account is confirmed. The plan is to insert an 'if statement' towards the end of the code, right after validating the user's email ...

node.jsexpresspassport.js

Is there a way to pass a token variable from the main page to an iframe without relying on a post message?

I am seeking assistance on how to transfer a variable from a parent window to an iframe. My situation is as follows: I am working with 2 Angular5 applications named A and B . Application B is loaded within Application A using an iframe. The aut ...

javascriptangulariframe

Switch out an item within a list of objects with a different item

I have a variable called this.rows that contains a collection of items. There is a real-time item being received from the server, which matches one of the items in the this.rows object collection. How can I replace an item with new values? Below is an ex ...

javascriptvue.jssocket.iolodash

Retrieving data from Node.js within an Angular application

I am currently working on retrieving data from MongoDB and displaying it on my website. However, I am facing an issue in sending the entire fetched object to a specific port (the response) so that I can retrieve it from Angular. I also need to know how to ...

javascriptnode.jsangular

Is it possible to utilize X-Y coordinates as repositories for values beyond just X-Y coordinates themselves?

I am in the process of creating a tile-based grid and I need to expand the X-Y coordinates to include additional values for determining characteristics of each tile, such as resources (for example, food, water, stone, lumber) within a game context. Conside ...

javascripthtmlhtml5-canvas

Monitoring user clicks using JavaScript code implemented

Currently, I am utilizing a JavaScript file to load on certain pages in order to track various events. Within this JavaScript file, there is a collection of selectors that have listeners attached to them. When these selectors are clicked, the tracking AP ...

javascripthtml

How can I intercept/manage the back button of the browser in React-router?

Utilizing Material-ui's Tabs, which are controlled, I am implementing them for (React-router) Links in the following manner: <Tab value={0} label="dashboard" containerElement={<Link to="/dashboard/home"/>}/> <Tab value={1} label="users ...

javascriptreactjsreact-routermaterial-ui

Create a mechanism in the API to ensure that only positive values greater than or equal to 0 are accepted

My goal is to process the API result and filter out any values less than 0. I've attempted to implement this feature, but so far without success: private handleChart(data: Object): void { const series = []; for (const [key, value] of Object.e ...

javascriptangulartypescripthighcharts

Reaching out to the Edge: Enhancing the jQuery Slider Experience

Alright, I'm really excited about using this amazing slider. What I love most is the "free mode" feature that creates this stunning sliding effect. The size and number of slides are absolutely perfect for me. But there's just one small adjustment ...

javascriptjqueryhtmlcssslider