Accessing a webpage using python requests to authenticate users

I'm attempting to access a website using python 3, requests, and lxml for logging in. However, despite sending a post request to the login page, I am unable to access the pages that are meant to be visible after logging in. What could be the issue?

import requests
from lxml import html

session_requests = requests.session()

login_URL = 'https://www.voetbal.nl/inloggen'
r = session_requests.get(login_URL)

tree = html.fromstring(r.text)
form_build_id = list(set(tree.xpath("//input[@name='form_build_id']/@value")))[0]

payload = {
    'email':'<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="94f9fbf9bae7fbf7f7f1e6d4f9f5fdf8baf7fbf9">[email protected]</a>',
    'password':'testaccount',
    'form_build_id':form_build_id
    }

headers = {
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-Encoding':'gzip, deflate, br',
    'Accept-Language':'nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7',
    'Cache-Control':'max-age=0',
    'Connection':'keep-alive',
    'Content-Type':'multipart/form-data; boundary=----WebKitFormBoundarymGk1EraI6yqTHktz',
    'Host':'www.voetbal.nl',
    'Origin':'https://www.voetbal.nl',
    'Referer':'https://www.voetbal.nl/inloggen',
    'Upgrade-Insecure-Requests':'1',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
    }

result = session_requests.post(
    login_URL,
    data = payload,
    headers = headers
)

pvc_url = 'https://www.voetbal.nl/club/BBCB10Z/overzicht'
result_pvc = session_requests.get(
    pvc_url,
    headers = headers
)

print(result_pvc.text)

The account mentioned here is active but it's just a test account created for this purpose. Feel free to test it out.

pythonpython-requestslxml

Answer №1

Solution:

There were several issues that needed to be addressed:

Payload: The 'form_id': 'voetbal_login_login_form' key-value pair was missing. Special thanks to @t.m.adam for pointing this out.

Cookies: The request cookies were missing and appeared to be static. I manually added them, following the guidance of @match and @Patrick Doyle, which resolved the issue.

Headers: I removed the 'content-type' line as it contained a dynamic element.

The login functionality is now working perfectly!

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Learn how to retrieve values from a .json file in real-time and then perform comparisons with user input using Python

I have a JSON file structured like this: [ { "name": { "common": "Aruba", "official": "Aruba", "native": { "nld": { "official ...

javascriptpythonarraysjson

Flask does not provide a direct boolean value for checkboxes

After struggling for a week, I am still lost on where to make changes in my code. I need the checkbox to return a boolean value in my Flask application. Below are snippets of the relevant code: mycode.py import os, sqlite3 from flask import Flask, flash ...

javascriptpythonhtmlcssflask

What's the best way to determine which of the two forms has been submitted in Django?

On my homepage, I have both a log_in and sign_up form. Initially, the log_in form is displayed by default, but when a user clicks on the Sign Up button, the sign_up form appears. These toggles switch depending on which button the user clicks. from django ...

javascriptjquerypythondjango

Exploring the capabilities of arrays within Ajax

Below is the original code I wrote in JavaScript: var wt_val = []; for (i = 0; i<human_wt.length; i++){ var mult; mult = data_list[basket_list[button_port_name][i]].map(x => x*(wt[i]/100)); wt_val.push(mult); ...

javascriptpythonajaxflask

Invoke a Python function from JavaScript

As I ask this question, I acknowledge that it may have been asked many times before. If I missed the answers due to my ignorance, I apologize. I have a hosting plan that restricts me from installing Django, which provided a convenient way to set up a REST ...

javascriptpythonangularjsclient-server

Encountered CSRF validation error while working with a Python Django backend in conjunction with React frontend using Axios for making POST requests

I recently completed a tutorial at and now I'm attempting to add a POST functionality to it. Despite obtaining the csrf from cookies and including it in the "csrfmiddlewaretoken" variable alongside a test message in json format for the axios function ...

javascriptpythondjangoreactjsaxios

Button click event is not being triggered by Ajax rendering

I am facing an issue with my Django template that showcases scheduled classes for our training department. Each item in the list has a roster button which, when clicked, should display the class roster in a div. This functionality works perfectly. However, ...

javascriptpythonajaxdjangodom

Error message stating that there is no property 'collection' in Firestore when using Firebase v9 modular syntax in Firebase Firestore

Working on a React application that makes use of Firebase Firestore for handling database operations, I recently upgraded to Firebase version 9 and adopted the modular syntax for importing Firebase services. Nevertheless, when attempting to utilize the co ...

javascriptpythonhtmlcssreactjs