As a beginner in the world of Python and Scrapy, I am struggling with the complexities of Scrapy documentation. Despite successfully creating a spider for my school project to scrape data, I am facing issues with the formatting in JSON export. Here is a snippet of my code;
def parse_links(self, response):
products = response.css('qwerty')
for product in products:
yield {
'Title' : response.xpath('/html/head/title/text()').get()
'URL' : response.url,
'Product' : response.css('product').getall(),
'Manufacturer' : response.xpath('Manufacturer').getall(),
'Description' : response.xpath('Description').getall(),
'Rating' : response.css('rating').getall(),
}.
The current JSON export appears like this;
[{"Title": "x", "URL": "https://y.com", "Product": ["a", "e"], "Manufacturer": ["b", "f"], "Description": ["c", "g"], "Rating": ["d", "h"]}]This is how it looks currently.
However, I aim to have my data exported in a different format;
[{"Products": [{"Title":"x","URL":"https://y.com", "Links":[{"Product":"a","Manufacturer":"b","Description":"c","Rating":"d"},{"Product":"e","Manufacturer":"f","Description":"g","Rating":"h"}]}]}]This is how I want the data to be formatted.
I have tried various solutions from the web but nothing has worked so far, and the explanations on the Scrapy site are difficult for me to comprehend as a newcomer. Any assistance would be greatly appreciated. While I managed to create the scraper easily, I have been stuck on this particular issue for a day now. Just to clarify, I am not utilizing any custom pipelines or items in this process.
Thank you in advance and have a wonderful day.