I will provide detailed information on the task at hand.
My objective is to establish a connection with an https URL that streams JSON data, look for specific keywords using REGEX as the data flows in, extract the matching JSON element, decode it, and then store it in a database.
The REGEX part of the process is not presenting any difficulties; it is relatively straightforward.
However, I am encountering challenges when it comes to reading the data line by line. I have experimented with various examples found online that utilize LWP, but the issue arises where the script hangs due to the stream loading continuously without stopping.
Below is the code snippet that shows my progress so far:
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
use IO::String;
my $handle = IO::String->new(get("https://stackoverflow.com"));
while (defined (my $line = <$handle>)) {
print $line; #Inserted for testing
#Decode and insert into DB here
}
close $handle;
Given that the data arrives at a rapid pace, efficiency is crucial in the design of the script.
If you have any advice or guidance on how to successfully complete this task, I would greatly appreciate it.
Thank you, Sean