[KLUG Members] Dynamic HTML/HTTP question

[Jamie McCarthy]

peter@killdevil.org (Peter Buxton) writes:

> Heck, it might not be a pain even using socket(1) and bash(1),
> maybe even chat(8).

(shudder :)


RGB writes:

> <form action="index.asp" method="post" name="dateSearch" 
>  onsubmit="return validateForm(this)">
> <input type="text" name="startDate" size="15">
> <input type="text" name="endDate" size="15">
> <input type="image" src="/images/button-images/search_button.gif"
>  width="50" height="10" border="0">
> </form>

>      http://www.yifnif.com/search/index.asp

Here's a one-liner that would do it (for large values of "one" :).
If the URL request fails, it prints to STDERR something about what
happened and exits with an error.  If it succeeds, it just outputs
the HTML that it retrieved.

    $ perl -MLWP -e '$ua = LWP::UserAgent->new;
    $response = $ua->post("http://www.yifnif.com/search/index.asp",
       { startDate => "foo", endDate => "bar" }
    );
    die $response->status_line unless $response->is_success;
    print $response->content'

The equivalent shell script is:

    #!/usr/bin/perl
    use LWP;
    $ua = LWP::UserAgent->new;
    $response = $ua->post("http://www.yifnif.com/search/index.asp",
       { startDate => "foo", endDate => "bar" }
    );
    die $response->status_line unless $response->is_success;
    print $response->content

Perl on your system may not have LWP installed.  If the above dies
with a complaint about a missing module, run this as root:

    perl -MCPAN -e 'CPAN::Shell->install("Bundle::LWP")'

If LWP is already installed, that won't hurt anything (it may
upgrade modules to the latest version though :).

> I'm going to rip theresult to shreds and look for some data,
> probably sections of pages that are tables will get processed
> into entry items for another database, or will go into XML
> files, or will just get muched on by the next thing in the
> pipe, or maybe the next funvtion, for computation and what-all.
> 
> I'm working on something that harvests data from a number of
> source, and does all kinds of analysis, and the rsults go out
> somewhere. I have most of the data locally, but occasionaly
> I'll need to reach out over the nt and query THAT database, or
> fetch stuff from yonder ftp site, or (here's where this
> started) ask a web0based app a question, and then parse the
> response, and keep on truckin'.

You are *so* going to want to use Perl for this.  :)