• Grabbing web documents with a unix script (i.e., automated access without a browser)

• Developing multi-part dialogs with a user (and remembering previous answers

• Generating on-the-fly graphics

• Restricting access to your page(s)

• Starting long jobs, and disconnecting from the browser

Manipulating web documents from a unix script

For instance, you can print the document, or store the document in a file on your local disk. (The latter is particularly helpful for binary files.)

Tutorial documentation on this package (brief) can be found at http://perl.com/perl/wwwman/libwww/lwpcook.html

More complete information on LWP, and other perl packages for the web, can be found at http://perl.com/perl/wwwman/index.html

Getting LWP

Perl software, including LWP and many other modules, can be downloaded from here.

If your sys-admin doesn't want to install some module in system directories, or if you just want to try it out without bothering him or her, you can install the module into your own directory, for instance

/home/mtn/my-home-directory/my-perl-library

use lib "/home/mtn/my-home-directory/my-perl-library"

Getting a document from the web

Here is an example of perl code you could run directly from Unix (for instance with a cron job), to grab a file from anywhere on the web:

#!/usr/local/perl5/bin/perl -wT

use LWP::Simple;

$url = "http://www.somewhere.gov/path/file-I-want.html";
$local_file = "xyz.txt";

getstore $url,$local_file; #store as a local file
#OR...
getprint $url;  #print it out

Let's try calling get_doc.pl from unix.

A better way to set the Expires: tag

#get seconds-since-1970
$current_time = time();

#load the Date portion of the LWP library	
use HTTP::Date;  

#make a nicely-formatted time
$stringGMT = time2str($current_time);

print <<"EOI";
Expires: $stringGMT
Content-type: text/html

[put your HTML stuff here!]
EOI

Maintaining state

There are several ways around this, and most involve sending state information back to the user in a way that it is re-sent to the server in the next communication.

For example, consider this multi-stage process:

• User requests data in a certain time range

• Server checks that the time range makes sense

• Server reports back to the user about the time range and, if it is OK, asks for more input

• User provides more input

• Server processes the final request

More on maintaining state

Note: hidden data isn't really hidden, the user can use view source to see it. But it doesn't clutter up the user's screen.

The most interesting aspect of this multi-part dialog is the script that responds to the first form, and creates the second.

Here is how it works.

(If you have a lot of information to save between calls to the server, it may be simpler to put it in a local file--with a unique file name--on the server computer, and simply send the file name back to the browser as a hidden variable.

On-the-fly graphics

Since we now know how to run processes from perl, on-the-fly graphics are simple.

You simply run a process that generates a .gif document, and create some html code that has a link to that .gif.

For example:

$plot_file = "/tmp/$$my_plot.gif";
open(GRAPH, "|/disk/my-graphics-program > $plot_file");
print GRAPH <<"EOI"
(commands for the graphics program go here)
...
EOI

<img src="$plot_file">

In this way, you can use any graphics package, for instance IDL, AVS, postscript, NCAR graphics, or anything else.

More on-the-fly graphics

This link to <a href="clock2.cgi"> generates graphics this way.

If you wanted to have the image appear directly in an html document, you'd include this in your html code:

<img src="clock2.cgi">

Generating graphics in perl

Two interesting perl graphics packages are

• pgperl, which is a perl extension of the (FORTRAN) PGPLOT language (used extensively in astronomy). This appears to include very nice ways of plotting data

• gd, which is a perl extension of the GD graphics library. This package is good for annotating existing .gif images, drawing lines, rectangles, arcs, doing flood fills, etc.

• But, beware of putting too much time into learning these. Java may be a better way to go because: on-the-fly graphics take time on the server computer. If you expect lots of hits to your site, wouldn't you rather have the work done on each browser's computer?

A look at gd

gd is currently installed in the web101/utilities directory.

Here is the code that generated the graphic we saw earlier. Here, again, is its output.

A more complex application (that probably would be better done in pgperl) is this start at plotting mesonet data.

Here is the script that does the work.

Homework: make the output nicer, so we can really generate mesonet time series plots from perl, and extend the script so that any variable can be plotted.

Here's an example of the kind of image more typically done with gd.

Warning: Sometimes on-the-fly graphics are not what you want. If you expect to generate the same graphic multiple times, it may be more efficient to generate the graphic once and store the gif image. Then it can be accessed multiple times with minimal CPU load on your server.

Restricting access to your page

But sometimes you want to restrict access to a particular cgi script. This is easy to do in perl.

Here is an example of some perl code that restricts access to particular domain names. (It is also a good example of using regular expressions.):

#get return address (name or number)
$returnAddress = ($ENV{'REMOTE_HOST'} || 
                  $ENV{'REMOTE_ADDR'} ||
                  "(Unknown requestor)");

#match the regular expression
#(Be sure to escape the dots!)
unless ($returnAddress =~ m|fsl\.noaa\.gov|) {
    print "Content-type: text/html\n\n" .
          "<h1>$returnAddress not allowed access.</h1>";
    die;
}
... only acceptable users get this far

Bypassing the server

Recall how the communication links usually work:

'nph-' is the prefix you need

Why would you want to do this?

• to avoid caching of the script's output by the server

• to enable your script to break its connection with the browser before it is done.

This is a script that does that, and here is its output.

You see, the server buffered all the output and sent it back in one chunk.

On the other hand, here is the output of the same script, but renamed nph-count.cgi

(We sent the status header because the server doesn't add this to the output from nph- scripts.)

Letting go of the browser

• accept some user input (such as from a form)

• start some long process (such as a model run)

• tell the user the process has started

• say goodby to the user

Here's a cgi script that starts a long job

After the script makes its announcement, it uses open to start a long job, then exits.

Here is the cgi script, and here is the script that it calls.

We execute this by calling unhook.cgi.

You see the problem: the server waits until the long job is completed before it sends anything back to us.

The way around this is to use nph-unhook.cgi.

The End

The class web server will stay up for two months more only, so be sure to move anything you want to keep to another place.

My lecture notes will remain, somewhere. I'll put a link on the internal documents page of the FSL homepage, so FSL people will be able to access them.

There wasn't much interest in another O'Reilly order, so I doubt that I'll send one in, unless some interest materializes in the next week or so.

We've obviously just scratched the surface. But if you put some time into studying Learning Perl, other books, and on-line documentation, you'll find that programming for the web isn't too hard, and can b a lot of fun.

If you have any questions about the web, cgi scripts, and perl, I'm happy to share what I know.