17.6. Creating a Web Client with WebRequest and WebResponse

As you would expect in a chapter on ASP.NET, the role of the Web client does not receive much attention. It's usually assumed to be a browser that requests a Web page specified by the user, and renders the returned HTML content. Despite the evolution of browser features, its primary role remains to display what it receives. It is of less use if you want to parse and extract portions of the received content—or want to examine the HTTP message headers. For this, you need a Web client—a program that communicates with an HTTP server and includes custom methods to perform operations on the response object. This section demonstrates how easy it is to write a Web client using .NET classes to handle the HTTP request/response duties.

WebRequest and WebResponse Classes

The System.Net namespace contains classes that are intended for applications involved with network communications. Included in these are the abstract WebRequest and WebResponse classes that contain generic properties to upload and download data given a specific Uniform Resource Identifier (URI). These classes are designed to support the response/request model—not a specific protocol. The details of that are left to descendant classes. To handle the HTTP protocol, we call on the HttpWebRequest and HttpWebResponse classes. Note that other classes are available to support the file:// URI scheme and FTP operations.

Web Client Example

This example accepts a URL, sends a request for its Web page, and uses the response object to display the server description, IP address(es) of the server, and source code for the requested Web page. Figure 17-10 shows the interface for the application.

Figure 17-10. Example using `HttpWebRequest` and `HttpWebResponse` classes

[View full size image]

The basic steps for communicating synchronously with the HTTP server are quite simple:

1.	Create an `HttpWebRequest` object that identifies the URL using the `Create` method: request = (HttpWebRequest) WebRequest.Create(url);
2.	Get a response from the Internet resource using the `WebRequest.GetResponse` method: response = (HttpWebResponse) request.GetResponse();
3.	Get the `Stream` used to read the body of the response and read its contents. Stream s = response.GetResponseStream(); string strContents = new StreamReader(s).ReadToEnd();

Note that request and response are cast to HttpWebRequest and HttpWebResponse types, respectively. As mentioned, these subclasses deal specifically with the HTTP protocol (see Listing 17-9).

Listing 17-9. Using `WebRequest` and `WebResponse` to Scrape a Web Page


private void btnURL_Click(object sender, System.EventArgs e)

{

   // Fetch web page for requested URL

   HttpWebRequest request; 

   HttpWebResponse response; 

   if(txt_URL.Text.Length>0)

   {

      lblServer.Text="";

      tbIP.Text="";

      string serverPath= txt_URL.Text;

      string url="http://"+serverPath;

      // create a request to the url 

      request = (HttpWebRequest) WebRequest.Create(url); 

      request.Timeout= 7000; // timeout after 7 seconds

      try

      {

         response = (HttpWebResponse) 

                  request.GetResponse(); 

         lblServer.Text= response.Server;

         // Get a stream to send the web page source 

         Stream s = response.GetResponseStream(); 

         string strContents = new 

                   StreamReader(s).ReadToEnd(); 

         // Place Web page source in text box

         HTMLViewer.Text= strContents;  

         s.Close();

         ListIP(serverPath);   // List IP address(es)

      } 

      catch ( Exception ex)

      {

         lblServer.Text= ex.Message;

      }

   }

   else 

   {

      lblServer.Text= "Please enter a domain name.";

   }

}

private void ListIP(string uri)

{

   // List IP addresses for this domain

   // Use only server name part of URI for IP resolution

   int ndx= uri.IndexOf("/");

   if(ndx>0) uri= uri.Substring(0,ndx);

   string ips="";

   // Get a list of IP addresses for the URI

   // Dns contacts the Internet Domain Name System

   IPHostEntry IPHost = Dns.GetHostByName(uri);

   foreach(IPAddress addr in IPHost.AddressList)

   ips+= addr.ToString()+"\r\n";

   tbIP.Text= ips;

}

You may encounter Web pages that check the UserAgent property of the request object and do not allow their pages to be downloaded unless they can identify the browser. You can assign a legitimate browser value to overcome this.