c# 2.0 - WebClient class throwing a WebException

[Keith R. Pinster]

I am instantiating a WebClient to scrap a web site. Here's my code:

string myURL = "http://www.bloomberg.com/markets/rates/index.html";
WebClient client = new WebClient();

string strPageData = client.DownloadString(myURL );
or
string strPageData = Encoding.ASCII.GetString(client.DownloadData(myURL));
or
HttpWebRequest webreq = (HttpWebRequest)WebRequest.Create(myURL );
webreq.MaximumAutomaticRedirections = 300;

HttpWebResponse webresp = (HttpWebResponse)webreq.GetResponse();

The problem is that when I hit the site, it throws this error:
"Too many automatic redirections were attempted."

Has anyone ever encountered something like this? I can't believe that
the site is redirecting me over 300 times. When you go to the site,
it does some things, but it loads pretty quickly, so there can't be
that many redirects. It was working up until yesterday and then all
of a sudden I get this error. Any thoughts?

[Keith R. Pinster]

After searching on the web for a couple of hours, I finally came
across this. I set the agent and now it works. - KRP

This error can occur when an application performs a web request to a
web app that's trying to interogate the Request.UserAgent - but the
requesting app has not set the user agent in the request.

To resolve either set the user agent before sending the request eg:

HttpWebRequest req = (HttpWebRequest) HttpWebRequest.Create(theUrl);
req.UserAgent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";

Alternatively, and preferrably (if you have access to it), fix the
server side web application to check Request.UserAgent for null or
blank values before performing any operations on it.

Post by Keith R. Pinster
string myURL = "http://www.bloomberg.com/markets/rates/index.html";
WebClient client = new WebClient();
string strPageData = client.DownloadString(myURL );
or
string strPageData =

Encoding.ASCII.GetString(client.DownloadData(myURL));

Post by Keith R. Pinster
or
HttpWebRequest webreq = (HttpWebRequest)WebRequest.Create(myURL );
webreq.MaximumAutomaticRedirections = 300;
HttpWebResponse webresp = (HttpWebResponse)webreq.GetResponse();
"Too many automatic redirections were attempted."
Has anyone ever encountered something like this? I can't believe that
the site is redirecting me over 300 times. When you go to the site,
it does some things, but it loads pretty quickly, so there can't be
that many redirects. It was working up until yesterday and then all
of a sudden I get this error. Any thoughts?