problem : using htmlagilitypack with c#

Topics: Developer Forum, Project Management Forum, User Forum
Mar 1, 2012 at 3:20 PM

problem : using htmlagilitypack with c#


problem : using htmlagilitypack with c#

I can not use the [ htmlagilitypack with c# ] to enter page protected with username and password.

if i use that code directly i have to use username and password to access this page , how i can use it with HtmlWeb ????

var getInfo = new HtmlWeb();
 var info= getInfo.Load("http://www.mysite.com/page.php");
 var userName = info.DocumentNode.SelectNodes("//div[@class='username']");
 if (userName != null) {

String uname= gotoAdventure[0].InnerText; MessageBox.Show(uname);

}

The previous method does not work



i use that code to request that page and pass (username and password) and then save that
response to temp.tmp file and then using HtmlWeb to load that file and getting the information , and that method is very bad and ineffective

sendGetRequest("http://www.mysite.com/page.php?username=ali&password=123123");
var getInfo = new HtmlWeb();
var info= getInfo.Load("e:/temp.tmp");


The previous method does not work



i use that code to request that page and pass (username and password) and then save that
response to temp.tmp file and then using HtmlWeb to load that file and getting the information , and that method is very bad and ineffective

sendGetRequest("http://www.mysite.com/page.php?username=ali&password=123123"

);

var getInfo = new HtmlWeb();
var info= getInfo.Load("e:/temp.tmp");
 

 

 public static void sendGetRequest(string link) {

 Random r = new Random(((int)DateTime.Now.Ticks));
 string filename = CreateRandomFileName(5, r);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.Method = "GET";
request.KeepAlive = true;
request.CookieContainer = cookies;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
 Stream receiveStream = response.GetResponseStream();
 StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
TextWriter tW = new StreamWriter("e:/temp.tmp");
 tW.Write(readStream.ReadToEnd());
tW.Close();

}

how i can slove the problem , and using htmlagilitypack without save any file ???
 
Mar 7, 2012 at 11:03 AM
Edited Mar 7, 2012 at 12:07 PM

It seems that HttpWebRequest doesn't store the cookies. I suggest that you use a webbrowser control (with the navigate function to post your credentials) instead so that you get the session cookie properly  and then get the resulting HtmlDocument.

Also, beware that WebBrowser is Async so you'll need to "wait" for the browser document to come back before passing it to the HtmlAgility.Load() method or you'll get an empty document.

Mar 8, 2012 at 1:03 AM
nexus1703 wrote:

It seems that HttpWebRequest doesn't store the cookies. I suggest that you use a webbrowser control (with the navigate function to post your credentials) instead so that you get the session cookie properly  and then get the resulting HtmlDocument.

Also, beware that WebBrowser is Async so you'll need to "wait" for the browser document to come back before passing it to the HtmlAgility.Load() method or you'll get an empty document.

HttpWebRequest has a cookie jar. It's just up to the developer to supply the cookies, for example by copying them from an HttpWebResponse obtained from a previous request.

HAP's HtmlWeb object has a UseCookies property. You could try setting that to true, posting to a login page using the PreRequest event to supply the form data, then going after the actual page. If that doesn't work, use the PostResponse event on the first request to extract the session cookie and the PreRequest event on the second to reinject it.

If the server's using HTTP authentication rather than forms & cookies, it's much easier: just use one of the overloads of HtmlWeb.Load() that allow you to supply a username and password: HtmlWeb.Load(string url, string method, WebProxy proxy, NetworkCredential credential) or HtmlWeb.Load(string url, string proxyHost, int proxyPort, string userId, string password)

Mar 10, 2012 at 6:04 PM
Edited Mar 10, 2012 at 6:04 PM

 

...

CookieContainer container = new CookieContainer();
...

//Previous query
response = request.GetResponse() as HttpWebResponse;
container.Add(response.Cookies);

//Next query with cookies
request = HttpWebRequest.Create(new Uri(url)) as HttpWebRequest; � � � � � � ��
request.CookieContainer = container;
response = request.GetResponse() as HttpWebResponse;
container.Add(response.Cookies);
...

 

...and so on...