DocumentNode.Descendants comes back with null values

Topics: Developer Forum, User Forum
Jun 12, 2013 at 11:01 PM
Edited Jun 12, 2013 at 11:28 PM
I am using HTML Agility Pack to scrape websites for my Windows Store app. The version for Windows 8 store apps does not support XPath nor does it have a selectNodes function, but it does support lambda expressions to look up the values needed.

After extensively searching the web the only solution I see (over and over again) is to use DocumentNode.Descendants() to get back a list of elements to search within. However, for me, DocumentNode.Descendants() always returns back null values. This happens whether or not I pass in a parameter to the function.

I am getting back a response though. I can see the html for the page inside of the HtmlDocument object.

Code Snippet:
    HttpContent loginContent = new FormUrlEncodedContent(new[]
    {
             new KeyValuePair<string, string>("name", username),
             new KeyValuePair<string, string>("password", password)
    }
    
    HttpResponseMessage response = await httpClient.GetAsync(httpClient.BaseAddress); //Gets the cookie
    
    response = await httpClient.PostAsync(loginPageAddress, loginContent); //Logs in
    HtmlDocument doc = new HtmlDocument();
    doc.Load(new StringReader(await response.Content.ReadAsStringAsync())); // Loads doc, can see html for the page in doc.text attribute now
                
    
    HtmlNode pointsNode = doc.DocumentNode.Descendants("div").Where(o => o.Attributes["class"].Value == "availableNumber").FirstOrDefault(); // fails because Descendants returns null values
From debugging in my immediate window:
    doc.DocumentNode.Descendants("div")
    {HtmlAgilityPack.HtmlNode.Descendants}
        name: null
        System.Collections.Generic.IEnumerator<HtmlAgilityPack.HtmlNode>.Current: null
        System.Collections.IEnumerator.Current: null
Any help solving this issue will be greatly appreciated! Thanks all.