- ago
I'm simply trying to scrape a webpage with HttpClient. Please tell me why this simple code doesn't work?

I would prefer to do this with a "static" httpClient instance, but I would also like to pass parameters back from the scraping task. If this requires a non-static implementation (Does it?), then so be it.

CODE:
using System; using System.Net.Http; using System.Threading.Tasks; using WealthLab.Backtest; using WealthLab.Core; namespace WealthScript3 { public class MyStrategy : UserStrategyBase {       static readonly HttpClient httpClient = new HttpClient();       private static async void DownloadPage(UserStrategyBase usb, string symbol)       //private static async Task<string> DownloadPage(UserStrategyBase usb, string symbol)       {          string responseString = null;          try          {             HttpResponseMessage response = new HttpResponseMessage();             response = await httpClient.GetAsync("<a href="https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=" target="_blank">https://digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=</a>" + symbol);             //response.EnsureSuccessStatusCode();             responseString = await response.Content.ReadAsStringAsync();          }          catch (Exception ex)          {             usb.WriteToDebugLog(ex.Message.ToString());          }          usb.WriteToDebugLog(responseString);          usb.WriteToDebugLog("hello world"); //doesn't work          //return responseString;       }        public override void Initialize(BarHistory bars) {          //bool isMutualFund;     //Task.Run(new Action(DownloadPage));     Task.Run(() => DownloadPage(this as UserStrategyBase, bars.Symbol));       }       public override void Execute(BarHistory bars, int idx) { } } }
0
256
Solved
6 Replies

Reply

Bookmark

Sort
Glitch8
 ( 12.08% )
- ago
#1
You’re calling the download method in a new Task, and then immediately exiting the method. So the Task isn’t even completed before the main code line hits your WriteToDebug method.
0
Best Answer
- ago
#2
CODE:
using System; using System.Net.Http; using System.Threading.Tasks; using WealthLab.Backtest; using WealthLab.Core; namespace WealthScript3 {    public class MyStrategy : UserStrategyBase    {       static readonly HttpClient httpClient = new HttpClient();       private string DownloadPage(UserStrategyBase usb, string symbol)       {          string uri = "https://" + $"digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=" + symbol;          string responseString = null;          try          {             HttpResponseMessage response = new HttpResponseMessage();             response = httpClient.GetAsync(uri).Result;             responseString = response.Content.ReadAsStringAsync().Result;          }          catch (Exception ex)          {             usb.WriteToDebugLog(ex.Message.ToString());          }          return responseString;       }       public override void Initialize(BarHistory bars)       {          WriteToDebugLog( DownloadPage(this as UserStrategyBase, bars.Symbol) );       }       public override void Execute(BarHistory bars, int idx) { }    } }
0
- ago
#3
Except for a small error in the URL, the code in Reply# 2 works. Interesting. I thought one was required to spin up a task to use HttpClient, but I guess I am wrong. Thanks a bunch for simplifying my solution. It's funny but all the examples of calling HttpClient employ a task somehow.

I think the error in the URL (Reply# 2) is created by the forum website and not the poster. For some reason the website is introducing additional code in the posted solution. Fixed as discussed in Reply# 6.
0
- ago
#4
The forum automatically recognizes URLs and wraps them into "a href", including CODE blocks.
0
- ago
#5
The forum website shouldn't adulterate CODE blocks under any circumstance. It simply breaks the code.
0
- ago
#6
It's too troublesome to fix that. Here's a workaround that would be way easier (amended my code above):
CODE:
string uri = "https://" + $"digital.fidelity.com/prgw/digital/research/quote/dashboard/summary?symbol=" + symbol;
1

Reply

Bookmark

Sort