Web Scraping Free Cash Flow from Marketwatch.com

richard1000

#1

3/3/2017 7:42 PM

Web scraping request. I wish to scrape the following web site and scrape the "Free Cash Flow" (the last line) for the latest quarter.

http://www.marketwatch.com/investing/stock/aapl/financials/cash-flow/quarter#

YCharts data has "Free Cash Flow" but has a significant delay.

Eugene

#2

3/3/2017 7:47 PM

I got it done earlier today after I had noticed your previous post in another thread. After pasting the code, check System.Xml in Strategy's References dialog:

CODE:
Please log in to see this code.

Note how it caches data in Wealth-Lab's global memory to speed up execution by avoiding repeated web requests.

Size:

Color:

I got it done earlier today after I had noticed [link=https://www.wealth-lab.com/Forum/Posts/Estimate-EPS-and-revenue-data-for-latest-quarter-33870/Page/1#198555]your previous post[/link] in another thread. After pasting the code, check [b]System.Xml[/b] in Strategy's References dialog:

[code]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;
using System.Net;
using System.Globalization;
using HtmlAgilityPack;
// In References dialog, check System.Xml

namespace WealthLab.Strategies
{
	public class MyStrategy : WealthScript
	{
		public double GetFreeCashFlow(string symbol)
		{
			double fcf = -1;
         
			string URI = string.Format("http://www.marketwatch.com/investing/stock/{0}/financials/cash-flow/quarter", symbol);
			string xpath = "//tr[@class='mainRow'][8]/td[@class='valueCell'][5]";

using (WebClient client = new WebClient())
			{
				client.Headers.Add("referer", "http://www.marketwatch.com/");
				string result = string.Empty;
				try
				{
					result = client.DownloadString(URI);

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
					doc.OptionFixNestedTags = true;
					doc.LoadHtml(result);
					CultureInfo c = CultureInfo.InvariantCulture;

if (doc.DocumentNode != null)
					{
						HtmlNode n = doc.DocumentNode.SelectSingleNode(xpath);
						if (n != null)
						{
							if (n != null)
							{
								char[] charsToTrim = { 'B', 'M', 'K', ' ', '\''};
								if( double.TryParse(n.InnerText.Trim(charsToTrim), NumberStyles.Any, c, out fcf) )
								{
									char last = n.InnerText[n.InnerText.Length - 1];
									fcf = last == 'B' ? fcf*1000000000 :
										last == 'M' ? fcf*1000000 : last == 'K' ? fcf*1000 : Double.NaN;
									return fcf;
								}                        
							}
						}
					}
				}
				catch
				{
				}
			}

return fcf;
		}
      
		protected override void Execute()
		{			
			var fcf = GetGlobal("FCF_"+Bars.Symbol);
			if( fcf == null )
			{
				fcf = (object)GetFreeCashFlow(Bars.Symbol);
				SetGlobal("FCF_"+Bars.Symbol, fcf);
			}

PrintDebug((double)fcf);
		}
	}
}[/code]

Note how it caches data in Wealth-Lab's global memory to speed up execution by avoiding repeated web requests.

richard1000

#3

3/5/2017 9:10 PM

Thanks. I guess previous code from Yahoo scraping did the trick.

I may want to pick another data from this table in the future so can you explain about how this part? (I don't know html coding or about HtmlAgilityPack.)

CODE:
Please log in to see this code.

When I opened the html source code, 'mainRow' showed up in 20 different rows and I can't find out why 8th row was used. If there was a reference to "Financing Activities" table, then 8th row would be correct but I don't see it in the code.

Eugene

#4

3/5/2017 9:23 PM

Sorry but teaching XPath and web developer tools goes beyond the quick help I intended to provide.

richard1000

#5

3/5/2017 10:06 PM

Ok. Can you link some reference so I may teach myself?

Eugene

#6

3/6/2017 6:07 AM

If I had references I'd give them but I learn these things (XPath, DOM, HTML, JSON, libraries for scraping...) here and there. Pointers are all around in Google search.

richard1000

#7

3/6/2017 10:09 AM

1. I was testing the above code and the symbols CVX and GE returns a value of 0 since FCF is negative. Marketwatch.com formats the negative value in parentheses rather than as a negative sign.

2. For some strange reason, symbols AMZN and VZ returns a value of -1 even though marketwatch.com shows them as positive and negative FCF respectively.

Eugene

#8

3/6/2017 10:44 AM

EDITED 12/04/2020 Fixed code broken due to changed HTML layout

Good catch, let's fix the parsing routine. Firstly I expected NumberStyles.AllowParentheses to handle numbers like "(460M)" but for some reason it doesn't. Probably this number format isn't built-in. Therefore the fix below is not elegant but works for me.

To clear global memory from -1s and 0s from the previous runs, uncomment ClearGlobals and do an Execute once, then comment ClearGlobals out and compile. Or copy/paste and restart WLP.

CODE:
Please log in to see this code.

P.S. Perhaps an easier approach would be to replace "(" with a "-" and trim ")" but this occurred to me only later.

Size:

Color:

[color=#ff00dd][b]EDITED 12/04/2020 Fixed code broken due to changed HTML layout[/b][/color]

Good catch, let's fix the parsing routine. Firstly I expected [i]NumberStyles.AllowParentheses[/i] to handle numbers like "(460M)" but for some reason it doesn't. Probably this number format isn't built-in. Therefore the fix below is not elegant but works for me.

To clear global memory from -1s and 0s from the previous runs, uncomment ClearGlobals and do an Execute once, then comment ClearGlobals out and compile. Or copy/paste and restart WLP.

[code]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using WealthLab.Indicators;
using System.Net;
using System.Globalization;
using HtmlAgilityPack;
// In References dialog, check System.Xml
using System.Text.RegularExpressions;

namespace WealthLab.Strategies
{
	public class MyStrategy : WealthScript
	{
		public double GetFreeCashFlow(string symbol)
		{
			double fcf = -1;
			string URI = string.Format("http://www.marketwatch.com/investing/stock/{0}/financials/cash-flow/quarter", symbol);
			string xpath = "(//table[contains(@class,'table')]//tr[contains(@class,'table__row')]/td[@class='overflow__cell']/div[@class='cell__content']/span)[280]";

using (WebClient client = new WebClient())
			{
				client.Headers.Add("referer", "http://www.marketwatch.com/");
				string result = string.Empty;
				try
				{
					result = client.DownloadString(URI);

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
					doc.OptionFixNestedTags = true;
					doc.LoadHtml(result);
					CultureInfo c = new CultureInfo("en-US");

if (doc.DocumentNode != null)
					{
						HtmlNode n = doc.DocumentNode.SelectSingleNode(xpath);
						if (n != null)
						{
							char[] charsToTrim = { 'B', 'M', 'K', ' ', '\''};
							bool positive = n.InnerText.Contains("(") | n.InnerText.Contains(")");
							int i = !positive ? 1 : 2;
							char last = n.InnerText[n.InnerText.Length - i];
							var InnerText = Regex.Replace(n.InnerText, "[BMK ]", "");

fcf = (double.Parse(InnerText.Trim(charsToTrim), 
								NumberStyles.AllowParentheses | NumberStyles.AllowLeadingWhite | NumberStyles.AllowTrailingWhite | 
								NumberStyles.AllowThousands | NumberStyles.AllowDecimalPoint | NumberStyles.AllowLeadingSign));
							fcf = last == 'B' ? fcf*1000000000 :
								last == 'M' ? fcf*1000000 : last == 'K' ? fcf*1000 : Double.NaN;
							return fcf;
						}
					}
				}
				catch
				{
				}
			}

return fcf;
		}
      
		protected override void Execute()
		{   
			//ClearGlobals();
			var fcf = GetGlobal("FCF_"+Bars.Symbol);
			if( fcf == null )
			{
				fcf = (object)GetFreeCashFlow(Bars.Symbol);
				SetGlobal("FCF_"+Bars.Symbol, fcf);
			}

PrintDebug((double)fcf);
		}
	}
}[/code]

P.S. Perhaps an easier approach would be to replace "(" with a "-" and trim ")" but this occurred to me only later.