Extracting Shares Related Info for Proshare ETFs

Harapa

#1

8/30/2015 8:25 AM

Having seen the code for downloading Symbols http://www.wealth-lab.com/Forum/Posts/Stocks-in-Russell-1000-and-Russell-3000-35500, I wonder if one can write similar code to extract shares outstanding & AUM from this link and best if such a code can use symbols from a data set, and save the data as .WL data file.

https://accounts.profunds.com/etfdata/ByFund/SSO-historical_nav.csv

In my practice, I append AUM as OHLC and shrs as volume in one file.

I am sure many others will be interested in it.

TIA

Cone

#2

8/31/2015 2:13 AM

Anything is possible, but it's unnecessary work for zero gain.

The file is perfect as csv for the ASCII provider, where you can specify the AUM field as Volume. In the Data Manager > ASCII Data tab, turn on "Cache ASCII Data" and the file will automatically be converted to WL format in cache whenever the csv file is updated.

Eugene

#3

8/31/2015 2:15 AM

Here you go. This code downloads the CSV and parses it on-the-fly. I leave saving to .WL file to you as coding this is explained in the QuickRef and creating a DataSet in the Wiki FAQ > Data, respectively. Or as Cone says, it's pretty pointless. You might also want to consider this technique: Access Data from the Internet on-the-fly.

Click "References..." in the Strategy Editor, then select "System.Core".

CODE:
Please log in to see this code.

Size:

Color:

Here you go. This code downloads the CSV and parses it on-the-fly. I leave saving to .WL file to you as coding this is explained in the QuickRef and creating a DataSet in the Wiki FAQ > [i]Data[/i], respectively. Or as Cone says, it's pretty pointless. You might also want to consider this technique: [link=http://www2.wealth-lab.com/WL5WIKI/kbAccessInternet.ashx]Access Data from the Internet on-the-fly[/link].

[color=#ff0000][b]Click "References..." in the Strategy Editor, then select "System.Core".[/b][/color]

[code]using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using WealthLab;
using System.Net;
using System.IO;
using System.Linq;
using System.Globalization;

namespace WealthLab.Strategies
{
	public class MyStrategy : WealthScript
	{
		private static IFormatProvider _frm = CultureInfo.InvariantCulture;// new CultureInfo("en-US");
		
		string getUrl(string symbol)
		{
			return string.Format(@"https://accounts.profunds.com/etfdata/ByFund/{0}-historical_nav.csv", symbol );
		}			
		
		string MakeRequest(string url)
		{
			string result = null;

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
			using (HttpWebResponse resp = (HttpWebResponse)request.GetResponse())
			{
				Stream stream = resp.GetResponseStream();
				StreamReader reader = new StreamReader(stream);
				result = reader.ReadToEnd();
				resp.Close();
			}

return result;
		}
      
		protected override void Execute()
		{
			// Skips the first 1 lines containing headers
			int skipFirstLines = 1;
			string iwb = MakeRequest(getUrl("SSO"));
			var lines = iwb.Split(new [] { '\r' });
			try 
			{
				var data = lines
					.Skip(skipFirstLines)
					.Where(item => !String.IsNullOrWhiteSpace(item))
					.Select(x => new {
						Date = DateTime.Parse( (x.Split(',').First()), _frm, DateTimeStyles.None ),
						SharesOutstanding = int.Parse(x.Split(',')[7]) * 1000, 
						AssetsUnderManagement = double.Parse(x.Split(',')[8])
						});
				
				PrintDebug("Bar: \t" + "Date:\t" + "Shares Outstanding:\t" + "Assets Under Management");
				foreach (var bar in data)
				{
					DateTime dt;
					PrintDebug(
						Bars.ConvertDateToBar(bar.Date,false) + "\t" +
						bar.Date.ToString("yyyy/MM/dd") + "\t" + 
						bar.SharesOutstanding + "\t" + 
						bar.AssetsUnderManagement);
				}
			}
			catch (Exception ex) {PrintDebug(ex.Message);}
		}
	}
}[/code]

Harapa

#4

8/31/2015 9:35 AM

Thank you Eugene.
I had looked at the link but was not able to see where I specify which header to import. Between this and WIKI example I will learn that part.

Cone, I will work on your suggestion, as I can download/save many data files in a zippo using a macro in Excel.

Harapa

#5

8/31/2015 3:49 PM

QUOTE:
The file is perfect as csv for the ASCII provider,

So I downloaded data for over hundred symbols (using my Excel Macro) and saved these as .csv When I import these via ASCII provider all data files are truncated after 1/28/2015 (this truncation is visible in the preview window of ASCI Data provider as well as on a Chart in WL). To share with you, I created one of the files as .txt which when imported also truncates after 1/28/2015. Looking at many files in note pad (where format errors are easier to spot) I couldn't see anything obvious.
Please try the enclosed file to replicate the issue at your end and tell me if you see the same and what can be causing it.
Thanks for you help.

SSO.txt

Eugene

#6

8/31/2015 4:03 PM

Obviously your settings are wrong i.e. you configured the ASCII provider incorrectly.

Harapa

#7

8/31/2015 4:05 PM

QUOTE:
Obviously your settings are wrong

Clarify, can you import data till 8/28/2015 using the above file...

Eugene

#8

8/31/2015 4:11 PM

For the discussion to stay productive, please provide screenshots of your ASCII DataSet configuration and Preview window. Use its "Copy format from a DataSet" feature to quickly rerpoduce the format used, or attach the DataSet's XML file.

Harapa

#9

8/31/2015 5:18 PM

QUOTE:
"For the discussion to stay productive, please provide screenshots of your ASCII DataSet configuration and Preview window. "

Attached

QUOTE:
"Use its "Copy format from a DataSet" feature to quickly rerpoduce the format used,"

Can't figure out this function
or

QUOTE:
"attach the DataSet's XML file."

.pdf of file is attached as Forum doesn't give an option to attach .XML

On a side note: A data file generated using your code also produces the same error ( off course the "field order" is chosen to fit the layout of that file)

ADCII-previw.png

ASCI import.png

PS_New_XML.pdf

Eugene

#10

8/31/2015 5:33 PM

The Date format is incorrect, "mm" stands for minutes. Should be "MM" for months.

Both that and "what you couldn't figure" are referenced in the User Guide: Data > Data Manager > Create New DataSet > Data Providers > ASCII > Format Options.

P.S. XML could be zipped to attach.

Harapa

#11

8/31/2015 5:41 PM

That fixed it. It is the last thing you will think of when 90% data reaches the destination correctly. Thanks for your help.

Would you mind if I were to post VBA code and instructions on how to download data for all instruments using Excel?"

Eugene

#12

8/31/2015 6:40 PM

Thanks but we're not an Excel board, and my code already does just that (you just need to add the file save capability which is trivial and exemplified: Wiki > Exporting to ASCII comma-separated files). Everybody can use their own download tool, be it WLP, wget, curl, some download manager, or even Excel.