Historical Data Provider API

This document details the API for building Historical Data Provider extensions for WealthLab 8. A Historical Data Provider allows WealthLab 8 to consume historical price/volume data from a specific source. Examples of such sources are:

• remote web services
• scraped web pages
• files in a specific format on the local file system.

Build Environment

You can create a Historical Data Provider in a .NET development tool such as Visual Studio 2022. Create a class library project that targets .NET8, then reference the WealthLab.Core library DLL that you'll find in the WL8 installation folder.

Note: If you are using Visual Studio 2022, it will need to be updated to at least version 17.8.6 to use .NET8.

Your Historical Data Provider will be a class in this library that descends from DataProviderBase, which is defined in the WealthLab.Core library, in the WealthLab.Data namespace. After you implement and build your library, simply copy the resulting assembly DLL into the WL8 installation folder. The next time WL8 starts up, it will discover your Historical Data Provider, making it available in appropriate locations of the WL8 user interface.

Accessing the Host (WL8) Environment

The IHost interface allows your extension to access information about the user's WealthLab environment. For example, the location of the user data folder, or obtaining a list of DataSets defined by the user. At any point in your extension's code base, you can access an instance of the IHost interface using the singleton class WLHost and its Instance property. Example:

//get user data folder
string folder = WLHost.Instance.DataFolder;

Configuration of a Historical Data Provider

DataProviderBase ultimately descends from the base class Configurable, which provides a way to allow the user to configure the Historical Data Provider. Consult the Configurable class reference for details.

By default, DataProviderBase assigns the enum value ParameterList to its ConfigurableType property, so the Historical Data Provider will use a ParameterList containing Parameter instances for configuration. You'll define these Parameters in the GenerateParameters method.

You can create Parameter instances to represent things like user names, passwords, or API keys. When the user configures your Provider, WL8 will show a dialog box allowing them to enter appropriate, typed, values for each Parameter instance.

You are free to change the ConfigurableType to the alternate option of VanillaString, in which case you'll need to work directly with the Configuration string rather than the Parameters instances. In this case, you'll need to create your own Editor, see the topic on Providing User Interfaces for WealthLab 8 Components for details.

Important Descriptive Properties

The Configurable class provides descriptive properties that control how your Historical Data Provider appears in WL8. The most important ones to override are Name, GlyphResource, Description, and URL.

Complete WL8 Data Subsystem Class Hierarchy

WL8's data subsystem has a hierarchy of several classes that derive from the Configurable base class. The DataProviderBase class follows a path that takes it from Configurable, to ProviderBase, to BulkUpdatableProviderBase, and finally to DataProviderBase.

Historical Data Provider Properties

Below are properties you can override to tell WL8 what features your Historical Data Provider supports.

public virtual bool SupportsUpdate

Returns true by default, indicating that the Provider supports data updates from the source. Return false if your Provider will not be updatable from within WL8. For example, it may be sourced from files on the local computer that are updated from some outside mechanism.

public virtual bool IsGeneralPurposeProvider

Returns true by default, indicating it can service ad-hoc symbol requests from Charts or Strategy windows. Return false to indicate your Provider cannot handle ad-hoc requests, but instead can provide data for only a fixed set of symbols. An example of a Provider that returns true here is WealthData. And an example of one that returns false is the MetaStock provider.

public virtual bool SupportsPartialBar

Returns the value of IsGeneralPurposeProvider by default. Indicates whether the Provider can return a "partial bar" of data. A partial bar consists of the trading session's open, high, low, volume and close up to the current point in time. If your Provider returns true here, you should implement the GetPartialBarInternal method described below.

public virtual bool ShowInNewDataSetList

Returns true by default, causing Providers that also returned true for IsGeneralPurposeProvider to appear in the list of available Providers in the New DataSet Wizard. Although the cases would be rare, you might want to prevent your Provider from appearing in the New DataSet Wizard list if has a very specific role. For example, if it provides data for DataSetProvider DataSets only and not user-created ones.

public virtual bool NeedsNewInstancesForEachDataSet

Returns false by default. Return true if WL8 should create a new instance of your Provider for each DataSet that is created based on it. You'd do this in cases where each DataSet requires special configuration in the New DataSet Wizard.

For example, the ASCII Data Provider requires the user to define the specific ASCII format, and the location of the ASCII files to load. So WL8 creates a new instance of the ASCII Data Provider for each DataSet created. WL8 places these specially generated instances in the general-purpose Data Provider list in the Data Manager so they can be selected as serviceable Data Providers by the user.

For example, imagine creating three different ASCII DataSets, each pointing to a different folder with different configurations. The end result would be three new ASCII Data Providers in the WL8 Data Manager, each assigned to its respective DataSet.

public virtual bool InjectNewProviderInstances

If a Provider returned true for NeedsNewInstancesForEachDataSet, then the instances will typically appear in the Data Manager's list of Historical Data Providers, as described above. Return false here to prevent your Provider's instances from being added to this list.

As an example, one Provider that returns false is the Stock Scanner Dynamic DataSet Provider. Its purpose is to return a list of symbols based on some filter criteria configured when the user creates a DataSet. The resulting DataSet appears in the DataSets list, but it doesn't make sense to show the corresponding generated Data Providers in the Data Manager/Providers list.

public string DSString

If NeedsNewInstancesForEachDataSet is true, this property will contain the DSString string that describes the DataSet that this Historical Data Provider instance was based on. For example, for the ASCII Provider, the DSString describes the ASCII configuration for the DataSet.

public DataSet DataSet

If NeedsNewInstancesForEachDataSet is true, this property will contain the DataSet instance that was used when the Provider instance was created.

public virtual bool ReadOnlySymbols

Returns false by default. Return true if the user should not be allowed to modify the symbols contained in a DataSet linked to your Provider. This would be the case, for example, if the symbols were based on a set of files in a specific folder on the local computer, as is the case with the ASCII and MetaStock Providers.

public virtual List<string> Symbols

If your Provider uses ReadOnlySymbols, and NeedsNewInstancesForEachDataSet is true, this implies that the symbols themselves may change outside of the WL8 environment. For example, perhaps the symbols are based on the files in a particular folder on the local computer. In this case, override the Symbols property to return the current symbols.

public virtual bool AllowConfigurationOfExistingDataSets

Returns true by default. Return false to disallow the user from re-configuring a DataSet linked to your Provider after the DataSet is created. Some Providers, like Finam, create DataSets with symbols from classification lists. Once the DataSet is created, the symbols can't be changed by the user because the Provider uses the classification to determine what symbols should be populated.

public virtual bool LookForIntradayWhenNoDaily 

Returns false by default. If true, causes WL8 to attempt to load and compress intraday data from your Provider when a daily or higher frequency request fails. Typically, you'd return true here only if your Provider provides intraday data only, and not native daily data. The ASCII and MetaStock Providers return true here, so that if you attempt to view Daily data from an intraday ASCII/MetaStock source, the Provider will compress the intraday data instead of relaying to request to another Provider that returns daily data.

public virtual bool Compress30MinuteBarsFor60Minute

WL8 by default requests 30-minute historical data and compresses it when 60-minute data is requested. This allows a partial, 30-minute, bar to be generated for the last 60-minute bar of the US market. If your provider does not require such logic, override this property to return false.

public virtual void SymbolsChanged(DataSet ds)

WL8 calls this method when the user changes the symbols of the DataSet in the DataSet panel or Data Manager.

Initialization

public virtual void Initialize()

If your Provider requires any kind of initialization before use, you can override this method and implement it here.

Returning Historical Data

This is the primary method of the Historical Data Provider, it returns historical data for a specified symbol, scale, and date range. Your Provider should override this method and return the historical data in the form of a BarHistory object.

protected abstract BarHistory GetHistoryInternal(string symbol, HistoryScale scale, DateTime startDate, DateTime endDate, int maxBars);

Interpret the parameters as follows:

• If maxBars is non-zero, return this many bars of historical data, up to the current date/time.
• If endDate is DateTime.MaxValue, return data up to the current date/time.
• if startDate is DateTime.MinValue, return data going back as far as possible into the past.
• Otherwise, use the date range specified in startDate and endDate.

Your implementation of GetHistoryInternal should do the following:

1. Check to see if the scale passed in the scale parameter (an instance of the HistoryScale class) is supported, and if not return null.
2. Determine if the symbol is one that your HDP supports, and if not return null.
3. Create a BarHistory instance, using the constructor that accepts the symbol and scale parameters.
4. Assign a value to the BarHistory's SecurityName property, if a security name can be obtained.
5. Obtain the historical data from the source, for as much of the requested range as possible.
6. Use the BarHistory Add method to add the date, open, high, low, close and volume to the instance.
7. Return the BarHistory instance.

Note: Your Provider should not return incomplete, partial daily or intraday bars in the result! If required, ignore these bars, or call RemovePartialBar, described below, before returning.

public BarData RemovePartialBar(BarHistory bh)

Call this helper method to remove a potentially incomplete, partial bar from the end of the BarHistory passed in bh. If a partial bar was removed, it will be returned as an instance of the BarData class. Prior to calling this method, ensure that the correct MarketDetails is assigned to the BarHistory's Market property.

public virtual string GetSecurityName(string symbol)

Ideally, you can set the BarHistory SecurityName property right in the call to GetHistory, but some providers do not return a security name along with the historical data. If your HDP is sourced from such a provider, but the provider has an alternate means of obtaining the security name, override this method to return the name.

public virtual bool SupportsScale(HistoryScale scale)

Override this method and return whether or not your Provider supports obtaining data from the source in the specified scale (a HistoryScale instance). Internally, WL8 can compress data. For example, if the user requests weekly data, WL8 can generate this from daily data. But do not consider compression here. Only return true if the specified scale can be returned natively from the source. Doing so correctly will cut down on the number of calls WL8 makes when requesting history, and improves performance.

protected virtual BarData GetPartialBarInternal(string symbol, HistoryScale scale)

Your GetHistoryInternal call above should return only complete bars of data, either intraday or daily+ scales. If WL8 requires a partial bar of data for a market that is still open, it will call GetPartialBarInternal. Here you can obtain a partial bar of data from the source, if possible, in the form of a BarData instance. If partial bar data is not available, return null.

For example, assume the Provider returns data for the US stock market, and it is currently a trading day, and the time is 2:00 PM EST. Since the market is still open for another two hours, there would be a partial daily bar of data for today, covering 9:30 AM EST until now. This partial bar should not be included in any GetHistoryInternal requests but should rather be returned in the GetPartialBarInternal request.

Returning Symbol MetaData

public virtual MarketDetails GetMarketForSymbol(string symbol)

Override this method to return an instance of the MarketDetails class that represents the market that this symbol trades in. You can query the MarketManager for MarketDetails instances, or even create a new MarketDetails and add it to the MarketManager in your Provider's Initialize method. If you do not implement this, the method defaults to returning the US stock market.

public virtual string GetSecurityName(string symbol)

Override this method to return the security name for the specified symbol. For example, "Microsoft Corp" for symbol MSFT.

public virtual SymbolInfo GetSymbolInfoForSymbol(string symbol, double price = 0.0)

Override this method to return a SymbolInfo instance with properties set to describe the specified symbol. The SymbolInfo class contains properties such as SecurityType (Stock, Crypto, etc) as well as decimal settings for instrument pricing and volume.

public virtual bool IsSymbolDelisted(string symbol)

Return true if your provider considers the specific symbol to be a delisted stock. The default implementation checks the symbol based on some rules for WealthData delisted symbols.

public virtual bool IsSymbolNonTradable(string symbol, bool excludeBroadMarketIndexes = true)

Return true if the specified symbol is considered to be non-tradable (an index or sentiment indicator for example.) If excludeBroadMarketIndexes is true, do not consider indices in the result.

Options Support

public virtual string GetOptionsSymbol(BarHistory underlierBars, OptionType optionType, double price, DateTime currentDate, int minDaysAhead = 0, bool useWeeklies = false, bool allowExpired = false, bool closestStrike = true, double multiplier = 100)

If your provider can return options data, override this method to return an option symbol per the specified parameters in the format required by the provider. If you have access to option chains, cache the chain in the background and select a valid symbol from it.

Interpret the parameters as follows:

public virtual DateTime GetSymbolExpiry(string optionSymbol)

Parse optionSymbol to return the expiration date as a DateTime.

public virtual double GetSymbolStrike(string optionSymbol)

Parse optionSymbol to return the strike as a double value.

public virtual OptionGreek GetGreeks(string optionSymbol)

Override this method to return an OptionGreek object. Assign only the OptionGreek fields that your provider supports.

public virtual OptionGreek GetGreeks(string optionSymbol, double impliedVolatility, double priceUnderlying)

Override this method if your provider can hypothetically calculate and return the Greeks based on user-specified IV and underlying price.

public virtual double CalculateIV(string optionSymbol, double priceOption, double priceUnderlying)

Override this method if your provider can calculate an implied volatility for an option given its price and the price of the underlying.

public virtual double CalculateOptionPrice(string optionSymbol, double impliedVolatility, double priceUnderlying)

Override this method if your provider can calculate an option price given the IV and the price of the underlying.

Returning Quotes

public virtual double GetQuote(string symbol)

Returns a current quote for the specified symbol. The default implementation checks to see if the market is currently open, and if so calls GetPartialBarInternal to get the latest price. If the market is closed, the method calls GetHistoryInternal to get the last daily closing price. You can override this method to customize or alter the logic.

Persistent Storage Options

public virtual bool UsesPersistentStorage

Returns false by default. Override this and return true to utilize the built-in persistent storage mechanism in your Provider. You'd typically do this if your Provider downloads data from a remote service, and you'd like to keep a copy of the data downloaded so far locally, so it does not have to request a full history each time.

public virtual BarHistory LoadFromStorage(string symbol, HistoryScale scale, DateTime? startDate = null, DateTime? endDate = null, int maxBars = 0)
public virtual void SaveToStorage(BarHistory bh)

By default, the persistent storage mechanism stores the historical data in a binary format in .QX files. You can override this default and implement your own storage scheme by overriding these methods.

protected virtual string ConstructSymbolFileName(string symbol, HistoryScale scale)

If you implement your own storage scheme as described above, you can call this method to obtain a file name for a symbol/scale combination. The file name will have a .QX extension, which you could then change to suit your purposes.

The method is also virtual, so you can override to change the default file naming that WL8 uses for your Provider.

public virtual BarHistory ReloadHistory(string symbol, HistoryScale scale)

WL8 calls this method when the user right clicks the chart and selects "Reload Chart Data". The method is already implemented, but you might want to augment the implementation by removing any other persistent data your Provider might save for the specified symbol and scale.

public virtual void DeleteLocalData()

WL8 calls this method when the user selects the option to "Delete all Local Data for this Provider" from the Data Manager. The method is already implemented, but you might want to augment the implementation by removing any Provider-specific persistent data that your Provider stores.

public virtual void ClearRequestLists()

WL8 calls this method when the user selects the option to "Clear Internal Tracking Info" from the Data Manager. The method is already implemented to clear certain internal lists that track which symbols were updated and at what time. You might want to augment the implementation in case your Provider manages any in-memory or file-based tracking data of its own.

Data Corrections

public virtual bool SupportsDataCorrections

If your Provider supports data corrections for previously downloaded data, return true here. WL8 will then call the method below when it requests updates for a previously saved history.

protected virtual ResponseWithCorrections GetHistoryInternalWithCorrections(string symbol, HistoryScale scale, DateTime startDate, DateTime endDate, int maxBars, DateTime lastRequestDate)

WL8 calls this method when it requests an update for a previously saved history, and your Provider returns true for the SupportsDataCorrections property. The method should emply the same logic as your GetHistoryInternal, but note that the return object is an instance of the ResponseWithCorrections class, instead of a BarHistory instance. Create a new instance of the ResponseWithCorrections class and assign values to its two properties:

• Bars - assign the result of the historical data request.
• Corrections - assign a new BarHistory instance that contains any corrected data points that occured since the DateTime that was passed in the lastRequestDate parameter.

protected virtual string CompanionEventProviderName

If your Provider has a corresponding Event Data Provider, you can override this property to return the Event Provider's Name property. WL8 can then check for dividends or splits during an update request, and reload the history when they occur, if the user has enabled this in their Data Preferences.

public virtual bool ReloadOnAnomaly(string symbol)

Override this method to determine if WL8 should apply data corrections based on a price gap greater than a certain percentage, or new dividends or splits being detected during the update period. Both of these conditions cause a complete reload of the symbol's historical data on a daily+ scale. The default return value is true.

Provider Update Option

 public override bool SupportsBulkUpdate

This property determines whether or not your Provider supports Provider-wide data updates from within the WL8 Data Manager. A "Provider Update" is intended to update all of the Provider's data at one time.

By default, the property returns the value of UsesPersistentStorage. If you use the built-in persistent storage mechanism, you can leave this as is and enjoy the benefit of the built-in Provider Update behavior. If, however, you do not use the built-in mechanism, but would still like to support Provider Update, override the methods below.

public override void PerformBulkUpdate(IBulkUpdateHost updateHost)

Implement the logic to update all of the data for your Provider here, communicating the status of the update by making callbacks to the updateHost instance of the IBulkUpdateHost interface that is passed as a parameter.

public override void CancelBulkUpdate()

WL8 calls this method when the user has canceled a Provider Update. Override this to cancel your Provider-specific update routine.

public virtual List<HistoryScale> ScalesWithData

If you implement your own persistent storage mechanism, and also support Provider Update, override this to return a list containing HistoryScale instances that represent all of the scales for which your HDP currently has persisted data.

public virtual List<string> GetSymbolsForScale(HistoryScale scale)

If you implement your own persistent storage mechanism, and also support Provider Update, override this to return the symbols that are persisted by your Provider for the specified scale.

 public override bool SupportsParallelRequests()

Override this optional property and return false if WL8 should restrict itself to single-threaded behavior and not make parallel data requests while performing a bulk update.

Parallel Processing Considerations

By default, WL8 will call your Provider's GetHistoryInternal method from multiple threads in a parallel processing mode. For this reason, you should generally avoid accessing class level variables in GetHistoryInternal. Rather, use variables that are defined in the method itself.

The SupportsParallelRequests property mentioned above prevents parallel requests during a Provider bulk update only. Since WL8 might leverage several Providers when delivering data for a DataSet, you should restrict parallel processing in your Provider if required.

You can use locking techniques to circumvent parallel processing. You would do this if, for some technical reason, your data source is not compatible with parallel requests. For example, the IQFeed and Interactive Broker Providers use sockets which send messages, making it more difficult to code these Providers in a parallel way.

The following pseudo-code demonstrates how locks can be used to make the parallel requests process sequentially.

protected override BarHistory GetHistoryInternal(string symbol, HistoryScale scale, DateTime startDate, DateTime endDate, int maxBars)
{
	//_requestLock is defined as a private variable of type object
	lock(_requestLock)
	{
		//normal Provider processing
	}
}

Tick Data Support

public virtual List<Tick> GetTicks(string symbol, DateTime startDate, DateTime endDate)

If your Provider supports tick data, override this method to return a List of Tick data instances for the specified symbol and date range.

public virtual bool UsesTickDataStore

Return true here if you want to use the built-in tick data persistent storage mechanism provided by WL8. If true, WL8 will store the downloaded tick data locally, and make subsequent GetTicks requests only when new updates are required.

Customizing Configuration and New DataSet UI

See the topic on Providing Editors for WealthLab 8 Components for information on how to provide Custom Settings Editor Panels, and New DataSet Wizard Pages for your Historical Data Provider.