Is ETF Bid / Ask data available in WL ?
Author: tuckandturn
Creation Date: 12/12/2011 1:40 PM
profile picture

tuckandturn

#1
My trading plan is designed to make a decision just before the close of day. If the most recent bid price is greater than some threshold then a transaction is to be generated. Question is, does Wealth Lab provide bid /ask data for ETFs/stocks? I seem to be unable to find much information on it.

My apologies if this has already been answered - if so then please just point me in the right direction.

profile picture

Eugene

#2
Wealth-Lab is designed to operate on bars, not with bid/ask data, and does so this way:

FAQ | Data and Data Providers > "Is it necessary to have access to intra-bar tick data to daytrade with Wealth-Lab?"
profile picture

tuckandturn

#3
Hi Eugene,

Thanks for your response, I take it then that bid /ask data is not available in WL?

My understanding of bars is that while they provide OHLC/V data down to the one minute interval, if there have been no trades for several minutes then the "last bar" will provide data that could be quite a distance from the current bid /ask spread. Thus prompting a decision on data from the "last bar" which could be significantly out of date. I'm I understanding this correctly?

Thanks
Doug
profile picture

Cone

#4
That's right. If there are no trades, there is no trading activity, which means you're just watching a bunch of market makers dance in Level II. I understand your proposition, but the bid/ask are not readily available to strategies. Even if they were it wouldn't make any difference in your scenario because the strategy won't execute anyway without at least one trade that created a new bar.

If you're a programmer capable of creating a brand new streaming feed, I could give you a suggestion how to proceed: re-map bid/ask to fields that are available to strategies and force the bar to close at whatever interval or time you specify.

So with some work, you can make it happen, but it would only be worthwhile if you already knew that your strategy actually made money. Have you tested it? If you can get ASCII data with bid/ask, then at least you can (and should) test your strategy first.
profile picture

tuckandturn

#5
Hi Cone,

You have restated the problem nicely.

The buy/sell decisions are to be implemented just before the close of day based on certain daily high/low thresholds.

If the last traded bar was several minutes away and the market maker(s) has created a market with a bid value that meets or exceeds the threshold, then I want to take the market But if I can't see "the market" then I can't make the appropriate decision.

I've only played with Wealth-Lab for a day or so, so I'm not clear what you mean by "re-map bid/ask to fields" Could you please elaborate on that for me?

Yes, I am a programmer and yes I have back-tested my strategy (in at least two different programming environments if that helps.)

Thanks for your help.

Doug





profile picture

Cone

#6
You just need a snapshot quote, and make it available to Wealth-Lab when the script runs. So, you just program a separate app that goes out and gets snapshots with bid/ask for all the symbols that you're trading. You store that info in one or more files for your script to read when it executes.

The problem is getting the script to execute in the last minute of the market day. The only "built in" way to do it (again, without the guarantee of a bar at 15:59) is to manually run the strategy on the DataSet after the snapshot process has completed.

Which data provider(s) do you use?
profile picture

tuckandturn

#7
Hi Cone,

If I understand what you're saying - I would need another program to get bid /ask data from some other source outside of WL for WL to read and to act upon.

Is that doable with Fidelity do you think?






profile picture

Cone

#8
No. Fidelity doesn't offer access to its data or trading APIs. Here's a hint for you:
http://download.finance.yahoo.com/d/quotes.csv?s=QQQ,DIA,SPY&f=sd1t1ohgl1vpba&e=.csv

Fields are: symbol, date,time,open,high,low,last,volume,previous close,bid,ask
profile picture

tuckandturn

#9
Interesting - I've been using Yahoo Finance data for quite some time, but never really went after the real time data.

This should add a new aspect to what I'm trying to accomplish.

Thanks!
profile picture

boreland

#10
Hi Cone & Eugene!

I know this thread is ancient but I was wondering if I could call upon you to find a creative way to pull in bid/ask data into my real time live trading code. The problem that I run into is some of my orders are submitted using a buy stop order. The entry value is determined algorithmically, but unfortunately many times the orders error out because the ask price is above the order stop price. I need a way to detect this and move my order stop value above the current ask. These orders are usually fired at the close of the 1st bar at the open of the trading day when the market is most frenetic.

From what I understand there is no bid/ask data at all in the Fidelity data feed. Pulling it from yahoo might be a possibility if its real time. This url for yahoo does still work: http://download.finance.yahoo.com/d/quotes.csv?s=QQQ,DIA,SPY&f=sd1t1ohgl1vpba&e=.csv Not sure about how quickly I can pull down the information. Perhaps 5sec before the close of the first 1min bar. Do you have a code snippet you could share for pulling down this information from yahoo so I might try to craft a working solution. I will of cause post back to the group the final working example.
profile picture

Eugene

#11
Hi Peter,

QUOTE:
Perhaps 5sec before the close of the first 1min bar.

Perhaps you intended to say 5sec after the close..? You might want to check out this FAQ: Is it necessary to have access to intra-bar tick data to daytrade with Wealth-Lab?

According to this source, to extract real-time bid and ask from Yahoo!, proposed URL should be more like this:

http://download.finance.yahoo.com/d/quotes.csv?s=QQQ&f=b2b3&e=.csv

CODE:
Please log in to see this code.


I leave experimenting with handling multiple symbols up to you.
profile picture

Cone

#12
I'm afraid Y! is going to be delayed.

Is your algorithm calculating a stop price higher than the close of the current bar?
It doesn't seem likely that the market Ask would be too far below the most recent close (but for sure it can happen). Maybe just checking the stop price against Close[bar] would be sufficient. If it's too close, just ExitAtMarket. Or, you could shave a nickel (or some percentage) off the price and place it...

CODE:
Please log in to see this code.
profile picture

Eugene

#13
QUOTE:
I'm afraid Y! is going to be delayed.

FWIW: according to Yahoo's data disclaimer page, data from BATS, NYSE, NASDAQ etc. is believed to be real-time.
profile picture

boreland

#14
Cone

Thkx. I want to check the ask price just before I set the buy stop order value. I guess I'll have to run a timer to capture this information from yahoo just prior to the close of the bar. It will be interesting to see if this runs under the strategy manager.

Price spreads at the open can be quite large, hence the reason for doing this and preventing fidelity from erroring out the order.

More like:

CODE:
Please log in to see this code.


A nickel on 2000 shares is a lot of slippage. We are scalping dims here:}

Unfortunately this is were the limitations of the platform really show themselves when there is no real time OnMarketData method.
profile picture

Cone

#15
I'd say more of a data feed (and broker) limitation than a platform limitation. You could use IQFeed to trade second bars, for example. Getting a quote every second should be fast enough? Also, the broker could convert WLP orders to market if the conditions were consistent for it.

You have to remember that a strategy isn't running continuously. You make decisions at the end of a complete bar, and whether that bar be 1 tick, 1 second, 30 seconds, 1 minute, 5 minutes, 30 minutes, or 1 day. the most recent price that just traded is Close[bar].

Previously you said, "These orders are usually fired at the close of the 1st bar at the open of the trading day."
But which is it? At the open? Or at the close first bar?
profile picture

boreland

#16
I agree, really a limitation of the data feed. You made some interested comments here. I'd like to explore this further if I may.

CODE:
Please log in to see this code.

What do you mean by this? Is there a way to force Fidelity to automatically convert a stop buy order to a market order if the ask is at or below the stop buy value at the time the order is placed?

OK, so my trade logic works on 1 min bars supplied by Fidelity. The orders are also placed through Fidelity of course. The decision to buy is made at the close of the 1st min bar at the open so 9.31 am. The orders are stop buy, but as you know if the ask price at the time the order is sent to Fidelity is at or below the buy stop price the order will be rejected by Fidelity and an error will be displayed in the SM. You can appreciate this is very frustrating as many good trades are missed.

What I was thinking of doing was running a timer triggered in some way to determine the ask price 5sec or so before the close of the 1 min bar, and use this information to adjust the value of the stop buy order. So if the ask price is at or below the calculated stop buy order value, it would be raised accordingly.

So first, could such a scheme be made to work, and second where to get the real time ask price data?
profile picture

Cone

#17
QUOTE:
What do you mean by this?
No, you can't do it, but if there were a Fidelity business requirement to implement that option.... anything is possible. (But at this stage, it's much less possible than when I suggested doing this years ago.)

QUOTE:
and an error will be displayed in the SM. You can appreciate this is very frustrating as many good trades are missed.
Definitely, but as I mentioned above, using the S. Monitor might be the main problem. There's too much delay with the S. Monitor for scalping.

1. How many symbols are you trading?
2. Did you check if the Y! link provides timely bid/ask information?

QUOTE:
determine the ask price 5sec or so before the close of the 1 min bar
That doesn't sound like a good idea. I think the problem is that the market is moving on you 5 to 10 seconds after the close of the bar, which is the delay you're getting in the S. Monitor. It would be much better to grab that data at the time the script executes.

Note that by modifying your signals using snapshot data instead of OHLC of the bar you risk the hypothetical script backtest being out of sync with reality. Of course
profile picture

boreland

#18
There is a noticeable delay with the S. Monitor. Is does mean that you do not always get the best fills if you use market orders instead of stop orders. It does not sound like that's ever likely to be improved upon. I generally run 8 to 10 symbols with a look back of 500 bars. Its reliability as to generating trade executions is so so. There are times when a signal should been generated and it is not as per the same script running as a strategy displayed through a chart. But I've learned to live with it. Generally the problem arises when you add a new sybol that it has not run before. I think it has to go out and perhaps get the data, I really do not know.

Y! link does still work and it's response time is a fraction of a second. The data appears real time. If the IQ feed or some other feed is not an option I'd like to give it a go. The question is what is the best way to code it?
profile picture

Eugene

#19
With regard to Y! bid / ask example above, what do you find challenging about implementing it in your solution?
profile picture

boreland

#20
So my first choice would be to use a different data feed like IQ is I could reduce the time frame down from 1 min bars. I do not know if this is possible? Cone semed to imply that is was.

Second choice is to pull in data from Y!. This issue here is how to trigger this event since it needs to occur several seconds before end of bar.
profile picture

Eugene

#21
Since this thread is one of very few dedicated to bid/ask data, please start a new thread for any other questions regarding IQFeed, Strategy Monitor etc.

Why pulling the bid/ask data from Y! should happen before end of bar? I do not follow it.
profile picture

boreland

#22
Pulling the data before the end of the bar is to allow time for the data to be returned so it might be used in the decision to set the buy stop value at bar close.
profile picture

Eugene

#23
But if your system doesn't process the stock asynchronously, why should there be a delay between getting the data and the buy decision?
profile picture

Cone

#24
Only 8 to 10 symbols? Use Streaming charts please!

Create a Workspace (or 2 if you have multiple monitors) and use Streaming. There's more setup involved because you have to go to each Alert tab to click "Auto-Stage/Place", but it's worth it for almost no delay getting your orders to market, and, it's far less likely for the Ask to be below the last close...

Light bulb moment!
I always assume a very liquid market, which is a mistake. It finally dawned on me that the last trade price could have occurred 45 or more seconds ago, for example, and the market has moved away from that price.

In this case, still use Streaming and just grab the Y! quotes to test your orders on the last bar. If the Ask is already below your calculated stop, make the script place a market order instead. Remember, when a normal stop is triggered, it executes as a market order anyway.


Here's a working example:

CODE:
Please log in to see this code.


Note!
It's possible that a solution like this makes your script become out of synch with reality, which would happen if the calculated stop price did not actually trigger (trade) on the bar that you exited at market.
profile picture

boreland

#25
I moved over to using the IQ Feed and shorter timeframe charts (15 sec). Is Bid/Ask data available within WLP from this feed?
profile picture

Eugene

#26
The IQFeed streaming provider returns live bid/ask prices with an instance of a Quote object. Can it be used in scripts? No, I don't think so.
profile picture

Cone

#27
By the way, just tried the script... the first URI doesn't work. The second one works, but there's definitely a delay. (Tested on AAPL just now, and modified the script above to show the Bid/Ask in the PricePane.)
profile picture

boreland

#28
Cone:

Here is a continuous update down version:
CODE:
Please log in to see this code.

Data download is fast for, but unfortunately the data appears to be delayed and is not real time. using the real time codes for bid and ask no longer returns any data. I wonder if there is another source to data?

What about a snapshot quote from Fidelity? If you have an account and are logged in I would assume its real time.
profile picture

boreland

#29
Cone:

Here is a very fast real time solution:

https://fastquote.fidelity.com/service/quote/json?productid=embeddedquotes&subproductid=eresearch&symbols=amat&dojo.preventCache=1478073465698&callback=dojo.io.script.jsonp_dojoIoScript3._jsonpCallback

Data includes a fetch duration time stamp. Returns ask and bid data in about 3ms. What is the best way to parse this do you think?
profile picture

Eugene

#30
Here's a quick example for you:

Prerequisites:

1. Check "System.Core" in References > NET Framework
2. In "Other assemblies to reference", navigate to C:\Windows\Microsoft.NET\Framework64\v.4.0.30319\ and select three assemblies: Microsoft.Csharp.dll, System.Web.Extensions.dll, System.Runtime.Serialization.Json.dll.



They are required for dynamic deserialization of Javascript (JSON) object that takes place in the code. Wealth-Lab will throw compiler errors if steps 1 and 2 aren't followed precisely.

CODE:

CODE:
Please log in to see this code.


EDIT: I've slightly improved your URL and adjusted the resulting code.
profile picture

boreland

#31
Got it much appreciated!

I do not think it is working. Bid and Ask is being returned as 0 value. Maybe its because the chart is not yet updating.
profile picture

Cone

#32
Excellent Eugene. Very useful!

Here's the routine wrapped in method using the example above:

CODE:
Please log in to see this code.
profile picture

Eugene

#33
QUOTE:
I do not think it is working. Bid and Ask is being returned as 0 value. Maybe its because the chart is not yet updating.

Perhaps your regional settings are non-standard in some way. For me it works all the time but try this just in case:

CODE:
Please log in to see this code.


profile picture

boreland

#34
Interesting, works only on some symbols but not others. No above code does not work for me unfortunately. Please try this symbol WLL. It works in the browser but no through the program.

I'm using Chrome. The issue is the user agent it needs to be chrome. Under Mozilla I do not get any data for the above symbol.
profile picture

Cone

#35
I use Chrome (only) too, and it works for me.

At this moment (08:49AM ET), these bid/ask data are returned for WLL:
CODE:
Please log in to see this code.


Wait until there's a market (0930) before trying the request.
profile picture

boreland

#36
Well still getting 0 value after the market opened. My Chrome browser is showing data however and it did before the market opened. What could be happening here?
profile picture

Eugene

#37
I do not get the bid/ask price for many instruments either. This may mean that their web table is not powered by the URI supplied but some other - and it's not evident to me which one. My guess is the page is static so bid/ask come from HTML. I leave hacking this up to you.
profile picture

Cone

#38
I think we can give up on the Fidelity or Yahoo! web-based bid/ask solution.

The Fidelity page returns data, but bid/ask come in only for Nasdaq stocks - and everything is 15 minutes delayed. Thinking about it, there's no way you'll get streaming data from Fidelity (or anyone legally) without a subscriber agreement and entitlement.

The second Yahoo! URI returns 15 minute delayed quotes as well.

It's not likely you'll find a free real time source for bid/ask prices, so the solution becomes much more custom. We (MS123) can't do anything about making bid/ask available to a script with the Fidelity stream, but we could try to invent something for the IQFeed Provider (but it would be another "in the future" project).
profile picture

boreland

#39
I do not want to give up quite yet. The data I am getting is real time not delayed from fidelity. I think I know how to crack this egg but need your help. I would assume folks using this will have a Fidelity account right.

When I down load using Chrome I get the real time data displayed in the HTML page for all symbols, but I had previously logged into fidelity.

I ran a test with Firefox, and did not get data, so I logged into my Fidelity account and run the url again, and this time I recieved data. Clearly there is session information being sent. Would it be a cookie being set or perhaps something else?

Perhaps the solution is to create an html fidelity login script?

So to test this idea I logged out from fidelity and sure enough the data went away.
profile picture

Eugene

#40
I second Cone's conclusion. As there's no streaming from this source without a Fidelity account, a "Fidelity login script" would go beyond the quick help we intended to provide. Please don't count on me.

Check this final try out. I discovered that freerealtimequotes.com is powered by a quotemedia.com call behind the scenes. If that fits then it's fine with me, if it doesn't then so be it.

P.S. No need to include any references - just a few "using" directives:

CODE:
Please log in to see this code.
profile picture

boreland

#41
Nice try but unfortunately the data is delayed. I did find this info on a fidelity login script but its not in C#:

http://marksweiss.github.io/sofine/docs/sofine/plugins/example/fidelity.m.html
profile picture

Eugene

#42
Hmm you're not the first one today who points me at some Python code (the first code was a Quantopian trading system). After looking at it, for sure it's clear in theory how to log in on the Fidelity page referred to in the script and what to do next. Problem is I don't have a Fidelity login. Furthermore as Cone have said it's really becoming a single-user solution.
profile picture

boreland

#43
Funny, I have a Quantopian tea shirt. I was one one there very early beta testers. In theory the solution should have broad appeal as people downloading wealthlab pro in the USA would have a Fidelity Login I assume. But I understand it puts you in a diificult situation. Perhaps you could post or email me what you think the solution ought to look like without testing it, and I'll try to do the rest and post it back it I get it to work. This is no an area where I have any real experience so I need a starting place. I did scrape up the login url it that helps.

You do not need to login into Fidelity through the application in order to get the real time data, just through a browser. I think you just need to send the right cookie information.

profile picture

Eugene

#44
Sorry but per our support policy we're not obliged to teach how to program and/or to engage in single-user solutions. While "in theory the solution should have broad appeal" as you have said, in practice you're the first in five years to reply in this thread. The ultimate task (Fidelity cookie) surely falls into the single-user solution category. You could find many online tutorials on how to handle cookies in C# programs as well as useful StackOverflow threads that would be more suitable for where this discussion is heading. Anyway, wish you good luck with your project.
profile picture

Eugene

#45
On a closing note. Since .NET uses built-in IE browser, in theory WebBrowser should share cookies with all other WebBrowser instances including IE. My final theory is that if you log in via IE and keep this browser instance while WL runs, the same Fidelity code might start working if rewritten to use the ActiveX WebBrowser control. Steps to try:

1. Log off in Chrome as we don't want its cookie to hamper. Log in to Fidelity via IE11. If on Windows 10 there is a way to launch IE11, too.
2. To run this code successfully you should take all the steps from my post #30.

CODE:
Please log in to see this code.


If it doesn't work then this is where it ends.
profile picture

boreland

#46
QUOTE:
On a closing note. Since .NET uses built-in IE browser, in theory WebBrowser should share cookies with all other Web Browser instances including IE.
I'm at a loss as to why this does not work it we are using the same browser engine. I can open up different instances are IE 11 and the data comes through when logged into fidelity. This is very frustrating, as it's a great idea.