Exporting data to CSV is very slow
Author: ruediger78
Creation Date: 6/18/2016 9:15 AM
profile picture

ruediger78

#1
I would like to write my data, that I have downloaded from IQFeed to a csv file. Therefore I created some code, which runs through a dataset and writes the data of any symbol to a file. This process is very slow. For one symbol with 1 minute data it takes one to two hours, while downloading the data from IQFeed lasts only some seconds. Any ideas how can speed up this process?

Here is my code:

CODE:
Please log in to see this code.
profile picture

Eugene

#2
Firstly, why won't you simply use the IQFeed provider? There doesn't seem to be anything unique in your script to justify it. The Market Manager (which the provider fully supports) already has a way to filter out data in a certain time range like 9:30 to 16:00 (see the Wiki link), if you're after it.

Second, in your code you're instantiating a StreamWriter on every bar. This is a tremendous waste of resources. Here's a proper way to do it - in the Wealth-Lab Wiki: Exporting data out of WL6 to ASCII and binary files. Feel free to modify the design pattern to suit your need. This should be sufficient for your task but for best performance on large data files you may even want to use StringBuilder.

P.S. I've renamed the title from something like "IQFeed data is very slow" which is confusing and incorrect and also moved it to the appropriate forum category. This is not the extension and/or Wealth-Lab issue.