Any-recommended-local-database-solutions-that-can-be-used-with-Wealth-Lab

Why is a db required? What's lacking with the built-in persistent storage mechanism for EventProviderBase?

In case it's not clear, in WealthLab, Event data is basically any data that isn't part of the BarHistory - like Fundamental data.
https://www.wealth-lab.com/Support/ExtensionApi/EventDataProvider

QUOTE:
if anyone here uses a self-built database,... using MySQL or

Back in the old days, RAM memory was expensive. So the database was stored on disk rather than cached in RAM memory. In this context, MySQL was tasked with indexing the data (which is cached in RAM) and locating the record on disk to bring that specific data record into RAM to satisfy the request. So the job of MySQL was to associate which record on disk held the requested data via the cached index.

Today RAM memory is cheap, so programmers just cache the entire database in RAM so we avoid all the indexing to disk records business a traditional disk-oriented database would do.

My suggestion is to forget the traditional disk-oriented database (e.g. MySQL), buy more RAM, and cache your entire database there. And periodically during and at the end of your WL session, you can simply write your RAM cached database to disk to save it. Keep it simple.

The next question is whether or not the WL internal event database, discussed in Post #1, is fast enough for you? Well is it? If not, why not? Perhaps you need to reorganize it into a different data structure to make lookup faster. The .NET framework does have a number of different data structures, such as Dictionary<key,reference>, you can reorganize the data into for more lookup speed.

I would get your solution running first, then worry about reorganizing the cached database (with a different .NET data type) for higher lookup speed later. But if you're interested, you can look over what .NET has to offer. https://learn.microsoft.com/en-us/dotnet/standard/collections/commonly-used-collection-types

Where are you sourcing the fundamental data?

I use PostgreSQL in my trading tool. But it's not used for Event data or price data. For that I depends on WL itself and data providers such as Norgate.

I use local db mainly for the following reasons. I use WL for backtesting and daily signal generation but use my own software for execution.
https://www.wealth-lab.com/Discussion/Add-Trading-Preference-to-exit-position-based-on-executed-order-size-12192
https://www.wealth-lab.com/Discussion/Automatically-save-Order-Manager-history-12155

@Cone: I see, the EventProvider in Wealth Lab itself will store a copy of the data, so all that needs to be done is to link Wealth Lab to the data source through the API, is that correct?

However, I also have some other uses for building my own database, such as creating a fundamental dashboard for stock.

@Glitch Now I am studying FMP. https://site.financialmodelingprep.com/

There are also some data sources in China that provide fundamental data for US stocks, but I have not verified their accuracy yet, such as:

Tushare: https://tushare.pro/
Akshare:https://akshare.akfamily.xyz/

@johnbens: It's described in the documentation: see Persistent Storage Option.

QUOTE:
I also have some other uses for building my own database

SQLite is highly recommended for such a purpose.

https://sqlite.org/

Another option is Microsoft SQL Server 2025 Developer Edition (it is free): https://www.microsoft.com/en-us/sql-server/sql-server-downloads

At the above page, scroll down to the download link titled "Download Standard Developer Edition".

For an IDE for SQL Server 2025 install SQL Server Management Studio: https://learn.microsoft.com/en-us/ssms/install/install

There are SQL Server extensions available for Visual Studio, and Visual Studio Code, as well as plugins for JetBrains Rider IDE.

Also, a nice tool for interactive LINQ is LINQPad: https://www.linqpad.net/

QUOTE:
However, I also have some other uses for building my own database, such as creating a fundamental dashboard for stock.

So do you need a disk-based relational database in the first place?

I see many suggestions for "SQL" type databases, which are relational. But I'm thinking a flat-file database would be good enough for this. Now if you need an "SQL Join" (Are you joining unlike data structures?), then you will need relational, but that would be unusual for stock analysis. If it's one uniform data structure, then a flat-file database would be much faster and simpler.

An example of a flat-file database would be an Excel spreadsheet. Could you fit your data application in something like that with a uniform data structure?

The other thing is are you going to have over 10,000 to 20,000 records? If yes, then an SQL disk-based database (with a cached index) maybe good. But if not, then I wouldn't use a disk-based database. Just cache your data in RAM memory like WL does. Keep it as simple as possible for speed. Picking data record-by-record off disk is slow.

Someone suggested SQLite in Post #8, which I'm unfamiliar with. Perhaps it's not disk-based or relational, which would make it much faster. You might research that.

WL uses cached C# data structures that I referenced in Post #2. You can then use LINQ queries to search them. This is the modern fastest approach promoted by Microsoft .NET.

QUOTE:
Someone suggested SQLite in Post #8, which I'm unfamiliar with. Perhaps it's not disk-based or relational,

SQLite is (as the name suggest) a SQL based (thus relational) database.
A SQLite database consists of one single file with zero administration efforts.
It is also possible to have it RAM based (i.e. completely in memory)

It is free, very fast, and very easy to drive from C# code.

BTW: The SQLite software is among the most heavily used pieces of software qorldwide. If you own a mobile phone or use a modern browser you in fact are using SQLite...

QUOTE:
However, I also have some other uses for building my own database, such as creating a fundamental dashboard for stock.

Considering uses such as creating a fundamental dashboard and those listed in the original post, I'd use a SQL solution. Consider data organization, data retrieval, indexing, filtering, data import and export, at least.

In the future you may want to add additional tables related to your stock data table(s) that allow for supporting additional capabilities.
The data you could import to a SQL database may need cleanup, and SQL would make that easy.
Backtesting may lead you to want to make queries against the data in your SQL database to validate outcomes, test ideas, etc.