I don't know Riak, other than its a distributed NoSQL key-value data store.
Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on.
How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf lang/db being developed by Kevin Lawler[3]?
Kevin also wrote kona, an opensource version of the "K programming language"[4].
Kdb is very fast at time series analysis on large datasets, and has many years of proven value in the financial industry.
[1] https://kx.com/ [2] http://www.jsoftware.com/jdhelp/overview.html [3] https://github.com/kevinlawler/kerf [4] https://github.com/kevinlawler/kona
Minor quibble: the article is about RiakTS, their time-series enhanced version of riak_core.
riak_core's main strength is that it does key-value in a distributed/resilient manner, spreading values in multiple copies (at least 3) all over a cluster of servers. Kill one server, no problem. Need more capacity, add servers and it will rebalance itself.
The TS part is just an optimization built on top of that, to make values which are near each other in time to be near each other in storage, for faster range-base retrieval.
Yes, exactly. (Author)
TS enhanced version of riak_core, got it.
Most TS, or tick DBs, are columnar-based, memory-mapped, fast and light systems.
Are there any benchmarks similar to STAC-M3, which is a year's worth of NYSE data run on different hardware to gauge kdb+'s effectiveness on different hardware configurations [1]? It's a great way to gauge performance and TCO.
Does it do both memory (streaming data) and disk-based (historical) storage for big data set analytics in realtime?
I'd be interested to see numbers there.
A lot of people think kdb+ is only for finance. There is a conference coming up in May that will have talks on q (the language for the kdb+ database) about natural language processing and machine learning in q to name a few. Another is about using it at a power plant to most efficiently route power based upon realtime data [2].
I only got into kdb+ and q with the free, non-commercial 32-bit version. I usually use J and sometimes APL, which had MapReduce since at least the 80s for APL.Check out this post from 2009 [3]. I guess the 'new shiny' bit threw me in your chosen title.
[1] https://stacresearch.com/news/2014/02/13/stac-reports-intel-...
[2] https://kxcon2016.com/agenda/
[3] http://blog.data-miners.com/2009/04/mapreduce-hadoop-everyth...
You might find these benchmarks interesting: http://kparc.com/q4/readme.txt
I was inquiring about benchmarks for RiakTS, but your link was perfect. I am a J/APL dabbler, and quite recently learning kdb+/q (I prefer k).
As much as I step away from these languages, I always find my way back to them in strange ways. I was studying music, and there was a great J article in Vector magazine written in August 2006 [1] that walks through scales, and other musical concepts in J.
A Forth-based music software called Sporth [2] has a kona ugen in it, so you can generate scales or other musical items in kona, and then use them in the stack-based Sporth audio language.
My interests in kdb+/q, k, J and APL are in applying them to mathematical investigations of music, visuals, doing data analysis, and then just code golfing, or toying around. They're so much fun!
I need more time on large streaming datasets (Time Series data), than large disk-based datasets to really test latencies. I am building a box much better suited for it than my current machine. The goal is to stay in RAM as much as possible.
https://github.com/JohnEarnest/ok/tree/gh-pages/ike
https://github.com/JohnEarnest/ok
And related APL/J/K subreddit: https://www.reddit.com/r/apljk/