Does anyone have recommendations for a database suited to log-like data? Write heavy, zero updates and will need to support moderately complex queries to facilitate some dashboards. PostgreSQL will probably be fine, but I thought it would be worthwhile to take a look around.
no plus ones
Shared publicly•View activity
View 12 previous comments
- Redis, depending on how complex the queries are.Dec 29, 2012
- What about not using a relational database, if it is really big data give a chance to map-reduce (i.e. hadoop)Dec 29, 2012
- Another recommendation for Splunk here. Too many reasons to list, but yes, log data is a typical starting point for leveraging Splunk.Dec 29, 2012
- Agreed Jason - the general idea of Splunk is that it is "ready to go" - Other solutions are possible - but you need to do more work to get the results ..
At least that is how I would word itDec 29, 2012
- Redis is great for many use cases too. Need to build your own plumbing, in this case. But there's a lot of interesting algo + infra work which Redis makes quite simple. The atomicity of macros, sorted sets, etc., are amazing. And in performance testing, we had to struggle to find ways to create a significant load... on a single process. Loading the NIC was more often the issue.Dec 29, 2012
- I have used the Infobright Community Edition (ICE) variant of MySQL very successfully for machine-generated data. To quote from http://www.infobright.org/:
"Infobright's column oriented, open-source analytic database provides both speed and efficiency. Coupled with 10:1 average compression, ICE takes your machine generated data and gives you the ultimate power to slice-n'-dice your data. Plus, it's free!"Dec 29, 2012