This question comes up all the time and there are many articles written about this that I think fail to come right out ans say it. There is a simple answer:

Splunk is better at indexing data. If you have a use case that is query intensive but not write intensive then Elasticsearch is probably best. Remember that Elasticsearch is famous for being the back end of websites like LinkedIn and eBay–imagine their read to write profile. If you need a system that can gobble up a lot of data per day, hour, minute, whatever then Splunk is your solution. Note: I am using “Elasticsearch” to imply the whole ELK stack solution.

What about the price, what about the price!! ES is free!  Splunk costs a fortune!! This is the next thing everyone jumps to. Here is the simple answer: If you are playing in the big data space with data that is critical and if you need a search engine for it then you are going to be spending a lot on storage and processing and you will absolutely need professional support (we said critical). ELK isn’t a free lunch and the corporate IT world has already figured out that Open Software has its advantages but, for critical applications, economy is an elusive advantage of Open Source. The difference in price for the total solution isn’t that interesting once you really get into the numbers and the risks. Elastic.co is going public (or has gone public by the time you read this). Part of that reason, IMHO, is to give them greater credibility with large Enterprise accounts that they aren’t “two open source hippies with a nice website and a dog” that characterize other Open Source support and development companies, but a real company to help address the risk issues of the Fortune 500 accounts.

What about the installations?? Elasticsearch downloads are exploding and overtaking Splunk!  Calm down! So what. What nobody who quotes these stats really looks at is market share of (a) actual production installations and (b) whether or not Elasticsearch is displacing Splunk in the markets that Splunk actually wants to be in. Regarding actual production installations, everyone has downloaded Elasticsearch including myself several times, all the tech minded employees of Splunk to play around with it and a zillion college students playing around with it. This has nothing to do with competition. Regarding markets, I am confident that Splunk isn’t going after the SMB market with data sets < 1TB. Just because Elasticsearch has >90% market-share there doesn’t mean much even though these may represent a few thousand installs. This article has it all wrong.

As long as there are going to be users with write intensive needs and big datasets and, as long as Elasticsearch’s write performance is going to continue to stink, Splunk is going to find a nice comfortable home in enterprises that have the budget and the need for a solution.

 

Leave a Reply