Hadoop: demonstrating Open Source benefits
The inaugural Strata + Hadoop World 2015 conference in Singapore is the first ever to be held in Asia, and opened to a sold-crowd from various companies in the big data and analytics industry and also various industries that used big data and analytics solutions and technologies.
Cloudera Chief Strategy Officer Mike Olson started the keynote session with a hypothetical question about whether Hadoop would disappear in a few years. Olson observed that there is possibility of this happening because “…it’s not that you won’t use Hadoop… but rather we won’t talk about it anymore because of the (more) interesting apps built on top.”
Today, Hadoop is already enabling management and querying of big data in industries like healthcare, banks, telcos and more.
A key founder of Hadoop, Doug Cutting, who is also Cloudera’s Chief Architect, commented, “Hadoop technology just exploded. Linux demonstrated value of open source, but at a very low level in the stack, whereas Hadoop is further up the stack.”
Hadoop, is now a very popular open source framework for storing data and running apps on scale-out commodity hardware, and Cloudera was one of the first companies in 2008 to bring Hadoop from the Silicon Valley where Hadoop first caught on with lots of Web companies, to enterprise businesses outside it.
But, why did Hadoop have to be open source? Cutting used to work on search and proprietary technologies at companies like Apple Inc., Excite, and Xerox PARC, a division in Xerox Corporation.
Cutting had learnt later on that open source can provide a technology that is adopted much more widely than proprietary solutions.“It’s not just because it’s cheaper. Companies are comfortable adopting it at the core of their business when it’s open source, because they are not giving control away to a tech vendor.
“They do not depend on the fate of another company.”
Addressing the question of whether open source can produce enterprise-grade solutions, Cutting observed, “That is not the case today, although it was true a decade ago.”
He said that Linux is the most popular operating system across industries, and that Hadoop is another strong example. “Enterprises are adopting it across their organisation, in industries like healthcare, banks, manufacturing and more.”
Another point is that more and more industries are generating revenue that is driven by data and IT these days.
Cutting pointed out, “These industries like transportation, manufacturing and agriculture, are becoming IT-based, and the technologies (they use) are open source. It’s not a coincidence, but it’s related. The appropriateness of technology is moving more quickly because it is not tied to tech vendors.”
In contrast, the technology that was used before Hadoop came along, relational database management (RDBM), was tied to vendors and Cutting observed, “They were not interested in making fundamental revolutions of that technology (RDBM)!”
It took Hadoop and a growing community that wanted to leverage big data and answer bigger questions, to do that instead.
(This journalist is a guest of Cloudera’s to Strata + Hadoop World 2015 in Singapore)