With Hadoop serving as both a scalable data platform and process engine, Data Science is currently re-emerging as a centerpiece of enterprise innovation, with applied data solutions like as on-line product recommendation, automatic fraud detection and client sentiment analysis.
What is Big Data & Hadoop?
Big data is a popular term used to describe the exponential growth of data. Big Data can be either structured data or unstructured data or a combination of both. It is nothing but an assortment of such huge and complex data that it becomes very tedious to capture, store, process, retrieve and analyze it. Thanks to on-hand database management tools or traditional data processing techniques, things have become easier now.
Big data could be a well-liked term to describe the exponential growth of information. Big Data is either structured data or unstructured data or a combination of both. It is nothing but an assortment of such huge and complex data that it becomes very tedious to capture, store, process, retrieve and analyze it. Thanks to on-hand database management tools or traditional data processing techniques, things have become easier now.
Hadoop could be a programming framework that supports the process of large data sets in a very distributed computing surroundings. Hadoop was the primary and still the simplest tool to handle Big Data. Technically speaking, Hadoop is an open-source software framework that supports data-intensive distributed applications. Hadoop is licensed under the Apache v2 license. Hadoop has been developed, based on a paper originally written by Google on MapReduce system and applies concepts of functional programming.
A brief history of Hadoop:
HDFS (Hadoop Distributed File System):
Apache HDFS springs from GFS (Google File System). HDFS is from the ‘Infrastructural’ purpose of read in Hadoop. Though HDFS is at present a subproject of Apache Hadoop, it was formally developed as an infrastructure for the Apache Nutch web search engine project.
HDFS could be a distributed and ascendable filing system designed for storing terribly giant files with streaming knowledge access patterns, running clusters on artifact hardware.
The following are some of the assumptions and Goals/Objectives behind HDFS:
- Large data sets
- Write once- read many Model
- Streaming data access
- Commodity hardware
- Data replication and fault tolerance
- Moving computation is better than moving data
- File system namespace
HDFS works on these assumptions associated goals so as to assist the user access or process large data sets within an incredibly short period of time.
It all started with Google applying the thought of useful programming to unravel the matter of a way to manage giant amounts of information on the net. MapReduce was created in 2004 and Yahoo stepped in to develop Hadoop so as to implement the MapReduce technique in Hadoop. The key parts of MapReduce square measure Job Tracker, Task Trackers and JobHistoryServer.
Key to Hadoop’s power:
- Reducing Time and Cost – Hadoop helps in dramatically reducing the Time and Cost of building large scale data products.
- Computation is co-located with Data – Data and Computation system is co-designed to work together.
- Affordable at Scale – Can use ‘commodity’ hardware nodes, is self-healing, excellent at batch processing of large datasets.
- Designed for one write and multiple reads – There are no random Writes and is Optimized for minimum seek on hard drives
What is a Data product?
A data product could be a set of measurements ensuing from associate observation that’s typically keep in a very file. A software system whose core functionality depends on the application of statistical analysis and machine learning to data.
Example #1: People you may know
Example #2: Spell Correction
The terribly term ‘Data’, uncalled-for to mention, refers to data or data, and also the term ‘science’ holds a key role here. Data science is that the study of extracting knowledge from data. Signal process, applied mathematics learning, machine learning, programming etc. square measure the numerous fields that come back underneath the class of information science.
In other words, Data Science does the following:
- Extracting deep meaning from data
- Building Data Products
Here are the common Data Science tasks:
Reason #1: Explore full datasets
Reason #2: Mining of larger datasets
Reason #3: Large-scale data preparation
Reason #4: Accelerate data-driven innovation
80% of data science work is data preparation