DAT475_Assignment_Hadoop and HBase Concepts

Introduction to Hadoop and HBase Concepts

As the volume and the speed at which data are being collected increase exponentially, the

approach to process and analyze these data are, equally, changed from the traditional computing

techniques to big data analysis. Hadoop is one of the platforms that can handle big data without

many challenges. With all the good news about Hadoop, there are some limitations, such as

inability to handle large volume of read and write requests simultaneously, and so on. To tackle

these limitations, the need for an application like HBase arises. i HBase is a column-oriented

database built on top of the Hadoop file system. It is an open-source project and is horizontally

scalable. This is designed to provide quick random access to huge amounts of data. Since Hadoop

File System (HDFS) has fault tolerance, HBase leverages this feature to provides random real-

time read/write access to data in the HDFS.

There are 3 key components of HBase; HMaster, Region Server (Slaves), and the Zookeeper.

HMaster: This is the master server, the “boss”. Just as its name, HMaster manages and assigns

regions to Region Servers. It controls load balancing, failover etc., manages the Region Server

and Hadoop cluster, handle the Data Defining Language (DDL) operations such as creating and

deleting tables, manages changes in metadata operations, and accepts requests and sends it to the

relevant Region Server.

configuration information, tracks Region Server and HMaster failure, maintains Region Server

information.

examples of each:

Row-Oriented Databases are the traditional databases, the Relational Database Management

Systems (RBDMS) like Oracle, MySQL and so on. While the Column-Oriented Databases are

database management systems like HBase, MongoDB etc. They are not only SQL (NoSQL)

databases. Unlike the RDBMS, Column-Oriented Databases store data tables by column instead

of rows.

Row-Oriented Databases are suitable for Online Transaction Process (OLTP). While Column-

Hive is best used for analytical querying of data, and HBase is primarily used to store or process

unstructured Hadoop data as a lake. Hive is not suitable for real-time querying, but HBase is used

for real-time querying or Big Data.

Hive is used to pull of the data and other basic things from the HDFS. For example, management

personal can use Hive to generate reports

HBase cluster.

2. create – this command will create table(s) and table schema(s) with row and column

families.

3. list - this command will list all the tables that are preset or created in HBase.

4. Describe – this command describes the table name, by displaying more information about

column families present in the mentioned-table.

5. Alter – to edit, or rename a table, we use the command alter

6. Put – to insert into a table, we use the command, put.

7. Get – to retrieve the information in a table, we use the get command.

HBase uses MapReduce jobs to process terabytes of data in a scalable fashion. It does not directly

support joins, but the join queries are implemented by retrieving data from HBase tables.

What is a namespace in HBase?

The namespace is used for logical table grouping into a database system. It helped to resource

management, Security, isolation.

What happens when you delete table cell(s) in HBase?

The short pseudocode example concludes with discussion of conceptual view after parsing 2

sample records. i i Show the conceptual view after parsing the following third sample record.

Member ID: 2011078; Member Name: Ozcan Ozan; Member since: 2011; Exercise: Exercise:

Bicep Curl, Weight: 25; Exercise: Triceps Deeps, Weight: Body Weight; Exercise: Plank,

Weight: Body Weight; Trainer 1: Heather Smith; Trainer 2: Barbara Eden; Trainer 3: James

Martin

Row_Key – Unique identifier

Column families and Columns

Member_Info – Member_ID, Member_Name, Member_Since

Weight_Shed_from_Exercise – a column for each exercise to be tracked. i New exercises can be

End of Part 1

Part 2

1. The Big Data Systems that will be used to solve the UMGC academic record system

problem is HBase. HBase uses Hadoop files as storage system to store the large amounts

of data. The UMGC academic records system data will be stored in HBase, which will be

Row_Key

Member_ID

3.

On the higher level,

the Column families and Columns

Row_Key – Unique identifier

Student_Info - Student_Name , EmplID, User_Name, Instructor_ID, Graduation_Status

Program_Info –Program, Specialization1, Specialization2, Specialization3

(a column for each program information to be tracked. A student enrolled in a program may

received on the last attempt overwrites.)

Course_Grades_DATA - DATA610, DATA620, DATA630, DATA630, DATA640, DATA650,

DATA670

(a column for each course grade in DATA to be tracked. Students can repeat class and the grade

received on the last attempt overwrites.)

Course_Instructors_ITEC - ITEC610 , ITEC620, ITEC630

Course_Instructors_DBST - DBST651, DBST651, DBST651_TA, DBST667

Course_Instructors_DATA - DATA610, DATA620, DATA630, DATA640, DATA650,

DATA650, DATA650_TA, DATA670, DATA670

And the table below shows the HBase table model showing column families and columns

5. The CAP theorem states that any distributed data store can only provide two of the

following three guarantees. Consistency, Availability, and Partition tolerance CAP.

Consistency means that when server receives reads, all clients see the same data at the

same time. Availability, a guarantee that every request sees a response whether it’s a

success or a failure. Partition tolerance means that the system will continue to function

when a node fails. A distributed system can only guarantee two of the three properties.

HBase is considered eventually consistent and implements consistency and partition

tolerance. When a node in a region is down, it is recoverable, meaning that the data will

still be available at the later time. While streaming a live video on Facebook or YouTube

and it was interrupted for some reason, you can still view the life streaming to see what