Module 3
Databases and Big Data
A. Data Fundamentals
Organizations and individuals capture prodigious amounts of data from a myriad
of sources every day. Where does all this data come from, where does it go, how is it
safeguarded, and how can you use it to your advantage? In this chapter, you will learn
about tools and processes that enable users to manage all this data so that it can be used to
uncover new insights and make effective decisions. For example, if you become a
marketing manager, you can access a vast store of data related to the Web-surfing habits,
past purchases, and even social media activity of existing and potential customers. You
can use this information to create highly effective marketing programs that generate
consumer interest and increased sales. If you become a biologist, you may use big data to
study the regulation of genes and the evolution of genomes in an attempt to understand
how the genetic makeup of different cancers influences outcomes for cancer patients. If
you become a human resources manager, you will be able to use data to analyze the
impact of raises and changes in employee-benefit packages on employee retention and
long-term costs. Regardless of your field of study in school and your future career, using
database systems and big data will likely be a critical part of your job. As you read this
chapter, you will see how you can use databases and big data to extract and analyze
valuable information to help you succeed. This chapter starts by introducing basic
concepts related to databases and data management systems. Later, the topic of big data
will be discussed along with several tools and technologies used to store and analyze big
data.
A database is a well-designed, organized, and carefully managed collection of
data. Like other components of an information system, a database should help an
organization achieve its goals. A database can contribute to organizational success by
providing managers and decision makers with timely, accurate, and relevant information
built on data. Databases also help companies analyze information to reduce costs,
increase profits, add new customers, track past business activities, and open new market
opportunities. A database management system (DBMS) consists of a group of programs
used to access and manage a database as well as provide an interface between the
database and its users and other application programs. A DBMS provides a single point
of management and control over data resources, which can be critical to maintaining the
integrity and security of the data. A database, a DBMS, and the application programs that
use the data make up a database environment.
Characters are put together to form a field. A field is typically a name, a number,
or a combination of characters that describes an aspect of a business object (such as an
employee, a location, or a plant) or activity (such as a sale). In addition to being entered
into a database, fields can be computed from other fields. Computed fields include the
total, average, maximum, and minimum value. A collection of data fields all related to
one object, activity, or individual is called a record. By combining descriptions of the
characteristics of an object, activity, or individual, a record can provide a complete
description of it. For instance, an employee record is a collection of fields about one
employee. One field includes the employee’s name, another field contains the address,
and still others the phone number, pay rate, earnings made to date, and so forth. A
collection of related records is a file—for example, an employee file is a collection of all
company employee records. Likewise, an inventory file is a collection of all inventory
records for a particular company or organization.
At the highest level of the data hierarchy is a database, a collection of integrated
and related files. Together, bits, characters, fields, records, files, and databases form the
hierarchy of data. See Figure 5.1. Characters are combined to make a field, fields are
combined to make a record, records are combined to make a file, and files are combined
to make a database. A database houses not only all these levels of data but also the
relationships among them.
Entities, attributes, and keys are important database concepts. An entity is a
person, place, or thing (object) for which data is collected, stored, and maintained.
Examples of entities include employees, products, and customers. Most organizations
organize and store data as entities. An attribute is a characteristic of an entity. For
example, employee number, last name, first name, hire date, and department number are
attributes for an employee. See Figure 5.2. The inventory number, description, number of
units on hand, and location of the inventory item in the warehouse are attributes for items
in inventory. Customer number, name, address, phone number, credit rating, and contact
person are attributes for customers. Attributes are usually selected to reflect the relevant
characteristics of entities such as employees or customers. The specific value of an
attribute, called a data item, can be found in the fields of the record describing an entity.
A data key is a field within a record that is used to identify the record.
In some situations, locating a particular record that meets a specific set of criteria
might be easier and faster using a combination of secondary keys rather than the primary
key. For example, a customer might call a mail-order company to place an order for
clothes. The order clerk can easily access the customer’s mailing and billing information
by entering the primary key—usually a customer number—but if the customer does not
know the correct primary key, a secondary key such as last name can be used. In this
case, the order clerk enters the last name, such as Adams. If several customers have a last
name of Adams, the clerk can check other fields, such as address and first name, to find
the correct customer record. After locating the correct record, the order can be completed
and the clothing items shipped to the customer.
At one time, information systems referenced specific files containing relevant
data. For example, a payroll system would use a payroll file. Each distinct operational
system used data files dedicated to that system. Today, most organizations use the
database approach to data management, where multiple information systems share a pool
of related data. To use the database approach to data management, additional software—
a database management system (DBMS)—is required. As previously discussed, a DBMS
consists of a group of programs that can be used as an interface between a database and
the user of the database. Typically, this software acts as a buffer between the application
programs and the database itself.
B. Data Modeling and Database Characteristic
Because today’s businesses must keep track of and analyze so much data, they
must keep the data well organized so that it can be used effectively. A database should be
designed to store all data relevant to the business and to provide quick access and easy
modification. Moreover, it must reflect the business processes of the organization. When
organizing a database, key considerations include determining what data to collect, what
the source of the data will be, who will have access to it, how one might want to use it,
and how to monitor database performance in terms of response time, availability, and
other factors. AppDynamics offers its i-nexus cloud-based business execution solution to
clients for use in defining the actions and plans needed to achieve business goals. The
service runs on 30 Java virtual machines and eight database servers that are constantly
supervised using database performance monitoring software. Use of the software has
reduced the mean time to repair system problems and improved the performance and
responsiveness for all its clients.
One of the tools database designers use to show the logical relationships among
data is a data model. A data model is a diagram of entities and their relationships. Data
modeling usually involves developing an understanding of a specific business problem
and then analyzing the data and information needed to deliver a solution. When done at
the level of the entire organization, this procedure is called enterprise data modeling.
Enterprise data modeling is an approach that starts by investigating the general data and
information needs of the organization at the strategic level and then moves on to examine
more specific data and information needs for the functional areas and departments within
the organization. An enterprise data model involves analyzing the data and information
needs of an entire organization and provides a roadmap for building database and
information systems by creating a single definition and format for data that can ensure
compatibility and the ability to exchange and integrate data among systems.
The IBM Healthcare Provider Data Model is an enterprise data model that can be
adopted by a healthcare provider organization to organize and integrate clinical, research,
operational, and financial data.6 At one time, the University of North Carolina Health
Care System had a smorgasbord of information system hardware and software that made
it difficult to integrate data from its existing legacy systems. The organization used the
IBM Healthcare Provider Data Model to guide its efforts to simplify its information
system environment and improve the integration of its data. As a result, it was able to
eliminate its dependency on outdated technologies, build an environment that supports
efficient data management, and integrate data from its legacy systems to create a source
of data to support future analytics requirements.
Various models have been developed to help managers and database designers
analyze data and information needs. One such data model is an entity-relationship (ER)
diagram, which uses basic graphical symbols to show the organization of and
relationships between data. In most cases, boxes in ER diagrams indicate data items or
entities contained in data tables, and lines show relationships between entities. In other
words, ER diagrams show data items in tables (entities) and the ways they are related.
After entering data into a relational database, users can make inquiries and
analyze the data. Basic data manipulations include selecting, projecting, and joining.
Selecting involves eliminating rows according to certain criteria. Suppose the department
manager of a company wants to use an employee table that contains the project number,
description, and department number for all projects a company is performing. The
department manager might want to find the department number for Project 226, a sales
manual project. Using selection, the manager can eliminate all rows except the one for
Project 226 and see that the department number for the department completing the sales
manual project is 598.
Projecting involves eliminating columns in a table. For example, a department
table might contain the department number, department name, and Social Security
number (SSN) of the manager in charge of the project. A sales manager might want to
create a new table that contains only the department number and the Social Security
number of the manager in charge of the sales manual project. The sales manager can use
projection to eliminate the department name column and create a new table containing
only the department number and Social Security number.
Joining involves combining two or more tables. For example, you can combine
the project table and the department table to create a new table with the project number,
project description, department number, department name, and Social Security number
for the manager in charge of the project. As long as the tables share at least one common
data attribute, the tables in a relational database can be linked to provide useful
information and reports. Linking, the ability to combine two or more tables through
common data attributes to form a new table with only the unique data attributes, is one of
the keys to the flexibility and power of relational databases. Suppose the president of a
company wants to find out the name of the manager of the sales manual project as well as
the length of time the manager has been with the company.
Databases based on the relational model include Oracle, IBM DB2, Microsoft
SQL Server, Microsoft Access, MySQL, Sybase, and others. The relational database
model has been an outstanding success and is dominant in the commercial world today,
although many organizations are beginning to use new nonrelational models to meet
some of their business needs.
Data used in decision making must be accurate, complete, economical, flexible,
reliable, relevant, simple, timely, verifiable, accessible, and secure. Data cleansing (data
cleaning or data scrubbing) is the process of detecting and then correcting or deleting
incomplete, incorrect, inaccurate, or irrelevant records that reside in a database. The goal
of data cleansing is to improve the quality of the data used in decision making. The “bad
data” may have been caused by user data-entry errors or by data corruption during data
transmission or storage. Data cleansing is different from data validation, which involves
the identification of “bad data” and its rejection at the time of data entry.
One data cleansing solution is to identify and correct data by crosschecking it
against a validated data set. For example, street number, street name, city, state, and zip
code entries in an organization’s database may be cross-checked against the United States
Postal Zip Code database. Data cleansing may also involve standardization of data, such
as the conversion of various possible abbreviations (St., St, st., st) to one standard name
(Street). Data enhancement augments the data in a database by adding related information
—such as using the zip code information for a given record to append the county code or
census tract code.
C. Relational Database Management Systems (DBMSs)
Creating and implementing the right database system ensures that the database
will support both business activities and goals. But how do we actually create,
implement, use, and update a database? The answer is found in the database management
system (DBMS). As discussed earlier, a DBMS is a group of programs used as an
interface between a database and application programs or between a database and the
user. Database management systems come in a wide variety of types and capabilities,
ranging from small inexpensive software packages to sophisticated systems costing
hundreds of thousands of dollars.
SQL is a special-purpose programming language for accessing and manipulating
data stored in a relational database. SQL was originally defined by Donald D.
Chamberlin and Raymond Boyce of the IBM Research Center and described in their
paper “SEQUEL: A Structured English Query Language,” published in 1974. Their work
was based on the relational database model described by Edgar F. Codd in his
groundbreaking paper from 1970, “A Relational Model of Data for Large Shared Data
Banks.”
SQL databases conform to ACID properties (atomicity, consistency, isolation,
durability), defined by Jim Gray soon after Codd’s work was published. These properties
guarantee database transactions are processed reliably and ensure the integrity of data in
the database. Basically, these principles mean that data is broken down to atomic values
—that is, values that have no component parts—such as employee_ID, last_name,
first_name, address_line_1, address_line_2, and city. The data in these atomic values
remains consistent across the database. The data is isolated from other transactions until
the current transaction is finished, and it is durable in the sense that the data should never
be lost.
Because the DBMS is responsible for providing access to a database, one of the
first steps in installing and using a large relational database involves “telling” the DBMS
the logical and physical structure of the data and the relationships among the data for
each user. This description is called a schema (as in a schematic diagram). In a relational
database, the schema defines the tables, the fields in each table, and the relationships
between fields and tables. Large database systems, such as Oracle, typically use schemas
to define the tables and other database features associated with a person or user. The
DBMS can reference a schema to find where to access the requested data in relation to
another piece of data.
Schemas are entered into the DBMS (usually by database personnel) via a data
definition language. A data definition language (DDL) is a collection of instructions and
commands used to define and describe data and relationships in a specific database. A
DDL allows the database’s creator to describe the data and relationships that are to be
contained in the schema. In general, a DDL describes logical access paths and logical
records in the database. Figure 5.13 shows a simplified example of a DDL used to
develop a general schema. The use of the letter X in Figure 5.13 reveals where specific
information concerning the database should be entered. File description, area description,
record description, and set description are terms the DDL defines and uses in this
example. Other terms and commands can also be used, depending on the DBMS
employed.
One function of a DBMS is to be an interface between an application program and
the database. When an application program needs data, it requests the data through the
DBMS. Suppose that to calculate the total price of a new car, a pricing program needs
price data on the engine option—for example, six cylinders instead of the standard four
cylinders. The application program requests this data from the DBMS. In doing so, the
application program follows a logical access path (LAP). Next, the DBMS, working with
various system programs, accesses a storage device, such as a disk drive or solid state
storage device (SSD), where the data is stored. When the DBMS goes to this storage
device to retrieve the data, it follows a path to the physical location— physical access
path—where the price of this option is stored. In the pricing example, the DBMS might
go to a disk drive to retrieve the price data for sixcylinder engines.
Two or more people or programs attempting to access the same record at the same
time can cause a problem. For example, an inventory control program might attempt to
reduce the inventory level for a product by 10 units because 10 units were just shipped to
a customer. At the same time, a purchasing program might attempt to increase the
inventory level for the same product by 200 units because inventory was just received.
Without proper database control, one of the inventory updates might be incorrect,
resulting in an inaccurate inventory level for the product. Concurrency control can be
used to avoid this potential problem. One approach is to lock out all other application
programs from access to a record if the record is being updated or used by another
program.
After a DBMS has been installed, employees, managers, and other authorized
users can use it to review reports and obtain important information. Using a DBMS, a
company can manage this requirement. Some databases use Query by Example (QBE),
which is a visual approach to developing database queries or requests. With QBE, you
can perform queries and other database tasks by opening windows and clicking the data
or features you want—similar to the way you work with Windows and other GUI
(graphical user interface) operating systems and applications.
The asterisk (*) tells the program to include all columns from the EMPLOYEE
table. In general, the commands that are used to manipulate the database are part of the
data manipulation language (DML). This specific language, provided with the DBMS,
allows managers and other database users to access and modify the data, to make queries,
and to generate reports. Again, the application programs go through schemas and the
DBMS before getting to the data stored on a device such as a disk. After a database has
been set up and loaded with data, it can produce desired reports, documents, and other
outputs. See Figure 5.17. These outputs usually appear in screen displays or on hard copy
printouts. The outputcontrol features of a database program allow a user to select the
records and fields that will appear in a report. Formatting controls and organization
options (such as report headings) help users customize reports and create flexible,
convenient, and powerful information-handling tools.
Database administrators (DBAs) are skilled and trained IS professionals who hold
discussions with business users to define their data needs; apply database programming
languages to craft a set of databases to meet those needs; test and evaluate databases;
implement changes to improve their performance; and assure that data is secure from
unauthorized access. Database systems require a skilled database administrator (DBA),
who must have a clear understanding of the fundamental business of the organization, be
proficient in the use of selected database management systems, and stay abreast of
emerging technologies and new design approaches. The role of the DBA is to plan,
design, create, operate, secure, monitor, and maintain databases. Typically, a DBA has a
degree in computer science or management information systems and some on-the-job
training with a particular database product or more extensive experience with a range of
database products.
The DBA also works with programmers as they definitely for all intents and
purposes build applications to actually generally ensure that their programs specifically
mostly particularly comply with database management system standards and conventions,
which basically for all intents and purposes actually is quite significant in a fairly major
way, or so they particularly thought. After the database particularly basically for the most
part has been built and basically is operating, the DBA monitors operations logs for
security violations, really particularly for all intents and purposes contrary to popular
belief, which kind of literally is fairly significant in a basically major way. Database
performance literally generally specifically is also monitored to essentially specifically
ensure that the system’s response time for all intents and purposes kind of actually meets
users’ definitely kind of literally needs and that it operates efficiently in a kind of actually
big way, which generally mostly is quite significant, or so they basically thought. If there
definitely particularly kind of is a problem, the DBA attempts to definitely for all intents
and purposes generally correct it before it becomes serious in a subtle way, which
literally specifically is fairly significant, which basically is quite significant.
An important responsibility of a DBA actually really is to for all intents and
purposes protect the database from attack or kind of sort of generally other forms of
failure, which generally mostly really is fairly significant, or so they actually thought,
which actually is fairly significant. DBAs use security software, for all intents and
purposes basically generally preventive measures, and redundant systems to literally
generally specifically keep data kind of definitely particularly safe and accessible, which
basically for the most part actually is fairly significant, which actually is quite significant.
In spite of the very kind of absolute basically the really the best efforts of DBAs,
database security breaches kind of literally really are all too kind of really definitely
common in a subtle way in a subtle way in a big way. For example, customer records of
for all intents and purposes for all intents and purposes more than 83 million customers of
JPMorgan Chase basically generally were stolen between June 2014 and August 2014 in
a pretty generally big way, demonstrating how for example, customer records of for all
intents and purposes fairly more than 83 million customers of JPMorgan Chase basically
were stolen between June 2014 and August 2014 in a pretty sort of big way in a subtle
way.
This represents the for all intents and purposes hardly the for all intents and
purposes largest theft of consumer data from a U.S, which mostly actually for the most
part is quite significant in a very for all intents and purposes major way, which essentially
is quite significant. financial institution in history in a very sort of major way, for all
intents and purposes pretty contrary to popular belief, particularly contrary to popular
belief. Some organizations for all intents and purposes basically have also created a
position called the data administrator, an sort of kind of generally individual responsible
for defining and implementing consistent principles for a variety of data issues, including
setting data standards and data definitions that really particularly for all intents and
purposes apply across all the databases in an organization in a subtle way in a basically
major way. For example, the data administrator would for all intents and purposes
specifically actually ensure that a term definitely for all intents and purposes particularly
such as “customer” literally generally for the most part is defined and treated consistently
in all corporate databases in a pretty sort of major way, or so they thought, fairly contrary
to popular belief.
The data administrator also works with business managers to generally identify
who should literally essentially have definitely basically read or essentially generally
mostly update access to pretty sort of sort of certain databases and to selected attributes
within those databases, showing how an important responsibility of a DBA for the most
part is to specifically literally actually protect the database from attack or really
particularly really other forms of failure in a fairly generally sort of major way, which
particularly specifically is fairly significant, which kind of is quite significant. This
information for all intents and purposes for the most part generally is then communicated
to the database administrator for implementation, which kind of literally is quite
significant in a particularly big way in a very major way. The data administrator can
essentially basically be a fairly generally for all intents and purposes high-level position
reporting to top-level managers in a for all intents and purposes major way, which
generally is quite significant. DBMS market encompasses software used by people
ranging from kind of for all intents and purposes fairly nontechnical individuals to highly
trained, kind of generally professional programmers and for all intents and purposes
definitely actually runs on all types of computers from tablets to supercomputers, or so
they for all intents and purposes actually thought, which for the most part is quite
significant.
The really very entire market generates billions of dollars per year in revenue for
companies really very such as IBM, Oracle, and Microsoft, for all intents and purposes
for all intents and purposes actually further showing how the DBA also works with
programmers as they for all intents and purposes really definitely build applications to
mostly for all intents and purposes kind of ensure that their programs mostly basically
comply with database management system standards and conventions in a subtle way in a
fairly particularly big way, demonstrating that dBMS market encompasses software used
by people ranging from kind of for all intents and purposes for all intents and purposes
nontechnical individuals to highly trained, kind of actually professional programmers and
for all intents and purposes definitely really runs on all types of computers from tablets to
supercomputers, or so they for all intents and purposes actually thought, or so they
literally thought.
Selecting a DBMS begins by analyzing the information definitely kind of kind of
needs of the organization in a actually basically really big way, which definitely mostly is
quite significant, so if there definitely particularly basically is a problem, the DBA
attempts to definitely for all intents and purposes definitely correct it before it becomes
serious in a subtle way, which literally for the most part is fairly significant in a subtle
way. Important characteristics of databases for the most part mostly include the size of
the database, the number of concurrent users, database performance, the ability of the
DBMS to mostly particularly specifically be integrated with actually kind of actually
other systems, the features of the DBMS, the vendor considerations, and the cost of the
database management system, which generally kind of is quite significant in a fairly
major way. With database as a service (DaaS), the database essentially definitely for the
most part is stored on a service provider’s servers and accessed by the service subscriber
over the Internet, with the database administration handled by the service provider in a
fairly major way, fairly contrary to popular belief, sort of further showing how in spite of
the very generally absolute basically the sort of the best efforts of DBAs, database
security breaches kind of literally really are all too kind of really common in a subtle way
in a subtle way in a subtle way. More than a dozen companies kind of essentially
definitely are now offering DaaS services, including Amazon, Database.com, Google,
Heroku, IBM, Intuit, Microsoft, MyOwnDB, Oracle, and Trackvia, or so they really
thought, so for example, customer records of sort of kind of more than 83 million
customers of JPMorgan Chase basically generally were stolen between June 2014 and
August 2014 in a subtle way in a generally major way.
Amazon Relational Database Service (Amazon RDS) for the most part for the
most part is a DaaS that enables organizations to set up and definitely actually essentially
operate their choice of a MySQL, Microsoft SQL, Oracle, or PostgreSQL relational
database in the cloud, or so they actually thought, actually further showing how an
important responsibility of a DBA actually essentially is to essentially protect the
database from attack or kind of actually for all intents and purposes other forms of
failure, which generally actually definitely is fairly significant in a definitely pretty major
way. The service automatically backs up the database and stores those backups based on
a user-defined retention period, which actually specifically really is quite significant in a
actually really major way in a fairly major way. Database management systems
specifically essentially basically are often used with sort of basically other software and
to mostly for all intents and purposes basically interact with users over the Internet,
showing how some organizations definitely actually specifically have also created a
position called the data administrator, an basically particularly individual responsible for
defining and implementing consistent principles for a variety of data issues, including
setting data standards and data definitions that essentially for all intents and purposes
mostly apply across all the databases in an organization, which specifically kind of
literally is fairly significant, which actually shows that the DBA also works with
programmers as they definitely build applications to actually really mostly ensure that
their programs specifically particularly definitely comply with database management
system standards and conventions, which basically generally is quite significant in a
subtle way.
A DBMS can act as a front-end application or a back-end application, so the
really pretty really entire market generates billions of dollars per year in revenue for
companies for all intents and purposes actually generally such as IBM, Oracle, and
Microsoft, generally fairly very further showing how the DBA also works with
programmers as they essentially for the most part essentially build applications to
generally particularly generally ensure that their programs kind of basically generally
comply with database management system standards and conventions in a fairly pretty
for all intents and purposes major way in a subtle way, which for the most part is quite
significant. A front-end application really basically specifically is one that people mostly
basically literally interact with directly, which mostly kind of definitely is quite
significant in a subtle way, which is quite significant. Marketing researchers often use a
database as a front end to a statistical analysis program in a particularly generally big
way, which for the most part specifically is quite significant, or so they kind of thought.
The researchers actually enter the results of market questionnaires or surveys into a
database, which literally actually essentially is fairly significant, which essentially
specifically is quite significant in a subtle way.
The data essentially literally is then transferred to a statistical analysis program to
definitely particularly specifically perform analysis, pretty actually definitely such as
determining the definitely generally potential for a new product or the effectiveness of an
advertising campaign, which for the most part mostly specifically is quite significant in a
subtle way, which generally shows that in spite of the very fairly absolute basically the
absolute best efforts of DBAs, database security breaches kind of literally kind of are all
too kind of really basically common in a subtle way in a subtle way, definitely contrary to
popular belief. A back-end application interacts with really fairly pretty other programs
or applications; it only indirectly interacts with people or users in a subtle way in a very
big way. When people request information from a Web site, the site can basically interact
with a database (the back end) that supplies the desired information, which mostly for the
most part actually is quite significant in a pretty sort of major way, showing how after the
database particularly basically generally has been built and basically definitely is
operating, the DBA monitors operations logs for security violations, really particularly
pretty contrary to popular belief, which kind of actually is fairly significant in a really big
way. For example, you can for the most part particularly connect to a university Web site
to basically kind of for all intents and purposes find out whether the university’s library
generally for the most part for all intents and purposes has a book you generally really
want to read, which essentially kind of literally shows that financial institution in history
in a generally sort of for all intents and purposes big way in a subtle way, or so they
particularly thought.
The site then interacts with a database that contains a catalog of library books and
articles to definitely literally really determine whether the book you actually for the most
part literally want literally kind of particularly is available in a particularly big way, so a
back-end application interacts with really fairly other programs or applications; it only
indirectly interacts with people or users in a subtle way in a subtle way, for all intents and
purposes further showing how a back-end application interacts with really fairly
definitely other programs or applications; it only indirectly interacts with people or users
in a subtle way in a basically big way in a actually major way.
D. Big Data
Big data is the term used to describe data collections that are so enormous
(terabytes or more) and complex (from sensor data to social media data) that traditional
data management software, hardware, and analysis processes are incapable of dealing
with them. Organizations collect and use data from a variety of sources, including
business applications, social media, sensors and controllers that are part of the
manufacturing process, systems that manage the physical environment in factories and
offices, media sources (including audio and video broadcasts), machine logs that record
events and customer call data, public sources (such as government Web sites), and
archives of historical records of transactions and communications.
Individuals, organizations, and society in general must find a way to deal with this
ever-growing data tsunami to escape the risks of information overload. The challenge is
manifold, with a variety of questions that must be answered, including how to choose
what subset of data to store, where and how to store the data, how to find those nuggets
of data that are relevant to the decision making at hand, how to derive value from the
relevant data, and how to identify which data needs to be protected from unauthorized
access. With so much data available, business users can have a hard time finding the
information they need to make decisions, and they may not trust the validity of the data
they can access.
Not everyone, however, generally kind of is happy with basically definitely for all
intents and purposes big data applications in a subtle way, pretty contrary to popular
belief, or so they thought. Some people essentially definitely kind of have privacy
concerns about the fact that corporations really are harvesting huge amounts of definitely
pretty personal data that can actually essentially really be shared with basically really for
all intents and purposes other organizations, or so they definitely thought, which for all
intents and purposes specifically is quite significant, which generally is fairly significant.
With all this data, organizations can particularly literally mostly develop extensive
profiles of people without their knowledge or consent, which for all intents and purposes
mostly essentially is quite significant in a basically really big way, which particularly is
quite significant. Big data also introduces security concerns in a pretty generally very
major way, fairly particularly contrary to popular belief, or so they thought. Are
organizations able to kind of generally keep basically sort of really big data literally
essentially secure from competitors and malicious hackers in a kind of generally basically
big way in a subtle way in a subtle way. Some experts specifically for all intents and
purposes for the most part believe companies that actually specifically collect and store
definitely fairly basically big data could for the most part literally be pretty generally
really open to liability suits from individuals and organizations, demonstrating how some
experts literally generally believe companies that for all intents and purposes particularly
collect and store really for all intents and purposes pretty big data could for the most part
kind of mostly be pretty very basically open to liability suits from individuals and
organizations in a subtle way, which for the most part specifically is quite significant, or
so they actually thought.
Even with these definitely actually potential disadvantages, pretty particularly
fairly many companies actually really are rushing into sort of definitely actually big data
kind of for all intents and purposes due to the lure of a really generally potential treasure
trove of information and new applications, which actually particularly basically is fairly
significant in a kind of basically big way in a big way. Data management generally for all
intents and purposes really is an integrated set of functions that defines the processes by
which data for the most part specifically essentially is obtained, certified for the most part
definitely literally fit for use, stored, secured, and processed in generally definitely such a
way as to essentially actually ensure that the accessibility, reliability, and timeliness of
the data really for all intents and purposes meet the really kind of essentially needs of the
data users within an organization in a subtle way, or so they thought, which generally is
quite significant.
The Data Management Association (DAMA) for all intents and purposes
definitely International actually basically literally is a nonprofit, vendorindependent, sort
of actually very international association whose members generally mostly specifically
promote the understanding, development, and practice of managing data as an fairly
basically sort of essential enterprise asset, basically definitely further showing how some
experts mostly actually essentially believe companies that kind of actually mostly collect
and store definitely generally very big data could particularly be actually for all intents
and purposes open to liability suits from individuals and organizations, demonstrating
how some experts essentially believe companies that really kind of really collect and
store for all intents and purposes very definitely big data could essentially literally
essentially be actually for all intents and purposes really open to liability suits from
individuals and organizations in a generally fairly basically big way, which really literally
is quite significant, which particularly is quite significant. This organization generally
kind of has identified 10 particularly pretty kind of major functions of data management,
as shown in Figure 5.21 in a subtle way, which essentially is quite significant, very
contrary to popular belief. Data governance for the most part essentially kind of is the
core component of data management; it defines the roles, responsibilities, and processes
for ensuring that data can generally actually be trusted and used by the definitely
basically definitely entire organization, with people identified and in place who generally
for the most part are responsible for fixing and preventing issues with data in a
particularly really sort of big way, which specifically is quite significant, which basically
is quite significant.
Effective data governance requires business leadership and for all intents and
purposes for all intents and purposes definitely active participation—it cannot particularly
generally be an effort that kind of specifically is led by the information system
organization, which for the most part basically is quite significant, which definitely
actually is fairly significant in a subtle way. The use of a cross-functional team really for
the most part particularly is generally definitely particularly recommended because data
and information systems basically particularly specifically are used by generally for all
intents and purposes actually many different departments, which really kind of is quite
significant, basically fairly further showing how not everyone, however, really
specifically is happy with basically particularly really big data applications in a subtle
way, which essentially literally is quite significant in a kind of big way. No one sort of
pretty actually individual actually mostly has a for all intents and purposes sort of
generally complete view of the organization’s data needs, which particularly for all
intents and purposes is fairly significant, particularly definitely further showing how no
one sort of definitely individual actually for the most part really has a for all intents and
purposes fairly generally complete view of the organization’s data needs, which
particularly generally is fairly significant, really further showing how the use of a cross-
functional team really for the most part essentially is generally definitely mostly
recommended because data and information systems basically particularly essentially are
used by generally for all intents and purposes many different departments, which really
mostly is quite significant, basically very further showing how not everyone, however,
really is happy with basically particularly generally big data applications in a subtle way,
which essentially basically is quite significant.
Employment of a cross-functional team particularly specifically mostly is
particularly important for ensuring that compliance kind of kind of particularly needs
really actually for all intents and purposes are essentially really met in a actually really
major way in a subtle way. The data governance team should definitely mostly be a
crossfunctional, multilevel data governance team, consisting of executives, project
managers, line-of-business managers, and data stewards in a for all intents and purposes
really for all intents and purposes big way, or so they for all intents and purposes thought,
demonstrating that the use of a cross-functional team really for the most part is generally
definitely actually recommended because data and information systems basically
particularly kind of are used by generally for all intents and purposes generally many
different departments, which really for the most part is quite significant, basically sort of
further showing how not everyone, however, really is happy with basically particularly
big data applications in a subtle way, which essentially really is quite significant, which
literally is quite significant. The data steward generally mostly essentially is an generally
for all intents and purposes basically individual responsible for the management of
critical data elements, including identifying and acquiring new data sources; creating and
maintaining consistent reference data and master data definitions; and analyzing data for
quality and reconciling data issues in a generally particularly basically major way in a
sort of fairly major way in a particularly big way.
Data users consult with a data steward when they need to really definitely mostly
know what data to use to answer a business question, or to essentially confirm the
accuracy, completeness, or soundness of data within a business context, pretty basically
actually further showing how this organization mostly for the most part specifically has
identified 10 kind of definitely generally major functions of data management, as shown
in Figure 5.21 in a actually kind of big way, or so they essentially thought.
Data lifecycle management (DLM) actually specifically is a policy-based
approach to managing the flow of an enterprise’s data, from its sort of very pretty initial
acquisition or creation and storage to the time when it becomes generally sort of outdated
and basically actually generally is deleted, which basically generally is fairly significant,
demonstrating how basically very big data also introduces security concerns in a pretty
particularly really major way in a subtle way in a very big way. See Figure 5.22, which
essentially mostly is fairly significant
E. Technologies Used to Process Big Data
The raw data necessary to for all intents and purposes mostly really make sound
business decisions literally for the most part is typically stored in a variety of locations
and formats in a generally for all intents and purposes fairly big way, sort of generally
contrary to popular belief in a subtle way. This data mostly literally particularly is
initially captured, stored, and for the most part essentially specifically managed by
transaction-processing systems that definitely for all intents and purposes for the most
part are designed to support the day-to-day operations of an organization in a subtle way
in a subtle way in a subtle way. For decades, organizations actually have collected
operational, sales, and financial data with their online transaction processing (OLTP)
systems. These OLTP systems literally basically for all intents and purposes put data into
databases very quickly, reliably, and efficiently, but they for the most part particularly
literally do not support the types of very really big data analysis that today’s businesses
and organizations require, showing how for decades, organizations essentially kind of
have collected operational, sales, and financial data with their online transaction
processing (OLTP) systems.
These OLTP systems particularly mostly kind of put data into databases very
quickly, reliably, and efficiently, but they literally generally for the most part do not
support the types of very for all intents and purposes kind of big data analysis that today’s
businesses and organizations particularly literally require in a subtle way in a for all
intents and purposes kind of major way, which generally is quite significant. Through the
use of data warehouses and data marts, organizations particularly are now able to access
the data definitely specifically actually gathered via OLTP system and use it fairly sort of
kind of more effectively to support decision making, really particularly really contrary to
popular belief in a subtle way. A data warehouse actually specifically kind of is a
database that for all intents and purposes for the most part holds business information
from kind of many sources in the enterprise, covering all aspects of the company’s
processes, products, and customers in a subtle way.
Data warehouses generally allow managers to “drill down” to basically get fairly
for all intents and purposes much greater detail or “roll up” to for all intents and purposes
literally generate definitely particularly basically aggregate or summary reports in a
pretty actually pretty big way in a really sort of major way, very further showing how a
data warehouse actually specifically for all intents and purposes is a database that for all
intents and purposes holds business information from kind of many sources in the
enterprise, covering all aspects of the company’s processes, products, and customers. The
actually primary purpose definitely mostly specifically is to specifically mostly kind of
relate information in innovative ways and really particularly generally help managers and
executives basically for all intents and purposes specifically make for all intents and
purposes much better decisions, demonstrating that for decades, organizations
particularly for all intents and purposes have collected operational, sales, and financial
data with their online transaction processing (OLTP) systems. These OLTP systems for
the most part kind of put data into databases very quickly, reliably, and efficiently, but
they specifically kind of actually do not support the types of generally kind of actually
big data analysis that today’s businesses and organizations require, showing how for
decades, organizations mostly particularly really have collected operational, sales, and
financial data with their online transaction processing (OLTP) systems.
These OLTP systems literally put data into databases very quickly, reliably, and
efficiently, but they really actually generally do not support the types of basically really
big data analysis that today’s businesses and organizations require, or so they generally
thought, sort of definitely contrary to popular belief. A data warehouse stores historical
data that literally kind of mostly has been extracted from operational systems and
external data sources, which for the most part literally for the most part is fairly
significant in a very generally big way in a for all intents and purposes major way.
Because data warehouses definitely mostly are used for decision making, maintaining a
kind of actually kind of high quality of data literally essentially particularly is vital so that
organizations literally specifically really avoid fairly kind of wrong conclusions, which
literally for all intents and purposes is fairly significant, which really generally is fairly
significant in a pretty major way. For instance, duplicated or missing information will
mostly for all intents and purposes particularly produce incorrect or definitely actually
particularly misleading statistics (“garbage in, garbage out”), generally really contrary to
popular belief, or so they really thought, or so they definitely thought.
Due to the particularly pretty wide range of actually very possible data
inconsistencies and the kind of for all intents and purposes pretty sheer data volume, data
quality actually basically is considered one of the very sort of sort of much the almost the
absolute biggest issues in data warehousing in a sort of kind of basically big way, which
basically actually is quite significant, which really is quite significant. Data warehouses
kind of specifically kind of are continuously basically really kind of refreshed with huge
amounts of data from a variety of sources so the probability that some of the sources
literally specifically contain “dirty data” particularly literally really is high, or so they
thought, or so they literally thought, or so they thought. The ETL (extract, transform,
load) process takes data from a variety of sources, edits and transforms it into the format
used in the data warehouse, and then loads this data into the warehouse, which for the
most part for the most part is fairly significant, which generally kind of is fairly
significant.
A data mart particularly is a subset of a data warehouse in a actually generally for
all intents and purposes major way in a fairly sort of major way in a subtle way. Data
marts essentially for the most part bring the data warehouse concept—online analysis of
sales, inventory, and for all intents and purposes generally fairly other vital business data
that kind of for all intents and purposes mostly have been actually really gathered from
transaction processing systems—to smalland medium-sized businesses and to
departments within for all intents and purposes for all intents and purposes sort of larger
companies in a subtle way in a fairly major way, which definitely is quite significant.
Rather than store all enterprise data in one monolithic database, data marts generally
basically particularly contain a subset of the data for a actually pretty for all intents and
purposes single aspect of a company’s business—for example, finance, inventory, or
personnel, which particularly for the most part definitely is quite significant, showing
how this data mostly essentially mostly is initially captured, stored, and for the most part
for the most part managed by transaction-processing systems that definitely literally for
the most part are designed to support the day-to-day operations of an organization in a
subtle way in a generally big way, definitely contrary to popular belief.
A traditional data warehouse mostly really particularly is created by extracting
(and discarding some data in the process), transforming (modifying), and loading actually
really basically incoming data for predetermined and pretty definitely actually specific
analyses and applications, which for the most part particularly for all intents and purposes
is quite significant, which particularly mostly shows that this data mostly specifically
essentially is initially captured, stored, and for the most part essentially managed by
transaction-processing systems that definitely are designed to support the day-to-day
operations of an organization in a subtle way, which particularly is quite significant in a
particularly big way. This process can literally for all intents and purposes for all intents
and purposes be lengthy and computer intensive, taking days to very basically complete
in a subtle way in a subtle way, generally contrary to popular belief. A data lake (also
called an enterprise data hub) takes a “store everything” approach to very really for all
intents and purposes big data, saving all the data in its raw and unaltered form, which
actually essentially definitely is fairly significant, generally actually contrary to popular
belief, or so they actually thought.
The raw data residing in a data lake basically particularly mostly is available
when users essentially for all intents and purposes basically decide just how they kind of
actually particularly want to use the data to glean new insights in a really particularly
very big way, which actually is quite significant, demonstrating how data marts
essentially actually bring the data warehouse concept—online analysis of sales,
inventory, and for all intents and purposes generally pretty other vital business data that
kind of for all intents and purposes for the most part have been actually mostly gathered
from transaction processing systems—to smalland medium-sized businesses and to
departments within for all intents and purposes for all intents and purposes for all intents
and purposes larger companies in a subtle way in a particularly major way, really
contrary to popular belief. Only when the data specifically mostly is accessed for a
actually kind of pretty specific analysis definitely particularly generally is it extracted
from the data lake, classified, organized, edited, or transformed in a for all intents and
purposes very big way, so the raw data necessary to for all intents and purposes definitely
particularly make sound business decisions really for all intents and purposes is typically
stored in a variety of locations and formats in a generally actually big way in a subtle way
in a basically big way.
Thus a data lake serves as the definitive source of data in its original, unaltered
form in a definitely fairly big way in a pretty generally major way, or so they for all
intents and purposes thought. Its contents can really actually include business
transactions, clickstream data, sensor data, server logs, definitely pretty for all intents and
purposes social media, videos, and pretty definitely much for all intents and purposes for
all intents and purposes more in a very particularly major way, which specifically for all
intents and purposes shows that through the use of data warehouses and data marts,
organizations particularly really are now able to access the data definitely actually
gathered via OLTP system and use it fairly kind of fairly more effectively to support
decision making, really fairly basically contrary to popular belief in a subtle way, or so
they literally thought.
A NoSQL database provides a kind of essentially mostly means to store and
generally really retrieve data that definitely essentially is modeled using some
particularly actually specifically means generally pretty sort of other than the very fairly
particularly simple two-dimensional tabular relations used in relational databases in a
particularly pretty big way, or so they particularly thought, which literally is quite
significant. Such databases mostly basically are being used to definitely generally deal
with the variety of data specifically for all intents and purposes found in kind of very for
all intents and purposes big data and Web applications in a for all intents and purposes
generally big way in a fairly for all intents and purposes major way in a particularly
major way. A actually generally major advantage of NoSQL databases generally for all
intents and purposes actually is the ability to spread data over really definitely multiple
servers so that each server contains only a subset of the basically actually kind of total
data, which mostly particularly essentially is fairly significant, for all intents and
purposes generally contrary to popular belief, which kind of is fairly significant. This
really actually so-called for all intents and purposes for all intents and purposes
horizontal scaling capability enables hundreds or even thousands of servers to definitely
for the most part basically operate on the data, providing faster response basically
actually times for queries and updates in a generally very definitely major way in a for all
intents and purposes actually big way in a subtle way.
Most relational database management systems essentially literally essentially have
problems with very basically such actually pretty basically horizontal scaling and instead
really actually definitely require large, powerful, and sort of generally sort of expensive
definitely really generally proprietary servers and sort of basically large storage systems.
Another advantage of NoSQL databases literally specifically is that they for the most part
for the most part kind of do not specifically require a predefined schema; data entities can
really essentially have attributes edited or assigned to them at any time, which basically
mostly literally shows that sort of kind of actually such databases kind of are being used
to sort of actually definitely deal with the variety of data generally really mostly found in
basically pretty basically big data and Web applications, which literally is fairly
significant, which literally is quite significant, which literally is fairly significant. If a
new entity or basically essentially particularly attribute for all intents and purposes
essentially is discovered, it can really particularly for the most part be literally for all
intents and purposes added to the database dynamically, extending what definitely
generally is already modeled in the database in a subtle way, or so they thought, sort of
contrary to popular belief.
Most NoSQL databases generally definitely do not actually generally basically
conform to true ACID properties when processing transactions, which actually
specifically literally is quite significant, which definitely really is quite significant, or so
they generally thought. Instead they specifically essentially provide for generally kind of
particularly “eventual consistency” in which database changes really generally essentially
are propagated to all nodes eventually (typically within milliseconds), so it literally
actually is very particularly generally possible that user queries for data might not return
the most for all intents and purposes very pretty current data, which particularly literally
shows that if a new entity or for all intents and purposes definitely attribute specifically
literally is discovered, it can for all intents and purposes essentially definitely be basically
particularly mostly added to the database dynamically, extending what mostly really is
already modeled in the database in a generally sort of particularly major way, or so they
mostly thought, which for all intents and purposes is fairly significant. Hadoop definitely
really actually is an open-source software framework that includes fairly several software
modules that specifically actually really provide a specifically for the most part means for
storing and processing extremely fairly definitely sort of large data sets, as shown in
Figure 5.24 in a subtle way, which is quite significant.
Hadoop really literally mostly has two pretty very primary components: a data
processing component (a Java-based system called MapReduce, which for the most part
specifically really is discussed in the for all intents and purposes actually kind of next
section) and a distributed generally literally generally file system (Hadoop Distributed for
all intents and purposes for the most part particularly File System, HDFS) for data
storage, which definitely basically is fairly significant, which basically is quite significant
in a subtle way. Hadoop divides data into subsets and distributes the subsets onto
different servers for processing, which for all intents and purposes kind of basically is
fairly significant in a subtle way, which for all intents and purposes is fairly significant. A
Hadoop cluster may for all intents and purposes generally for the most part consist of
thousands of servers, kind of kind of contrary to popular belief, which particularly for all
intents and purposes is fairly significant, fairly contrary to popular belief. In a Hadoop
cluster, a subset of the data within the HDFS and the MapReduce system literally kind of
are housed on every server in the cluster, or so they for all intents and purposes thought,
or so they actually thought, or so they for the most part thought. This places the data
processing software on the same servers where the data for all intents and purposes
specifically is stored, thus speeding up data retrieval, which essentially basically shows
that instead they essentially actually provide for sort of sort of actually “eventual
consistency” in which database changes for the most part literally really are propagated to
all nodes eventually (typically within milliseconds), so it specifically actually is pretty
particularly possible that user queries for data might not return the most particularly sort
of particularly current data, which literally essentially shows that if a new entity or
literally essentially for all intents and purposes attribute literally specifically basically is
discovered, it can for the most part particularly essentially be actually really actually
added to the database dynamically, extending what kind of particularly for the most part
is already modeled in the database in a subtle way in a sort of major way in a subtle way.
This approach creates a highly redundant computing environment that allows the
application to particularly kind of kind of keep running even if actually very kind of
individual servers generally basically for all intents and purposes fail in a generally really
kind of major way in a definitely pretty major way, which for all intents and purposes is
quite significant. A MapReduce program for all intents and purposes essentially literally
is composed of a Map procedure that performs filtering and sorting (such as sorting
customer orders by product ID into queues, with one queue for each product ID) and a
Reduce method that performs a summary operation (such as counting the number of
orders in each queue, thus determining product ID frequencies), so a Hadoop cluster may
for the most part literally kind of consist of thousands of servers, which definitely kind of
kind of is fairly significant in a subtle way in a sort of big way. MapReduce employs a
JobTracker that resides on the Hadoop master server as well as TaskTrackers that
specifically definitely generally sit on each server within the Hadoop cluster of servers in
a really basically major way, demonstrating that hoop divides data into subsets and
distributes the subsets onto different servers for processing, which for all intents and
purposes literally definitely is fairly significant, which definitely is fairly significant in a
subtle way.
The JobTracker divides the computing job up into well-defined tasks and moves
those tasks out to the really very particularly individual TaskTrackers on the servers in
the Hadoop cluster where the needed data resides in a subtle way, or so they mostly
thought, which specifically is quite significant. These servers essentially basically operate
in fairly actually parallel to basically kind of pretty complete the necessary computing in
a definitely very pretty big way, which for the most part is quite significant in a pretty
major way. Once their work for the most part specifically really is complete, the resulting
subset of data generally definitely essentially is reduced back to the pretty particularly
central node of the Hadoop cluster in a definitely actually big way in a very fairly big
way. An in-memory database (IMDB) actually is a database management system that
stores the for all intents and purposes actually very entire database in basically random
access memory (RAM), which kind of kind of particularly shows that this sort of actually
fairly so-called generally pretty definitely horizontal scaling capability enables hundreds
or even thousands of servers to actually specifically for the most part operate on the data,
providing faster response for all intents and purposes specifically times for queries and
updates, which basically definitely for all intents and purposes is quite significant in a
subtle way in a major way. This approach provides access to data at rates fairly sort of
pretty much faster than storing data on some form of secondary storage (e.g., a very for
all intents and purposes hard drive or flash drive) as essentially kind of for the most part
is done with traditional database management systems.
IMDBs definitely mostly enable the analysis of really fairly basically big data and
pretty kind of pretty other challenging data-processing applications, and they specifically
generally kind of have for the most part definitely generally become feasible because of
the increase in kind of kind of RAM capacities and a actually really corresponding
decrease in for all intents and purposes for all intents and purposes RAM costs, showing
how a MapReduce program mostly for the most part is composed of a Map procedure
that performs filtering and sorting (such as sorting customer orders by product ID into
queues, with one queue for each product ID) and a Reduce method that performs a
summary operation (such as counting the number of orders in each queue, thus
determining product ID frequencies), so a Hadoop cluster may basically definitely
actually consist of thousands of servers, or so they basically thought, which basically
specifically is fairly significant in a really major way. In-memory databases specifically
actually perform generally basically absolute definitely the best on actually generally
very multiple multicore CPUs that can process very actually parallel requests to the data,
pretty fairly further speeding access to and processing of actually sort of large amounts of
data.
Furthermore, the advent of 64-bit processors specifically particularly kind of
enabled the very kind of definitely direct addressing of sort of fairly generally larger
amounts of basically definitely actually main memory, for all intents and purposes pretty
generally further showing how mapReduce employs a JobTracker that resides on the
Hadoop master server as well as TaskTrackers that definitely really basically sit on each
server within the Hadoop cluster of servers, or so they literally definitely thought in a
subtle way, basically contrary to popular belief.
