Data: can be Qualitative = descriptive and conceptual and cannot be measured OR
Quantitative = can be counted, measured, and expressed using numbers.
Can be Numerical or Categorical: Numerical > measured when continuous and counted when discrete > Interval or Ratio : Interval = aka
“scale”, “quantitative”, or “parametric” data. Distances between consecutive integers are equal. Data is always numerical. Differences between
numbers are comparable. Location of origin, “zero”, is arbitrary. Zero has no meaning : Ratio = has a true zero point. Scale is commonly used
for values that are measured in numbers, such as length, height, weight, volume, or monetary values like cost, profit and loss, and revenue.
Zero has a meaning : Categorical > contains a finite number of categories or distinct groups > Nominal or Ordinal : Nominal = numerical values
just “name” and attribute. Numbers are used to classify (male or female) or categorize (color) – can be stored as “word”, “text”, or “nominal
code”. We cannot find average value of nominal data, we can only compare if the data is equal or not equal : Ordinal = on an ordinal scale.
Numbers are used to indicate rank or order. Differences between numbers are not comparable. Order is important. Also not able to find
average or mean of ordinal data.
Descriptive Statistics = study data with entirety
3 Principles of describing data: Center, Spread, Shape
Population = the whole population. A collection of persons, objects, or items under study : Sample = a portion of the whole/population. A
subset of the population; must be large enough to represent the whole
Measuring Data Centrality > yield information about the center, or middle part, of a group of numbers. Can be measured by mean, median,
mode, percentiles, and quartiles. Mean = the average of a group of numbers. Computed by summing all values in the data set and dividing the
sum by the number of values in the data set. Affected by each value in the data set, including extreme values. Median = middle value in an
ordered array of numbers. Applicable for ordinal, interval, and ratio data but not for nominal data. Unaffected by extremely large and
extremely small values. Mode = the most frequently occurring value in a data set. Applicable to all levels of data measurement.
Measuring Data Variability (Spread) > describe the spread or dispersion of a set of data. Can be measured by range, interquartile
range(difference between 1st and 3rd quartiles)(25-75), mean absolute deviation, variance, and standard deviation. Range = the difference
between the largest and the smallest values in a set of data. Ignores all data points except the two extreme values. Mean absolute deviation
= average of the absolute deviations from the mean. Variance = average of the squared deviations from the arithmetic mean. To compute
sample variance, n-1 = Bessel’s correction : Standard deviation = square root of the variance
Measuring Data Shape > describe the skewness of a set of data. Commonly measured by kurtosis – a measure of whether the data are
peaked or flat relative to a normal distribution. High kurtosis = tend to have a distinct peak near the mean. Low kurtosis = tend to have a flat
top near the mean rather than a sharp peak. If the data peaks above normal mean = high kurtosis, if below = low k.
Two variables have a strong statistical relationship with one another if they appear to move together. Positive and Negative
Covariance = a measure of the linear association(correlation) between 2 variables. The covariance between X and Y is the average of the
product of the deviations of each pair of observations from their respective means. Correlation coefficient= between -1 and 1
Correlation = a measure of the linear relationship between two variables, which does not depend on the units of measurement. Measured by
the correlation coefficient. -1 is negative, 1 is positive, 0 means no correlation. Compute by dividing covariance
Inferential Statistics = study sample data. Has to be representative of the population and randomly drawn.
Confidence level (CI) = you will be able to find out if the claim is correct or not with confidence level – 95% with a margin of error of 5%. For
example, CI is 0.930 when the level of significance is 0.07
Random sampling = part of the sampling technique in which each sample has an equal probability of being chosen.
Central Limit Theorem – the sample mean distribution forms a normal distribution, the mean of the sample mean distribution is the same as
the population mean. Standard deviation of the sample mean=standard error. Standard error decreases as sample size increases. Sample of a
continuous random variable = the time to complete a specific task.
Confidence interval – upper 95% mean and lower 95% mean shown in summary statistics
Z-statistics – for larger samples, n>30, aka standard normal distribution
T-statistics – for samples n<30, used when population SD is unknown. Degrees of freedom n-1. Has a larger variance
Null hypothesis – Ho – a statement that generally assumes nothing has changed, an existing theory or belief
Alternative hypothesis – Ha – opposite of null, typically your claim
There are two outcomes for any hypothesis test: reject the null hypothesis or fail to reject the null hypothesis
When p-value is lower than a (alpha), then reject the null hypothesis. a = 1 – CL = 5%
If CL increases, the CI gets wider. If sample size increases, CI decreases and becomes narrower=more precise measurement.
Null hypothesis=true > Reject null hypothesis >> Type-I Error (should not reject null hypothesis to be correct)
Research hypothesis=true > Don’t reject null hypothesis >> Type-II Error (should reject the null hypothesis to be correct)
Fail to reject the null = we have insufficient data to reject the null hypothesis
One way Analysis of Variance (ANOVA) – used to compare (test) more than two sample means. This process investigates variances to
establish if the means are significantly different. Will tell you that at least one of the means is significantly different from the others. If p-value
is less than .05 we reject the null. You can perform follow-up test to figure out which groups are different – called Tukey’s HSD (Honestly
Significant Difference), gives us a connecting letters report with differences
Correlation coefficient – statistic that shows the strength of the relationship. Perfect negative correlation -1, perfect positive correlation +1.
EX: The correlation coefficient between the age of a vehicle and the money spent to repair it is 0.9. This means 81% of the variation in the
money spent on repairs is explained by the age of the vehicle.
Linear Regression – Ho = there is no significant correlation between the variables. Ha = null hypothesis is not true (they’re different). If p-
value is less than .05, we reject the null, means that there is a correlation between the variables
Dependent variable (Y) vs. Independent variable (X)
Regression line – the line of “best fit” through a scatter plot. Summarizes the relationship between independent and dependent variable.
Ex: What is the null hypothesis for testing a linear regression model with profit as the dependent variable and sales as the independent
variable? There is no linear relationship between profit and sales.
Least square fit approach – draws a perpendicular line from the points to the line (in the y-direction) then you square the distance and add
them up. Whatever gives you the minimum distance of sum of square distance = line that will best fit
Residual Variance (unexplained variance) – based on difference between scatter plot points and predicted point on the regression line. This
represents what regression line cannot predict.
Regression Variance (explained variance) – based on the differences between predicted data points and the mean of Y.
Total Variance – based on deviations of data points from the mean of Y.
Regression equation = (intercept(dependent coefficient=beta 0)(7.902) + (aptitude score=beta1)(0.753) * Aptitude score – this means one
unit change in aptitude score will change the performance score by 0.753
R-squared – a statistical measure of how close the data are to the fitted regression line, aka the coefficient of determination. R^2=0.694,
means that the model explains 69.4% of the observed variation of the response data (Y) around its mean. Always fall between 0 and 1,
inclusive. Higher R-squared = better predictability of the regression model.
Multiple linear regression: There must be a linear relationship between the dependent variable and independent variables. Multivariate
normality = the residuals are normally distributed. No multi-collinearity – independent variables are not highly correlated with each other.
Tested using Variance Inflation Factor (VIF) – indicate the degree that the variances in the regression estimates are increased due to multi-
collinearity. VIF values higher than 5 indicate that multi-collinearity is a problem. You should remove those variables and run model again.
Logistic Regression a specialized type of regression analysis that is designed to predict binary categorical variables. 0/1 outcome
In logistic regression analysis, instead of Y as a dependent variable, we use a function of Y called Logit.
The dependent variable Y is defined as Log (p/1-p). Odds ratio is defined as p/1-p, where p is the probability of success.
In classification analysis, we are determining the probability of an observation to be part of a certain class or not. Ex: buy/won’t buy,
true/false. We also typically split the data into two mutually exclusive sets, known as training and validation/testing, to investigate the
strength of the developed model.
In classification problems, the primary source for accuracy estimation of the model is the confusion matrix.
Ex: A loan officer wants to know if the next customer is likely to default or not on a loan. How can she assess the risk of extending the loan to
that customer? By utilizing a multiple logistic regression model developed by an in-house analyst.
Data visualization – process of transforming the input data into a set of features which can well represent the input data. When the data is
presented visually, one can grasp difficult concepts or identify new patterns more easily. Lets you see things that would go unnoticed. Visual
data enables the reader to see trends, behavior patterns and dependencies on that information. Visualizing data is a kind of descriptive
analytical technique. William Playfair is credited for inventing the line and bar chart. 1786 line char: National debt vs. Time. Rose diagram:
plots data using a circular historical plot: 1858 Florence Nightingale: poor sanitary conditions in hospitals during war. 1869: Map of Napoleon’s
disastrous Russian campaign: Charles Minard: notable for representation in two dimensions of six types of data: # of Napoleon troops,
distance, temperature, latitude, longitude, direction of the travel and the location relative to the specific dates. 1854: John Snow’s map of
deaths caused by cholera in Soho, London: bulk of deaths used same municipal water pump
Explanatory visualization – have historical data, trying to explore and see what the data is telling you, exploring for answers. Confirmatory
visualization – you are expecting a correlation and you’re looking at the visualization to confirm this. Production visualization – you know
exactly what to look for because you already know and understand your production environment. Ex: dashboard graphics. Dynamic
visualization – dynamic graphic showing change over time. Spatial data visualization – most illustrative when shown using maps
Interpretation – a continued process throughout the analysis. Have expectations prior to performing the interpretation. Not completed until
we consider the results in totality. 4 principles. (1) Revisit our original question (2) The primary statistical model result should focus on the
directionality of the relationship between variables, the magnitude of the relationship, and the uncertainty of the relationship, which can be
measured with correlation, p-value, and confidence interval. (3) Develop an overall interpretation, includes but not limited to the totality of
your analysis or the content of what is already known about the subject matter. (4) Consider the implications
Many model results can combine to provide an answer to our question. We can refine a model after interpreting the results
When we utilize a visualization on paper/screen, that visualization is limited to exploring as many variables as we can coherently communicate
in 2 dimensions
Basic Principles of data visualizations: (1) The chart should tell a story. The graph suggests a possible true effect. (2) The chart should have
graphical integrity: it shouldn’t lie. Lie factor should be close to 1. < 1 = understated effect. > 1 = exaggerated effect: means the size of the
effect shown in the graph is bigger than the actual effect in the data. (3) The chart should minimize graphical complexity. The simpler the
better. Tables carry more information in less space than a chart, and is more precise, but they cannot easily show trends. Data ink – amount of
“ink” devoted to data in a chart. Data-ink ratio should be close to 1. <1 = more non-data related ink in graphic. = 1 implies all ink is devoted to
data. Erase ink whenever possible. Can sometimes help tell a richer story. Deleting grid lines in a chart increases the data-ink ratio. When
data-ink ratio is low, chartjunk is likely to be high. Chartjunk – unnecessary visual clutter that doesn’t provide additional insight. Distraction
from the story the chart is supposed to convey. Moiré effects – creates illusion of movement. Stands out in a bad way
In developing spam filter algorithms, we need labeled data of both spam and non-spam emails. Supervised learning require labeled data for
training. Techniques include classification and regression. ML algorithms=linear/logistic regression & decision tree. Unsupervised learning
require unlabeled data, example, clustering. Techniques include clustering and association. Association rule learning – how frequently an item
set occurs in a transaction. AI – a broad science of mimicking human abilities. No embraced everywhere because it can be operationally
expensive. An ideal machine learning process needs highly granular, large volume, and extremely diverse data. Artificial neural networks: their
learning process is similar to our brain. The network is modeled after the human brain in which brain cells work in a network. The input layer
in the network receives the data. Modern update of artificial neural networks is deep learning, not modern are decision tree, logistic
regression, and clustering.
