Statistics tasks need done

91 comparing two populations Part 1 11/03/2020 Part 2 11/04/2020 Homework Due date 11/12/2020 Consider the data on crude oil production in California between 1981 and 2020. Compare X = production in February and Y = production in September, for the years 1981 to 1995 following our example below. Find 99% confidence intervals. First, assume that population variances are known and that X and Y are independent. Second, determine if the data gives support for the idea that X and Y may be independent. And lastly, assume the population variances are unknown and that X and Y are dependent.

Part 1 random variables X = random variable giving class median for class/frequency data X[1], X[2], X[3], … X[N] N experiments EX = random variable giving the expectation values for each experiment EX[1], EX[2], EX[3], … EX[N] expectation values for each experiment VarX = random variable giving the variance for each experiment VarX[1], VarX[2], VarX[3], … VarX[N] variances for each experiment P(X) = probability distribution for X based on a fit to data or empirical frequencies N[k] = total number of data values for experiment k = 1, 2, … N f[k] = frequency for value X[k] of experiment k P(X[k]) = f[k]/N[k] empirical probabilities EX[k] = sum[X[k]] P(X[k]) X[k] X values from experiment k EX[k]2 = sum[X[k]] P(X[k]) X[k]2 VarX[k] = EX[k]2 – (EX[k])2 SDevX[k] = sqrt(VarX[k]) P(EX) = probability distribution for EX EEX = sum[EX[k]] P(EX[k]) EX[k] EEX2 = sum[EX[k]] P(EX[k]) EX[k]2 VarEX = EEX2 – (EEX)2 experimental values EX[k] = (1/N[k]) sum[X[k]] X[k] VarX[k] = (1/(N[k] – 1)) sum[X[k]] (X[k] – EX[k])2 central limit theorem The central limit theorem, when N is sufficiently large, log(P(EX)/w(EX)) = - (1/2) log(VarEX) – (1/2) log(2𝜋) – (1/2) log(e) Z2 where w(Y) is assumed to be a small width for each class median value Y, and the z-value is given as Z = (X – EEX)/(VarEX) Furthermore, EEX = EX VarEX = VarX/N two independent random variables X,Y = two independent random variables comparison of experimental results EX – EY apply central limit theorem E(EX – EY) = EX – EY (when the number of experiments for each of X and Y is large enough) Var(EX – EY) = VarEX + VarEY = VarX/NX + VarY/NY NX = total experiments for X NY = total experiments for Y (consequence of symmetry) P(EX – EY) = normal distribution

This indicates in outline how to find a confidence interval. We give an example problem on the next page.

Part 2 data table (same as for Sec 8.5 on chi square distribution)

our study An obvious problem with the above data is that probably nothing about it can be regarded as associated with a couple of independent random variables, except maybe trivialities. Everything with this data is totally intertwined with a big complex actual picture of evolution in California. Bear this in mind: Our example assumes the two random variables X and Y are independent, and this, at least in this case, is but the wildest fantasy. We also make assumptions to get the above picture applied to the comparison of the two random variables. In this instance, this is an obvious pretty meaningless fabrication. the two random variables for comparison Let’s make a 95% confidence interval for EX – EY, where X = data value for January and Y = data value for July. Here is the relevant data table data table (see above data table for more information) Year January value July value

X Y LIST L1 L2 1981 3.0297 3.1383 1982 3.1352 3.1758 1983 3.1302 3.1880 1984 3.1894 3.2193 1985 3.2492 3.3154 1986 3.4258 3.1790 1987 3.0621 3.0897 1988 3.0547 3.0208 1989 2.8431 2.8143 1990 2.7322 2.6962 step 1 in constructing the confidence interval: find the point estimate (consult your text) E(X – Y) = (1/N) sum[X – Y] X – Y = (1/10) ((3.1383 – 3.0297) + (3.1758 – 3.1352) + …) = (1/N) sum(L1 – L2) = 0.00148 comment We can see that because this is only about 1/3000th the value of X or Y, these two variables are HIGHLY correlated and unlikely to be independent. computation of the population standard deviations We assume the given data values constitute both our “sample” and our “population”. Therefore, we KNOW the exact population standard deviation in this case, which is necessary to use the text set up for the confidence interval in this case. expectation and variance X EX = (1/N) sum(L1) = 3.08516 EX2 = (1/N) sum(L12) = 9.55282 VarX = EX2 – (EX)2 = 0.04361 SDevX = sqrt(VarX) = 0.1860 population standard deviation

SDevX = sqrt((1/(N – 1)) sum[X] (X – EX)2) = sqrt((1/9) sum((L1 – 3.08516)2 ) = 0.1961 sample standard deviation

comment We are seeing a slight difference between population and sample standard deviation due to the N – 1 in place of N in the sample standard deviation. expectation and variance Y EY = (1/N) sum(L2) = 3.08368 EY2 = (1/N) sum(L22) = 9.54210 VarY = EY2 – (EY)2 = 0.03302 SDevY = sqrt(VarY) = 0.1817 population standard deviation SDevY = sqrt((1/(N – 1)) sum[Y] (Y – EY)2)

= sqrt((1/9) sum((L2 – 3.08368)2 ) = 0.1915 sample standard deviation

the standard deviation for E(X – Y) (via the central limit theorem) SDevE(X – Y) = sqrt(VarX/N + VarY/N) = 0.08754 step 2 in constructing the confidence interval: find the margin of error (consult your text) margin of error = SDevE(X – Y) x (critical z-value for the ideal normal distribution) critical z-value for a 95% confidence interval alpha = probability of a type I error = probability that a valid EX[k] for some experiment k will fall outside

the confidence interval = 1 – (decimal percent for confidence interval) = 1 – 0.95 alpha/2 = 0.025 upper and lower critical z-values

Probability of exceeding this z-value must be alpha/2 (and likewise the probability of getting less than the lower critical z-value must also be alpha/2)

Upper critical z-value = - (Lower critical z-value) algorithm for finding the lower critical z-value using the TI calculator

(we use the Solver, but you obviously also can use invNorm: see YouTube videos for instruction.) MATH > up arrow > 0:Solver > up arrow > 2nd VARS (DISTR) > 2:normalcdf(

> -1000, X) – 0.025 > ENTER > -2 (guess) > ALPHA ENTER estimated lower critical z-value = - 1.95996 upper critical z-value = 1.95996 margin of error = (0.08754)(1.95996) = 0.1716 step 3 in constructing the confidence interval: lower and upper limits of the interval lower limit of the interval = E(X – Y) – (margin of error) = 0.00148 – 0.1716 = -0.1701

upper limit of the interval = E(X – Y) + (margin of error) = 0.1731 confidence interval -0.1701 < E(X – Y) < 0.1731

(the true value, apart from our estimate, 0.00148, lies in (-0.1701, 0.1731) at a 95% confidence level.) comment Since we have the actual population in this case, we can compute the actual value of the variance of E(X – Y) and check to see if our independence assumption was reasonable E(X – Y) = 0.00148 E(X – Y)2 = (1/10) sum((L1 – L2)2) = 0.00870 Var(X – Y) = E(X – Y)2 – (E(X – Y))2 = 0.008698 SDev(X – Y) = sqrt(Var(X – Y)) = 0.09327 The SDev for X – Y based on independence is sqrt(VarX + VarY) = sqrt(0.04361 + 0.03302) = 0.2768 This is three times larger than the actual. Therefore, the assumption of independence in this case is not valid. But we cautioned you about this. This is merely a toy problem example. We continue discussion of comparison of X and Y next time.

Part 3 the loss of quality in experimental data For various reasons, some of which we have discussed, one usually cannot just construct a confidence interval as a result of experimental data and truly expect it to be very meaningful without usually investing significantly in skill, resources and understanding to obtain ideal or near ideal conditions. The key problem is the philosophical background of the central limit theorem: It is usually a very unrealistic idealization from actual experimental situations. You typically need a great amount of extremely high quality data and usually it must be accumulated over a fairly short time. These conditions force us to acknowledge inherent weaknesses in experimental data and surveys. t distribution We have noted that one way to accommodate errors, poor experimental design or equipment, lack of adequate skill or understanding, just plain stupidity and ignorance, presence of extraneous signals, outliers of unknown significance, etc. is to use a random variable with potentially broader tails such as the hypergeometric distribution or the t-distribution. This approach attempts to capture a fair signal strength in the presence of a non-ideal data spread and significant noise. steps with the t-distribution (X,Y independent random variables) These are identical to those of the normal distribution given above. step 1 in constructing the confidence interval: find the point estimate, namely E(X – Y) step 2 in constructing the confidence interval: find the margin of error

margin of error = SDevE(X – Y) x (critical z-value for the ideal t distribution) Z = (X – E(X – Y))/SDevE(X – Y) VarE(X – Y) = VarX/NX + VarY/NY

SDevE(X – Y) = sqrt(VarE(X – Y)) critical z-value for alpha/2 = 0.025 (95% confidence interval)

algorithm for finding the lower critical z-value using the TI calculator (we use the Solver, but you obviously also can use invNorm: see YouTube videos for instruction.) MATH > up arrow > 0:Solver > up arrow > 2nd VARS (DISTR) > 6:tcdf(

> -1000, X, df) – 0.025 > ENTER > -2 (guess) > ALPHA ENTER (use df = smaller of NX – 1 and NY – 1 to represent weakness of experimental data. The text gives a subtler formula as well that you might want to use:

df = ((NX – 1)-1WX + (NY – 1)-1WY)-1 WX = (VarEX/VarE(X – Y))2 WY = (VarEY/VarE(X – Y))2 VarE(X – Y) = VarEX + VarEY

(Independence of X and Y, carries over to EX and EY) VarEX = VarX/NX (central limit theorem) VarEY = VarY/NY

(This represents an averaging of NX – 1 and NY – 1 since WX + WY ≤ 1. We have the following result:

(smaller of NX – 1 and NY – 1) ≤ ((NX – 1)-1WX + (NY – 1)-1WY)-1 First, this inequalitiy tells us that the smaller of NX – 1 and NY – 1 sets the lower limit on loss of information in an experiment. Second, the emphasis on the particular value NX – 1 or NY – 1 results from the weighting factors, WX and WY. It is clear that WX relates to VarEX and WY to VarEY, because

as we have mentioned, many aspects of loss of information from data relates to how broad the distribution, and this is measured by the SDevX, i.e. the square root of the variance. Why are we squaring VarEX, or VarEY, and why do we divide by VarE(X – Y)? Partly, this is because VarEX/VarE(X – Y) is less than 1, i.e. does not magnify the loss of information. Nevertheless, the whole philosophy here, while it has a mathematical context in some depth, is worth stating more explicitly: It IS philosophy, and a rather arbitrary choice in the end. But we will discuss it no further here.

step 3 in constructing the confidence interval: lower and upper limits of the interval

lower limit of the interval = E(X – Y) – (margin of error) upper limit of the interval = E(X – Y) + (margin of error) pooled variance There is a bit of a different perspective too, on the experimental comparison of X and Y, when EX may differ from EY but in terms of sample or experimental variance, we suspect ideally that the variances, VarEX and VarEY are equal. This is not an uncommon phenomenon. Here is the usual experimental variance VarX = (1/(NX – 1)) sum[X] (X – EX)2 VarY = (1/(NY – 1)) sum[Y] (Y – EY)2 The pooled variance, in the case in which we expect population variances to be equal (an ideal but sometimes reasonable simplification), is VarPooled = (1/((NX – 1) + (NY – 1))) (sum[X] (X – EX)2 + sum[Y] (Y – EY)2) = ((NX – 1) VarX + (NY – 1) VarY)/(NX + NY – 2) We use this pooled variance as a common approximation for VarX and VarY, since we believe the population variances of X and Y to be equal. The central limit theorem yields VarE(X – Y) = VarX/NX + VarY/NY = VarPooled x (1/NX + 1/NY) Thus, in construction of a confidence interval with pooled variance we would use margin of error = SDevE(X – Y) x (critical z-value for the ideal t distribution) = sqrt(VarPooled) x sqrt(1/NX + 1/NY) the case when X and Y are not independent It is of course to address this case, as this is the situation that we are often in. The main points about independence are E(X – Y) = EX – EY Var(X – Y) = VarX + VarY i.e. we make a separation where we can consider P(X) and P(Y) separately. On the other hand, with the case of loss of independence we have E(X – Y) = EX - EY E(X – Y)2 = EX2 – 2EXY + EY2 Var(X – Y) = E(X – Y)2 – (EX – EY)2 = VarX + VarY – 2 CovXY CovXY = EXY – (EX)(EY) covariance of X and Y = 0 when X and Y are independent The definition of EXY is EXY = sum[XY] P(XY) XY We know the values of X and Y, and therefore the product XY. But, how does the probability distribution P(XY) relate to P(X) and P(Y)? When we lack independence, there will be whole aspects of correlations to the distribution of the random product XY that we will not be able to detect from just P(X) and P(Y) alone. In general, P(XY = c) = P(Y = c/X) = (P(Y = c/X)/P(X)) x P(X) = P(Y|X) P(X), Y = c = one of its data values

i.e. this probability distribution depends on conditional probability and when X and Y are independent, P(Y|X) = P(Y) and P(XY) = P(X)P(Y) reduces to knowledge of just P(X) and P(Y) separately. When we lose independence P(Y|X) is dependent on the particular values of X, as well as Y. modeling X and Y dependence for X – Y comparison We assume that P(X – Y) exists, the probability distribution for the difference X – Y. Then, the whole apparatus of confidence intervals we discussed above works. The sole difference is that we must use as variance, Var(X – Y) = VarX + VarY – 2 CovXY Plus the experimental variance is Var(X – Y) = (1/(N – 1)) sum[X – Y] (X – Y – E(X – Y))2 and this is what we use in the determination of the margin of error for confidence intervals. So, next time, we will finish up this section, give you homework for this section, and finish up.

Part 4 comment about dependence The bottom line about independence is P(XY) = P(X)P(Y) This is equivalent to P(X|Y) = P(X) or P(Y|X) = P(Y). In the case of dependence we can only say P(XY) = P(X|Y)P(Y) or P(XY) = P(Y|X)P(X) When we previously studied Independence in classical probability its significance was unclear. Now, in the context of fits to data distributions, we see quite clearly that this does not relate to peak values: E(X – Y) = EX – EY or E(X + Y) = EX + EY But for variance Var(X – Y) = VarX + VarY – 2 CovXY or Var(X + Y) = VarX + VarY + 2 CovXY and CovXY is a result of dependence CovXY = EXY – (EX)(EY) (=0 in the case that X and Y are independent) EXY = sum[XY] P(XY) XY Thus dependence is entirely associated with the variance, and unrelated to expectation values. three dependent random variables W, X and Y For this, we have P(XY), P(YW) and P(WX) to consider. For these each we have P(XY) = P(X|Y)P(Y) for example, as above. Now, we can consider P(WXY = c) = (P(W = c/XY)/P(XY)) x P(XY) = P(W|XY)P(X|Y)P(Y), WXY = c Also, E(W + X + Y) = EW + EX + EY Var(W + X + Y) = VarW + VarX + VarY + 2 CovWX + 2 CovWY + 2 CovXY The relation P(WXY) = P(W|XY)P(X|Y)P(Y) If W, X and Y are independent, then P(W|XY) = P(W), P(X|Y) = P(X) and P(WXY) = P(X)P(Y)P(W) Thus the framework we developed can be used for comparing any number of random variables. confidence interval, with X and Y assumed to be dependent We have covered a lot of ground, but for our example, we just want to repeat the analysis and this time assume dependence of X and Y, which is MUCH better supported by the actual data. data table (see above data table for more information) Year January value July value

X Y LIST L1 L2 1981 3.0297 3.1383 1982 3.1352 3.1758 1983 3.1302 3.1880 1984 3.1894 3.2193 1985 3.2492 3.3154 1986 3.4258 3.1790 1987 3.0621 3.0897 1988 3.0547 3.0208 1989 2.8431 2.8143

1990 2.7322 2.6962 95% confidence interval using the t distribution (recall N = 10) step 1 in constructing the confidence interval: find the point estimate As remarked, the point estimate is not affected by independence vs dependence: E(X – Y) = (1/N) sum[X – Y] X – Y = (1/10) ((3.1383 – 3.0297) + (3.1758 – 3.1352) + …) = (1/N) sum(L1 – L2) = 0.00148 This is the same as before. step 2 in constructing the confidence interval: find the margin of error margin of error = SDevE(X – Y) x (critical z-value for the ideal t distribution) Z = (X – E(X – Y))/SDevE(X – Y) VarE(X – Y) = Var(X – Y)/N

Var(X – Y) = (1/(N – 1)) sum[X – Y] (X – Y – E(X – Y))2 = (1/9) sum((L1 – L2 – 0.00148)2) = 0.009665

SDevE(X – Y) = sqrt(VarE(X – Y)) = 0.09915 critical z-value for alpha/2 = 0.025 (95% confidence interval)

algorithm for finding the lower critical z-value using the TI calculator (we use the Solver, but you obviously also can use invNorm: see YouTube videos for instruction.) MATH > up arrow > 0:Solver > up arrow > 2nd VARS (DISTR) > 6:tcdf(

> -1000, X, df) – 0.025 > ENTER > -2 (guess) > ALPHA ENTER (use df = smaller of NX – 1 and NY – 1 to represent weakness of experimental data. df = 9 lower critical z-value = -2.2622 upper critical z-value = 2.2622

margin of error = 0.09915 x 2.2622 = 0.2243 step 3 in constructing the confidence interval: lower and upper limits of the interval lower limit of the interval = E(X – Y) – (margin of error) = 0.00148 – 0.2243 = -0.2228 upper limit of the interval = E(X – Y) + (margin of error) = 0.2258 95% confidence interval -0.2228 < E(X – Y) < 0.2258 This is a more appropriate result, as we saw before that the data just does not support the idea that X and Y are independent.