Finite Sample Results for Autoregressive Processes
Introduction to Autoregressive Processes
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 −𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
• Autoregressive models are simple yet widely used in time series analysis
• Classical theory mainly focuses on asymptotic behavior
• Key point: These models are more subtle than commonly believed
Definition of Autoregressive Process
• Let 𝐵 be the backward shift operator on vectors
• For 𝑢 = (𝑢1,…,𝑢𝑛) in ℝ𝑛, 𝐵𝑢 =(0,𝑢1,…,𝑢𝑛)
• 𝜖 is a mean zero random vector with i.i.d. components
• Definition: 𝑋 in ℝ𝑛 is an autoregressive process of order 𝑝 if:
𝑋 = ∑ 𝜃𝑖
1≤𝑖≤𝑝 𝐵𝑖𝑋 +𝜖
• where 𝜃 = (𝜃1,…,𝜃𝑝) in ℝ𝑝, with 𝜃𝑝≠ 0
Explicit Form of Autoregressive Process
𝑋1= 𝜖1
𝑋2= 𝜃1𝑋1+𝜖2
⋮
𝑋𝑝+1 = 𝜃1𝑋𝑝+𝜃2𝑋𝑝−1 +⋯+𝜃𝑝𝑋1+ 𝜖𝑝+1
⋮
𝑋𝑛= 𝜃1𝑋𝑛−1 +𝜃2𝑋𝑛−2 + ⋯+𝜃𝑝𝑋𝑛−𝑝 +𝜖𝑛
Statistical Questions
• Main focus: Estimation and test procedures
• Goal: Estimate 𝜃 or perform tests on 𝜃 based on observed 𝑋
Least Squares Estimator
• Define matrix 𝐗 = (𝐵𝑋,…,𝐵𝑝𝑋)
• Equation becomes: 𝑋 = 𝐗𝜃 +𝜖
• Least squares estimator 𝜃𝐿𝑆:
𝜃𝐿𝑆 =(𝐗𝑇𝐗)−1𝐗𝑇𝑋
• Note: Can be calculated solely from observed 𝑋
Empirical Autocovariances
• Define empirical autocovariance of order 𝑘 < 𝑛:
𝛾𝑛(𝑘)=1
𝑛∑ 𝑋𝑟
𝑘+1≤𝑟≤𝑛 𝑋𝑟−𝑘
• Important relation: For fixed 𝑖 −𝑗, as 𝑛 → ∞,
⟨𝐵𝑖𝑋,𝐵𝑗𝑋⟩ = 𝑛𝛾𝑛(|𝑖 − 𝑗|)+ 𝑂𝑃(1)
Popular Estimator
• Define matrix 𝛤𝑛= (𝛾𝑛(|𝑖 − 𝑗|))1≤𝑖,𝑗≤𝑝
• Define vector 𝛾𝑛= (𝛾𝑛(𝑖))1≤𝑖≤𝑝
• Popular estimator: 𝜃
𝑛= 𝛤𝑛
−1𝛾𝑛
• Note: 𝜃
𝑛 is asymptotically equivalent to 𝜃𝐿𝑆
Connection to Previous Results
• 𝑋 is a linear function of 𝜖
• Autocovariance 𝛾𝑛(𝑘) is a quadratic form in 𝑋 and 𝜖
• Key insight: Tail probabilities of 𝛾𝑛(𝑘) are relevant to statistics
• Previous results from Chapter 8 provide the right estimates
•
