EXAM-Introduction to scientific computing math

A First Course in NumeriCAl methods

CS07_Ascher-Greif_FM-a.indd 1 5/13/2011 1:43:24 PMDownloaded 17 Feb 2012 to 129.174.55.245. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

the siAm series on Computational science and engineering publishes research monographs, advanced undergraduate- or graduate-level textbooks, and other volumes of interest to an interdis- ciplinary Cs&e community of computational mathematicians, computer scientists, scientists, and engineers. the series includes both introductory volumes aimed at a broad audience of mathematically motivated readers interested in understanding methods and applications within computational science and engineering and monographs reporting on the most recent developments in the field. the series also includes volumes addressed to specific groups of professionals whose work relies extensively on computational science and engineering.

siAm created the Cs&e series to support access to the rapid and far-ranging advances in computer modeling and simulation of complex problems in science and engineering, to promote the interdis- ciplinary culture required to meet these large-scale challenges, and to provide the means to the next generation of computational scientists and engineers.

Computational Science & Engineering

Editor-in-Chief donald estep Colorado state university Editorial Board omar Ghattas university of texas at Austin

max Gunzburger Florida state university

des higham university of strathclyde

michael holst university of California, san diego

max d. morris iowa state university

Alex Pothen Purdue university

Padma raghavan Pennsylvania state university

Karen Willcox massachusetts institute of technology

Series Volumes

Ascher, uri m. and Greif, Chen, A First Course in Numerical Methods

layton, William, Introduction to the Numerical Analysis of Incompressible Viscous Flows

Ascher, uri m., Numerical Methods for Evolutionary Differential Equations

Zohdi, t. i., An Introduction to Modeling and Simulation of Particulate Flows

Biegler, lorenz t., omar Ghattas, matthias heinkenschloss, david Keyes, and Bart van Bloemen Waanders, editors, Real-Time PDE-Constrained Optimization

Chen, Zhangxin, Guanren huan, and Yuanle ma, Computational Methods for Multiphase Flows in Porous Media

shapira, Yair, Solving PDEs in C++: Numerical Methods in a Unified Object-Oriented Approach

CS07_Ascher-Greif_FM-a.indd 2 5/13/2011 1:43:24 PMDownloaded 17 Feb 2012 to 129.174.55.245. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

A First Course in NumeriCAl methods

uri m. Ascher Chen Greif the university of British Columbia Vancouver, British Columbia, Canada

society for industrial and Applied mathematics Philadelphia

CS07_Ascher-Greif_FM-a.indd 3 5/13/2011 1:43:24 PMDownloaded 17 Feb 2012 to 129.174.55.245. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

Copyright © 2011 by the society for industrial and Applied mathematics.

10 9 8 7 6 5 4 3 2 1

All rights reserved. Printed in the united states of America. No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the publisher. For information, write to the society for industrial and Applied mathematics, 3600 market street, 6th Floor, Philadelphia, PA 19104-2688 usA.

trademarked names may be used in this book without the inclusion of a trademark symbol. these names are used in an editorial context only; no infringement of trademark is intended. mAtlAB is a registered trademark of the mathWorks, inc. For mAtlAB product information, please contact the mathWorks, inc., 3 Apple hill drive, Natick, mA 01760-2098 usA, 508-647-7000, Fax: 508-647-7001, [email protected], www.mathworks.com.

Figures 4.9a and 11.16 reprinted with permission from ACm. Figure 7.1 reproduced with permission from World scientific Publishing Co. Pte. ltd. Figure 13.7 reprinted with permission from mos and springer. Figures 16.12 and 16.13 reprinted with permission from elsevier.

Library of Congress Cataloging-in-Publication Data

Ascher, u. m. (uri m.), 1946- A first course in numerical methods / uri m. Ascher, Chen Greif. p. cm. -- (Computational science and engineering series) includes bibliographical references and index. isBN 978-0-898719-97-0 1. Numerical calculations--data processing. 2. Numerical analysis. 3. Algorithms. i. Greif, Chen, 1965- ii. title. QA297.A748 2011 518’.4--dc22 2011007041

is a registered trademark.

CS07_Ascher-Greif_FM-a.indd 4 5/13/2011 1:43:24 PMDownloaded 17 Feb 2012 to 129.174.55.245. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

We dedicate this book to our tolerant families.

v

CS07_Ascher-Greif_FM-a.indd 5 5/13/2011 1:43:24 PMDownloaded 17 Feb 2012 to 129.174.55.245. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php

✐

✐

✐

✐

✐

✐

✐

✐

Contents

List of Figures xi

List of Tables xix

Preface xxi

1 Numerical Algorithms 1 1.1 Scientific computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Numerical algorithms and errors . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Algorithm properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.5 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2 Roundoff Errors 17 2.1 The essentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 Floating point systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3 Roundoff error accumulation . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4 The IEEE standard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.6 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3 Nonlinear Equations in One Variable 39 3.1 Solving nonlinear equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 Bisection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3 Fixed point iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.4 Newton’s method and variants . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.5 Minimizing a function in one variable . . . . . . . . . . . . . . . . . . . . . . 55 3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.7 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4 Linear Algebra Background 65 4.1 Review of basic concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.2 Vector and matrix norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.3 Special classes of matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.4 Singular values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.5 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.7 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

vii

✐

✐

✐

✐

✐

✐

✐

✐

viii Contents

5 Linear Systems: Direct Methods 93 5.1 Gaussian elimination and backward substitution . . . . . . . . . . . . . . . . 94 5.2 LU decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.3 Pivoting strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 5.4 Efficient implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.5 The Cholesky decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.6 Sparse matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 5.7 Permutations and ordering strategies . . . . . . . . . . . . . . . . . . . . . . 122 5.8 Estimating errors and the condition number . . . . . . . . . . . . . . . . . . . 127 5.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 5.10 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

6 Linear Least Squares Problems 141 6.1 Least squares and the normal equations . . . . . . . . . . . . . . . . . . . . . 141 6.2 Orthogonal transformations and QR . . . . . . . . . . . . . . . . . . . . . . . 151 6.3 Householder transformations and Gram–Schmidt orthogonalization . . . . . . 157 6.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 6.5 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

7 Linear Systems: Iterative Methods 167 7.1 The need for iterative methods . . . . . . . . . . . . . . . . . . . . . . . . . 167 7.2 Stationary iteration and relaxation methods . . . . . . . . . . . . . . . . . . . 173 7.3 Convergence of stationary methods . . . . . . . . . . . . . . . . . . . . . . . 179 7.4 Conjugate gradient method . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 7.5 *Krylov subspace methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 7.6 *Multigrid methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 7.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.8 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

8 Eigenvalues and Singular Values 219 8.1 The power method and variants . . . . . . . . . . . . . . . . . . . . . . . . . 219 8.2 Singular value decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 229 8.3 General methods for computing eigenvalues and singular values . . . . . . . . 236 8.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 8.5 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

9 Nonlinear Systems and Optimization 251 9.1 Newton’s method for nonlinear systems . . . . . . . . . . . . . . . . . . . . . 251 9.2 Unconstrained optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 9.3 *Constrained optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 9.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 9.5 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293

10 Polynomial Interpolation 295 10.1 General approximation and interpolation . . . . . . . . . . . . . . . . . . . . 295 10.2 Monomial interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 10.3 Lagrange interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 10.4 Divided differences and Newton’s form . . . . . . . . . . . . . . . . . . . . . 306 10.5 The error in polynomial interpolation . . . . . . . . . . . . . . . . . . . . . . 313 10.6 Chebyshev interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 10.7 Interpolating also derivative values . . . . . . . . . . . . . . . . . . . . . . . 319

✐

✐

✐

✐

✐

✐

✐

✐

Contents ix

10.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 10.9 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

11 Piecewise Polynomial Interpolation 331 11.1 The case for piecewise polynomial interpolation . . . . . . . . . . . . . . . . 331 11.2 Broken line and piecewise Hermite interpolation . . . . . . . . . . . . . . . . 333 11.3 Cubic spline interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 11.4 Hat functions and B-splines . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 11.5 Parametric curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 11.6 *Multidimensional interpolation . . . . . . . . . . . . . . . . . . . . . . . . 353 11.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 11.8 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

12 Best Approximation 365 12.1 Continuous least squares approximation . . . . . . . . . . . . . . . . . . . . 366 12.2 Orthogonal basis functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 12.3 Weighted least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 12.4 Chebyshev polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377 12.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 12.6 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382

13 Fourier Transform 383 13.1 The Fourier transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 13.2 Discrete Fourier transform and trigonometric interpolation . . . . . . . . . . . 388 13.3 Fast Fourier transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 13.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 13.5 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406

14 Numerical Differentiation 409 14.1 Deriving formulas using Taylor series . . . . . . . . . . . . . . . . . . . . . . 409 14.2 Richardson extrapolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 14.3 Deriving formulas using Lagrange polynomial interpolation . . . . . . . . . . 415 14.4 Roundoff and data errors in numerical differentiation . . . . . . . . . . . . . 420 14.5 *Differentiation matrices and global derivative approximation . . . . . . . . . 426 14.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 14.7 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438

15 Numerical Integration 441 15.1 Basic quadrature algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 442 15.2 Composite numerical integration . . . . . . . . . . . . . . . . . . . . . . . . 446 15.3 Gaussian quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 15.4 Adaptive quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 15.5 Romberg integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 15.6 *Multidimensional integration . . . . . . . . . . . . . . . . . . . . . . . . . . 472 15.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 15.8 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479

16 Differential Equations 481 16.1 Initial value ordinary differential equations . . . . . . . . . . . . . . . . . . . 481 16.2 Euler’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 16.3 Runge–Kutta methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493

✐

✐

✐

✐

✐

✐

✐

✐

x Contents

16.4 Multistep methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500 16.5 Absolute stability and stiffness . . . . . . . . . . . . . . . . . . . . . . . . . 507 16.6 Error control and estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 515 16.7 *Boundary value ODEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520 16.8 *Partial differential equations . . . . . . . . . . . . . . . . . . . . . . . . . . 524 16.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 16.10 Additional notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537

Bibliography 539

Index 543

✐

✐

✐

✐

✐

✐

✐

✐

List of Figures

1.1 Scientific computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 A simple instance of numerical differentiation: the tangent f ′(x0) is approximated

by the chord ( f (x0 +h)− f (x0))/h. . . . . . . . . . . . . . . . . . . . . . . . 6 1.3 The combined effect of discretization and roundoff errors. The solid curve inter-

polates the computed values of | f ′(x0)− f (x0+h)− f (x0)h | for f (x) = sin(x), x0 = 1.2. Also shown in dash-dot style is a straight line depicting the discretization error without roundoff error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.4 An ill-conditioned problem of computing output values y given in terms of input values x by y = g(x): when the input x is slightly perturbed to x̄ , the result ȳ = g(x̄) is far from y. If the problem were well-conditioned, we would be expecting the distance between y and ȳ to be more comparable in magnitude to the distance between x and x̄ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.5 An instance of a stable algorithm for computing y = g(x): the output ȳ is the exact result, ȳ = g(x̄), for a slightly perturbed input, i.e., x̄ which is close to the input x . Thus, if the algorithm is stable and the problem is well-conditioned, then the computed result ȳ is close to the exact y. . . . . . . . . . . . . . . . . . . . 12

2.1 A double word (64 bits) in the standard floating point system. The blue bit is for sign, the magenta bits store the exponent, and the green bits are for the fraction. 19

2.2 The “almost random” nature of roundoff errors. . . . . . . . . . . . . . . . . . 21 2.3 Picture of the floating point system described in Example 2.8. . . . . . . . . . . 26

3.1 Graphs of three functions and their real roots (if there are any): (i) f (x) = sin(x) on [0,4π], (ii) f (x)= x3−30x2+2552 on [0,20], and (iii) f (x)= 10cosh(x/4)− x on [−10,10]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2 Fixed point iteration for x = e−x , starting from x0 = 1. This yields x1 = e−1, x2 = e−e−1 , . . . . Convergence is apparent. . . . . . . . . . . . . . . . . . . . . . 47

3.3 The functions x and 2cosh(x/4) meet at two locations. . . . . . . . . . . . . . 48 3.4 Newton’s method: the next iterate is the x-intercept of the tangent line to f at the

current iterate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.5 Graph of an anonymous function; see Exercise 18. . . . . . . . . . . . . . . . . 63

4.1 Intersection of two straight lines: a11x1 +a12x2 = b1 and a21x1 +a22x2 = b2. . 66 4.2 Eigenvalues in the complex plane, Example 4.5. Note that the complex ones

arrive in pairs: if λ is an eigenvalue, then so is λ̄. Also, here all eigenvalues have nonpositive real parts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.3 The “unit circle” according to the three norms, �1, �2, and �∞. Note that the diamond is contained in the circle, which in turn is contained in the square. . . . 75

xi

✐

✐

✐

✐

✐

✐

✐

✐

xii List of Figures

4.4 The Cartesian coordinate system as a set of orthonormal vectors. . . . . . . . . 80 4.5 Singular value decomposition: Am×n = Um×m�m×n V Tn×n . . . . . . . . . . . . 81 4.6 Data fitting by a cubic polynomial using the solution of a linear system for the

polynomial’s coefficients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.7 A discrete mesh for approximating a function and its second derivative in Exam-

ple 4.17. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.8 Recovering a function v satisfying v(0) = v(1) = 0 from its second derivative.

Here N = 507. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.9 Example 4.18: a point cloud representing (a) a surface in three-dimensional space,

and (b) together with its unsigned normals. . . . . . . . . . . . . . . . . . . . . 89

5.1 Gaussian elimination for the case n = 4. Only areas of potentially nonzero entries are shown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.2 LU decomposition for the case n = 10. Only areas of potentially nonzero entries in the square matrices L and U are shown. . . . . . . . . . . . . . . . . . . . . 102

5.3 Compressed row form of the matrix in Example 5.14. Shown are the vectors i, j,v. The arrows and the “eof” node are intended merely for illustration of where the elements of the vector i point to in j and correspondingly in v. . . . . . . . . . . 118

5.4 A banded matrix for the case n = 10, p = 2, q = 3. Only areas of potentially nonzero entries are shaded in green. . . . . . . . . . . . . . . . . . . . . . . . . 120

5.5 Graphs of the matrices A (left) and B (right) of Example 5.15. . . . . . . . . . 125 5.6 Sparsity pattern of a certain symmetric positive definite matrix A (left), its RCM

ordering (middle), and approximate minimum degree ordering (right). . . . . . 126 5.7 Sparsity pattern of the Cholesky factors of A (left), its RCM ordering (middle),

and approximate minimum degree ordering (right). . . . . . . . . . . . . . . . 127 5.8 Stretching the unit circle by a symmetric positive definite transformation. . . . . 133

6.1 Matrices and vectors and their dimensions in �2 data fitting. . . . . . . . . . . . 143 6.2 Discrete least squares approximation. . . . . . . . . . . . . . . . . . . . . . . . 145 6.3 Linear regression curve (in blue) through green data. Here, m = 25, n = 2. . . . 148 6.4 The first 5 best polynomial approximations to f (t) = cos(2π t) sampled at 0 : .05 :

1. The data values appear as red circles. Clearly, p4 fits the data better than p2, which in turn is a better approximation than p0. Note p2 j+1 = p2 j . . . . . . . . 150

6.5 Ratio of execution times using QR vs. normal equations. The number of rows for each n is 3n +1 for the upper curve and n +1 for the lower one. . . . . . . . . 157

6.6 Householder reflection. Depicted is a 20×5 matrix A after 3 reflections. . . . . 160 7.1 A two-dimensional cross section of a three-dimensional domain with a square

grid added. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 7.2 A two-dimensional grid with grid function values at its nodes, discretizing the

unit square. The length of each edge of the small squares is the grid width h, while their union is a square with edge length 1. The locations of ui, j and those of its neighbors that are participating in the difference formula (7.1) are marked in red. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.3 The sparsity pattern of A for Example 7.1 with N = 10. Note that there are nz= 460 nonzeros out of 10,000 matrix locations. . . . . . . . . . . . . . . . . 172

7.4 Red-black ordering. First sweep simultaneously over the black points, then over the red. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

7.5 Example 7.10, with N = 31: convergence behavior of various iterative schemes for the discretized Poisson equation. . . . . . . . . . . . . . . . . . . . . . . . 185

✐

✐

✐

✐

✐

✐

✐

✐

List of Figures xiii

7.6 Sparsity patterns for the matrix of Example 7.1 with N = 8: top left, the IC factor F with no fill-in (IC(0)); top right, the product F FT ; bottom left, the full Cholesky factor G; bottom right, the IC factor with drop tolerance .001. See Figure 7.3 for the sparsity pattern of the original matrix. . . . . . . . . . . . . . 189

7.7 Iteration progress for CG, PCG with the IC(0) preconditioner and PCG with the IC preconditioner using drop tolerance tol= 0.01. . . . . . . . . . . . . . . . 190

7.8 Convergence behavior of restarted GMRES with m = 20, for a 10,000× 10,000 matrix that corresponds to the convection-diffusion equation on a 100×100 uni- form mesh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

7.9 The polynomials that are constructed in the course of three CG iterations for the small linear system of Examples 7.9 and 7.14. The values of the polynomials at the eigenvalues of the matrix are marked on the linear, quadratic, and cubic curves. 202

7.10 An illustration of the smoothing effect, using damped Jacobi with ω= 0.8 applied to the Poisson equation in one dimension. . . . . . . . . . . . . . . . . . . . . 205

7.11 Convergence behavior of various iterative schemes for the Poisson equation (see Example 7.17) with n = 2552. . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

7.12 Example 7.17: number of iterations (top panel) and CPU times (bottom panel) required to achieve convergence to tol= 1.e-6 for the Poisson problem of Exam- ple 7.1 (page 168) with N = 2l −1, l = 5,6,7,8,9. . . . . . . . . . . . . . . . . 209

8.1 Convergence behavior of the power method for two diagonal matrices, Exam- ple 8.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

8.2 A toy network for the PageRank Example 8.3. . . . . . . . . . . . . . . . . . . 224 8.3 Things that can go wrong with the basic model: depicted are a dangling node

(left) and a terminal strong component featuring a cyclic path (right). . . . . . . 225 8.4 Convergence behavior of the inverse iteration in Example 8.4. . . . . . . . . . . 229 8.5 Original 200×320 pixel image of a clown. . . . . . . . . . . . . . . . . . . . . 232 8.6 A rank-20 SVD approximation of the image of a clown. . . . . . . . . . . . . . 232 8.7 The result of the first stage of a typical eigensolver: a general upper Hessenberg

matrix for nonsymmetric matrices (left) and a tridiagonal form for symmetric matrices (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

8.8 The result of the first stage of the computation of the SVD is a bi-diagonal matrix C (left). The corresponding tridiagonal matrix CT C is given on the right. . . . . 243

8.9 Mandrill image and a drawing by Albrecht Dürer; see Example 11. . . . . . . . 248

9.1 A parabola meets a circle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 9.2 The point x, the direction p, and the point x+p. . . . . . . . . . . . . . . . . . 254 9.3 Two solutions for a boundary value ODE. . . . . . . . . . . . . . . . . . . . . . 257 9.4 The function x21 + x42 +1 has a unique minimum at the origin (0,0) and no maxi-

mum. Upon flipping it, the resulting function −(x21 + x42 +1) would have a unique maximum at the origin (0,0) and no minimum. . . . . . . . . . . . . . . . . . . 258

9.5 The function of Example 9.5 has a unique minimum at x∗ = (3, .5)T as well as a saddle point at x̂ = (0,1)T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

9.6 Convergence behavior of gradient descent and conjugate gradient iterative schemes for the Poisson equation of Example 7.1. . . . . . . . . . . . . . . . . . . . . . 266

9.7 Nonlinear data fitting; see Example 9.8. . . . . . . . . . . . . . . . . . . . . . 270 9.8 Equality and inequality constraints. The feasible set consists of the points on

the thick red line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 9.9 A feasible set with a nonempty interior, and level sets of φ; larger ellipses

signify larger values of φ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

✐

✐

✐

✐

✐

✐

✐

✐

xiv List of Figures

9.10 One equality constraint and the level sets of φ. At x∗ the gradient is orthogonal to the tangent of the constraint. . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

9.11 Center path in the LP primal feasibility region. . . . . . . . . . . . . . . . . . . 278 9.12 Two minimum norm solutions for an underdetermined linear system of equations.

The �1 solution is nicely sparse. . . . . . . . . . . . . . . . . . . . . . . . . . . 286 9.13 A depiction of the noisy function (in solid blue) and the function to be recovered

(in red) for Exercise 17. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

10.1 Different interpolating curves through the same set of points. . . . . . . . . . . 296 10.2 Quadratic and linear polynomial interpolation. . . . . . . . . . . . . . . . . . . 299 10.3 The quadratic Lagrange polynomials L0(x), L1(x), and L2(x) based on points

x0 = 1, x1 = 2, x2 = 4, used in Example 10.2. . . . . . . . . . . . . . . . . . . 303 10.4 The Lagrange polynomial L2(x) for n = 5. Guess what the data abscissae xi are. 304 10.5 The interpolating polynomials p2 and p3 for Example 10.4. . . . . . . . . . . . 310 10.6 Global polynomial interpolation at uniformly spaced abscissae can be bad. Here

the blue curve with one maximum point is the Runge function of Example 10.6, and the other curves are polynomial interpolants of degree n. . . . . . . . . . . 317

10.7 Polynomial interpolation at Chebyshev points, Example 10.6. Results are much improved as compared to Figure 10.6, especially for larger n. . . . . . . . . . . 317

10.8 Top panel: the function of Example 10.7 is indistinguishable from its polynomial interpolant at 201 Chebyshev points. Bottom panel: the maximum polynomial interpolation error as a function of the polynomial degree. When doubling the degree from n = 100 to n = 200 the error decreases from unacceptable (> 1) to almost rounding unit level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318

10.9 A quadratic interpolant p2(x) satisfying p2(0) = 1.5, p′2(0) = 1, and p2(5) = 0. . 320 10.10 The osculating Hermite cubic for ln(x) at the points 1 and 2. . . . . . . . . . . . 322 10.11 Quadratic and linear polynomial interpolation. . . . . . . . . . . . . . . . . . . 324

11.1 A piecewise polynomial function with break points ti = i , i = 0,1, . . . ,6. . . . . 333 11.2 Data and their broken line interpolation. . . . . . . . . . . . . . . . . . . . . . 334 11.3 Matching si ,s′i , and s′′i at x = xi with values of si−1 and its derivatives at the same

point. In this example, xi = 0 and yi = 1. . . . . . . . . . . . . . . . . . . . . . 338 11.4 Not-a-knot cubic spline interpolation for the Runge Example 10.6 at 20 equidis-

tant points. The interval has been rescaled to be [0,1]. . . . . . . . . . . . . . . 340 11.5 The interpolant of Example 11.6. . . . . . . . . . . . . . . . . . . . . . . . . . 341 11.6 Hat functions: a compact basis for piecewise linear approximation. . . . . . . . 346 11.7 Basis functions for piecewise Hermite polynomials. . . . . . . . . . . . . . . . 348 11.8 B-spline basis for the C2 cubic spline. . . . . . . . . . . . . . . . . . . . . . . 349 11.9 Parametric broken-line interpolation. . . . . . . . . . . . . . . . . . . . . . . . 350 11.10 Parametric polynomial interpolation. . . . . . . . . . . . . . . . . . . . . . . . 351 11.11 A simple curve design using 11 Bézier polynomials. . . . . . . . . . . . . . . . 352 11.12 Bilinear interpolation. Left green square: data are given at the unit square’s cor-

ners (red points), and bilinear polynomial values are desired at mid-edges and at the square’s middle (blue points). Right blue grid: data are given at the coarser grid nodes (blue points) and a bilinear interpolation is performed to obtain val- ues at the finer grid nodes, yielding values fi, j for all i = 0,1,2, . . . , Nx , j = 0,1,2, . . . , Ny . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355

11.13 Triangle mesh in the plane. This one is MATLAB’s data set trimesh2d. . . . 356 11.14 Linear interpolation over a triangle mesh. Satisfying the interpolation conditions

at triangle vertices implies continuity across neighboring triangles. . . . . . . . 357

✐

✐

✐

✐

✐

✐

✐

✐

List of Figures xv

11.15 Triangle surface mesh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 11.16 RBF interpolation of an upsampling of a consolidated point cloud. . . . . . . . 359

12.1 The first five best polynomial approximations to f (x) = cos(2πx). The approxi- mated function is in solid green. Note the similarity to Figure 6.4 of Example 6.3. 369

12.2 The first five Legendre polynomials. You should be able to figure out which curve corresponds to which polynomial. . . . . . . . . . . . . . . . . . . . . . . . . . 372

12.3 Chebyshev polynomials of degrees 4, 5, and 6. . . . . . . . . . . . . . . . . . . 378

13.1 A linear combination of sines and cosines with various k-values up to 110 (top panel), filtered by taking its best least squares trigonometric polynomial approx- imation with l = 10 (bottom panel). This best approximation simply consists of that part of the given function that involves the first 20 basis functions φ j ; thus the higher frequency contributions disappear. . . . . . . . . . . . . . . . . . . . 385

13.2 For the discrete Fourier transform, imagine the red mesh points located on a blue circle with x0 = xn+1 closing the ring. Thus, xn is the left (clockwise) neighbor of x0, x1 is the right (counterclockwise) neighbor of xn+1, and so on. . . . . . . 390

13.3 Trigonometric polynomial interpolation for the hat function with p3(x). . . . . 391 13.4 Trigonometric polynomial interpolation for a smooth function with p3(x) (top

left), p7(x) (top right), p15(x) (bottom left), and p31(x) (bottom right). The ap- proximated function is plotted in dashed green. . . . . . . . . . . . . . . . . . . 393

13.5 Trigonometric polynomial interpolation for the square wave function on [0,2π ] with n = 127. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396

13.6 The unit circle in the complex plane is where the values of eıθ reside. The m = 8 roots of the polynomial equation θ8 = 1, given by e−ı2 jπ/8, j = 0,1, . . . ,7, are displayed as red diamonds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398

13.7 Example 13.9: an observed blurred image b, and the result of a deblurring algo- rithm applied to this data. (Images courtesy of H. Huang [43].) . . . . . . . . . . 402

13.8 Cosine basis interpolation for the function ln(x +1) on [0,2π] with n = 31. . . 404 14.1 Actual error using the three methods of Example 14.1. Note the log-log scale

of the plot. The order of the methods is therefore indicated by the slope of the straight line (note that h is decreased from right to left). . . . . . . . . . . . . . 413

14.2 The measured error roughly equals truncation error plus roundoff error. The for- mer decreases but the latter grows as h decreases. The “ideal roundoff error” is just η/h. Note the log-log scale of the plot. A red circle marks the “optimal h” value for Example 14.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422

14.3 Numerical differentiation of noisy data. On the left panel, sin(x) is perturbed by 1% noise. On the right panel, the resulting numerical differentiation is a disaster. 425

14.4 An image with noise and its smoothed version. Numerical differentiation must not be applied to the original image. . . . . . . . . . . . . . . . . . . . . . . . 426

14.5 Maximum absolute errors for the first and second derivatives of f (x)= ex sin(10x) on the interval [0,π] at the Chebyshev extremum points, i.e., using the Chebyshev differentiation matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429

14.6 Maximum absolute errors for the first and second derivatives of f (x) = e−5x2 on the interval [−2π ,2π]. The two left subplots use FFT, and the right subplots use the Chebyshev differentiation matrix. The errors are evaluated at the correspond- ing data locations (also called collocation points). . . . . . . . . . . . . . . . . 431

14.7 This “waterfall” plot depicts the progress of two initial pulses in time; see Ex- ample 14.14. The two solitons merge and then split with their form intact. Rest assured that they will meet (and split) again many times in the future. . . . . . . 434

✐

✐

✐

✐

✐

✐

✐

✐

xvi List of Figures

15.1 Area under the curve. Top left (cyan): for f (x) that stays nonnegative, I f equals the area under the function’s curve. Bottom left (green): approximation by the trapezoidal rule. Top right (pink): approximation by the Simpson rule. Bottom right (yellow): approximation by the midpoint rule. . . . . . . . . . . . . . . . 444

15.2 Composite trapezoidal quadrature with h = 0.2 for the integral of Figure 15.1. . 448 15.3 Numerical integration errors for the composite trapezoidal and Simpson methods;

see Example 15.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 15.4 Numerical integration errors for the composite trapezoidal and Simpson methods;

see Example 15.4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 15.5 The discontinuous integrand of Example 15.10. . . . . . . . . . . . . . . . . . 459 15.6 The integrand f (x) = 200

2x3−x2 (5sin( 20 x ))

2 with quadrature mesh points along the

x-axis for tol= 5×10−3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 15.7 Approximating an iterated integral in two dimensions. For each of the quadrature

points in x we integrate in y to approximate g(x). . . . . . . . . . . . . . . . . 474 15.8 Gaussian points; see Exercise 10. . . . . . . . . . . . . . . . . . . . . . . . . . 477

16.1 A simple pendulum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 16.2 Two steps of the forward Euler method. The exact solution is the curved solid

line. The numerical values obtained by the Euler method are circled and lie at the nodes of a broken line that interpolates them. The broken line is tangential at the beginning of each step to the ODE trajectory passing through the corresponding node (dashed lines). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486

16.3 The sixth solution component of the HIRES model. . . . . . . . . . . . . . . . 490 16.4 RK method: repeated evaluations of the function f in the current mesh subinterval

[ti , ti+1] are combined to yield a higher order approximation yi+1 at ti+1. . . . . 493 16.5 Predator-prey model: y1(t) is number of prey, y2(t) is number of predator. . . . 497 16.6 Predator-prey solution in phase plane: the curve of y1(t) vs. y2(t) yields a limit

cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 16.7 Multistep method: known solution values of y or f (t , y) at the current location ti

and previous locations ti−1, . . . , ti+1−s are used to form an approximation for the next unknown yi+1 at ti+1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500

16.8 Stability regions for q-stage explicit RK methods of order q , q = 1,2,3,4. The inner circle corresponds to forward Euler, q = 1. The larger q is, the larger the stability region. Note the “ear lobes” of the fourth-order method protruding into the right half plane. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 508

16.9 Solution of the heat problem of Example 16.17 at t = 0.1. . . . . . . . . . . . . 512 16.10 The exact solution y(ti ), which lies on the lowest of the three curves, is approxi-

mated by yi , which lies on the middle curve. If we integrate the next step exactly, starting from (ti , yi ), then we obtain (ti+1, ȳ(ti+1)), which also lies on the middle curve. But we don’t: rather, we integrate the next step approximately as well, obtaining (ti+1, yi+1), which lies on the top curve. The difference between the two curve values at the argument ti+1 is the local error. . . . . . . . . . . . . . 516

16.11 Astronomical orbit using ode45; see Example 16.20. . . . . . . . . . . . . . . 518 16.12 Oscillatory energies for the FPU problem; see Example 16.21. . . . . . . . . . 520 16.13 Oscillatory energies for the FPU problem obtained using MATLAB’s ode45with

default tolerances. The deviation from Figure 16.12 depicts a significant, nonran- dom error. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520

✐

✐

✐

✐

✐

✐

✐

✐

List of Figures xvii

16.14 The two solutions of Examples 16.22 and 9.3 obtained using simple shooting starting from two different initial guesses c0 for v′(0). Plotted are the trajectories for the initial guesses (dashed magenta), as well as the final BVP solutions (solid blue). The latter are qualitatively the same as in Figure 9.3. Note the different vertical scales in the two subfigures. . . . . . . . . . . . . . . . . . . . . . . . 524

16.15 Hyperbolic and parabolic solution profiles starting from a discontinuous initial value function. For the advection equation the exact solution is displayed. For the heat equation we have integrated numerically, using homogeneous BC at x =±π and a rather fine discretization mesh. . . . . . . . . . . . . . . . . . . . . . . . 526

16.16 “Waterfall” solutions of the classical wave equation using the leapfrog scheme; see Example 16.25. For α = 1 the solution profile is resolved well by the dis- cretization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530

✐

✐

✐

✐

✐

✐

✐

✐

List of Tables

7.1 Errors for basic relaxation methods applied to the model problem of Example 7.1 with n = N2 = 225. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

9.1 Convergence of Newton’s method to the two roots of Example 9.1. . . . . . . . 256 9.2 Example 9.5. The first three columns to the right of the iteration counter track

convergence starting from x0 = (8, .2)T : Newton’s method yields the minimum quickly. The following (rightmost) three columns track convergence starting from x0 = (8, .8)T : Newton’s method finds a critical point, but it’s not the minimizer. . 262

9.3 Example 9.14. Tracking progress of the primal-dual LP algorithm. . . . . . . . . 284

14.1 Errors in the numerical differentiation of f (x) = ex at x = 0 using methods of order 1, 2, and 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413

14.2 Errors in the numerical differentiation of f (x) = ex at x = 0 using three-point methods on a nonuniform mesh, Example 14.6. The error eg in the more elaborate method is second order, whereas the simpler method yields first order accuracy es . The error value 0 at the bottom of the second column is a lucky break. . . . . 419

16.1 Absolute errors using the forward Euler method for the ODE y ′ = y. The values ei = y(ti )− ei are listed under Error. . . . . . . . . . . . . . . . . . . . . . . . . 487

16.2 Errors and calculated observed orders (rates) for the forward Euler, the explicit midpoint (RK2), and the classical Runge–Kutta (RK4) methods. . . . . . . . . . 496

16.3 Example 16.12: errors and calculated rates for Adams–Bashforth methods; (s,q) denotes the s-step method of order q . . . . . . . . . . . . . . . . . . . . . . . . 503

16.4 Example 16.12: errors and calculated rates for Adams–Moulton methods; (s,q) denotes the s-step method of order q . . . . . . . . . . . . . . . . . . . . . . . . 503

16.5 Coefficients of BDF methods up to order 6. . . . . . . . . . . . . . . . . . . . . 505 16.6 Example 16.14: errors and calculated rates for BDF methods; (s,q) denotes the

s-step method of order q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505

xix

✐

✐

✐

✐

✐

✐

✐

✐

Preface

This book is designed for students and researchers who seek practical knowledge of modern tech- niques in scientific computing. The text aims to provide an in-depth treatment of fundamental issues and methods, and the reasons behind success and failure of numerical software. On one hand, we avoid an extensive, encyclopedic, heavily theoretical exposition, and try to get to current methods, issues, and software fairly quickly. On the other hand, this is by no means a quick recipe book, since we feel that the introduction of algorithms requires adequate theoretical foundation: having a solid basis enables those who need to apply the techniques to successfully design their own solution approach for any nonstandard problems they may encounter in their work.

There are many books on scientific computing and numerical analysis, and a natural question here would be why we think that yet another text is necessary. It is mainly because we feel that in an age where yesterday’s concepts are not necessarily today’s truths, scientific computing is constantly redefining itself and is now positioned as a discipline truly at the junction of mathematics, computer science, and engineering. Books that rely heavily on theory, or on algorithm recipes, do not quite capture the current state of this broad and dynamic area of research and application. We thus take an algorithmic approach and focus on techniques of a high level of applicability to engineering, com- puter science, and industrial mathematics practitioners. At the same time, we provide mathematical justification throughout for the methods introduced. While we refrain from a theorem–proof type of exposition, we construct the theory behind the methods in a systematic and fairly complete fashion. We make a strong effort to emphasize computational aspects of the algorithms discussed by way of discussing their efficiency and their computational limitations.

This book has been developed from notes for two courses taught by the Department of Com- puter Science at the University of British Columbia (UBC) for over 25 years. The first author developed a set of notes back in the late 1970s. These were based on the books by Conte and de Boor [13], Dahlquist and Bjorck [15], and others, as well as on notes by Jim Varah. Improvements to these notes were subsequently made by Ian Cavers, by the second author, and by Dhavide Aru- liah. A substantial addition of material was made by the two authors in the last few years. A few of our colleagues (Jack Snoeyink, Oliver Dorn, Ian Mitchell, Kees van den Doel and Tyrone Rees at UBC, Jim Lambers at Stanford University, Marsha Berger and Michael Overton at NYU, Edmond Chow at Columbia University, André Weidemann at Stellenbosch, and others) have used the notes and provided us with invaluable feedback.

Most of the material contained in this book can be covered in two reasonably paced semesterial courses. A possible model here is the one that has been adopted at UBC: one course covers Chapters 3 to 9, while another concentrates on Chapters 10 to 16. The two parts are designed so that they do not heavily rely on each other, although we have found it useful to include the material of Chapter 1 and a subset of Chapter 2 in both semesterial courses. More advanced material, contained in sections denoted by an asterisk, is included in the text as a natural extension. However, with one or two exceptions, we have not taught the advanced material in the two basic courses.

Another use of this text is for a breadth-type introductory course on numerical methods at the graduate level. The target audience would be beginning graduate students in a variety of disci-

xxi

✐

✐

✐

✐

✐

✐

✐

✐

xxii Preface

plines including computer science, applied mathematics, and engineering, who start graduate school without having taken undergraduate courses in this area. Such a course would also use our more advanced sections and possibly additional material.

The division of the material into chapters can be justified by the type of computational errors encountered. These concepts are made clear in the text. We start off with roundoff error, which appears in all the numerical processes considered in this book. Chapters 1 and 2 systematically cover it, and later chapters, such as 5 and 6, contain assessments of its effect. Chapter 3 and Chapters 7 to 9 are concerned mainly with iterative processes, and as such, the focus is on the iteration, or convergence error. Finally, control and assessment of discretization error is the main focus in Chapters 10 to 16, although the other error types do arise there as well.

We illustrate our algorithms using the programming environment of MATLAB and expect the reader to become gradually proficient in this environment while (if not before) learning the material covered in this book. But at the same time, we use the MATLAB language and programming environment merely as a tool, and we refrain from turning the book into a language tutorial.

Each chapter contains exercises, ordered by section. In addition, there is an exercise numbered 0 which consists of several review questions intended for self-testing and for punctuating the process of reading and absorbing the presented material. Instructors should resist the temptation to answer these review questions for the students.

We have made an effort to make the chapters fairly independent of one another. As a result, we believe that readers who are interested in specific topics may often be able to settle for reading almost exclusively only the chapters that are relevant to the material they wish to pursue.

We are maintaining a project webpage for this book which can be found at

http://www.siam.org/books/cs07

It contains links to our programs, solutions to selected exercises, errata, and more. Last but not at all least, many generations of students at UBC have helped us over the years to

shape, define, and debug this text. The list of individuals who have provided crucial input is simply too long to mention. This book is dedicated to you, our students.

✐

✐

✐

✐

✐

✐

✐

✐

Chapter 1

Numerical Algorithms

This opening chapter introduces the basic concepts of numerical algorithms and scientific comput- ing.

We begin with a general, brief introduction to the field in Section 1.1. This is followed by the more substantial Sections 1.2 and 1.3. Section 1.2 discusses the basic errors that may be encountered when applying numerical algorithms. Section 1.3 is concerned with essential properties of such algorithms and the appraisal of the results they produce.

We get to the “meat” of the material in later chapters.

1.1 Scientific computing Scientific computing is a discipline concerned with the development and study of numerical al- gorithms for solving mathematical problems that arise in various disciplines in science and engin- eering.

Typically, the starting point is a given mathematical model which has been formulated in an attempt to explain and understand an observed phenomenon in biology, chemistry, physics, eco- nomics, or any other scientific or engineering discipline. We will concentrate on those mathematical models which are continuous (or piecewise continuous) and are difficult or impossible to solve ana- lytically; this is usually the case in practice. Relevant application areas within computer science and related engineering fields include graphics, vision and motion analysis, image and signal processing, search engines and data mining, machine learning, and hybrid and embedded systems.

In order to solve such a model approximately on a computer, the continuous or piecewise continuous problem is approximated by a discrete one. Functions are approximated by finite arrays of values. Algorithms are then sought which approximately solve the mathematical problem effi- ciently, accurately, and reliably. This is the heart of scientific computing. Numerical analysis may be viewed as the theory behind such algorithms.

The next step after devising suitable algorithms is their implementation. This leads to ques- tions involving programming languages, data structures, computing architectures, etc. The big pic- ture is depicted in Figure 1.1.

The set of requirements that good scientific computing algorithms must satisfy, which seems elementary and obvious, may actually pose rather difficult and complex practical challenges. The main purpose of this book is to equip you with basic methods and analysis tools for handling such challenges as they arise in future endeavors.

1

✐

✐

✐

✐

✐

✐

✐

✐

2 Chapter 1. Numerical Algorithms

Observed phenomenon

Mathematical model

Discretization Solution algorithm

Efficiency Accuracy Robustness

Implementation

Programming environment

Data structures

Computing architecture

Figure 1.1. Scientific computing.

Problem solving environment

As a computing tool, we will be using MATLAB: this is an interactive computer language, which for our purposes may best be viewed as a convenient problem solving environment.

MATLAB is much more than a language based on simple data arrays; it is truly a complete environment. Its interactivity and graphics capabilities make it more suitable and convenient in our context than general-purpose languages such as C++, Java, Scheme, or Fortran 90. In fact, many of the algorithms that we will learn are already implemented in MATLAB. . . So why learn them at all? Because they provide the basis for much more complex tasks, not quite available (that is to say, not already solved) in MATLAB or anywhere else, which you may encounter in the future.

Rather than producing yet another MATLAB tutorial or introduction in this text (there are several very good ones available in other texts as well as on the Internet) we will demonstrate the use of this language on examples as we go along.

✐

✐

✐

✐

✐

✐

✐

✐

1.2. Numerical algorithms and errors 3

1.2 Numerical algorithms and errors The most fundamental feature of numerical computing is the inevitable presence of error. The result of any interesting computation (and of many uninteresting ones) is typically only approximate, and our goal is to ensure that the resulting error is tolerably small.

Relative and absolute errors

There are in general two basic types of measured error. Given a scalar quantity u and its approxima- tion v:

• The absolute error in v is |u − v|.

• The relative error (assuming u �= 0) is |u − v| |u| .

The relative error is usually a more meaningful measure. This is especially true for errors in floating point representation, a point to which we return in Chapter 2. For example, we record absolute and relative errors for various hypothetical calculations in the following table:

u v Absolute Relative error error

1 0.99 0.01 0.01

1 1.01 0.01 0.01

−1.5 −1.2 0.3 0.2 100 99.99 0.01 0.0001

100 99 1 0.01

Evidently, when |u| ≈ 1 there is not much difference between absolute and relative error measures. But when |u| 1, the relative error is more meaningful. In particular, we expect the approximation in the last row of the above table to be similar in quality to the one in the first row. This expectation is borne out by the value of the relative error but is not reflected by the value of the absolute error.

When the approximated value is small in magnitude, things are a little more delicate, and here is where relative errors may not be so meaningful. But let us not worry about this at this early point.

Example 1.1. The Stirling approximation

v = Sn = √

2πn · (n

e

)n is used to approximate u = n! = 1 ·2 · · ·n for large n. The formula involves the constant e = exp(1)= 2.7182818 . . .. The following MATLAB script computes and displays n! and Sn , as well as their absolute and relative differences, for 1 ≤ n ≤ 10: e=exp(1); n=1:10; % array Sn=sqrt(2*pi*n).*((n/e).^n); % the Stirling approximation.

✐

✐

✐

✐

✐

✐

✐

✐

4 Chapter 1. Numerical Algorithms

fact_n=factorial(n); abs_err=abs(Sn-fact_n); % absolute error rel_err=abs_err./fact_n; % relative error format short g [n; fact_n; Sn; abs_err; rel_err]’ % print out values

Given that this is our first MATLAB script, let us provide a few additional details, though we hasten to add that we will not make a habit out of this. The commands exp, factorial, and abs use built-in functions. The command n=1:10 (along with a semicolon, which simply suppresses screen output) defines an array of length 10 containing the integers 1,2, . . . ,10. This illustrates a fundamental concept in MATLAB of working with arrays whenever possible. Along with it come array operations: for example, in the third line “.*” corresponds to elementwise multiplication of vectors or matrices. Finally, our printing instructions (the last two in the script) are a bit primitive here, a sacrifice made for the sake of simplicity in this, our first program.

The resulting output is

1 1 0.92214 0.077863 0.077863 2 2 1.919 0.080996 0.040498 3 6 5.8362 0.16379 0.027298 4 24 23.506 0.49382 0.020576 5 120 118.02 1.9808 0.016507 6 720 710.08 9.9218 0.01378 7 5040 4980.4 59.604 0.011826 8 40320 39902 417.6 0.010357 9 3.6288e+005 3.5954e+005 3343.1 0.0092128

10 3.6288e+006 3.5987e+006 30104 0.008296

The values of n! become very large very quickly, and so are the values of the approximation Sn . The absolute errors grow as n grows, but the relative errors stay well behaved and indicate that in fact the larger n is, the better the quality of the approximation is. Clearly, the relative errors are much more meaningful as a measure of the quality of this approximation.

Error types

Knowing how errors are typically measured, we now move to discuss their source. There are several types of error that may limit the accuracy of a numerical calculation.

1. Errors in the problem to be solved.

These may be approximation errors in the mathematical model. For instance:

• Heavenly bodies are often approximated by spheres when calculating their properties; an example here is the approximate calculation of their motion trajectory, attempting to answer the question (say) whether a particular asteroid will collide with Planet Earth before 11.12.2016.

• Relatively unimportant chemical reactions are often discarded in complex chemical modeling in order to obtain a mathematical problem of a manageable size.

It is important to realize, then, that often approximation errors of the type stated above are deliberately made: the assumption is that simplification of the problem is worthwhile even if it generates an error in the model. Note, however, that we are still talking about the math- ematical model itself; approximation errors related to the numerical solution of the problem are discussed below.

✐

✐

✐

✐

✐

✐

✐

✐

1.2. Numerical algorithms and errors 5

Another typical source of error in the problem is error in the input data. This may arise, for instance, from physical measurements, which are never infinitely accurate.

Thus, it may be that after a careful numerical simulation of a given mathematical problem, the resulting solution would not quite match observations on the phenomenon being examined.

At the level of numerical algorithms, which is the focus of our interest here, there is really nothing we can do about the above-described errors. Nevertheless, they should be taken into consideration, for instance, when determining the accuracy (tolerance with respect to the next two types of error mentioned below) to which the numerical problem should be solved.

2. Approximation errors

Such errors arise when an approximate formula is used in place of the actual function to be evaluated.

We will often encounter two types of approximation errors:

• Discretization errors arise from discretizations of continuous processes, such as inter- polation, differentiation, and integration.

• Convergence errors arise in iterative methods. For instance, nonlinear problems must generally be solved approximately by an iterative process. Such a process would con- verge to the exact solution in infinitely many iterations, but we cut it off after a finite (hopefully small!) number of such iterations. Iterative methods in fact often arise in linear algebra.

3. Roundoff errors

Any computation with real numbers involves roundoff error. Even when no approximation error is produced (as in the direct evaluation of a straight line, or the solution by Gaussian elimination of a linear system of equations), roundoff errors are present. These arise because of the finite precision representation of real numbers on any computer, which affects both data representation and computer arithmetic.

Discretization and convergence errors may be assessed by an analysis of the method used, and we will see a lot of that in this text. Unlike roundoff errors, they have a relatively smooth structure which may occasionally be exploited. Our basic assumption will be that approximation errors dominate roundoff errors in magnitude in actual, successful calculations.

Theorem: Taylor Series. Assume that f (x) has k +1 derivatives in an interval containing the points x0 and x0 +h. Then

f (x0 +h) = f (x0)+h f ′(x0)+ h 2

2 f ′′(x0)+·· ·+ h

k

k! f (k)(x0)

+ h k+1

(k +1)! f (k+1)(ξ ),

where ξ is some point between x0 and x0 +h.

Discretization errors in action

Let us show an example that illustrates the behavior of discretization errors.

✐

✐

✐

✐

✐

✐

✐

✐

6 Chapter 1. Numerical Algorithms

Example 1.2. Consider the problem of approximating the derivative f ′(x0) of a given smooth function f (x) at the point x = x0. For instance, let f (x) = sin(x) be defined on the real line −∞ < x < ∞, and set x0 = 1.2. Thus, f (x0) = sin(1.2) ≈ 0.932 . . . .

Further, consider a situation where f (x) may be evaluated at any point x near x0, but f ′(x0) may not be directly available or is computationally expensive to evaluate. Thus, we seek ways to approximate f ′(x0) by evaluating f at x near x0.

A simple algorithm may be constructed using Taylor’s series. This fundamental theorem is given on the preceding page. For some small, positive value h that we will choose in a moment, write

f (x0 +h) = f (x0)+h f ′(x0)+ h 2

2 f ′′(x0)+ h

3

6 f ′′′(x0)+ h

4

24 f ′′′′(x0)+·· · .

Then

f ′(x0) = f (x0 +h)− f (x0) h

− (

h

2 f ′′(x0)+ h

2

6 f ′′′(x0)+ h

3

24 f ′′′′(x0)+·· ·

) .

Our algorithm for approximating f ′(x0) is to calculate f (x0 +h)− f (x0)

h .

The obtained approximation has the discretization error∣∣∣∣ f ′(x0)− f (x0 +h)− f (x0)h ∣∣∣∣= ∣∣∣∣h2 f ′′(x0)+ h26 f ′′′(x0)+ h324 f ′′′′(x0)+·· ·

∣∣∣∣ . Geometrically, we approximate the slope of the tangent at the point x0 by the slope of the chord through neighboring points of f . In Figure 1.2, the tangent is in blue and the chord is in red.

0 x

f

x0 x0 +h

Figure 1.2. A simple instance of numerical differentiation: the tangent f ′(x0) is approxi- mated by the chord ( f (x0 +h)− f (x0))/h.

If we know f ′′(x0), and it is nonzero, then for h small we can estimate the discretization error by ∣∣∣∣ f ′(x0)− f (x0 +h)− f (x0)h

∣∣∣∣≈ h2 ∣∣ f ′′(x0)∣∣ . Using the notation defined in the box on the next page we notice that the error is O(h) so long as

✐

✐

✐

✐

✐

✐

✐

✐

1.2. Numerical algorithms and errors 7

f ′′(x0) is bounded, and it is �(h) if also f ′′(x0) �= 0. In any case, even without knowing f ′′(x) we expect the discretization error to decrease at least as fast as h when h is decreased.

Big-O and � Notation Throughout this text we consider various computational errors depending on a discretiza- tion step size h > 0 and ask how they decrease as h decreases. In other instances, such as when estimating the efficiency of a particular algorithm, we are interested in a bound on the work estimate as a parameter n increases unboundedly (e.g., n = 1/h).

For an error e depending on h we denote

e = O(hq ) if there are two positive constants q and C such that

|e| ≤ Chq

for all h > 0 small enough. Similarly, for w = w(n) the expression

w = O(n logn) means that there is a constant C > 0 such that

w ≤ Cn logn as n → ∞. It will be easy to figure out from the context which of these two meanings is the relevant one.

The � notation signifies a stronger relation than the O notation: a function φ(h) for small h (resp., φ(n) for large n) is �(ψ(h)) (resp., �(ψ(n))) if φ is asymptotically bounded both above and below by ψ .

For our particular instance, f (x) = sin(x), we have the exact value f ′(x0) = cos(1.2) = 0.362357754476674 . . .. Carrying out our short algorithm we obtain for h = 0.1 the approxi- mation f ′(x0) ≈ (sin(1.3)− sin(1.2))/0.1 = 0.315 . . . . The absolute error thus equals approximately 0.047. The relative error is not qualitatively different here.

This approximation of f ′(x0) using h = 0.1 is not very accurate. We therefore apply the same algorithm using several increasingly smaller values of h. The resulting errors are as follows:

h Absolute error

0.1 4.716676e-2

0.01 4.666196e-3

0.001 4.660799e-4

1.e-4 4.660256e-5

1.e-7 4.619326e-8

Indeed, the error appears to decrease like h. More specifically (and less importantly), using our explicit knowledge of f ′′(x) = f (x) = −sin(x), in this case we have that 12 f ′′(x0) ≈ −0.466. The quantity 0.466h is seen to provide a rather accurate estimate for the above-tabulated absolute error values.

✐

✐

✐

✐

✐

✐

✐

✐

8 Chapter 1. Numerical Algorithms

The damaging effect of roundoff errors

The calculations in Example 1.2, and the ones reported below, were carried out using MATLAB’s standard arithmetic. Let us stay for a few more moments with the approximation algorithm featured in that example and try to push the envelope a little further.

Example 1.3. The numbers in Example 1.2 might suggest that an arbitrary accuracy can be achieved by the algorithm, provided only that we take h small enough. Indeed, suppose we want∣∣∣∣cos(1.2)− sin(1.2+h)− sin(1.2)h

∣∣∣∣< 10−10. Can’t we just set h ≤ 10−10/0.466 in our algorithm?

Not quite! Let us record results for very small, positive values of h:

h Absolute error

1.e-8 4.361050e-10

1.e-9 5.594726e-8

1.e-10 1.669696e-7

1.e-11 7.938531e-6

1.e-13 4.250484e-4

1.e-15 8.173146e-2

1.e-16 3.623578e-1

A log-log plot1 of the error versus h is provided in Figure 1.3. We can clearly see that as h is decreased, at first (from right to left in the figure) the error decreases along a straight line, but this trend is altered and eventually reversed. The MATLAB script that generates the plot in Figure 1.3 is given next.

x0 = 1.2; f0 = sin(x0); fp = cos(x0); i = -20:0.5:0; h = 10.^i; err = abs (fp - (sin(x0+h) - f0)./h ); d_err = f0/2*h; loglog (h,err,’-*’); hold on loglog (h,d_err,’r-.’); xlabel(’h’) ylabel(’Absolute error’)

Perhaps the most mysterious line in this script is that defining d_err: it calculates 1 2 | f ′′(x0)|h.

1Graphing error values using a logarithmic scale is rather common in scientific computing, because a logarithmic scale makes it easier to trace values that are close to zero. As you will use such plotting often, let us mention at this early stage the MATLAB commands plot, semilogy, and loglog.

✐

✐

✐

✐

✐

✐

✐

✐

1.3. Algorithm properties 9

10 −20

10 −15

10 −10

10 −5

10 0

10 −15

10 −10

10 −5

10 0

h

A bs

ol ut

e er

ro r

Figure 1.3. The combined effect of discretization and roundoff errors. The solid curve interpolates the computed values of | f ′(x0)− f (x0+h)− f (x0)h | for f (x)= sin(x), x0 = 1.2. Also shown in dash-dot style is a straight line depicting the discretization error without roundoff error.

The reason the error “bottoms out” at about h = 10−8 in the combined Examples 1.2–1.3 is that the total, measured error consists of contributions of both discretization and roundoff errors. The discretization error decreases in an orderly fashion as h decreases, and it dominates the roundoff error when h is relatively large. But when h gets below approximately 10−8 the discretization error becomes very small and roundoff error starts to dominate (i.e., it becomes larger in magnitude).

Roundoff error has a somewhat erratic behavior, as is evident from the small oscillations that are present in the graph in a few places.

Moreover, for the algorithm featured in the last two examples, overall the roundoff error in- creases as h decreases. This is one reason why we want it always dominated by the discretization error when solving problems involving numerical differentiation such as differential equations.

Popular theorems from calculus

The Taylor Series Theorem, given on page 5, is by far the most cited theorem in numerical anal- ysis. Other popular calculus theorems that we will use in this text are gathered on the following page. They are all elementary and not difficult to prove: indeed, most are special cases of Taylor’s Theorem.

Specific exercises for this section: Exercises 1–3.

1.3 Algorithm properties In this section we briefly discuss performance features that may or may not be expected from a good numerical algorithm, and we define some basic properties, or characteristics, that such an algorithm should have.

✐

✐

✐

✐

✐

✐

✐

✐

10 Chapter 1. Numerical Algorithms

Theorem: Useful Calculus Results.

• Intermediate Value If f ∈ C[a,b] and s is a value such that f (â) ≤ s ≤ f (b̂) for two numbers â, b̂ ∈ [a,b], then there exists a real number c ∈ [a,b] for which f (c) = s.

• Mean Value If f ∈ C[a,b] and f is differentiable on the open interval (a,b), then there exists a real number c ∈ (a,b) for which f ′(c) = f (b)− f (a)b−a .

• Rolle’s If f ∈ C[a,b] and f is differentiable on (a,b), and in addition f (a) = f (b) = 0, then there is a real number c ∈ (a,b) for which f ′(c) = 0.

Criteria for assessing an algorithm

An assessment of the quality and usefulness of an algorithm may be based on a number of criteria:

• Accuracy This issue is intertwined with the issue of error types and was discussed at the start of Sec- tion 1.2 and in Example 1.2. (See also Exercise 3.) The important point is that the accuracy of a numerical algorithm is a crucial parameter in its assessment, and when designing numerical algorithms it is necessary to be able to point out what magnitude of error is to be expected when the computation is carried out.

• Efficiency A good computation is one that terminates before we lose our patience. A numerical algo- rithm that features great theoretical properties is useless if carrying it out takes an unreason- able amount of computational time. Efficiency depends on both CPU time and storage space requirements. Details of an algorithm implementation within a given computer language and an underlying hardware configuration may play an important role in yielding code efficiency. Other theoretical properties yield indicators of efficiency, for instance, the rate of conver- gence. We return to this in later chapters.

Often a machine-independentestimate of the number of elementary operations required, namely, additions, subtractions, multiplications, and divisions, gives an idea of the algorithm’s effi- ciency. Normally, a floating point representation is used for real numbers and then the costs of these different elementary floating point operations, called flops, may be assumed to be roughly equal to one another.

Example 1.4. A polynomial of degree n, given as

pn(x) = c0 + c1x +·· ·+ cnxn , requires O(n2) operations2 to evaluate at a fixed point x , if done in a brute force way without intermediate storing of powers of x . But using the nested form, also known as Horner’s rule and given by

pn(x) = (· · · ((cnx + cn−1)x + cn−2)x · · · )x + c0, 2See page 7 for the O notation.

✐

✐

✐

✐

✐

✐

✐

✐

1.3. Algorithm properties 11

suggests an evaluation algorithm which requires only O(n) elementary operations, i.e., re- quiring linear (in n) rather than quadratic computation time. A MATLAB script for nested evaluation follows:

% Assume the polynomial coefficients are already stored % in array c such that for any real x, % p(x) = c(1) + c(2)x + c(3)x^2 + ... + c(n+1)x^n p = c(n+1); for j = n:-1:1

p = p*x + c(j); end

The “onion shell” evaluation formula thus unravels quite simply. Note also the manner of introducing comments into the script.

It is important to note that while operation counts as in Example 1.4 often give a rough idea of algorithm efficiency, they do not give the complete picture regarding execution speed, since they do not take into account the price (speed) of memory access which may vary consider- ably. Furthermore, any setting of parallel computing is ignored in a simple operation count as well. Curiously, this is part of the reason the MATLAB command flops, which had been an integral part of this language for many years, was removed from further releases several years ago. Indeed, in modern computers, cache access, blocking and vectorization features, and other parameters are crucial in the determination of execution time. The computer language used for implementation can also affect the comparative timing of algorithm implementations. Those, unfortunately, are much more difficult to assess compared to an operation count. In this text we will not get into the gory details of these issues, despite their relevance and im- portance.

• Robustness Often, the major effort in writing numerical software, such as the routines available in MAT- LAB for solving linear systems of algebraic equations or for function approximation and in- tegration, is spent not on implementing the essence of an algorithm but on ensuring that it would work under all weather conditions. Thus, the routine should either yield the correct result to within an acceptable error tolerance level, or it should fail gracefully (i.e., terminate with a warning) if it does not succeed to guarantee a “correct result.”

There are intrinsic numerical properties that account for the robustness and reliability of an algorithm. Chief among these is the rate of accumulation of errors. In particular, the algorithm must be stable; see Example 1.6.

Problem conditioning and algorithm stability

In view of the fact that the problem and the numerical algorithm both yield errors, a natural question arises regarding the appraisal of a given computed solution. Here notions such as problem sensitivity and algorithm stability play an important role. If the problem is too sensitive, or ill-conditioned, meaning that even a small perturbation in the data produces a large difference in the result,3 then no algorithm may be found for that problem which would meet our requirement of solution robustness; see Figure 1.4 for an illustration. Some modification in the problem definition may be called for in such cases.

3Here we refer to intuitive notions of “large” vs. “small” quantities and of values being “close to” vs. “far from” one another. While these notions can be quantified and thus be made more precise, such a move would typically make definitions cumbersome and harder to understand at this preliminary stage of the discussion.

✐

✐

✐

✐

✐

✐

✐

✐

12 Chapter 1. Numerical Algorithms

x

y

x̄

ȳ

g

g

Figure 1.4. An ill-conditioned problem of computing output values y given in terms of input values x by y = g(x): when the input x is slightly perturbed to x̄, the result ȳ = g(x̄) is far from y. If the problem were well-conditioned, we would be expecting the distance between y and ȳ to be more comparable in magnitude to the distance between x and x̄.

For instance, the problem of numerical differentiation depicted in Examples 1.2 and 1.3 turns out to be ill-conditioned when extreme accuracy (translating to very small values of h) is required.

The job of a stable algorithm for a given problem is to yield a numerical solution which is the exact solution of an only slightly perturbed problem; see the illustration in Figure 1.5. Thus, if the algorithm is stable and the problem is well-conditioned (i.e., not ill-conditioned), then the computed result ȳ is close to the exact y.

x x̄

ȳ

g

Figure 1.5. An instance of a stable algorithm for computing y = g(x): the output ȳ is the exact result, ȳ = g(x̄), for a slightly perturbed input, i.e., x̄ which is close to the input x. Thus, if the algorithm is stable and the problem is well-conditioned, then the computed result ȳ is close to the exact y.

Example 1.5. The problem of evaluating the square root function for an argument near the value 1 is well-conditioned, as we show below.

✐

✐

✐

✐

✐

✐

✐

✐

1.3. Algorithm properties 13

Thus, let g(x) = √1+ x and note that g′(x) = 1 2 √

1+x . Suppose we fix x so that |x | � 1, and consider x̄ = 0 as a small perturbation of x . Then ȳ = g(x̄) = 1, and y − ȳ =√1+ x −1.

If we approximate √

1+ x by the first two terms of its Taylor series expansion (see page 5) about the origin, namely, g(x) ≈ 1+ x2 , then

y − ȳ ≈ (

1+ x 2

) −1 = x

2 = 1

2 (x − x̄).

Qualitatively, the situation is fortunately not as in Figure 1.4. For instance, if x = .001, then y − ȳ ≈ .0005.

We can say that the conditioning of this problem is determined by g′(0) = 12 , because g′(0) ≈ g(x)−g(0)

x−0 for x small. The problem is well-conditioned because this number is not large. On the other hand, a function whose derivative wildly changes may not be easily evaluated

accurately. A classical example here is the function g(x)= tan(x). Evaluating it for x near zero does not cause difficulty (the problem being well-conditioned there), but the problem of evaluating the same function for x near π2 is ill-conditioned. For instance, setting x = π2 − .001 and x̄ = π2 − .002 we obtain that |x − x̄ | = .001 but | tan(x)− tan(x̄)| ≈ 500. A look at the derivative, g′(x) = 1

cos2(x) ,

for x near π2 explains why.

Error accumulation

We will have much to say in Chapter 2 about the floating point representation of real numbers and the accumulation of roundoff errors during a calculation. Here let us emphasize that in general it is impossible to prevent linear accumulation, meaning the roundoff error may be proportional to n after n elementary operations such as addition or multiplication of two real numbers. However, such an error accumulation is usually acceptable if the linear rate is moderate (i.e., the constant c0 below is not very large). In contrast, exponential growth cannot be tolerated.

Explicitly, if En measures the relative error at the nth operation of an algorithm, then

En � c0nE0 for some constant c0 represents linear growth, and En � cn1 E0 for some constant c1 > 1 represents exponential growth.

An algorithm exhibiting relative exponential error growth is unstable. Such algorithms must be avoided!

Example 1.6. Consider evaluating the integrals

yn = ∫ 1

0

xn

x +10dx

for n = 1,2, . . . ,30. Observe at first that analytically

yn +10yn−1 = ∫ 1

0

xn +10xn−1 x +10 dx =

∫ 1 0

xn−1dx = 1 n

.

Also

y0 = ∫ 1

0

1

x +10dx = ln(11)− ln(10).

✐

✐

✐

✐

✐

✐

✐

✐

14 Chapter 1. Numerical Algorithms

An algorithm which may come to mind is therefore as follows:

1. Evaluate y0 = ln(11)− ln(10). 2. For n = 1, . . . ,30, evaluate

yn = 1 n −10 yn−1.

Note that applying the above recursion formula would give exact values if roundoff errors were not present.

However, this algorithm is in fact unstable, as the magnitude of roundoff errors gets multiplied by 10 each time the recursion is applied. Thus, there is exponential error growth with c1 = 10. In MATLAB (which automatically employs the IEEE double precision floating point arithmetic; see Section 2.4) we obtain y0 = 9.5310e− 02, y18 = −9.1694e+ 01, y19 = 9.1694e+ 02, . . . , y30 = −9.1694e+ 13. It is not difficult to see that the exact values all satisfy 0 < yn < 1, and hence the computed solution, at least for n ≥ 18, is meaningless!

Thankfully, such extreme instances of instability as illustrated in Example 1.6 will not occur in any of the algorithms developed in this text from here on.

Specific exercises for this section: Exercises 4–5.

1.4 Exercises 0. Review questions

(a) What is the difference, according to Section 1.1, between scientific computing and nu- merical analysis?

(b) Give a simple example where relative error is a more suitable measure than absolute error, and another example where the absolute error measure is more suitable.

(c) State a major difference between the nature of roundoff errors and discretization errors.

(d) Explain briefly why accumulation of roundoff errors is inevitable when arithmetic opera- tions are performed in a floating point system. Under which circumstances is it tolerable in numerical computations?

(e) Explain the differences between accuracy, efficiency, and robustness as criteria for eval- uating an algorithm.

(f) Show that nested evaluation of a polynomial of degree n requires only 2n elementary operations and hence has O(n) complexity.

(g) Distinguish between problem conditioning and algorithm stability.

1. Carry out calculations similar to those of Example 1.3 for approximating the derivative of the function f (x)= e−2x evaluated at x0 = 0.5. Observe similarities and differences by comparing your graph against that in Figure 1.3.

2. Carry out derivation and calculations analogous to those in Example 1.2, using the expression

f (x0 +h)− f (x0 −h) 2h

for approximating the first derivative f ′(x0). Show that the error is O(h2). More precisely, the leading term of the error is − h23 f ′′′(x0) when f ′′′(x0) �= 0.

✐

✐

✐

✐

✐

✐

✐

✐

1.5. Additional notes 15

3. Carry out similar calculations to those of Example 1.3 using the approximation from Exer- cise 2. Observe similarities and differences by comparing your graph against that in Fig- ure 1.3.

4. Following Example 1.5, assess the conditioning of the problem of evaluating

g(x) = tanh(cx) = exp(cx)− exp(−cx) exp(cx)+ exp(−cx)

near x = 0 as the positive parameter c grows. 5. Consider the problem presented in Example 1.6. There we saw a numerically unstable proce-

dure for carrying out the task.

(a) Derive a formula for approximately computing these integrals based on evaluating yn−1 given yn .

(b) Show that for any given value ε > 0 and positive integer n0, there exists an integer n1 ≥ n0 such that taking yn1 = 0 as a starting value will produce integral evaluations yn with an absolute error smaller than ε for all 0 < n ≤ n0.

(c) Explain why your algorithm is stable.

(d) Write a MATLAB function that computes the value of y20 within an absolute error of at most 10−5. Explain how you choose n1 in this case.

1.5 Additional notes The proliferation in the early years of the present century of academic centers, institutes, and special programs for scientific computing reflects the coming of age of the discipline in terms of exper- iments, theory, and simulation. Fast, available computing hardware, powerful programming en- vironments, and the availability of numerical algorithms and software libraries all make scientific computing an indispensable tool in modern science and engineering. This tool allows for an inter- play with experiment and theory. On the one hand, improvements in computing power allow for experimentation and computations in a scale that could not have been imagined just a few years ago. On the other hand, the great progress in the theoretical understanding of numerical methods, and the availability of convenient computational testing environments, have given scientists and practi- tioners the ability to make predictions of and conclusions about huge-scale phenomena that today’s computers are still not (and may never be) powerful enough to handle.

A potentially surprising amount of attention has been given throughout the years to the defini- tions of scientific computing and numerical analysis. An interesting account of the evolution of this seemingly esoteric but nevertheless important issue can be found in Trefethen and Bau [70].

The concept of problem conditioning is both fundamental and tricky to discuss so early in the game. If you feel a bit lost somewhere around Figures 1.4 and 1.5, then rest assured that these concepts eventually will become clearer as we gain experience, particularly in the more specific contexts of Sections 5.8, 8.2, and 14.4.

Many computer science theory books deal extensively with O and � notations and complexity issues. One widely used such book is Graham, Knuth, and Patashnik [31].

There are many printed books and Internet introductions to MATLAB. Check out wikipedia and what’s in Mathworks. One helpful survey of some of those can be found at http://www.cs.ubc. ca/∼mitchell/matlabResources.html .

✐

✐

✐

✐

✐

✐

✐

✐

Chapter 2

Roundoff Errors

As observed in Chapter 1, various errors may arise in the process of calculating an approximate solution for a mathematical model. In this chapter we introduce and discuss in detail the most fundamental source of imperfection in numerical computing: roundoff errors. Such errors arise due to the intrinsic limitation of the finite precision representation of numbers (except for a restricted set of integers) in computers.

Different audiences may well require different levels of depth and detail in the present topic. We therefore start our discussion in Section 2.1 with the bare bones: a collection of essential facts related to floating point systems and roundoff errors that may be particularly useful for those wishing to concentrate on the last seven chapters of this text.

Note: If you do not require detailed knowledge of roundoff errors and their propagation during a computation, not even the essentials of Section 2.1, then you may skip this chap- ter (not recommended), at least upon first reading. What you must accept, then, is the notion that each number representation and each elementary operation (such as + or ∗) in standard floating point arithmetic introduces a small, random relative error: up to about 10−16 in today’s standard floating point systems.

In Section 2.2 we get technical and dive into the gory details of floating point systems and floating point arithmetic. Several issues only mentioned in Section 2.1 are explained here.

The small representation errors as well as errors that arise upon carrying out elementary op- erations such as addition and multiplication are typically harmless unless they accumulate or get magnified in an unfortunate way during the course of a possibly long calculation. We discuss round- off error accumulation as well as ways to avoid or reduce such effects in Section 2.3.

Finally, in Section 2.4, the IEEE standard for floating point arithmetic, which is implemented in any nonspecialized hardware, is briefly described.

2.1 The essentials This section summarizes what we believe all our students should know as a minimum about floating point arithmetic. Let us start with a motivating example.

Example 2.1. Scientists and engineers often wish to believe that the numerical results of a computer calculation, especially those obtained as output of a software package, contain no error—at least not a significant or intolerable one. But careless numerical computing does occasionally lead to trouble.

17

✐

✐

✐

✐

✐

✐

✐

✐

18 Chapter 2. Roundoff Errors

Note: The word “essential” is not synonymous with “easy.” If you find some part of the description below too terse for comfort, then please refer to the relevant section in this chapter for more motivation, detail, and explanation.

One of the more spectacular disasters was the Patriot missile failure in Dhahran, Saudi Arabia, on February 25, 1991, which resulted in 28 deaths. This failure was ultimately traced to poor han- dling of roundoff errors in the missile’s software. The webpage http://www.ima.umn.edu/∼arnold/ disasters/patriot.html contains the details of this story. For a large collection of software bugs, see http://wwwzenger.informatik.tu-muenchen.de/persons/huckle/bugse.html.

Computer representation of real numbers

Computer memory has a finite capacity. This obvious fact has far-reaching implications on the representation of real numbers, which in general do not have a finite uniform representation. How should we then represent a real number on the computer in a way that can be sensibly implemented in hardware?

Any real number x ∈ R is accurately representable by an infinite sequence of digits.4 Thus, we can write

x =±(1.d1d2d3 · · ·dt−1dt dt+1 · · · )×2e, where e is an integer exponent. The (possibly infinite) set of binary digits {di} each have a value 0 or 1. The decimal value of this mantissa is

1.d1d2d3 · · · = 1+ d1 2

+ d2 22

+ d3 23

+·· · .

For instance, the binary representation x =−(1.10100 · · ·)×21 has in decimal the value x =−(1+1/2+1/8)×2=−3.25.

Of course, it should be clear that the choice of a binary representation is just one of many possibilities, indeed a convenient choice when it comes to computers. To represent any real number on the computer, we associate to x a floating point representation fl(x) of a form similar to that of x but with only t digits, so

fl(x) = sign(x)× (1.d̃1d̃2d̃3 · · · d̃t−1d̃t )×2e

for some t that is fixed in advance and binary digits d̃i that relate to di in a manner that will soon be specified. Storing this fraction in memory thus requires t bits. The exponent e must also be stored in a fixed number of bits and therefore must be bounded, say, between a lower bound L and an upper bound U . Further details are given in Section 2.2, which we hope will provide you with much of what you’d like to know.

Rounding unit and standard floating point system

Without any further details it should already be clear that representing x by fl(x) necessarily causes an error. A central question is how accurate the floating point representation of the real numbers is.

4It is known from calculus that the set of all rational numbers in a given real interval is dense in that interval. This means that any number in the interval, rational or not, can be approached to arbitrary accuracy by a sequence of rational numbers.

✐

✐

✐

✐

✐

✐

✐

✐

2.1. The essentials 19

Specifically, we ask how large the relative error in such a representation, defined by

|fl(x)− x | |x | ,

can be. Modern floating point systems, such as the celebrated IEEE standard described in detail in

Section 2.4, guarantee that this relative error is bounded by

η = 1 2 ×2−t .

The important quantity η, which will be more generally defined in Section 2.2, has been called rounding unit, machine precision, machine epsilon, and more. We will use the term rounding unit throughout.

The usual floating point word in the standard floating point system, which is what MATLAB uses by default, has 64 bits. Of these, 52 bits are devoted to the mantissa (or fraction) while the rest store the sign and the exponent. Hence the rounding unit is

η = 2−53 ≈ 1.1×10−16. This is called double precision; see the schematics in Figure 2.1.

Figure 2.1. A double word (64 bits) in the standard floating point system. The blue bit is for sign, the magenta bits store the exponent, and the green bits are for the fraction.

If the latter name makes you feel a bit like starting house construction from the second floor, then rest assured that there is also a single precision word, occupying 32 bits. This one obviously has a much smaller number of digits t , hence a significantly larger rounding unit η. We will not use single precision for calculations anywhere in this book except for Examples 2.2 and 14.6, and neither should you.

Roundoff error accumulation

Even if number representations were exact in our floating point system, arithmetic operations in- volving such numbers introduce roundoff errors. These errors can be quite large in the relative sense, unless guard digits are used. These are extra digits that are used in interim calculations. The IEEE standard requires exact rounding, which yields that the relative error in each arithmetic operation is also bounded by η.

Given the above soothing words about errors remaining small after representing a number and performing an arithmetic operation, can we really put our minds at ease and count on a long and intense calculation to be as accurate as we want it to be?

Not quite! We have already seen in Example 1.3 that unpleasant surprises may arise. Let us mention a few potential traps. The fuller version is in Section 2.3.

Careless computations can sometimes result in division by 0 or another form of undefined numerical result. The corresponding variable name is then assigned the infamous designation NaN. This is a combination of letters that stands for “not a number,” which naturally one dreads to see in

✐

✐

✐

✐

✐

✐

✐

✐

20 Chapter 2. Roundoff Errors

one’s own calculations, but it allows software to detect problematic situations, such as an attempt to divide 0 by 0, and do something graceful instead of just halting. We have a hunch that you will inadvertently encounter a few NaN’s before you finish implementing all the algorithms in this book.

There is also a potential for an overflow, which occurs when a number is larger than the largest that can be represented in the floating point system. Occasionally this can be avoided, as in Example 2.9 and other examples in Section 2.3.

We have already mentioned in Section 1.3 that a roundoff error accumulation that grows lin- early with the number of operations in a calculation is inevitable. Our calculations will never be so long that this sort of error would surface as a practical issue. Still, there are more pitfalls to watch out for. One painful type of error magnification is a cancellation error, which occurs when two nearly equal numbers are subtracted from one another. There are several examples of this in Section 2.3. Furthermore, the discussion in this chapter suggests that such errors may consistently arise in practice.

The rough appearance of roundoff error

Consider a smooth function g, sampled at some t and at t + h for a small (i.e., near 0) value h. Continuity then implies that the values g(t) and g(t + h) are close to each other. But the rounding errors in the machine representation of these two numbers are unrelated to each other: they are random for all we know! These rounding errors are both small (that’s what η is for), but even their signs may in fact differ. So when subtracting g(t + h)− g(t), for instance, on our way to estimate the derivative of g at t , the relative roundoff error becomes significantly larger as cancellation error naturally arises. This is apparent already in Figure 1.3.

Let us take a further look at the seemingly unstructured behavior of roundoff error as a smooth function is being sampled.

Example 2.2. We evaluate g(t)= e−t (sin(2π t)+2) at 501 equidistant points between 0 and 1, using the usual double precision as well as single precision and plotting the differences. This essentially gives the rounding error in the less accurate single precision evaluations. The following MATLAB instructions do the job:

t = 0:.002:1; tt = exp(-t) .* (sin(2*pi*t)+2); rt = single(tt); round_err = (tt - rt) ./tt ; plot (t,round_err,’b-’); title (’error in sampling exp(-t)(sin(2\pi t)+2) single precision’) xlabel(’t’) ylabel(’roundoff error’)

% relative error should be about eta = eps(single)/2 rel_round_err = max(abs(round_err)) / (eps(’single’)/2)

Thus, the definition of the array values tt is automatically implemented in double precision, while the instruction singlewhen defining the array rt records the corresponding values in single precision.

The resulting plot is depicted in Figure 2.2. Note the disorderly, “high frequency” oscillation of the roundoff error. This is in marked contrast to discretization errors, which are usually “smooth,” as we have seen in Example 1.2. (Recall the straight line drop of the error in Figure 1.3 for relatively large h, which is where the discretization error dominates.)

The output of this program indicates that, as expected, the relative error is at about the rounding unit level. The latter is obtained by the (admittedly unappealing) function call

✐

✐

✐

✐

✐

✐

✐

✐

2.2. Floating point systems 21

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 −8

−6

−4

−2

0

2

4

6 x 10

−8 error in sampling exp(−t)(sin(2π t)+2) in single precision

t

ro un

do ff

er ro

r

Figure 2.2. The “almost random” nature of roundoff errors.

eps(’single’)/2, which yields roughly the value ηsingle = 6e-8. The last line in the script produces approximately the value 0.97, which is indeed close to 1.

Specific exercises for this section: Exercises 1–2.

2.2 Floating point systems

Note: Here is where we get more detailed and technical in the present chapter. Note that we are not assuming the IEEE standard before discussing it in Section 2.4.

A floating point system can be characterized by four values (β, t , L,U ), where

β = base of the number system; t = precision (# of digits);

L = lower bound on exponent e; U = upper bound on exponent e.

Generalizing the binary representation in Section 2.1, we write

fl(x) =± (

d̃0 β0

+ d̃1 β1

+·· ·+ d̃t−1 β t−1

) ×βe,

where β is an integer larger than 1 referred to as the base and d̃i are integer digits in the range 0 ≤ d̃i ≤ β−1. The number fl(x) is an approximation of x . To ensure uniqueness of this representation, it is normalized to satisfy d̃0 �= 0 by adjusting the exponent e so that leading zeros are dropped. Note

✐

✐

✐

✐

✐

✐

✐

✐

22 Chapter 2. Roundoff Errors

that unless β = 2 this does not fix the value of d̃0, which therefore must be stored, too. This is why the last stored digit has index t − 1 and not t as in Section 2.1. In addition, e must be in the range L ≤ e ≤ U .

Chopping and rounding

To store x = ±(d0.d1d2d3 · · ·dt−1dt dt+1 · · · )×βe using only t digits, it is possible to use one of a number of strategies. The two basic ones are

• chopping: ignore digits dt ,dt+1,dt+2,dt+3 . . ., yielding d̃i = di and fl(x) =± d0.d1d2d3 · · ·dt−1 ×βe;

• rounding: consult dt to determine the approximation

fl(x) = { ± d0.d1d2d3 · · ·dt−1 ×βe, dt < β/2, ±(d0.d1d2d3 · · ·dt−1 +β1−t)×βe, dt > β/2.

In case of a tie (dt = β/2), round to the nearest even number.

Example 2.3. Here are results of chopping and rounding with β = 10, t = 3:

x Chopped to Rounded to 3 digits 3 digits

5.672 5.67 5.67

−5.672 −5.67 −5.67 5.677 5.67 5.68

−5.677 −5.67 −5.68 5.692 5.69 5.69

5.695 5.69 5.70

Example 2.4. Since humans are used to decimal arithmetic, let us set β = 10. Consider 8

3 = 2.6666 . . .=

( 2

100 + 6

101 + 6

102 + 6

103 + 6

104 +·· ·

) ×100.

This number gives an infinite series in base 10, although the digit series has finite length in base 3. To represent it using t = 4, chopping gives

8

3 � (

2

100 + 6

101 + 6

102 + 6

103

) ×100 = 2.666×100.

On the other hand, with rounding note that dt = 6 > 5, so β1−t = 10−3 is added to the number before chopping, which gives 2.667×100.

This floating point representation is not unique; for instance, we have

2.666×100 = 0.2666×101. Therefore, we normalize the representation by insisting that d0 �= 0, so

1 ≤ d0 ≤ 9, 0 ≤ di ≤ 9, i = 1, . . . , t −1, which eliminates any ambiguity.

✐

✐

✐

✐

✐

✐

✐

✐

2.2. Floating point systems 23

The number 0 cannot be represented in a normalized fashion. It and the limits ±∞ are repre- sented as special combinations of bits, according to an agreed upon convention for the given floating point system.

Example 2.5. Consider a (toy) decimal floating point system with t = 4, U = 1, and L =−2. Thus, the decimal number 2.666 is precisely representable because it has four digits in its mantissa and L < e < U .

The largest number here is 99.99� 102 = 100, the smallest is −99.99�−100, and the small- est positive number is 10−2 = 0.01.

How many different numbers do we have? The first digit can take on 9 different values, the other three digits 10 values each (because they may be zero, too). Thus, there are 9×10×10×10 = 9,000 different normalized fractions possible. The exponent can be one of U − L + 1 = 4 values, so in total there are 4×9,000 = 36,000 different positive numbers possible. There is the same total of negative numbers, and then there is the number 0. So, there are 72,001 different numbers in this floating point system.

What about the choice of a base? Computer storage is in an integer multiple of bits, hence all computer representations we know of have used bases that are powers of 2. In the 1970s there were architectures with bases 16 and 8. Today, as we have observed in Section 2.1, the standard floating point system uses base β = 2; see Figure 2.1 and Section 2.4.

The error in floating point representation

The relative error is generally a more meaningful measure than absolute error in floating point rep- resentation, because it is independent of a change of exponent. Thus, if in a decimal system (i.e., with β = 10) we have that u = 1,000,000 = 1×106 and v = fl(u) = 990,000 = 9.9×105, we expect to be able to work with such an approximation as accurately as with u = 10 and v = fl(u) = 9.9. This is borne out by the value of the relative error.

Let us denote the floating point representation mapping by x �→ fl(x), and suppose rounding is used. Then the quantity η in the formula on the current page is fundamental as it expresses a bound on the relative error when we represent a number in our prototype floating point system. We have already mentioned in Section 2.1 the rounding unit η. Furthermore, the negative of its exponent, t −1 (for the rounding case), is often referred to as the number of significant digits.

Rounding unit. For a general floating point system (β, t , L,U ) the rounding unit is

η = 1 2 β1−t .

Recall that for the double precision word in the standard floating point system, a bound on the relative error is given by |x −fl(x)|/|x | ≤ η ≈ 1.1e-16.

Floating point arithmetic

As briefly mentioned in Section 2.1, the IEEE standard requires exact rounding, which means that the result of a basic arithmetic operation must be identical to the result obtained if the arith- metic operation was computed exactly and then the result rounded to the nearest floating point

✐

✐

✐

✐

✐

✐

✐

✐

24 Chapter 2. Roundoff Errors

Theorem: Floating Point Representation Error. Let x �→ fl(x) = g ×βe, where x �= 0 and g is the normalized, signed mantissa.

Then the absolute error committed in using the floating point representation of x is bounded by

|x −fl(x)| ≤  β1−t ·βe for chopping,1

2β 1−t ·βe for rounding,

whereas the relative error satisfies

|x −fl(x)| |x | ≤

 β1−t for chopping,1 2β

1−t for rounding.

number. With exact rounding, if fl(x) and fl(y) are machine numbers, then

fl(fl(x)±fl(y)) = (fl(x)±fl(y))(1+ �1), fl(fl(x)×fl(y)) = (fl(x)×fl(y))(1+ �2), fl(fl(x)÷fl(y)) = (fl(x)÷fl(y))(1+ �3),

where |�i | ≤ η. Thus, the relative errors remain small after each such operation.

Example 2.6. Consider a floating point system with decimal base β = 10 and four digits, i.e., t = 4. Thus, the rounding unit is η = 12 ×10−3. Let

x = .1103 = 1.103×10−1, y = 9.963×10−3. Subtracting these two numbers, the exact value is x − y = .100337. Hence, exact rounding yields .1003. Obviously, the relative error is

|.100337− .1003| .100337

≈ .37×10−3 < η.

However, if we were to subtract these two numbers without guard digits we would obtain .1103− .0099= .1004. Now the obtained relative error is not below η, because

|.100337− .1004| .100337

≈ .63×10−3 > η.

Thus, guard digits must be used to produce exact rounding.

Example 2.7. Generally, proper rounding yields fl(1+α) = 1 for any number α that satisfies |α| ≤ η. In particular, the MATLAB commands

eta = .5*2^(-52), beta = (1+eta)-1

produce the output eta= 1.1102e-16, beta= 0. Returning to Example 1.3 and to Figure 1.3, we can now explain why the curve of the error is flat for the very, very small values of h. For such values, fl( f (x0 +h)) = fl( f (x0)), so the approximation is precisely zero and the recorded values are those of fl( f ′(x0)), which is independent of h.

✐

✐

✐

✐

✐

✐

✐

✐

2.2. Floating point systems 25

Spacing of floating point numbers

If you think of how a given floating point system represents the real line you’ll find that it has a somewhat uneven nature. Indeed, very large numbers are not represented at all, and the distance between two neighboring, positive floating point values is constant in the relative but not in the absolute sense.

Example 2.8. Let us plot the decimal representation of the numbers in the system specified by (β, t , L,U ) = (2,3,−2,3). The smallest possible number in the mantissa d0.d1d2 is 1.00 and the largest possible number is 1.11 in binary, which is equal to 1+ 1/2+ 1/4 = 1.75. So, we will run in a loop from 1 to 1.75 in increments of β1−t = 2−2 = 0.25. This is the basis for one loop. The exponent runs from −2 to 3, which is what we loop over in the second (nested) loop. The resulting double loop builds up an array with all the positive elements of the system.

Here is a MATLAB script that plots the numbers of the system. Notice the format in the plot command.

x = [];

% Generate all positive numbers of the system (2,3,-2,3) for i = 1:0.25:1.75

for j = -2:3 x = [x i*2^j];

end end

x=[x -x 0]; % Add all negative numbers and 0 x = sort(x); % Sort y = zeros(1,length(x)); plot(x,y,’+’)

The resulting plot is in Figure 2.3. Note that the points are not distributed uniformly along the real line.

The rounding unit for this system is

η = 1 2 β1−t = 1

2 ×21−3 = 1

8 ,

which is half the spacing between 1 = β0 and the next number in the system. (Can you see why?)

Most annoying in Figure 2.3 is the fact that the distance between the value 0 and the smallest positive number is significantly larger than the distance between that same smallest positive number and the next smallest positive number! A commonly used way to remedy this in modern floating point systems is by introducing next to 0 additional, subnormal numbers which are not normalized. We will leave it at that.

MATLAB has a parameter called eps that signifies the spacing of floating point numbers. In particular, eps(1) gives twice the rounding unit η. Enter help eps to learn more about this parameter.

Specific exercises for this section: Exercises 3–8.

✐

✐

✐

✐

✐

✐

✐

✐

26 Chapter 2. Roundoff Errors

−15 −10 −5 0 5 10 15 −1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Figure 2.3. Picture of the floating point system described in Example 2.8.

2.3 Roundoff error accumulation Because many operations are being performed in the course of carrying out a numerical algorithm, many small errors unavoidably result. We know that each elementary floating point operation may introduce a small relative error, but how do these errors accumulate?

In general, as we have already mentioned in Chapter 1, error growth that is linear in the number of operations is unavoidable. Yet, there are a few things to watch out for.

• Error magnification: 1. If x and y differ widely in magnitude, then x + y has a large absolute error. 2. If |y| � 1, then x/y may have large relative and absolute errors. Likewise for xy if

|y| 1. 3. If x � y, then x − y has a large relative error (cancellation error).

• Overflow and underflow: An overflow is obtained when a number is too large to fit into the floating point system in use, i.e., when e > U . An underflow is obtained when e < L. When overflow occurs in the course of a calculation, this is generally fatal. But underflow is nonfatal: the system usually sets the number to 0 and continues. (MATLAB does this, quietly.)

It is also worth mentioning again the unfortunate possibility of running into NaN, described in Section 2.1.

Design of library functions

You may wonder what the fuss is all about, if the representation of a number and basic arithmetic operations cause an error as small as about 10−16 in a typical floating point system. But taking this

✐

✐

✐

✐

✐

✐

✐

✐

2.3. Roundoff error accumulation 27

issue lightly may occasionally cause surprisingly serious damage. In some cases, even if the com- putation is long and intensive it may be based on or repeatedly use a short algorithm, and a simple change in a formula may greatly improve performance with respect to roundoff error accumulation. For those operations that are carried out repeatedly in a typical calculation, it may well be worth obtaining a result as accurate as possible, even at the cost of a small computational overhead.

Example 2.9. The most common way to measure the size of a vector x = (x1, x2, . . . , xn)T is the Euclidean norm (also called �2-norm) defined by

‖x‖2 = √

x21 + x22 +·· ·+ x2n .

We discuss vector norms and their properties in Section 4.2. Any standard numerical linear algebra software package has a library function that computes the norm, and often the input vector is very large; see, for instance, Chapter 7 for relevant situations. One of the dangers to avoid in computing the norm is the possibility of overflow. To illustrate this point, let us look at a small vector with just two but widely differing components.

Consider computing c =√a2 +b2 in a floating point system with four decimal digits and two exponent digits, for a = 1060 and b = 1. Thus, c is the Euclidean norm of the vector x = (1060,1)T . The correct result here is c = 1060. But overflow will result during the course of calculation, because squaring a gives 10120, which cannot be represented in this system. Yet, this overflow can easily be avoided if we rescale ahead of time, noting that c = s√(a/s)2 + (b/s)2 for any s �= 0. Thus, using s = a = 1060 gives an underflow when b/s is squared, which is set to zero. This yields the correct answer. (It is important to stress that “correct answer” here refers not necessarily to the “true,” precise answer on the real line, but rather to the most accurate answer we can obtain given this particular floating point system.)

The above principle can be generalized to the computation of the norm of a vector of more than two components, simply scaling by the largest one.

Another issue is the order in which the values x2i are summed. Doing this in increasing order of magnitude reduces error accumulation; see, for instance, Exercise 17.

In Exercise 18 you are asked to design a library function for computing the �2-norm of a vector.

Avoiding cancellation error

Another trouble spot to watch out for is cancellation error, mentioned in Section 2.1. Some insight into the damaging effect of subtracting two nearly equal numbers in a floating point system can be observed by deriving a bound on the error. Suppose z = x − y, where x ≈ y. Then

|z −fl(z)| ≤ |x −fl(x)|+ |y−fl(y)| ,

from which it follows that the relative error satisfies

|z −fl(z)| |z| ≤

|x −fl(x)|+ |y−fl(y)| |x − y| .

The numerator of the bound could be tight since it merely depends on how well the floating point system can represent the numbers x and y. But the denominator is very close to zero if x ≈ y regardless of how well the floating point system can work for these numbers, and so the relative error in z could become large.

✐

✐

✐

✐

✐

✐

✐

✐

28 Chapter 2. Roundoff Errors

Example 2.10. The roots of the quadratic equation

x2 −2bx + c = 0 with b2 > c are given by

x1,2 = b± √

b2 − c. The following simple script gives us the roots according to the formula:

x1 = b + sqrt(b^2-c); x2 = b - sqrt(b^2-c);

Unfortunately, if b2 is significantly larger than c, to the point that √

b2 − c ≈ |b|, then one of the two calculated roots is bound to be approximately zero and inaccurate. (There is no problem with the other root, which would be approximately equal to 2b in this case.)

To our aid comes the formula x1x2 = c. We can then avoid the cancellation error by using a modified algorithm, given by the script

if b > 0 x1 = b + sqrt(b^2-c); x2 = c / x1;

else x2 = b - sqrt(b^2-c); x1 = c / x2;

end

The algorithm can be further improved by also taking into consideration the possibility of an

overflow. Thus, if b2 c, we replace √b2 − c by |b| √

1− c b2

, in a way similar to the technique

described in Example 2.9. In Exercise 16 you are asked to design and implement a robust quadratic equation solver.

Cancellation error deserves special attention because it often appears in practice in an identi- fiable way. For instance, recall Example 1.2. If we approximate a derivative of a smooth (differen- tiable) function f (x) at a point x = x0 by the difference of two neighboring values of f divided by the difference of the arguments with a small step h, e.g., by

f ′(x0) ≈ f (x0 +h)− f (x0) h

,

then there is a cancellation error in the numerator, which is then magnified by the denominator. Recall also the discussion in Section 2.1, just before Example 2.2.

When cancellation errors appear in an identifiable way, they can often be avoided by a simple modification in the algorithm. An instance is illustrated in Exercise 2. Here is another one.

Example 2.11. Suppose we wish to compute y = √x +1 −√x for x = 100,000 in a five-digit decimal arithmetic. Clearly, the number 100,001 cannot be represented in this floating point system exactly, and its representation in the system (when either chopping or rounding is used) is 100,000. In other words, for this value of x in this floating point system, we have x + 1 = x . Thus, naively computing

√ x +1−√x results in the value 0.

We can do much better if we use the identity

( √

x +1−√x)(√x +1+√x) ( √

x +1+√x) = 1√

x +1+√x .

✐

✐

✐

✐

✐

✐

✐

✐

2.4. The IEEE standard 29

Applying this formula (i.e., computing the right-hand expression in 5-digit decimal arithmetic) yields 1.5811×10−3, which happens to be the correct value to 5 decimal digits.

There are also other ways to avoid cancellation error, and one popular technique is the use of a Taylor expansion (page 5).

Example 2.12. Suppose we wish to design a library function for computing

y = sinh(x) = 1 2

(ex − e−x ).

Think, for example, of having a scientific calculator: we want to be sure that it gives an accurate result for any argument x .

While the above formula can be straightforwardly applied (assuming we have a decent library function for computing exponents), the going may get a little tougher for values of x near 0. Directly using the formula for computing y may be prone to severe cancellation errors, due to the subtraction of two quantities that are approximately equal to 1. On the other hand, using the Taylor expansion

sinh(x) = x + x 3

6 + ξ

5

120

for some ξ satisfying |ξ | ≤ |x | may prove useful. The simple cubic expression x + x36 should give an effective approximation if |x | is sufficiently small, as the discretization error can be bounded by |x |5 120 and the roundoff error is no longer an issue.

Employing 5-digit decimal arithmetic for our cubic approximation, we compute sinh(0.1) = 0.10017 and sinh(0.01) = 0.01. (Believe us! Or otherwise please feel free to verify.) These are the “exact” values in this floating point system. On the other hand, using the formula that involves the exponential functions, which is the exact formula and would produce the exact result had we not had roundoff errors, we obtain 0.10018 and 0.010025 for these two values of x , respectively, so there are pronounced errors.

For x = 0.01 it is in fact sufficient to use only one term of the Taylor expansion, i.e., approxi- mate sinh(x) by x in this toy floating point system.

Specific exercises for this section: Exercises 9–19.

2.4 The IEEE standard During the dawn of the modern computing era there was chaos. Then, in 1985, came the IEEE standard, slightly updated in 2008. Today, except for special circumstances such as a calculator or a dedicated computer on a special-purpose device, every software designer and hardware manufacturer follows this standard. Thus, the output of one’s program no longer varies upon switching laptops or computing environments using the same language. Here we describe several of the nuts and bolts (from among the less greasy ones) of this standard floating point system.

Recall from Section 2.1 that the base is β = 2. In decimal value we have

fl(x) =± (

1+ d̃1 2

+ d̃2 4

+·· ·+ d̃t 2t

) ×2e,

where d̃i are binary digits each requiring one bit for storing.

✐

✐

✐

✐

✐

✐

✐

✐

30 Chapter 2. Roundoff Errors

The standard and the single precision floating point words

The standard floating point word used exclusively for all calculations in this book except Exam- ples 2.2 and 14.7 requires 64 bits of storage and is called double precision or long word. This is the MATLAB default. Of these 64 bits, one is allocated for sign s (the number is negative if and only if s = 1), 11 for the exponent, and t = 52 for the fraction:

Double precision (64-bit word)

s =± b = 11-bit exponent f = 52-bit fraction β = 2, t = 52, L =−1022, U = 1023

Since the fraction f contains t digits the precision is 52+1 = 53. Thus, a given bit pattern in the last 52 bits of a long word is interpreted as a sequence of binary digits d1d2 · · ·d52. A given bit pattern in the exponent part of the long word is interpreted as a positive integer b in binary representation that yields the exponent e upon shifting by L, i.e., e = b−1023.

Example 2.13. The sequence of 64 binary digits

0100000001111110100000000000000000000000000000000000000000000000

considered as a double precision word can be interpreted as follows:

• Based on the first digit 0 the sign is positive by convention. • The next 11 digits,

10000000111,

form the exponent: in decimal, b = 1×210 +1×22 +1×21 +1×20 = 1031. So in decimal, e = b−1023 = 1031−1023 = 8.

• The next 52 bits, 1110100000000000000000000000000000000000000000000000,

form the decimal fraction: f = 12 + 14 + 18 + 132 = 0.90625. • The number in decimal is therefore

s · (1+ f )×2e = 1.90625×28 = 488.

There is also a single precision arrangement of 32 bits, as follows:

Single precision (32-bit word)

s =± b = 8-bit exponent f = 23-bit fraction β = 2, t = 23, L =−126, U = 127

Storing special values

Upon carefully examining the possible range of exponents you will notice that it is not fully utilized. For double precision 211 = 2048 different exponents could be distinguished, and for single precision 28 = 256 are available. But only 2046 and 254, respectively, are used in practice. This is because

✐

✐

✐

✐

✐

✐

✐

✐

2.4. The IEEE standard 31

the IEEE standard reserves the endpoints of the exponent range for special purposes. The largest number precisely representable in a standard long word is therefore[

1+ 52∑

i=1

( 1

2

)i] ×21023 = (2−2−52)×21023 ≈ 21024 ≈ 10308,

and the smallest positive number is

[1+0]×2−1022 = 2−1022 ≈ 2.2×10−308. How are 0 and ∞ stored? Here is where the endpoints of the exponent are used, and the

conventions are simple and straightforward:

• For 0, set b = 0, f = 0, with s arbitrary; i.e., the minimal positive value representable in the system is considered 0.

• For ±∞, set b = 1 · · ·1, f = 0. • The pattern b = 1 · · ·1, f �= 0 is by convention NaN.

In single precision, or short word, the largest number precisely representable is[ 1+

23∑ i=1

( 1

2

)i] ×2127 = (2−2−23)×2127 ≈ 2128 ≈ 3.4×1038,

and the smallest positive number is

[1+0]×2−126 = 2−126 ≈ 1.2×10−38.

Rounding unit

The formulas for the machine precision or rounding unit η were introduced on page 19 in Section 2.1. Note again the shift from t −1 to t in the power of the base as compared to the general formula given on page 23.

Using the word arrangement for single and double precision, η is therefore calculated as fol- lows:

• For single precision, η = 12 ·β−t = 12 · 2−23 ≈ 6.0× 10−8 (so, there are 23 significant binary digits, or about 7 decimal digits).

• For double precision, η = 12 ·β−t = 12 ·2−52 ≈ 1.1×10−16 (so, there are 52 significant binary digits, or about 16 decimal digits).

Typically, single and double precision floating point systems as described above are imple- mented in hardware. There is also quadruple precision (128 bits), often implemented in software and thus considerably slower, for applications that require very high precision (e.g., in semiconduc- tor simulation, numerical relativity and astronomical calculations).

The fundamentally important exact rounding, mentioned in both Sections 2.1 and 2.2, has a rather lengthy definition for its implementation, which stands in contrast to the cleanly stated requirement of its result. We will not dive deeper into this.

Specific exercises for this section: Exercises 20–21.

✐

✐

✐

✐

✐

✐

✐

✐

32 Chapter 2. Roundoff Errors

2.5 Exercises 0. Review questions

(a) What is a normalized floating point number and what is the purpose of normalization?

(b) A general floating point system is characterized by four values (β, t , L,U ). Explain in a few brief sentences the meaning and importance of each of these parameters.

(c) Write down the floating point representation of a given real number x in a decimal sys- tem with t = 4, using (i) chopping and (ii) rounding.

(d) Define rounding unit (or machine precision) and explain its importance.

(e) Define overflow and underflow. Why is the former considered more damaging than the latter?

(f) What is a cancellation error? Give an example of an application where it arises in a natural way.

(g) What is the rounding unit for base β = 2 and t = 52 digits? (h) Under what circumstances could nonnormalized floating point numbers be desirable?

(i) Explain the storage scheme for single precision and double precision numbers in the IEEE standard.

1. The fraction in a single precision word has 23 bits (alas, less than half the length of the double precision word).

Show that the corresponding rounding unit is approximately 6×10−8. 2. The function f1(x0,h) = sin(x0+h)−sin(x0) can be transformed into another form, f2(x0,h),

using the trigonometric formula

sin(φ)− sin(ψ) = 2cos ( φ+ψ

2

) sin

( φ−ψ

2

) .

Thus, f1 and f2 have the same values, in exact arithmetic, for any given argument values x0 and h.

(a) Derive f2(x0,h).

(b) Suggest a formula that avoids cancellation errors for computing the approximation ( f (x0 + h)− f (x0))/h to the derivative of f (x) = sin(x) at x = x0. Write a MATLAB pro- gram that implements your formula and computes an approximation of f ′(1.2), for h = 1e-20,1e-19, . . .,1.

(c) Explain the difference in accuracy between your results and the results reported in Ex- ample 1.3.

3. (a) How many distinct positive numbers can be represented in a floating point system using base β = 10, precision t = 2 and exponent range L =−9, U = 10? (Assume normalized fractions and don’t worry about underflow.)

(b) How many normalized numbers are represented by the floating point system (β , t , L,U )? Provide a formula in terms of these parameters.

4. Suppose a computer company is developing a new floating point system for use with their machines. They need your help in answering a few questions regarding their system. Fol- lowing the terminology of Section 2.2, the company’s floating point system is specified by (β, t , L,U ). Assume the following:

✐

✐

✐

✐

✐

✐

✐

✐

2.5. Exercises 33

• All floating point values are normalized (except the floating point representation of zero). • All digits in the mantissa (i.e., fraction) of a floating point value are explicitly stored. • The number 0 is represented by a float with a mantissa and an exponent of zeros. (Don’t

worry about special bit patterns for ±∞ and NaN.) Here is your part:

(a) How many different nonnegative floating point values can be represented by this floating point system?

(b) Same question for the actual choice (β, t , L,U ) = (8,5,−100,100) (in decimal) which the company is contemplating in particular.

(c) What is the approximate value (in decimal) of the largest and smallest positive numbers that can be represented by this floating point system?

(d) What is the rounding unit?

5. (a) The number 87 = 1.14285714285714 . . . obviously has no exact representation in any decimal floating point system (β = 10) with finite precision t . Is there a finite floating point system (i.e., some finite integer base β and precision t) in which this number does have an exact representation? If yes, then describe such a system.

(b) Answer the same question for the irrational number π .

6. Write a MATLAB program that receives as input a number x and a parameter n and returns x rounded to n decimal digits. Write your program so that it can handle an array as input, returning an array of the same size in this case.

Use your program to generate numbers for Example 2.2, demonstrating the phenomenon de- picted there without use of single precision.

7. Prove the Floating Point Representation Error Theorem on page 24.

8. Rewrite the script of Example 2.8 without any use of loops, using vectorized operations in- stead.

9. Suggest a way to determine approximately the rounding unit of your calculator. State the type of calculator you have and the rounding unit you have come up with. If you do not have a calculator, write a short MATLAB script to show that your algorithm works well on the standard IEEE floating point system.

10. The function f1(x ,δ) = cos(x + δ)− cos(x) can be transformed into another form, f2(x ,δ), using the trigonometric formula

cos(φ)− cos(ψ) =−2sin ( φ+ψ

2

) sin

( φ−ψ

2

) .

Thus, f1 and f2 have the same values, in exact arithmetic, for any given argument values x and δ.

(a) Show that, analytically, f1(x ,δ)/δ or f2(x ,δ)/δ are effective approximations of the func- tion −sin(x) for δ sufficiently small.

(b) Derive f2(x ,δ).

(c) Write a MATLAB script which will calculate g1(x ,δ)= f1(x ,δ)/δ+sin(x) and g2(x ,δ)= f2(x ,δ)/δ+ sin(x) for x = 3 and δ = 1.e-11.

(d) Explain the difference in the results of the two calculations.

✐

✐

✐

✐

✐

✐

✐

✐

34 Chapter 2. Roundoff Errors

11. (a) Show that

ln (

x − √

x2 −1 ) =− ln

( x +

√ x2 −1

) .

(b) Which of the two formulas is more suitable for numerical computation? Explain why, and provide a numerical example in which the difference in accuracy is evident.

12. For the following expressions, state the numerical difficulties that may occur, and rewrite the formulas in a way that is more suitable for numerical computation:

(a) √

x + 1x − √

x − 1x , where x 1.

(b)

√ 1

a2 + 1

b2 , where a ≈ 0 and b ≈ 1.

13. Consider the linear system  a b b a

 x y

=  1

0

 with a,b > 0.

(a) If a ≈ b, what is the numerical difficulty in solving this linear system? (b) Suggest a numerically stable formula for computing z = x + y given a and b. (c) Determine whether the following statement is true or false, and explain why:

“When a ≈ b, the problem of solving the linear system is ill-conditioned but the problem of computing x + y is not ill-conditioned.”

14. Consider the approximation to the first derivative

f ′(x) ≈ f (x +h)− f (x) h

.

The truncation (or discretization) error for this formula is O(h). Suppose that the absolute error in evaluating the function f is bounded by ε and let us ignore the errors generated in basic arithmetic operations.

(a) Show that the total computational error (truncation and rounding combined) is bounded by

Mh

2 + 2ε

h ,

where M is a bound on | f ′′(x)|. (b) What is the value of h for which the above bound is minimized?

(c) The rounding unit we employ is approximately equal to 10−16. Use this to explain the behavior of the graph in Example 1.3. Make sure to explain the shape of the graph as well as the value where the apparent minimum is attained.

✐

✐

✐

✐

✐

✐

✐

✐

2.5. Exercises 35

(d) It is not difficult to show, using Taylor expansions, that f ′(x) can be approximated more accurately (in terms of truncation error) by

f ′(x) ≈ f (x +h)− f (x −h) 2h

.

For this approximation, the truncation error is O(h2). Generate a graph similar to Fig- ure 1.3 (please generate only the solid line) for the same function and the same value of x , namely, for sin(1.2), and compare the two graphs. Explain the meaning of your results.

15. Suppose a machine with a floating point system (β, t , L,U ) = (10,8,−50,50) is used to cal- culate the roots of the quadratic equation

ax2 +bx + c = 0, where a, b, and c are given, real coefficients.

For each of the following, state the numerical difficulties that arise if one uses the standard formula for computing the roots. Explain how to overcome these difficulties (when possible).

(a) a = 1 ; b =−105 ; c = 1. (b) a = 6 ·1030 ; b = 5 ·1030 ; c =−4 ·1030. (c) a = 10−30 ; b = −1030 ; c = 1030.

16. Write a quadratic equation solver. Your MATLAB script should get a,b,c as input, and accurately compute the roots of the corresponding quadratic equation. Make sure to check end cases such as a = 0, and consider ways to avoid an overflow and cancellation errors. Implement your algorithm and demonstrate its performance on a few cases (for example, the cases mentioned in Exercise 15). Show that your algorithm produces better results than the standard formula for computing roots of a quadratic equation.

17. Write a MATLAB program that

(a) sums up 1/n for n = 1,2, . . . ,10,000; (b) rounds each number 1/n to 5 decimal digits and then sums them up in 5-digit decimal

arithmetic for n = 1,2, . . . ,10,000; (c) sums up the same rounded numbers (in 5-digit decimal arithmetic) in reverse order, i.e.,

for n = 10,000, . . . ,2,1. Compare the three results and explain your observations. For generating numbers with the requested precision, you may want to do Exercise 6 first.

18. (a) Explain in detail how to avoid overflow when computing the �2-norm of a (possibly large in size) vector.

(b) Write a MATLAB script for computing the norm of a vector in a numerically stable fashion. Demonstrate the performance of your code on a few examples.

19. In the statistical treatment of data one often needs to compute the quantities

x̄ = 1 n

n∑ i=1

xi , s2 = 1 n

n∑ i=1

(xi − x̄)2,

✐

✐

✐

✐

✐

✐

✐

✐

36 Chapter 2. Roundoff Errors

where x1, x2, . . . , xn are the given data. Assume that n is large, say, n = 10,000. It is easy to see that s2 can also be written as

s2 = 1 n

n∑ i=1

x2i − x̄2.

(a) Which of the two methods to calculate s2 is cheaper in terms of overall computational cost? Assume x̄ has already been calculated and give the operation counts for these two options.

(b) Which of the two methods is expected to give more accurate results for s2 in general?

(c) Give a small example, using a decimal system with precision t = 2 and numbers of your choice, to validate your claims.

20. With exact rounding, we know that each elementary operation has a relative error which is bounded in terms of the rounding unit η; e.g., for two floating point numbers x and y, fl(x + y) = (x + y)(1+ �), |�| ≤ η. But is this true also for elementary functions such as sin, ln, and exponentiation?

Consider exponentiation, which is performed according to the formula

x y = ey ln x (assuming x > 0). Estimate the relative error in calculating x y in floating point, assuming fl(ln z) = (lnz)(1+ �), |�| ≤ η, and that everything else is exact. Show that the sort of bound we have for elemen- tary operations and for ln does not hold for exponentiation when x y is very large.

21. The IEEE 754 (known as the floating point standard) specifies the 128-bit word as having 15 bits for the exponent.

What is the length of the fraction? What is the rounding unit? How many significant decimal digits does this word have?

Why is quadruple precision more than twice as accurate as double precision, which is in turn more than twice as accurate as single precision?

2.6 Additional notes A lot of thinking in the early days of modern computing went into the design of floating point sys- tems for computers and scientific calculators. Such systems should be economical (fast execution in hardware) on one hand, yet they should also be reliable, accurate enough, and free of unusual exception-handling conventions on the other hand. W. Kahan was particularly instrumental in such efforts (and received a Turing award for his contributions), especially in setting up the IEEE stan- dard. The almost universal adoption of this standard has significantly increased both reliability and portability of numerical codes. See Kahan’s webpage for various interesting related documents: http://www.cs.berkeley.edu/∼wkahan/.

A short, accessible textbook that discusses floating point systems in great detail is Over- ton [58]. A comprehensive and thorough treatment of roundoff errors and many aspects of numerical stability can be found in Higham [40].

The practical way of working with floating point arithmetic, which is to attempt to keep errors “small enough” so as not be a bother, is hardly satisfactory from a theoretical point of view. Indeed, what if we want to use a floating point calculation for the purpose of producing a mathematical proof?! The nature of the latter is that a stated result should always—not just usually—hold true. A

✐

✐

✐

✐

✐

✐

✐

✐

2.6. Additional notes 37

more careful approach uses interval arithmetic. With each number are associated a lower and an upper bound, and these are propagated with the algorithm calculations on a “worst case scenario” basis. The calculated results are then guaranteed to be within the limits of the calculated bounds. See Moore, Kearfott, and Cloud [54] and also [58] for an introduction to this veteran subject. Nat- urally, such an approach is expensive and can be of limited utility, as the range between the bounds typically grows way faster than the accumulated error itself. But at times this approach does work for obtaining truly guaranteed results.

A move to put (more generally) complexity theory for numerical algorithms on firmer foun- dations was initiated by S. Smale and others in the 1980s; see [9]. These efforts have mushroomed into an entire organization called Foundations of Computational Mathematics (FOCM) that is in- volved with various interesting activities which concentrate on the more mathematically rich aspects of numerical computation.

✐

✐

✐

✐

✐

✐

✐

✐

Chapter 3

Nonlinear Equations in One Variable

This chapter is concerned with finding solutions to the scalar, nonlinear equation

f (x) = 0, where the variable x runs in an interval [a,b]. The topic provides us with an opportunity to discuss various issues and concepts that arise in more general circumstances.

There are many canned routines that do the job of finding a solution to a nonlinear scalar equation; the one in MATLAB is called fzero; type help fzero to see how this function is used and what input parameters it requires.

Following an extended introduction to our topic in Section 3.1 are three sections, 3.2–3.4, each devoted to a different method or a class of methods for solving our nonlinear equation. A closely related problem is that of minimizing a function in one variable, and this is discussed in Section 3.5.

3.1 Solving nonlinear equations Referring to our prototype problem introduce above, f (x)= 0, let us further assume that the function f is continuous on the interval, denoted f ∈ C[a,b]. Throughout our discussion we denote a solution of the equation (called root, or zero) by x∗.

Note: Why introduce nonlinear equations before introducing their easier comrades, the linear ones?! Because one linear equation in one unknown is just too easy and not particu- larly illuminating, and systems of equations bring a wave of complications with them. We have a keen interest in solving scalar nonlinear equations not only because such problems arise frequently in many applications, but also because it is an opportunity to discuss var- ious issues and concepts that arise in more general circumstances and highlight ideas that are extensible well beyond just one equation in one unknown.

Example 3.1. Here are a few simple functions and their roots.

1. f (x) = x −1, on the interval [a,b] = [0,2]. Obviously there is one root for this linear equation: x∗ = 1.

39

✐

✐

✐

✐

✐

✐

✐

✐

40 Chapter 3. Nonlinear Equations in One Variable

0 2 4 6 8 10 12 14 −1

0

1

x

f( x)

0 2 4 6 8 10 12 14 16 18 20 −2000

0

2000

4000

x

f( x)

−10 −8 −6 −4 −2 0 2 4 6 8 10 0

50

100

x

f( x)

Figure 3.1. Graphs of three functions and their real roots (if there are any): (i) f (x) = sin(x) on [0,4π], (ii) f (x) = x3 − 30x2 + 2552 on [0,20], and (iii) f (x) = 10cosh(x/4)− x on [−10,10].

2. f (x) = sin(x). Since sin(nπ) = 0 for any integer n, we have

(a) On the interval [a,b] = [π2 , 3π2 ] there is one root, x∗ = π . (b) On the interval [a,b] = [0,4π] there are five roots; see Figure 3.1.

3. f (x) = x3 −30x2 +2552, 0 ≤ x ≤ 20. In general, a cubic equation with complex coefficients has three complex roots. But if the polynomial coefficients are real and x is restricted to be real and lie in a specific interval, then there is no general a priori rule as to how many (real) roots to expect. A rough plot of this function on the interval [0,20] is given in Figure 3.1. Based on the plot we suspect there is precisely one root x∗ in this interval.

4. f (x) = 10cosh(x/4)− x , −∞< x < ∞, where the function cosh is defined by cosh(t) = et+e−t2 . Not every equation has a solution. This one has no real roots.

Figure 3.1 indicates that the function φ(x) = 10cosh(x/4)− x has a minimum. To find the minimum we differentiate and equate to 0. Let f (x)=φ′(x)= 2.5sinh(x/4)−1 (where, analogously to cosh, we define sinh(t) = et−e−t2 ). This function should have a zero in the interval [0,4]. For more on finding a function’s minimum, see Section 3.5.

✐

✐

✐

✐

✐

✐

✐

✐

3.1. Solving nonlinear equations 41

Example 3.2. Here is the MATLAB script that generates Figure 3.1.

t = 0:.1:4*pi; tt = sin(t); ax = zeros(1,length(t)); xrt = 0:pi:4*pi; yrt = zeros(1,5); subplot(3,1,1) plot(t,tt,’b’,t,ax,’k’,xrt,yrt,’rx’); xlabel(’x’) ylabel(’f(x)’)

t = 0:.1:20; tt = t.^3 - 30*t.^2 + 2552; ax = zeros(1,length(t)); subplot(3,1,2) plot(t,tt,’b’,t,ax,’k’,11.8615,0,’rx’); xlabel(’x’) ylabel(’f(x)’)

t = -10:.1:10; tt = 10 * cosh(t ./4) - t; ax = zeros(1,length(t)); subplot(3,1,3) plot(t,tt,’b’,t,ax,’k’); xlabel(’x’) ylabel(’f(x)’)

This script should not be too difficult to read like text. Note the use of array arguments, for instance, in defining the array tt in terms of the array t.

Iterative methods for finding roots

It is not realistic to expect, except in special cases, to find a solution of a nonlinear equation by using a closed form formula or a procedure that has a finite, small number of steps. Indeed, it is enough to consider finding roots of polynomials to realize how rare closed formulas are: they practically exist only for very low order polynomials. Thus, one has to resort to iterative methods: starting with an initial iterate x0, the method generates a sequence of iterates x1, x2, . . . , xk , . . . that (if all works well) converge to a root of the given, continuous function. In general, our methods will require rough knowledge of the root’s location. To find more than one root, we can fire up the same method starting from different initial iterates x0.

To find approximate locations of roots we can roughly plot the function, as done in Exam- ple 3.1. (Yes, it sounds a little naive, but sometimes complicated things can be made simple by one good picture.) Alternatively, we can probe the function, i.e., evaluate it at several points, looking for locations where f (x) changes sign.

If f (a) · f (b) < 0, i.e., f changes sign on the interval [a,b], then by the Intermediate Value Theorem given on page 10 there is a number c = x∗ in this interval for which f (c) = s = 0. To see this intuitively, imagine trying to graph a scalar function from a positive to a negative value, without lifting the pen (because the function is continuous): somewhere the curve has to cross the x-axis!

Such simple procedures for roughly locating roots are unfortunately not easy to generalize to several equations in several unknowns.

✐

✐

✐

✐

✐

✐

✐

✐

42 Chapter 3. Nonlinear Equations in One Variable

Stopping an iterative procedure

Typically, an iterative procedure starts with an initial iterate x0 and yields a sequence of iterates x1, x2, . . . , xk , . . . . Note that in general we do not expect the iterative procedure to produce the exact solution x∗ exactly. We would conclude that the series of iterates converges if the values of | f (xk)| and/or of |xk − xk−1| decrease towards 0 sufficiently fast as the iteration counter k increases.

Correspondingly, one or more of the following general criteria are used to terminate such an iterative process successfully after n iterations:

|xn − xn−1| < atol and/or |xn − xn−1| < rtol|xn| and/or

| f (xn)| < ftol, where atol, rtol, and ftol are user-specified constants.

Usually, but not always, the second, relative criterion is more robust than the first, absolute one. A favorite combination uses one tolerance value tol, which reads

|xn − xn−1| < tol(1+|xn|). The third criterion is independent of the first two; it takes the function value into account. The function f (x) may in general be very flat, or very steep, or neither, near its root.

Desired properties of root finding algorithms

When assessing the qualities of a given root finding algorithm, a key property is its efficiency. In determining an algorithm’s efficiency it is convenient to concentrate on the number of function evaluations, i.e., the evaluations of f (x) at the iterates {xk}, required to achieve convergence to a given accuracy. Other details of the algorithm, which may be considered as overhead, are then generally neglected.5 Now, if the function f (x) is as simple to evaluate as those considered in Example 3.1, then it is hard to understand why we concentrate on this aspect alone. But in these circumstances any of the algorithms considered in this chapter is very fast indeed. What we really keep in the back of our minds is a possibility that the evaluation of f (x) is rather costly. For instance, think of simulating a space shuttle returning to earth, with x being a control parameter that affects the distance f (x) of the shuttle’s landing spot from the location of the reception committee awaiting this event. The calculation of f (x) for each value of x involves a precise simulation of the flight’s trajectory for the given x , and it may then be very costly. An algorithm that does not require too many such function evaluations is then sought.

Desirable qualities of a root finding algorithm are the following:

• Efficient—requires a small number of function evaluations. • Robust—fails rarely, if ever. Announces failure if it does fail. • Requires a minimal amount of additional data such as the function’s derivative. • Requires f to satisfy only minimal smoothness properties. • Generalizes easily and naturally to many equations in many unknowns.

No algorithm we are aware of satisfies all of these criteria. Moreover, which of these is more important to honor depends on the application. So we study several possibilities in the following sections.

5An exception is the number of evaluations of the derivative f ′(x) required, for instance, by Newton’s method. See Section 3.3.

✐

✐

✐

✐

✐

✐

✐

✐

3.2. Bisection method 43

3.2 Bisection method The method developed in this section is simple and safe and requires minimal assumptions on the function f (x). However, it is also slow and hard to generalize to higher dimensions.

Suppose that for a given f (x) we know an interval [a,b] where f changes sign, i.e., f (a) · f (b) < 0. The Intermediate Value Theorem given on page 10 then assures us that there is a root x∗ such that a ≤ x∗ ≤ b. Now evaluate f (p), where p = a+b2 is the midpoint, and check the sign of f (a) · f (p). If it is negative, then the root is in [a, p] (by the same Intermediate Value Theorem), so we can set b ← p and repeat; else f (a) · f (p) is positive, so the root must be in [ p,b], hence we can set a ← p and repeat. (Of course, if f (a) · f (p) = 0 exactly, then p is the root and we are done.)

In each such iteration the interval [a,b] where x∗ is trapped shrinks by a factor of 2; at the kth step, the point xk is the midpoint p of the kth subinterval trapping the root. Thus, after a total of n iterations, |x∗ − xn| ≤ b−a2 · 2−n . Therefore, the algorithm is guaranteed to converge. Moreover, if required to satisfy

|x∗ − xn| ≤ atol for a given absolute error tolerance atol> 0, we may determine the number of iterations n needed by demanding b−a2 ·2−n ≤ atol. Multiplying both sides by 2n/atol and taking log, we see that it takes

n = ⌈

log2

( b−a

2 atol

)⌉ iterations to obtain a value p = xn that satisfies

|x∗ − p| ≤ atol. The following MATLAB function does the job:

function [p,n] = bisect(func,a,b,fa,fb,atol) % % function [p,n] = bisect(func,a,b,fa,fb,atol) % % Assuming fa = func(a), fb = func(b), and fa*fb < 0, % there is a value root in (a,b) such that func(root) = 0. % This function returns in p a value such that % | p - root | < atol % and in n the number of iterations required.

% check input if (a >= b) | (fa*fb >= 0) | (atol <= 0)

disp(’something wrong with the input: quitting’); p = NaN; n=NaN; return

end

n = ceil ( log2 (b-a) - log2 (2*atol)); for k=1:n

p = (a+b)/2; fp = feval(func,p); if fa * fp < 0

b = p; fb = fp;

else a = p;

✐

✐

✐

✐

✐

✐

✐

✐

44 Chapter 3. Nonlinear Equations in One Variable

fa = fp; end

end p = (a+b)/2;

Example 3.3. Using our MATLAB function bisect for two of the instances appearing in Exam- ple 3.1, we find

• for func(x) = x3 − 30x2 + 2552, starting from the interval [0,20] with tolerance 1.e-8 gives x∗ ≈ 11.86150151 in 30 iterations;

• for func(x)= 2.5sinh(x/4)−1, starting from the interval [−10,10] with tolerance 1.e-10 gives x∗ ≈ 1.5601412791 in 37 iterations.

Here is the MATLAB script for the second function instance:

format long g [x,n] = bisect(’fex3’,-10,10,fex3(-10),fex3(10),1.e-10) function f = fex3(x) f = 2.5 * sinh (x/4) - 1;

Please make sure that you can produce the corresponding script and results for the first instance of this example.

The stopping criterion in the above implementation of the bisection method is absolute, rather than relative, and it relates to the values of x rather than to those of f .

Note that in the function bisect, if an evaluated p happens to be an exact root, then the code can fail. Such an event would be rather rare in practice, unless we purposely, not to say maliciously, aim for it (e.g., by starting from a =−1, b = 1, for f (x) = sin(x)). Adding the line if abs(fp) < eps, n = k; return, end

just after evaluating fp would handle this exception. We note here that the situation where a root is known to be bracketed so decisively as with the

bisection method is not common.

Recursive implementation

The bisection method is a favorite example in elementary computer programming courses, because in addition to its conceptual simplicity it admits a natural presentation in recursive form. In MAT- LAB this can look as follows:

function [p] = bisect_recursive (func,a,b,fa,fb,atol) p = (a+b)/2; if b-a < atol return

else fp = feval(func,p); if fa * fp < 0

b = p; fb = fp;

else a = p;

✐

✐

✐

✐

✐

✐

✐

✐

3.3. Fixed point iteration 45

fa = fp; end p = bisect_recursive (func,a,b,fa,fb,atol);

end

Here then is an incredibly short, yet complete, method implementation. However, what makes recursion unappealing for effective implementation in general is the fact that it is wasteful in terms of storage and potentially suboptimal in terms of CPU time. A precise characterization of the reasons for that is beyond the scope of our discussion, belonging more naturally in an introductory book on programming techniques.

Specific exercises for this section: Exercises 1–2.

3.3 Fixed point iteration The methods discussed in the present section and in the next two, unlike the previous one, have direct extensions to more complicated problems, e.g., to systems of nonlinear equations and to more complex functional equations.

Our problem

f (x) = 0

can be written as

x = g(x).

We will discuss how this can be done in a moment. Given the latter formulation, we are looking for a fixed point, i.e., a point x∗ satisfying

x∗ = g(x∗).

In the fixed point iteration process we define a sequence of iterates x1, x2, . . . , xk , . . . by

xk+1 = g(xk), k = 0,1, . . . ,

starting from an initial iterate x0. If such a sequence converges, then the limit must be a fixed point. The convergence properties of the fixed point iteration depend on the choice of the function g.

Before we see how, it is important to understand that for a given problem f (x) = 0, we can define many functions g (not all of them “good,” in a sense that will become clear soon). For instance, we can consider any of the following, and more:

• g(x) = x − f (x), • g(x) = x +2 f (x), • g(x) = x − f (x)/ f ′(x) (assuming f ′ exists and f ′(x) �= 0).

Thus, we are really considering not one method but a family of methods. The algorithm of fixed point iteration for finding the roots of f (x) that appears on the following page includes the selection of an appropriate g(x).

✐

✐

✐

✐

✐

✐

✐

✐

46 Chapter 3. Nonlinear Equations in One Variable

Algorithm: Fixed Point Iteration. Given a scalar continuous function in one variable, f (x), select a function g(x) such that x satisfies f (x) = 0 if and only if g(x) = x . Then:

1. Start from an initial guess x0.

2. For k = 0,1,2, . . . , set xk+1 = g(xk)

until xk+1 satisfies termination criteria.

Suppose that we have somehow determined the continuous function g ∈ C[a,b], and let us consider the fixed point iteration. Obvious questions arise:

1. Is there a fixed point x∗ in [a,b]?

2. If yes, is it unique?

3. Does the sequence of iterates converge to a root x∗?

4. If yes, how fast?

5. If not, does this mean that no root exists?

Fixed point theorem

To answer the first question above, suppose that there are two values a < b such that g(a) ≥ a and g(b) ≤ b. If g(a) = a or g(b) = b, then a fixed point has been found, so now assume g(a) > a and g(b) < b. Then for the continuous function

φ(x) = g(x)− x we have φ(a) > 0, φ(b) < 0. (Note that φ does not have to coincide with the function f that we have started with; there are lots of φ’s for a given f .) Hence, by the Intermediate Value Theorem given on page 10, just as before, there is a root a < x∗ < b such that φ(x∗) = 0. Thus, g(x∗) = x∗, so x∗ is a fixed point. We have established the existence of a root: f (x∗) = 0.

Next, suppose that g is not only continuous but also differentiable and that there is a positive number ρ < 1 such that

|g′(x)| ≤ ρ, a < x < b. Then the root x∗ is unique in the interval [a,b], for if there is also y∗ which satisfies y∗ = g(y∗), then

|x∗ − y∗| = |g(x∗)− g(y∗)| = |g′(ξ )(x∗ − y∗)| ≤ ρ|x∗ − y∗|, where ξ is an intermediate value between x∗ and y∗. Obviously, this inequality can hold with ρ < 1 only if y∗ = x∗.

We can summarize our findings so far as the Fixed Point Theorem.

✐

✐

✐

✐

✐

✐

✐

✐

3.3. Fixed point iteration 47

Theorem: Fixed Point. If g ∈ C[a,b] and a ≤ g(x) ≤ b for all x ∈ [a,b], then there is a fixed point x∗ in the interval [a,b].

If, in addition, the derivative g′ exists and there is a constant ρ < 1 such that the derivative satisfies

|g′(x)| ≤ ρ ∀ x ∈ (a,b), then the fixed point x∗ is unique in this interval.

Convergence of the fixed point iteration

Turning to the fixed point iteration and the third question on the preceding page, similar arguments establish convergence (now that we know that there is a unique solution): under the same assump- tions we have

|xk+1 − x∗| = |g(xk)− g(x∗)| ≤ ρ|xk − x∗|. This is a contraction by the factor ρ. So

|xk+1 − x∗| ≤ ρ|xk − x∗| ≤ ρ2|xk−1 − x∗| ≤ · · · ≤ ρk+1|x0 − x∗|. Since ρ < 1, we have ρk → 0 as k →∞. This establishes convergence of the fixed point iteration.

Example 3.4. For the function g(x)= e−x on [0.2,1.2], Figure 3.2 shows the progression of iterates towards the fixed point x∗ satisfying x∗ = e−x∗ .

0 0.2 0.4 0.6 0.8 1 1.2 1.4 0

0.2

0.4

0.6

0.8

1

1.2

1.4

x

y=x

g(x)=e−x

x 0

x 1

x 2

Figure 3.2. Fixed point iteration for x = e−x , starting from x0 = 1. This yields x1 = e−1, x2 = e−e−1 , . . . . Convergence is apparent.

Note the contraction effect.

To answer the question about the speed of convergence, it should be clear from the bounds derived before Example 3.4 that the smaller ρ is, the faster the iteration converges. In case of the

✐

✐

✐

✐

✐

✐

✐

✐

48 Chapter 3. Nonlinear Equations in One Variable

bisection method, the speed of convergence does not depend on the function f (or any g, for that matter), and it is in fact identical to the speed of convergence obtained with ρ ≡ 1/2. In contrast, for the fixed point iteration, there is dependence on |g′(x)|. This, and the (negative) answer to our fifth question on page 46, are demonstrated next.

Example 3.5. Consider the function

f (x) = α cosh(x/4)− x , where α is a parameter. For α = 10 we saw in Example 3.1 that there is no root. But for α = 2 there are actually two roots. Indeed, setting

g(x) = 2cosh(x/4) and plotting g(x) and x as functions of x , we obtain Figure 3.3, which suggests not only that there are two fixed points (i.e., roots of f )—let’s call them x∗1 and x∗2 —but also that we can bracket them, say, by

2 ≤ x∗1 ≤ 4, 8 ≤ x∗2 ≤ 10. Next we apply our trusted routine bisect, introduced in the previous section, to f (x) with α = 2. This yields

x∗1 ≈ 2.35755106, x∗2 ≈ 8.50719958, correct to an absolute tolerance 1.e-8, in 27 iterations for each root. (You should be able to explain why precisely the same number of iterations was required for each root.)

0 1 2 3 4 5 6 7 8 9 10 0

2

4

6

8

10

12

14

x

y=x y=2cosh(x/4)

Figure 3.3. The functions x and 2cosh(x/4) meet at two locations.

✐

✐

✐

✐

✐

✐

✐

✐

3.3. Fixed point iteration 49

Now, it is very tempting, and rather natural here, to use the same function g defined above in a fixed point iteration, thus defining

xk+1 = 2cosh(xk/4). For the first root, on the interval [2,4] we have the conditions of the Fixed Point Theorem

holding. In fact, near x∗1 , |g′(x)| < 0.4, so we expect faster convergence than with the bisection method (why?). Indeed, starting from x0 = 2 the method requires 16 iterations, and starting from x0 = 4 it requires 18 iterations to get to within 1.e-8 of the root x∗1 .

For the second root, however, on the interval [8,10] the conditions of the Fixed Point Theorem do not hold! In particular, |g′(x∗2 )| > 1. Thus, a fixed point iteration using this g will not converge to the root x∗2 . Indeed, starting with x0 = 10 the iteration diverges quickly, and we obtain overflow after 3 iterations, whereas starting this fixed point iteration with x0 = 8 we do obtain convergence, but to x∗1 rather than to x∗2 .

It is important to realize that the root x∗2 is there, even though the fixed point iteration with the “natural g” does not converge to it. Not everything natural is good for you! There are other choices of g for the purpose of fixed point iteration that perform better here.

Note: A discussion of rates of convergence similar to that appearing here is also given in Section 7.3. There it is more crucial, because here we will soon see methods that converge faster than a rate of convergence can suitably quantify.

Rate of convergence

Suppose now that at a particular root of a given fixed point iteration, ρ = |g′(x∗)| satisfies 0 <ρ < 1. Starting with x0 sufficiently close to x∗ we can write xk − x∗ ≈ g′(x∗)(xk−1 − x∗). Hence we get

|xk − x∗| ≈ ρ|xk−1 − x∗| ≈ · · · ≈ ρk |x0 − x∗|. To quantify the speed of convergence in terms of ρ we can ask, how many iterations does it take to reduce the error by a fixed factor, say, 10?

To answer this we set |xk − x∗| ≈ 0.1|x0 − x∗|, obtaining ρk ≈ 0.1.

Taking log10 of both sides yields k log10 ρ ≈−1. Let us define the rate of convergence as rate =− log10 ρ. (3.1)

Then it takes about k = �1/rate iterations to reduce the error by more than an order of magni- tude. Obviously, the smaller ρ the higher the convergence rate and correspondingly, the smaller the number of iterations required to achieve the same error reduction effect.

Example 3.6. The bisection method is not exactly a fixed point iteration, but it corresponds to an iterative method of a similar sort with ρ = 0.5. Thus its convergence rate according to (3.1) is

rate =− log10 0.5 ≈ 0.301. This yields k = 4, and indeed the error reduction factor with this many bisections is 16, which is more than 10.

✐

✐

✐

✐

✐

✐

✐

✐

50 Chapter 3. Nonlinear Equations in One Variable

For the root x∗1 of Example 3.5 we have ρ ≈ 0.3121 and rate =− log10 0.3121 ≈ 0.506.

Here it takes only two iterations to roughly reduce the error by a factor of 10 if starting close to x∗. For the root x∗2 of Example 3.5 we obtain from (3.1) a negative rate of convergence, as is

appropriate for a case where the method does not converge.

Of course, it makes no sense to calculate ρ or the convergence rate so precisely as in Ex- ample 3.6: we did this only so that you can easily follow the algebra with a calculator. Indeed, such accurate estimation of ρ would require knowing the solution first! Moreover, there is nothing magical about an error reduction by a particularly precise factor. The rate of convergence should be considered as a rough indicator only. It is of importance, however, to note that the rate becomes negative for ρ > 1, indicating no convergence of the fixed point iteration, and that it becomes infinite for ρ = 0, indicating that the error reduction per iteration is faster than by any constant factor. We encounter such methods next.

Specific exercises for this section: Exercises 3–4.

3.4 Newton’s method and variants The bisection method requires the function f to be merely continuous (which is good) and makes no further use of further information on f such as availability of its derivatives (which causes it to be painfully slow at times). At the other end of the scale is Newton’s method, which requires more knowledge and smoothness of the function f but which converges much faster in appropriate circumstances.

Newton’s method

Newton’s method is the most basic fast method for root finding. The principle we use below to derive it can be directly extended to more general problems.

Assume that the first and second derivatives of f exist and are continuous: f ∈ C2[a,b]. Assume also that f ′ can be evaluated with sufficient ease. Let xk be a current iterate. By Taylor’s expansion on page 5 we can write

f (x) = f (xk)+ f ′(xk)(x − xk)+ f ′′(ξ (x))(x − xk)2/2, where ξ (x) is some (unknown) point between x and xk .

Now, set x = x∗, for which f (x∗) = 0. If f were linear, i.e., f ′′ ≡ 0, then we could find the root by solving 0 = f (xk)+ f ′(xk)(x∗ − xk), yielding x∗ = xk − f (xk)/ f ′(xk). For a nonlinear function we therefore define the next iterate by the same formula, which gives

xk+1 = xk − f (xk) f ′(xk)

, k = 0,1,2, . . . .

This corresponds to neglecting the term f ′′(ξ (x∗))(x∗− xk)2/2 when defining the next iterate; how- ever, if xk is already close to x∗, then (x∗− xk)2 is very small, so we would expect xk+1 to be much closer to x∗ than xk is.

A geometric interpretation of Newton’s method is that xk+1 is the x-intercept of the tangent line to f at xk ; see Figure 3.4.

✐

✐

✐

✐

✐

✐

✐

✐

3.4. Newton’s method and variants 51

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 −1

−0.5

0

0.5

1

1.5

x

x 0

x 1

x 2

Figure 3.4. Newton’s method: the next iterate is the x-intercept of the tangent line to f at the current iterate.

Algorithm: Newton’s Method. Given a scalar differentiable function in one variable, f (x):

1. Start from an initial guess x0.

2. For k = 0,1,2, . . . , set

xk+1 = xk − f (xk) f ′(xk)

,

until xk+1 satisfies termination criteria.

Example 3.7. Consider the same function as in Example 3.5, given by

f (x) = 2cosh(x/4)− x . The Newton iteration here is

xk+1 = xk − 2cosh(xk/4)− xk 0.5sinh(xk/4)−1.

We use the same absolute tolerance of 1.e-8 and the same four initial iterates as in Example 3.5.

• Starting from x0 = 2 requires 4 iterations to reach x∗1 to within the given tolerance. • Starting from x0 = 4 requires 5 iterations to reach x∗1 to within the given tolerance. • Starting from x0 = 8 requires 5 iterations to reach x∗2 to within the given tolerance. • Starting from x0 = 10 requires 6 iterations to reach x∗2 to within the given tolerance.

✐

✐

✐

✐

✐

✐

✐

✐

52 Chapter 3. Nonlinear Equations in One Variable

The values of f (xk), starting from x0 = 8, are displayed below:

k 0 1 2 3 4 5

f (xk) −4.76e-1 8.43e-2 1.56e-3 5.65e-7 7.28e-14 1.78e-15 This table suggests (indirectly) that the number of significant digits essentially doubles at each iter- ation. Of course, when roundoff level is reached, no meaningful improvement can be obtained upon heaping more floating point operations, and the improvement from the 4th to the 5th iteration in this example is minimal.

Speed of convergence

How fast does a nonlinear iteration converge, if it does? Let us assume that the method indeed converges and that xk is already “close enough” to the root x∗. We define convergence orders as follows. The method is said to be

• linearly convergent if there is a constant ρ < 1 such that |xk+1 − x∗| ≤ ρ|xk − x∗|

for all k sufficiently large;

• quadratically convergent if there is a constant M such that |xk+1 − x∗| ≤ M|xk − x∗|2

for all k sufficiently large;

• superlinearly convergent if there is a sequence of constants ρk → 0 such that |xk+1 − x∗| ≤ ρk |xk − x∗|

for all k sufficiently large.

Note that for quadratic convergence we can set ρk = M|xk − x∗|→k→∞ 0; thus the quadratic order implies superlinear order. Superlinear convergence may require less than this but more than linear convergence.

Higher convergence orders (e.g., cubic) may be defined similarly to the quadratic order. How- ever, normally there is really no practical reason to consider such higher convergence orders: meth- ods of this sort would typically require more work per iteration, and roundoff error sets in quickly even for quadratically convergent methods, as we have seen in Example 3.7.

For Newton’s method we have already noted that in the derivation a quadratic term was dropped. Indeed, it turns out that if f ∈ C2[a,b] and there is a root x∗ in [a,b] such that f (x∗) = 0, f ′(x∗) �= 0, then there is a number δ such that, starting with x0 from anywhere in the neighborhood [x∗ − δ, x∗+ δ], Newton’s method converges quadratically. See Exercise 9.

How is the general fixed point iteration xk+1 = g(xk), considered in the previous section, related to Newton’s method? We have seen that the speed of convergence may strongly depend on the choice of g (recall Example 3.5, for instance). It is not difficult to see (Exercise 3) that if, in addition to the assumptions that guarantee convergence in the Fixed Point Theorem, also g′(x∗) �= 0, then the method converges linearly. In this case the size of |g′(x∗)| matters and this is quantified by the rate of convergence, defined in Section 3.3. The method may converge faster than linearly if g′(x∗) = 0. This is precisely the case with Newton’s method. Here g(x) = x − f (x)/ f ′(x), and

✐

✐

✐

✐

✐

✐

✐

✐

3.4. Newton’s method and variants 53

hence g′ = 1 − ( f ′)2− f f ′′ ( f ′)2 = f f

′′ ( f ′)2 . Now, since x = x∗ is the root of f it follows immediately that

g′(x∗) = 0. Newton’s method does have two disadvantages: (i) the need to know not only that the deriva-

tive f ′ exists but also how to evaluate it, and (ii) the local nature of the method’s convergence. The first of these two aspects is addressed (or, rather, circumvented) in what follows.

Secant method

The requirement that not only the function f but also its derivative f ′ be supplied by the user of Newton’s method is at times simple to satisfy, as in our Example 3.7. But at other instances it can be a drag. An extra effort is required by the user, and occasionally the evaluation of f ′ can be much more costly than the evaluation of f at a given argument value. Moreover, sometimes the derivative is simply not available, for example, in certain experimental setups where it is possible to evaluate only the function itself.

The secant method is a variant of Newton’s method, where f ′(xk) is replaced by its finite difference approximation based on the evaluated function values at xk and at the previous iterate xk−1. Assuming convergence, observe that near the root

f ′(xk) ≈ f (xk)− f (xk−1) xk − xk−1 .

(A similar formula was used for different purposes in Example 1.2.) Substitution of this approxi- mation into the formula for Newton’s method yields the Secant method, also referred to as quasi- Newton, given by

xk+1 = xk − f (xk)(xk − xk−1) f (xk)− f (xk−1) , k = 0,1,2, . . . .

Algorithm: Secant Method. Given a scalar differentiable function in one variable, f (x):

1. Start from two initial guesses x0 and x1.

2. For k = 1,2, . . . , set

xk+1 = xk − f (xk)(xk − xk−1) f (xk)− f (xk−1)

until xk+1 satisfies termination criteria.

It is worthwhile to compare at this point the Secant algorithm to the Newton algorithm on page 51.

Example 3.8. Consider again the same function as in Example 3.5, given by

f (x) = 2cosh(x/4)− x . We apply the secant method with the same stopping criterion and the same tolerance as in Examples 3.5 and 3.7. Note that the secant method requires two initial iterates, x0 and x1.

• Starting from x0 = 2 and x1 = 4 requires 7 iterations to reach x∗1 to within the given tolerance. • Starting from x0 = 10 and x1 = 8 requires 7 iterations to reach x∗2 to within the given tolerance.

✐

✐

✐

✐

✐

✐

✐

✐

54 Chapter 3. Nonlinear Equations in One Variable

The values of f (xk), starting from x0 = 10 and x1 = 8, are displayed below:

k 0 1 2 3 4 5 6

f (xk) 2.26 −4.76e-1 −1.64e-1 2.45e-2 −9.93e-4 −5.62e-6 1.30e-9 This table suggests that the number of significant digits increases more rapidly as we get nearer

to the root (i.e., better than a linear order), but not as fast as with Newton’s method. Thus we observe indeed a demonstration of superlinear convergence.

Theorem: Convergence of the Newton and Secant Methods. If f ∈ C2[a,b] and there is a root x∗ in [a,b] such that f (x∗) = 0, f ′(x∗) �= 0, then there is a number δ such that, starting with x0 (and also x1 in case of the secant method) from anywhere in the neighborhood [x∗− δ, x∗ + δ], Newton’s method converges quadratically and the secant method converges superlinearly.

This theorem specifies the conditions under which Newton’s method is guaranteed to converge quadratically, while the secant method is guaranteed to converge superlinearly.

Constructing secant-like modifications for Newton’s method for the case of systems of equa- tions is more involved but possible and practically useful; we will discuss this in Section 9.2.

The case of a multiple root

The fast local convergence rate of both Newton and secant methods is predicated on the assumption that f ′(x∗) �= 0. What if f ′(x∗) = 0? This is the case of a multiple root. In general, if f (x) can be written as

f (x) = (x − x∗)mq(x), where q(x∗) �= 0, then x∗ is a root of multiplicity m. If m > 1, then obviously f ′(x∗) = 0. As it turns out, Newton’s method may still converge, but only at a linear (rather than quadratic) rate.

Example 3.9. For the monomial f (x) = xm , m > 1, we get f (x)

f ′(x) = x

m .

Newton’s method reads

xk+1 = xk − xk m

= m −1 m

xk .

Since the root is x∗ = 0, we can somewhat trivially write

|xk+1 − x∗| = m −1 m

|xk − x∗|,

so the method is linearly convergent, with the contraction factor ρ = m−1m . The corresponding rate of convergence is rate = − log10 m−1m , and it is not difficult to calculate these values and see that they deteriorate as m grows. If f ′′(x∗) �= 0, i.e., m = 2, then ρ = 0.5. See Example 3.6.

✐

✐

✐

✐

✐

✐

✐

✐

3.5. Minimizing a function in one variable 55

Globalizing Newton’s method

Both Newton’s method and the secant method are guaranteed to converge under appropriate con- ditions only locally: there is the assumption in the theorem statement that the initial iterate x0 is already “close enough” to an isolated root. This is in contrast to the situation for the bisection method, which is guaranteed to converge provided only that a bracketing interval is found on which f changes sign. The δ-neighborhood for which Newton’s method is guaranteed to converge may be large for some applications, as is the case for Example 3.7, but it may be small for others, and it is more involved to assess ahead of time which is the case for a given problem.

A natural idea, then, is to combine two methods into a hybrid one. We start with a rough plot or probe of the given function f in order to bracket roots. Next, starting from a given bracket with f (x0) · f (x1) < 0, initiate a Newton or a secant method, monitoring the iterative process by requiring sufficient improvement at each iteration. This “sufficient improvement” can, for instance, be that | f (xk+1)|< 0.5| f (xk)| or the bisection guaranteed improvement ratio |xk+1 − xk | < 0.5|xk − xk−1|. If a Newton or secant iteration is applied but there is no sufficient improvement, then we conclude that the current iterate must be far from where the superior fire power of the fast method can be utilized, and we therefore return to x0 and x1, apply a few bisection iterations to obtain new x0 and x1 which bracket the root x∗ more tightly, and restart the Newton or secant routine.

This hybrid strategy yields a robust solver for scalar nonlinear equations; see Exercise 13.

Convergence and roundoff errors

Thus far in this chapter we have scarcely noticed the possible existence of roundoff errors. But rest assured that roundoff errors are there, as promised in Chapter 2. They are simply dominated by the convergence errors, so long as the termination criteria of our iterative methods are well above rounding unit, and so long as the problem is well-conditioned. In the last entry of the table in Example 3.7, the convergence error is so small that roundoff error is no longer negligible. The result is certainly not bad in terms of error magnitude, but the quadratic convergence pattern (which does not hold for roundoff errors, barring a miracle) is destroyed.

Our essential question here is, given that the error tolerances are well above rounding unit in magnitude, do we ever have to worry about roundoff errors when solving a nonlinear equation? The answer is that we might, if the problem is such that roundoff error accumulates significantly.

When could ill-conditioning occur for the problem of finding roots of a scalar nonlinear func- tion? This can happen if the function f (x) is very flat near its root, because small changes to f may then significantly affect the precise location of the root. Exercise 2 illustrates this point. The general case is that of a multiple root.

Specific exercises for this section: Exercises 5–12.

3.5 Minimizing a function in one variable A major source of applications giving rise to root finding is optimization. In its one-variable version we are required to find an argument x = x̂ that minimizes a given objective function φ(x).6 For a simple instance, we have already briefly examined in Example 3.1 the minimization of the function

φ(x) = 10cosh(x/4)− x

over the real line.

6We arbitrarily concentrate on finding minimum rather than maximum points. If a given problem is naturally formulated as finding a maximum of a function ψ(x), say, then define φ(x) = −ψ(x) and consider minimizing φ.

✐

✐

✐

✐

✐

✐

✐

✐

56 Chapter 3. Nonlinear Equations in One Variable

Another, general, instance is obtained from any root finding or fixed point problem, g(x) = x , by setting

φ(x) = [g(x)− x]2. But here we concentrate on cases where the given objective is not necessarily that of root finding. Note that in general φ(x∗) �= 0.

Conditions for a minimum point

Assume that φ ∈ C2[a,b], i.e., that φ and its first two derivatives are continuous on a given interval [a,b]. Denote

f (x) = φ′(x). An argument x∗ satisfying a < x∗ < b is called a critical point if

f (x∗) = 0. Now, for a parameter h small enough so that x∗ + h ∈ [a,b] we can expand in Taylor’s series on page 5 and write

φ(x∗ +h) = φ(x∗)+hφ′(x∗)+ h 2

2 φ′′(x∗)+·· ·

= φ(x∗)+ h 2

2 [φ′′(x∗)+O(h)].

(See the discussion on page 7 regarding order O notation.) Since |h| can be taken arbitrarily small, it is now clear that at a critical point:

• If φ′′(x∗) > 0, then x̂ = x∗ is a local minimizer of φ(x). This means that φ attains minimum at x̂ = x∗ in some neighborhood which includes x∗.

• If φ′′(x∗) < 0, then x∗ is a local maximizer of φ(x). This means that φ attains maximum at x̂ = x∗ in some neighborhood which includes x∗.

• If φ′′(x∗) = 0, then a further investigation at x∗ is required.

Algorithms for function minimization

It is not difficult to see that if φ(x) attains a minimum (or maximum) at a point x̂ , then this point must be critical, i.e., f (x̂) = 0. Thus, the problem of finding all minima of a given function φ(x) can be solved by finding all the critical roots7 and then checking for each if it is a minimum by examining the sign of the second derivative of φ.

Example 3.10. For the function φ(x) = 10cosh(x/4)− x we have

φ′(x) = f (x) = 10 4

sinh(x/4)−1,

φ′′(x) = f ′(x) = 10 16

cosh(x/4).

7Note that for quadratic convergence of Newton’s method, φ(x) must have three continuous derivatives.

✐

✐

✐

✐

✐

✐

✐

✐

3.5. Minimizing a function in one variable 57

Since φ′′(x) > 0 for all x , we can have only one minimum point. (Think of how the curve must look like if we had more than one minimum.) Applying any of the algorithms described in Sections 3.2–3.4, we obtain the unique minimum point as the root of f (x), given by

x̂ ≈ 1.5601412791. At this point, φ(x̂) ≈ 9.21018833518615.

Example 3.11. For an example with many critical points, consider φ(x) = cos(x). The critical points are where f (x) = φ′(x) =−sin(x) = 0; thus

x∗ = jπ , j = 0,±1,±2, . . . . Since

φ′′(x∗) =−cos( jπ) = (−1) j+1, j = 0,±1,±2, . . . , we have minimum points where j is odd, i.e., at every second critical point: the local minimizers are

x̂ = (2l +1)π , l = 0,±1,±2, . . . . The other critical points are maximum points.

For simple problems this could mark the end of this section. However, for problems where function evaluations are really dear we note that there are two imperfections in the simple procedure outlined above:

• We are investing a considerable effort into finding all critical points, even those that end up being local maxima. Is there a way to avoid finding such unwanted points?

• No advantage is taken of the fact that we are minimizing φ(x). Can’t we gauge how good an iterate xk is by checking φ(xk)?

These objections become more of an issue when considering the minimization of functions of several variables. Nonetheless, let us proceed to quickly outline some general ideas with Newton’s method on page 51 for the scalar case.

The kth iteration of Newton’s method applied to f (x) = φ′(x) can be written here as xk+1 = xk +αkξk , where

ξk = − φ ′(xk)

φ′′(xk) , αk = 1.

Now, if φ′′(xk) > 0 and the step size αk > 0 is small enough, then

φ(xk+1) ≈ φ(xk)+αkξkφ′(xk) = φ(xk)−αkφ′(xk)2/φ′′(xk) < φ(xk). Therefore, we have a decrease in φ, i.e., a step in the right direction.

Thus, it is possible to design a procedure where we insist that φ(x) decrease in each iteration. This would prevent convergence to critical points which are local maxima rather than minima. In such a procedure we monitor φ′′(xk), replacing it by max{φ′′(xk),�}, where � > 0 is a parameter chosen not large. We then select a step size αk by trying αk = 1 first and decreasing it if necessary until a sufficient decrease is obtained in φ(xk+1) relatively to φ(xk). We shall have more to say about this sort of procedure when considering minimizing functions in several variables.

✐

✐

✐

✐

✐

✐

✐

✐

58 Chapter 3. Nonlinear Equations in One Variable

Local and global minimum

The procedures outlined above find local minima. The problem of finding a global minimizer, i.e., a point x̂ such that φ(x̂) ≤ φ(x) for all x ∈ [a,b], is significantly more difficult in general. In the worst situations, where not much is known about the objective function φ, a suitable algorithm may involve a complete enumeration, i.e., finding all local minima and then comparing them. At the other end of the scale, if the objective function is known to be convex, which means that

φ(θx + (1− θ )y)≤ θφ(x)+ (1− θ )φ(y) ∀ 0 ≤ θ ≤ 1 for any two points x , y ∈ [a,b], then there is a unique minimum and the problems of finding local and global minima coincide. Such is the instance considered in Example 3.10, but not that considered in Example 3.11.

Specific exercises for this section: Exercises 21–23.

3.6 Exercises 0. Review questions

(a) What is a nonlinear equation?

(b) Is the bisection method (i) efficient? (ii) robust? Does it (iii) require a minimal amount of additional knowledge? (iv) require f to satisfy only minimum smoothness properties? (v) generalize easily to several functions in several variables?

(c) Answer similar questions for the Newton and secant methods.

(d) State at least one advantage and one disadvantage of the recursive implementation of the bisection method over the iterative nonrecursive implementation.

(e) In what way is the fixed point iteration a family of methods, rather than just one method like bisection or secant?

(f) What is the basic condition for convergence of the fixed point iteration, and how does the speed of convergence relate to the derivative of the iteration function g?

(g) Suppose a given fixed point iteration does not converge: does this mean that there is no root in the relevant interval? Answer a similar question for the Newton and secant methods.

(h) State at least two advantages and two disadvantages of Newton’s method.

(i) What are order of convergence and rate of convergence, and how do they relate?

(j) State at least one advantage and one disadvantage of the secant method over Newton’s.

(k) In what situation does Newton’s method converge only linearly?

(l) Explain the role that roundoff errors play in the convergence of solvers for nonlinear equations, and explain their relationship with convergence errors.

(m) State a similarity and a difference between the problem of minimizing a function φ(x) and that of solving the nonlinear equation φ′(x) = 0.

(n) State what a convex function is, and explain what happens if an objective function is convex.

1. Apply the bisection routine bisect to find the root of the function

f (x) = √x −1.1 starting from the interval [0,2] (that is, a = 0 and b = 2), with atol = 1.e-8.

✐

✐

✐

✐

✐

✐

✐

✐

3.6. Exercises 59

(a) How many iterations are required? Does the iteration count match the expectations, based on our convergence analysis?

(b) What is the resulting absolute error? Could this absolute error be predicted by our con- vergence analysis?

2. Consider the polynomial function8

f (x) = (x −2)9 = x9 −18x8 +144x7 −672x6+2016x5−4032x4+5376x3−4608x2

+ 2304x −512. (a) Write a MATLAB script which evaluates this function at 161 equidistant points in the

interval [1.92,2.08] using two methods:

i. Apply nested evaluation (cf. Example 1.4) for evaluating the polynomial in the expanded form x9 −18x8 +·· · .

ii. Calculate (x −2)9 directly. Plot the results in two separate figures.

(b) Explain the difference between the two graphs.

(c) Suppose you were to apply the bisection routine from Section 3.2 to find a root of this function, starting from the interval [1.92,2.08] and using the nested evaluation method, to an absolute tolerance 10−6. Without computing anything, select the correct outcome:

i. The routine will terminate with a root p satisfying |p−2| ≤ 10−6. ii. The routine will terminate with a root p not satisfying |p−2| ≤ 10−6.

iii. The routine will not find a root.

Justify your choice in one short sentence.

3. Consider the fixed point iteration xk+1 = g(xk), k = 0,1, . . . , and let all the assumptions of the Fixed Point Theorem hold. Use a Taylor’s series expansion to show that the order of convergence depends on how many of the derivatives of g vanish at x = x∗. Use your result to state how fast (at least) a fixed point iteration is expected to converge if g′(x∗) = ·· · = g(r)(x∗) = 0, where the integer r ≥ 1 is given.

4. Consider the function g(x) = x2 + 316 . (a) This function has two fixed points. What are they?

(b) Consider the fixed point iteration xk+1 = g(xk) for this g. For which of the points you have found in (a) can you be sure that the iterations will converge to that fixed point? Briefly justify your answer. You may assume that the initial guess is sufficiently close to the fixed point.

(c) For the point or points you found in (b), roughly how many iterations will be required to reduce the convergence error by a factor of 10?

5. Write a MATLAB script for computing the cube root of a number, x = 3√a, with only basic arithmetic operations using Newton’s method, by finding a root of the function f (x) = x3 − a. Run your program for a = 0,2,10. For each of these cases, start with an initial guess reasonably close to the solution. As a stopping criterion, require the function value whose root you are searching to be smaller than 10−8. Print out the values of xk and f (xk) in each iteration. Comment on the convergence rates and explain how they match your expectations.

8This beautiful example is inspired by Demmel [21]. We gave a modified version of it as an exam question once. Unfortunately, not everyone thought it was beautiful.

✐

✐

✐

✐

✐

✐

✐

✐

60 Chapter 3. Nonlinear Equations in One Variable

6. (a) Derive a third order method for solving f (x) = 0 in a way similar to the derivation of Newton’s method, using evaluations of f (xn), f ′(xn), and f ′′(xn). The following remarks may be helpful in constructing the algorithm:

• Use the Taylor expansion with three terms plus a remainder term. • Show that in the course of derivation a quadratic equation arises, and therefore two

distinct schemes can be derived.

(b) Show that the order of convergence (under the appropriate conditions) is cubic.

(c) Estimate the number of iterations and the cost needed to reduce the initial error by a factor of 10m .

(d) Write a script for solving the problem of Exercise 5. To guarantee that your program does not generate complex roots, make sure to start sufficiently close to a real root.

(e) Can you speculate what makes this method less popular than Newton’s method, despite its cubic convergence? Give two reasons.

7. Consider Steffensen’s method

xk+1 = xk − f (xk) g(xk)

, k = 0,1, . . . ,

where

g(x) = f (x + f (x))− f (x) f (x)

.

(a) Show that in general the method converges quadratically to a root of f (x).

(b) Compare the method’s efficiency to the efficiency of the secant method.

8. It is known that the order of convergence of the secant method is p = 1+ √

5 2 = 1.618 . . . and

that of Newton’s method is p = 2. Suppose that evaluating f ′ costs approximately α times the cost of approximating f . Determine approximately for what values of α Newton’s method is more efficient (in terms of number of function evaluations) than the secant method. You may neglect the asymptotic error constants in your calculations. Assume that both methods are starting with initial guesses of a similar quality.

9. This exercise essentially utilizes various forms of Taylor’s expansion and relies on expertise in calculus.

(a) Prove that if f ∈ C2[a,b] and there is a root x∗ in [a,b] such that f (x∗) = 0, f ′(x∗) �= 0, then there is a number δ such that, starting with x0 from anywhere in the neighborhood [x∗ − δ, x∗+ δ], Newton’s method converges quadratically.

(b) This is more challenging: prove the same conclusions under the same assumptions, except that now f is only assumed to have a first Lipschitz continuous derivative. Thus, while f ′′ may not exist everywhere in [a,b], there is a constant γ such that for any x , y ∈ [a,b]

| f ′(x)− f ′(y)| ≤ γ |x − y|. 10. The function

f (x) = (x −1)2ex has a double root at x = 1.

✐

✐

✐

✐

✐

✐

✐

✐

3.6. Exercises 61

(a) Derive Newton’s iteration for this function. Show that the iteration is well-defined so long as xk �= −1 and that the convergence rate is expected to be similar to that of the bisection method (and certainly not quadratic).

(b) Implement Newton’s method and observe its performance starting from x0 = 2. (c) How easy would it be to apply the bisection method? Explain.

11. In general, when a smooth function f (x) has a multiple root at x∗, the function

ψ(x) = f (x) f ′(x)

has a simple root there.

(a) Show that a Newton method for ψ(x) yields the iteration

xk+1 = xk − f (xk) f ′(xk)

[ f ′(xk)]2 − f (xk) f ′′(xk) .

(b) Give one potential advantage and two potential disadvantages for this iteration as com- pared to Newton’s method for f (x).

(c) Try this method on the problem of Exercise 10. What are your observations?

12. Given a > 0, we wish to compute x = lna using addition, subtraction, multiplication, division, and the exponential function, ex .

(a) Suggest an iterative formula based on Newton’s method, and write it in a way suitable for numerical computation.

(b) Show that your formula converges quadratically.

(c) Write down an iterative formula based on the secant method.

(d) State which of the secant and Newton’s methods is expected to perform better in this case in terms of overall number of exponential function evaluations. Assume a fair comparison, i.e., same floating point system, “same quality” initial guesses, and identical convergence criterion.

13. Write a MATLAB program to find all the roots of a given, twice continuously differentiable, function f ∈ C2[a,b]. Your program should first probe the function f (x) on the given interval to find out where it changes sign. (Thus, the program has, in addition to f itself, four other input arguments: a, b, the number nprobe of equidistant values between a and b at which f is probed, and a tolerance tol.)

For each subinterval [ai ,bi ] over which the function changes sign, your program should then find a root as follows. Use either Newton’s method or the secant method to find the root, monitoring decrease in | f (xk)|. If an iterate is reached for which there is no sufficient decrease (e.g., if | f (xk)| ≥ 0.5| f (xk−1)|), then revert back to [ai ,bi ], apply three bisection steps and restart the Newton or secant method.

The i th root is deemed “found” as xk if both

|xk − xk−1| < tol(1+|xk|) and | f (xk)| < tol hold.

✐

✐

✐

✐

✐

✐

✐

✐

62 Chapter 3. Nonlinear Equations in One Variable

Verify your program by finding the two roots (given in Example 3.5) of the function

f (x) = 2cosh(x/4)− x , starting your search with [a,b] = [0,10] and nprobe = 10. Then use your program to answer other questions in this section where relevant.

14. Find all the roots of the function

f (x) = {

sin(x) x , x �= 0,

1, x = 0, in the interval [−10,10] for tol= 10−7. [This function is special enough to have a name. It is called the sinc function.]

15. For x > 0 consider the equation x + ln x = 0.

It is a reformulation of the equation of Example 3.4.

(a) Show analytically that there is exactly one root, 0 < x∗ < ∞. (b) Plot a graph of the function on the interval [0.1,1].

(c) As you can see from the graph, the root is between 0.5 and 0.6. Write MATLAB routines for finding the root, using the following:

i. The bisection method, with the initial interval [0.5,0.6]. Explain why this choice of the initial interval is valid.

ii. A linearly convergent fixed point iteration, with x0 = 0.5. Show that the conditions of the Fixed Point Theorem (for the function g you have selected) are satisfied.

iii. Newton’s method, with x0 = 0.5. iv. The secant method, with x0 = 0.5 and x1 = 0.6.

For each of the methods:

• Use |xk − xk−1| < 10−10 as a convergence criterion. • Print out the iterates and show the progress in the number of correct decimal digits

throughout the iteration. • Explain the convergence behavior and how it matches theoretical expectations.

16. We saw in Example 3.5 that the function

f (x) = α cosh(x/4)− x has two roots for α = 2 and none for α = 10. Is there an α for which there is precisely one root? If yes, then find such an α and the corresponding root; if not, then justify.

17. The derivative of the sinc function is given by

f (x) = x cos(x)− sin(x) x2

.

(a) Show that near x = 0, this function can be approximated by f (x) ≈−x/3.

The error in this approximation gets smaller as x approaches 0.

(b) Find all the roots of f in the interval [−10,10] for tol= 10−8.

✐

✐

✐

✐

✐

✐

✐

✐

3.6. Exercises 63

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 −0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

x

f

Figure 3.5. Graph of an anonymous function; see Exercise 18.

18. Consider finding the root of a given nonlinear function f (x), known to exist in a given interval [a,b], using one of the following three methods: bisection, Newton, and secant. For each of the following instances, one of these methods has a distinct advantage over the other two. Match problems and methods and justify briefly.

(a) f (x) = x −1 on the interval [0,2.5]. (b) f (x) is given in Figure 3.5 on [0,4].

(c) f ∈ C5[0.1,0.2], the derivatives of f are all bounded in magnitude by 1, and f ′(x) is hard to specify explicitly or evaluate.

19. You are required by a computer manufacturer to write a library function for a given floating point system to find the cube root y1/3 of any given positive number y. Any such relevant floating point number can be represented as y = a × 2e, where a is a normalized fraction (0.5 ≤ a < 1) and e is an integer exponent. This library function must be very efficient and it should always work. For efficiency purposes it makes sense to store some useful constants ahead of computation time, e.g., the constants 21/3, 23 , and a/3, should these prove useful.

(a) Show how y1/3 can be obtained, once a1/3 has been calculated for the corresponding fraction, in at most five additional flops.

(b) Derive the corresponding Newton iteration. What is the flop count per iteration?

(c) How would you choose an initial approximation? Roughly how many iterations are needed? (The machine rounding unit is 2−52.)

[You might find this exercise challenging.]

✐

✐

✐

✐

✐

✐

✐

✐

64 Chapter 3. Nonlinear Equations in One Variable

20. Suppose that the division button of your calculator has stopped working, and you have ad- dition, subtraction, and multiplication only. Given a real number b �= 0, suggest a quadrati- cally convergent iterative formula to compute 1b , correct to a user-specified tolerance. Write a MATLAB routine that implements your algorithm, using |xk − xk−1| < 10−10 as a conver- gence criterion, and apply your algorithm to b = π (that is, we compute 1

π ), with two different

initial guesses: (a) x0 = 1 ; and (b) x0 = 0.1. Explain your results.

21. Write a MATLAB program to minimize a smooth, scalar function in one variable φ(x) over a given interval. The specifications are similar to those in Exercise 13. Your program should first find and display all critical points, i.e., zeros of φ′(x). Then it should determine which of these correspond to a local minimum by checking the sign of φ′′, and display the local minimum points. Finally, it should determine the global minimum point(s), value, and arguments by minimizing over the local minimum values.

22. Apply your program from Exercise 21 to find the (unique) global minimum of the function φ(x) = cos(x)+ x/10 over the interval [−10,10]. What are x̂ and φ(x̂)?

23. Use your program from Exercise 21 to minimize

(a) φ(x) = sin(x) and (b) φ(x) = −sinc(x) = − sin(x)x

over the interval [−10,10] with tol= 10−8. You might find that determining the global minimum of the second function is significantly more challenging, even though the global minimum for the first one is not unique. Explain the source of difficulties and how you have addressed them.

[Exercises 17 and 14 are relevant here.]

3.7 Additional notes Just about any introductory text on numerical methods or numerical analysis covers the topics intro- duced in this chapter, although the goals and the manner of these treatments may vary significantly. For different flavors, see Heath [39], Cheney and Kincaid [12], and Burden and Faires [11], as well as the classics Conte and de Boor [13] and Dahlquist and Björck [15].

For us this chapter is mainly an opportunity to introduce and demonstrate, within a simple context, several basic notions and issues arising also in much more complex numerical computations. We have thus avoided methods that do not seem to shed much light on other problems as well.

There is a (somewhat more complicated) analogue to the bisection method for function min- imization. This classical algorithm is called golden section search and is described, together with extensions, in Press et al. [59]; alternatively, see the crisper Heath [39].

✐

✐

✐

✐

✐

✐

✐

✐

Chapter 4

Linear Algebra Background

Systems of linear equations and algebraic eigenvalue (or singular value) problems arise very fre- quently in large numerical computations, often as a subproblem. Computational tasks associated with models arising in computer science, mathematics, statistics, the natural sciences, social sci- ences, various engineering disciplines, and business and economics all require the solution of such linear algebra systems. The simplest techniques for solving these problems have been known for centuries, and yet their efficient implementation and the investigation of modern methods in large, complex situations is an active area of research.

Numerical methods that relate directly to linear algebra problems are covered in several chap- ters following this one. Here we set up the mathematical preliminaries and tools used later on, and we provide some typical examples of simple applications that give rise to common systems of this sort. What this chapter barely contains are actual algorithms or their analysis. We lightly touch upon some of those in Section 4.5 but leave the more systematic treatment of linear algebra algorithms for later chapters.

Note: A large portion of this chapter is a review of material that can be found in an intro- ductory linear algebra text. We do, however, make a point of spicing things up by giving a preliminary taste of numerical issues and by highlighting notions that are particularly im- portant for computations. Readers who feel comfortable with the basic concepts and are familiar with applications that yield such problems may skip this chapter. But we have a feeling that those who do stick around will be able to learn a new thing or two, particularly in the later parts of our discussion.

In Section 4.1 we quickly review basic concepts that are covered in any text on linear alge- bra. Tools for measuring sizes of vectors and matrices and distances between them are provided in Section 4.2. In Section 4.3 we describe two very important special classes of matrices that you will encounter again and again: symmetric positive definite and orthogonal matrices. Section 4.4 introduces the singular value decomposition. Finally, the examples in Section 4.5 provide “teasers” for what may arise in applications.

4.1 Review of basic concepts In this section we provide a brief (and by no means comprehensive) review of basic concepts in linear algebra. The two fundamental problems that we consider are linear systems and eigenvalue problems.

65

✐

✐

✐

✐

✐

✐

✐

✐

66 Chapter 4. Linear Algebra Background

0 x1

x2

Figure 4.1. Intersection of two straight lines: a11x1 +a12x2 = b1 and a21x1 +a22x2 = b2.

Basics of linear system solution

Example 4.1. Consider the simple example of finding the point of intersection of two straight lines in a plane; see Figure 4.1. If one line is defined by a11x1 + a12x2 = b1 and the other by a21x1 + a22x2 = b2, then the point of intersection, lying on both lines, must satisfy both equations

a11x1 +a12x2 = b1, a21x1 +a22x2 = b2.

For instance, let a11 = 1, a12 = −4, a21 = .5, a22 = −1, b1 = −10, b2 = −2. Then the point of intersection (x1, x2) is the solution of the 2×2 system

x1 −4x2 =−10, .5x1 − x2 =−2.

How would we go about finding a solution x = (x1 x2

) that satisfies these equations? Recall that we

can multiply any row in such a system by a scalar and add to another row. Here, if we multiply the first row by −0.5 and add to the second, then x1 is eliminated and we obtain

0 · x1 + x2 ≡ x2 = 3. Substituting in the first equation then yields x1 = −10+12 = 2. Hence, the point of intersection is (2,3).

Singularity and linear independence

It is surprising how directly Example 4.1, which is trivial by all accounts, extends in many ways. Let us stay with it for another brief moment and ask whether there is always a unique intersection point for the two lines in the plane. Geometrically, the answer is that there is a unique solution if and only if the lines are not parallel. Here the term “parallel” includes also the case of coinciding lines. In the case of parallel lines we have either no intersection point, i.e., no solution to the linear

✐

✐

✐

✐

✐

✐

✐

✐

4.1. Review of basic concepts 67

system of equations, as for the system

x1 + x2 = 5, 3x1 +3x2 = 16,

or infinitely many “intersection points” if the two straight lines coincide, as for the system

x1 + x2 = 5, 3x1 +3x2 = 15.

When are the lines parallel? When their slopes are equal, expressed algebraically by the condition − a11a12 = −

a21 a22

, or det(A) = a11a22 −a21a12 = 0.

When the determinant9 of a square matrix A equals 0 we say that A is singular. Equivalently, A is nonsingular if and only if its columns (or rows) are linearly independent.10,11

Range space and nullspace

Generally, we can write Ax = b as12

a1x1 +a2x2 +·· ·+anxn = b, where a j , or A(:, j ) in MATLAB-speak, is the j th column of A. Pictorially, we can write

...

...

a1 ... ...

 x1 +



...

...

a2 ... ...

 x2 +·· ·+



...

...

an ... ...

 xn =



...

...

b ... ...

 and x =



x1

x2 ...

xn−1 xn

 .

If A is nonsingular, then the columns a j are linearly independent. Then there is a unique way of writing any vector b as a linear combination of these columns. Hence, if A is nonsingular, then there is a unique solution for the system Ax = b.

9Please refer to the nearest text on linear algebra for the definition of a determinant. Our algorithms will not use determinants.

10Recall that a set of vectors {vi } are linearly independent if ∑

i αi vi = 0 necessarily implies that all coefficients αi are equal to zero.

11For a given set {vi }mi=1 of m real, linearly independent vectors vi ∈Rn (where necessarily m ≤ n), the linear space V = span{v1, . . . ,vm } consists of all linear combinations

∑m i=1 αi vi . Obviously V is a subspace of Rn , i.e., the space

V is contained in the space Rn , because any v ∈ V also satisfies v ∈ Rn . 12Throughout this text a given matrix A is by default square and real, i.e., it can be written as

A =

 a11 a12 · · · a1n a21 a22 · · · a2n

... ...

. . . ...

an1 an2 · · · ann

 ,

and the elements ai, j are real numbers. Exceptions where A is rectangular or complex are noted in context.

✐

✐

✐

✐

✐

✐

✐

✐

68 Chapter 4. Linear Algebra Background

On the other hand, if A is singular, or if it actually has more rows than columns, then it may or may not be possible to write b as a linear combination of the columns of A. Let the matrix generally have m rows and n columns, and denote

• range(A) ≡ all vectors that can be written as a linear combination of the columns of A, i.e., all vectors y that can be written as y = Ax, for some vector x (note that y ∈ Rm while x ∈ Rn) ;

• null(A) ≡ nullspace of A ≡ all vectors z ∈ Rn for which Az = 0, where 0 denotes a vector of zeros.13

If A is n ×n and singular, then there are vectors in Rn which do not belong to range(A), and there are nonzero vectors which belong to A’s nullspace, null(A). So, in this case, for a particular given right-hand-side vector b the equation Ax = b will not have a solution if b /∈ range(A). On the other hand, if A is singular and b ∈ range(A), then there are infinitely many solutions. This is because if some particular x̂ satisfies the linear system of equations Ax̂ = b, then x = x̂+ x̃ is also a solution for any x̃ ∈ null(A): A(x̂+ x̃) = b+0 = b.

To summarize, the following statements are equivalent:

• A is nonsingular. • det(A) �= 0. • The columns of A are linearly independent. • The rows of A are linearly independent. • There exists a matrix which we denote by A−1, which satisfies A−1 A = I = AA−1. • range(A) = Rn . • null(A) = {0}.

Example 4.2. The matrix A = (1 1 3 3

) is singular, as noted earlier. The nullspace of A can be found

in a straightforward way by considering a linear system of the form Ax = 0. It readily follows that the elements of x are characterized by the relation x1 + x2 = 0; hence the nullspace is given by

null(A) = α

 1 −1

 ∀α ∈ R  .

Next, observe that if b∈ range(A), then it can be written as a linear combination of the columns of A; hence

range(A) = β

1 3

 ∀β ∈ R  .

For the case b = (23) there is clearly no solution because b /∈ range(A). By the same token, for the case b = ( 26) = 2(13) we can be sure that b ∈ range(A). By inspection there is a solution x̂ = ( 11). Now, any vector of the form x = (1+α1−α ) for any scalar α is also a solution. In fact, for this particular choice of b any vector x whose elements satisfy x1 + x2 = 2 is a solution. For instance, x =

( 1001−999

) is a solution, although intuitively it is quite far from x̂.

13Mathematicians often prefer the term kernel over nullspace and use ker(A) rather than null(A) to denote it.

✐

✐

✐

✐

✐

✐

✐

✐

4.1. Review of basic concepts 69

Almost singularity

In practice we often (though certainly not always) encounter situations where the matrix A for a given linear system is known to be nonsingular. However, what if A is “almost singular”?

Example 4.3. Consider the previous example, slightly perturbed to read1+ ε 1 3 3

x1 x2

= 2+ ε

6

 , where ε is a small but positive number, 0 < ε � 1. (You can think of ε = 10−10, for instance.)

The matrix in this example is nonsingular, so there is a unique solution, x̂ = ( 11). However, upon a slight perturbation (removing ε) the matrix can become singular and there are many solutions, some of which are rather far from x̂; for instance, set α = 1000 in Example 4.2.

The simple Examples 4.2 and 4.3 indicate that if the matrix A is “almost singular,” in the sense that a slight perturbation makes it singular, then a small change in the problem of solving linear systems with this matrix may cause a large change in the solution. Such problems are ill- conditioned.

Basics of eigenvalue problems

The algebraic eigenvalue problem Ax = λx

is fundamental. Basically, one wants to characterize the action of the matrix A in simple terms. Here the product of A and the eigenvector x equals the product of the scalar eigenvalue λ and the same vector. We require x �= 0 in order not to say something trivial. Together, (λ,x) is an eigenpair, and the set of eigenvalues forms the spectrum of A.14 If there are n eigenpairs (λ j ,x j ) and if the eigenvectors are linearly independent, then any other vector y ∈ Rn can be written as their linear combination, i.e., there are coefficients α j such that

y = n∑

j=1 α j x j ,

and hence the operation of our matrix A on this arbitrary vector can be characterized in terms of the eigenpairs as

Ay = n∑

j=1 α j Ax j =

n∑ j=1

α jλ j x j .

The coefficient α j in y is thus replaced by α jλ j in Ay and a powerful decomposition is obtained.

The n eigenvalues of an n × n matrix Does every real n ×n matrix have n eigenvalues? We can write the eigenvalue problem as a homo- geneous linear system

(λI − A)x = 0. 14The eigenvectors x are sometimes referred to as right eigenvectors, to distinguish between them and left eigenvec-

tors, which are given by wT A = λwT .

✐

✐

✐

✐

✐

✐

✐

✐

70 Chapter 4. Linear Algebra Background

Since we want a nontrivial x, this means that λI − A must be singular. Therefore we can find λ by forming the characteristic polynomial and finding its roots. That is, in principle we solve

det(λI − A) = 0.

Example 4.4. For a 2×2 matrix we have

det(λI − A) = det λ−a11 −a12

−a21 λ−a22

= (λ−a11)(λ−a22)−a12a21 = λ2 − (a11 +a22)λ+ (a11a22 −a12a21) .

We therefore have to solve a quadratic equation of the form

λ2 −bλ+ c = 0. From high school we know that to solve this we should form the discriminant � = b2 − 4c.

A short calculation shows that here � = (a11 −a22)2 + 4a12a21. If � > 0, then there are two real solutions given by

λ1,2 = 1 2

(b±√�) = 1 2

(a11 +a22 ± √ �);

if � = 0, then there is one real solution; and if � < 0, then there are no real solutions. This is also the spirit of Chapter 3, where only distinct real solutions are sought for nonlinear equations.

Here, however, there is no reason to restrict ourselves to real numbers: we want two eigen- values rain or shine, and this means we must be ready for a situation where the eigenvalues may be complex numbers.15

In the complex plane we now have that in case of � < 0 the eigenvalues are

λ1,2 = 1 2

(a11 +a22 ± ı √−�),

and if � = 0, then there is the double root 12 (a11 +a22). Thus, for the matrix A = (1 2

2 4

) we have � = 32 + 42 = 25; hence λ1 = (5 + 5)/2 = 5,

λ2 = (5 − 5)/2 = 0. For the matrix A = (

0 1 −1 0

) we have � = −4, and it follows that λ1 = ı ,

λ2 = −ı .

The lessons of Example 4.4, if not its simplicity, generalize. Clearly, the degree of the charac- teristic polynomial is equal to the dimension of the matrix. A polynomial of degree n has n generally complex and not necessarily distinct roots, and we can write

det(λI − A) = (λ−λ1)(λ−λ2) · . . . · (λ−λn) = 0, where the roots λ1,λ2, . . . ,λn are the eigenvalues of A. If there are k roots that are equal to each other, call their value λ j ; then we say that λ j is an eigenvalue with algebraic multiplicity k. The sum of the algebraic multiplicities of all the eigenvalues of A must be n, the dimension of A.

15Complex numbers make a rare appearance in our text, but when they do their role can be critical. This is so in eigenvalue problems (Chapter 8), Fourier transform (Chapter 13), and differential equations (Chapter 16). Their basic properties are summarized on the facing page.

✐

✐

✐

✐

✐

✐

✐

✐

4.1. Review of basic concepts 71

Complex Numbers. A complex number z can be written as

z = x + ı y, where x and y are two real scalars and ı = √−1. We write x = !(z), y = "(z). (In MATLAB, x = real(z) and y = imag(z).)

It is customary to view z as a point (a vector from the origin) on the complex plane with !(z) on the x-axis and "(z) on the y-axis. The Euclidean length of the vector z is then the magnitude

|z| = √

x2 + y2 =√(x + ı y)(x − ı y) =√zz̄, where z̄ is the conjugate of z. Complex eigenvalues, for instance, appear in conjugate pairs.

The fundamental Euler identity reads

eıθ = cos(θ )+ ı sin(θ ) (4.1) for any real angle θ (in radians). In polar coordinates (r ,θ ) we can then write a complex number z as

z = reıθ = r cos(θ )+ ır sin(θ ) = x + ı y, where r = |z| and tan(θ ) = y/x .

Example 4.5. The matrix

A =



−0.9880 1.8000 −0.8793 −0.5977 −0.7819 −1.9417 −0.5835 −0.1846 −0.7250 1.0422 0.6003 −0.0287 −0.5446 −2.0667 −0.3961 0.8222 1.4453 1.3369 −0.6069 0.8043 −0.4187 −0.2939 1.4814 −0.2119 −1.2771

 has eigenvalues given approximately by λ1 = −2, λ2 = −1+ 2.5ı , λ3 = −1 − 2.5ı , λ4 = 2ı , and λ5 =−2ı . These eigenvalues are plotted in Figure 4.2.

The connection between eigenvalues and the roots of the characteristic polynomial has further implications. It is known that closed form formulas for the roots of a polynomial do not generally exist if the polynomial is of degree 5 or higher. Thus, unlike the solution of a linear system of equations, we may not expect to be able to solve the eigenvalue problem to infinite precision, even in the absence of roundoff errors.

Diagonalizability and spectral decomposition

Note that for a given eigenvalue λ j the corresponding eigenvector x j may be complex valued if λ j is not real. Moreover, the eigenvector is fixed only up to a constant.

✐

✐

✐

✐

✐

✐

✐

✐

72 Chapter 4. Linear Algebra Background

−5 −4 −3 −2 −1 0 1 −5

−4

−3

−2

−1

0

1

2

3

4

5

ℜ(λ)

ℑ (λ

)

Figure 4.2. Eigenvalues in the complex plane, Example 4.5. Note that the complex ones arrive in pairs: if λ is an eigenvalue, then so is λ̄. Also, here all eigenvalues have nonpositive real parts.

Let us mention a few basic properties of eigenvalues:

1. If Ax = λx, then for any complex scalar α (A+α I )x = (λ+α)x.

2. For any positive integer k Akx = λkx.

3. If B = S−1 AS and Ax = λx, then By = λy, where x = Sy. In other words, B has the same eigenvalues as A. We say that B is similar to A and call the transformation A → S−1 AS a similarity transformation of A.

4. Spectral decomposition: Let X be the matrix whose columns are eigenvectors of A, and suppose that X is square and nonsingular. Then

AX = A[x1,x2, . . . ,xn] = [λ1x1,λ2x2, . . . ,λnxn] = X�,

where � is a diagonal matrix with the eigenvalues on its diagonal, � = diag(λ1,λ2, . . . ,λn). Therefore, we can write

A = X�X−1.

✐

✐

✐

✐

✐

✐

✐

✐

4.2. Vector and matrix norms 73

This decomposition is the spectral decomposition of A, and any matrix A that can be decom- posed in this fashion is diagonalizable.

Unfortunately, the last property above does not apply to just any matrix; there are matrices that do not have a complete, linearly independent set of eigenvectors. Such matrices are not diag- onalizable and are labeled by less than flattering adjectives, such as defective matrices. Given an eigenvalue, its geometric multiplicity is defined as the dimension of the space that its associated eigenvectors span; in other words, it is the number of linearly independent eigenvectors associ- ated with this eigenvalue. It is enough to have one eigenvalue whose geometric multiplicity is smaller than its algebraic multiplicity for the matrix to lose its diagonalizability. In the sense de- scribed earlier, eigenvalues of a defective matrix capture less of its action than in the diagonalizable case.

Example 4.6. The eigenvalues of any upper triangular or lower triangular matrix can be read off its main diagonal. For example, the matrix A = (4 1

0 4

) is 2×2 and has the eigenvalue 4 with algebraic

multiplicity 2 and eigenvector (1,0)T . But no matter how hard you look, you will not be able to find another (linearly independent) eigenvector. The geometric multiplicity of the eigenvalue 4 is, then, only 1.

Take next the matrix A = (4 0 0 4

) . Trivially, this is just the identity matrix multiplied by 4.

Here we can easily find two linearly independent eigenvectors, the simplest choice being (1,0)T and (0,1)T , so the geometric multiplicity of the eigenvalue 4 equals 2.

A question that we may ask ourselves is in which circumstances we should be concerned with potential difficulties when solving an eigenvalue problem. It turns out that nondiagonalizable matrices are such a source for trouble, as illustrated below in a very simple setting.

Example 4.7. Take the first matrix from Example 4.6, and perturb its off-diagonal elements slightly so that now A = ( 4 1.01

0.01 4

) . Using the MATLAB function eig we find that the eigenvalues of our

(diagonalizable) matrix are approximately 4.1005 and 3.8995. This result is somewhat alarming: a perturbation of magnitude 0.01 has produced a change of magnitude 0.1005 in the eigenvalues! Such a magnification is precisely the type of ill-conditioning mentioned previously for linear systems, and it can cause a considerable headache if we do not detect it early enough.

On the other hand, applying the same perturbation to the second, diagonal matrix from Exam- ple 4.6 produces the eigenvalues 4.01 and 3.99, so the matrix perturbation has not been magnified in the eigenvalues. The eigenvalue 4 in this case is well-conditioned.

Specific exercises for this section: Exercises 1–3.

4.2 Vector and matrix norms In previous chapters we have defined relative and absolute errors for scalars and have seen their use for assessing the quality of an algorithm. Errors measure the distance between the true solution and the computed solutions, so for solutions of linear systems and other problems in linear algebra we need to equip ourselves with the notion of a distance between vectors and between matrices. This gives rise to the notion of a norm, generalizing the concept of absolute value (or magnitude) of a scalar.

✐

✐

✐

✐

✐

✐

✐

✐

74 Chapter 4. Linear Algebra Background

Vector norms

For a vector x ∈ Rn the Euclidean length is defined as ‖x‖2 =

√ xT x

= (x21 + x22 +·· ·+ x2n)1/2

= (

n∑ i=1

x2i

)1/2 .

This is called the �2-norm, and it is the usual way for measuring a vector’s length. But there are others:

• The �∞-norm is the magnitude of the largest element in x, defined by ‖x‖∞ = max

1≤i≤n |xi |.

• The �1-norm is the sum of magnitudes of the elements of x, defined by

‖x‖1 = n∑

i=1 |xi |.

But what exactly is a norm? It is a function with a range of scalar values, denoted ‖ · ‖, that satisfies the essential properties of the magnitude of a scalar:

1. ‖x‖ ≥ 0; ‖x‖ = 0 if and only if x = 0, 2. ‖αx‖ = |α|‖x‖ ∀ α ∈ R, 3. ‖x+y‖ ≤ ‖x‖+‖y‖ ∀ x,y ∈ Rn .

The three norms introduced above (�1, �2, and �∞) are all special cases of a family of �p- norms defined by

‖x‖p = (

n∑ i=1

|xi |p )1/p

, 1 ≤ p ≤∞.

It is possible to verify that �p-norm is a norm by confirming that it satisfies the three properties listed above that define a norm function. (The third property, however, may be nontrivial to show.) Inci- dentally, this definition of an �p-norm is good also for complex-valued vectors (such as eigenvectors can be). The maximum norm can be viewed as a limiting case for p → ∞.

Example 4.8. Suppose we want to measure the distance between the two vectors

x =

 11

12

13

 and y = 

12

14

16

 . Let

z = y−x =

 1

2

3

 .

✐

✐

✐

✐

✐

✐

✐

✐

4.2. Vector and matrix norms 75

Then

‖z‖1 = 1+2+3 = 6, ‖z‖2 =

√ 1+4+9 ≈ 3.7417,

‖z‖∞ = 3. In MATLAB we get the above results by typing

zz = [12,14,16]’ - [11,12,13]’; norm(zz,1) norm(zz) norm(zz,inf)

Although the values of the different norms are different, they have the same order of magni- tude.

We see that in Example 4.8, ‖z‖1 ≥ ‖z‖2 ≥ ‖z‖∞. This is true in general. Moreover, it is easy to see (and worth remembering) that for any vector x ∈ Rn

‖x‖∞ ≤ ‖x‖2 ≤ √

n‖x‖∞, ‖x‖∞ ≤ ‖x‖1 ≤ n‖x‖∞.

Example 4.9. Figure 4.3 displays plots of all vectors in the plane R2 whose norm equals 1, accord- ing to each of the three norms we have seen.

1 1 1

‖x‖1 ‖x‖2 ‖x‖∞

Figure 4.3. The “unit circle” according to the three norms, �1, �2, and �∞. Note that the diamond is contained in the circle, which in turn is contained in the square.

Our usual geometric notion of a circle is captured by the �2-norm, less so by the other two.16

Indeed, even though the �2-norm is harder to compute (especially for matrices, as becomes clear below) we will use it in the sequel, unless otherwise specifically noted.

Matrix norms

To be able to answer questions such as how far a matrix is from being singular, and to be able to examine quantitatively the stability of our algorithms, we need matrix norms.

16The �1-norm is occasionally called the “Manhattan norm.” Figure 4.3 should help explain why.

✐

✐

✐

✐

✐

✐

✐

✐

76 Chapter 4. Linear Algebra Background

Matrix norms (i.e., functions which take a matrix as an argument, produce a scalar value, and satisfy the above three norm conditions) can be obtained in many different ways. But of particular interest to us is the induced or natural norm associated with each vector norm. Given an m × n matrix A and a vector norm ‖ · ‖, define

‖A‖ = max x �=0

‖Ax‖ ‖x‖

= max‖x‖=1‖Ax‖.

Note that for all vectors x which are not identically equal to the zero-vector, ‖x‖> 0. Also, for each such vector, dividing by its norm we can normalize it. Thus, the above definition makes sense.

It is not difficult to show that induced matrix norms satisfy the three norm conditions, as well as the consistency condition ‖AB‖ ≤ ‖A‖‖B‖. In particular, for any vector x of an appropriate dimension, we have

‖Ax‖ ≤ ‖A‖‖x‖. Here, then, are the four properties that our matrix norms satisfy:

1. ‖A‖ ≥ 0; ‖A‖ = 0 if and only if A = 0 (elementwise); 2. ‖αA‖ = |α|‖A‖ ∀α ∈ R; 3. ‖A+ B‖ ≤ ‖A‖+‖B‖ ∀A, B ∈ Rm×n ; 4. ‖A · B‖ ≤ ‖A‖‖B‖ ∀A ∈ Rm×n , B ∈ Rn×l .

When referring to the usual induced norms, a common convention is to omit the � and re- fer to the analogues of the �1-, �2-, �∞-norms for vectors as the 1-norm, 2-norm, and ∞-norm, respectively, for matrices.

Calculating induced matrix norms

Calculating ‖A‖∞ is simple. Letting x be any vector with ‖x‖∞ = 1 we have |x j | ≤ 1, j = 1, . . . ,n, and hence

‖Ax‖∞ = max i

∣∣∣∣∣∣ n∑

j=1 ai j x j

∣∣∣∣∣∣ ≤ max

i

n∑ j=1

|ai j ||x j | ≤ max i

n∑ j=1

|ai j |,

so

‖A‖∞ = max‖x‖∞=1‖Ax‖∞ ≤ maxi n∑

j=1 |ai j |.

But equality can be achieved by a judicious choice of x: if p is the maximizing row in the above expression, i.e.,

∑n j=1 |apj | = maxi

∑n j=1 |ai j |, then choose x̃ j = sign(apj ) for each j . This yields

that apj x̃ j = |apj |, so for this particular vector x = x̃ we have

‖Ax̃‖∞ = max i

∣∣∣∣∣∣ n∑

j=1 ai j x̃ j

∣∣∣∣∣∣≥ ∣∣∣∣∣∣

n∑ j=1

apj x̃ j

∣∣∣∣∣∣= n∑

j=1 |apj |.

✐

✐

✐

✐

✐

✐

✐

✐

4.2. Vector and matrix norms 77

Combining the two estimates, we obtain

n∑ j=1

|apj | ≤ ‖Ax̃‖∞ ≤ ‖A‖∞ ≤ n∑

j=1 |apj |,

and this implies that there is equality throughout in the derivation. We conclude that in general

‖A‖∞ = max 1≤i≤m

n∑ j=1

|ai j |.

An expression of similar simplicity is available also for the 1-norm and is given by

‖A‖1 = max 1≤ j≤n

m∑ i=1

|ai j |.

No such simple expression is available for the 2-norm, though!

Example 4.10. Consider

A =  1 3 7 −4 1.2725 −2

 . Then

‖A‖∞ = max{11,7.2725}= 11, ‖A‖1 = max{5,4.2725,9}= 9, ‖A‖2 = max

xT x=1 ‖Ax‖2 = max

xT x=1

√ (Ax)T Ax = max

xT x=1

√ xT (AT A)x.

We can calculate

AT A =

 17 −2.09 15

−2.09 10.6193 18.455 15 18.455 53

 , but we still need to figure out how to maximize the quadratic form xT (AT A)x. Please stay tuned.

The 2-norm and spectral radius

To calculate the 2-norm of a matrix, we need to consider the eigenvalues of AT A. Let B be a square, n ×n real matrix. From our discussion in the previous section, recall that B has n eigenvalues. We define the spectral radius of B as

ρ(B) = max{|λ|; λ is an eigenvalue of B}. Obviously for any induced matrix norm, if (λ̃, x̃) is the eigenvalue-eigenvector pair which maximizes |λ| for B , with x̃ normalized to satisfy ‖x̃‖ = 1, then

ρ(B) = |λ̃| = ‖λ̃x̃‖ = ‖B x̃‖ ≤ ‖B‖.

✐

✐

✐

✐

✐

✐

✐

✐

78 Chapter 4. Linear Algebra Background

So, any induced norm is always bounded below by the spectral radius! Note that the spectral radius of a general square matrix is not a norm. For example, for C = (0 1

0 0

) we have ρ(C) = 0 even

though C �= 0. The spectral radius is often a pretty good lower bound for a norm. Further, it can be shown

that even for rectangular (i.e., not necessarily square) matrices, we have

‖A‖2 = √ ρ(AT A).

Example 4.11. Continuing with Example 4.10 the eigenvalues of the matrix AT A, found by the MATLAB command eig(A’*A), are (approximately) 0, 16.8475, and 63.7718. Thus, we finally get

‖A‖2 ≈ √

63.7718 ≈ 7.9857. This approximate value is in fact much more accurate than we would ever want for a norm in practice.

The square roots of the eigenvalues of B = AT A (which are all real and nonnegative) are called the singular values of A. The 2-norm of a matrix is therefore equal to its largest singular value. We return to singular values in full force in Section 4.4; see especially page 81. For now, let us just say that for general square matrices (not symmetric positive definite ones) the singular values can be significantly more friendly than eigenvalues: the singular values are real and positive, and they can be computed without forming AT A.

Specific exercises for this section: Exercises 4–6.

4.3 Special classes of matrices There are several important classes of matrices for which the algorithms for solution of linear sys- tems or eigenvalue problems can be simplified or made significantly more efficient than in the gen- eral case. We will see such matrices in detail in the next few chapters, where their importance and treatment become relevant in a local context. In this section we define two particularly important classes of matrices: symmetric positive definite matrices and orthogonal matrices.

Why single out these particular two classes? Because of their fundamental nature. Symmetric positive definite matrices extend the notion of a positive scalar, arise in many real-world applications, and virtually divide numerical linear algebra techniques into two groups—those that are designed for these specific matrices, and those that are not. Orthogonal matrices form a particularly useful class of matrix transformations which are the basis of many modern methods, in a sense that will become clearer in Chapters 6–8.

Symmetric positive definite matrices

The square matrix A is symmetric if AT = A. Furthermore, a matrix A is positive definite if xT Ax > 0 ∀ x �= 0.

Thus, for any column vector x = (x1, . . . , xn)T we require∑ni, j=1 ai, j xi x j > 0, provided that at least one component x j �= 0.

Symmetric positive definite matrices may be considered an extension of positive scalars to n dimensions. They arise often in practice.

✐

✐

✐

✐

✐

✐

✐

✐

4.3. Special classes of matrices 79

Example 4.12. Suppose we wish to find a minimum of a nonlinear C2 function of n variables, φ(x) = φ(x1, . . . , xn). This task extends minimization in one variable, considered in Section 3.4. Recall that for one variable the necessary condition for a minimum at x = x∗ is φ′(x∗) = 0, and a sufficient condition is that also φ′′(x∗) > 0.

In n variables the necessary condition for a minimum of φ at a point x∗ is that the vector of first derivatives ∂φ

∂xi , called the gradient, vanish at x∗; a sufficient condition that such a point

actually yields a minimum (and not a maximum or a saddle point) is that the matrix A of second

derivatives of φ, having the elements ai, j = ∂2φ∂xi∂x j , be symmetric positive definite at x∗. We study such problems in Section 9.2.

A symmetric n × n matrix has n real eigenpairs. If A is also positive definite, then for any eigenvalue λ = λ j and its associated eigenvector x = x j we can write

0 < xT Ax = λxT x = λ n∑

i=1 x2i ,

implying that λ j > 0. Hence all eigenvalues are positive, and we can order them as

λ1 ≥ λ2 ≥ ·· · ≥ λn > 0. It is easy to see that if A is symmetric positive definite, then all principal minors A[ j :k, j :k] are

also symmetric positive definite (simply consider all vectors x that have zeros in the first j −1 and the last n − k places). The determinants of these principal minors are all positive as well.

Note that for any rectangular matrix B with linearly independent columns the matrix A = BT B is symmetric positive definite, because for any nontrivial vector x we can write

xT Ax = xT BT Bx = ‖Bx‖22 > 0. Recall that

‖B‖2 = √ ρ(A).

Orthogonal vectors and matrices

Two vectors u and v of the same length are orthogonal to each other if their inner product vanishes, i.e.,

uT v = 0. If in addition each vector has a unit length, i.e., ‖u‖2 = ‖v‖2 = 1, then we say that the vectors are orthonormal.

In geometric terms, orthogonality of two vectors simply means that the vectors form a right angle between them. Of course, the notion of orthogonality extends to more than two vectors: a set of vectors are orthogonal (orthonormal) if each pair among them satisfies the above-mentioned orthogonality (orthonormality) relation.

Example 4.13. The simplest example of a set of orthogonal vectors is those which form the axes of the standard coordinate system. In three dimensions we have e1 = (1,0,0)T , e2 = (0,1,0)T , and e3 = (0,0,1)T , scaling the vectors so they are orthonormal; see Figure 4.4.

In this system, any vector w = (x , y, z)T can be expressed as w = xe1 + ye2 + ze3. We then say that x , y, and z are the coordinates of w with respect to the orthogonal basis (e1,e2,e3). All this may seem trivial, because these basis vectors (e1,e2,e3) form an identity matrix I . But in fact, expressing a vector in terms of its coordinates with respect to a given orthogonal basis is quite useful in general for more complicated settings.

✐

✐

✐

✐

✐

✐

✐

✐

80 Chapter 4. Linear Algebra Background

x

y

z

Figure 4.4. The Cartesian coordinate system as a set of orthonormal vectors.

Orthogonal matrices

A real square matrix Q is orthogonal if its columns are orthonormal, i.e.,

QT Q = I . It is straightforward to verify that for any vector x we have in the �2-norm, ‖Qx‖ = ‖x‖,

because ‖Qx‖2 = (Qx)T (Qx) = xT QT Qx = xT x = ‖x‖2.

This property of norm preservation is important in numerical computations. For one thing it means that applying an orthogonal transformation (read, “multiplying by Q”) in a floating point environ- ment does not (in principle) amplify errors. Moreover, this mathematical property can be conve- niently used for transforming matrices in certain situations, for example, to devise effective solution methods for least squares problems. Details on this are provided in Section 6.2.

In Example 4.13, where a standard coordinate system is considered, if we line up e1, e2, and e3 as columns of a matrix, then we get the identity matrix. The latter is the simplest example of an orthogonal matrix. It is straightforward to observe that orthogonal matrices remain orthogonal under row and column permutations.

Example 4.14. Rigid rotations are another example of an orthogonal linear operator. The matrix cos(θ ) sin(θ ) −sin(θ ) cos(θ )

 is orthogonal for any θ and corresponds simply to a rotation of a two-dimensional vector by an angle θ . In other words, multiplying a given vector by this matrix results in a vector whose length is the same but whose angle with the axes is shifted by θ compared to the original vector.

Specific exercises for this section: Exercises 7–13.

4.4 Singular values The singular value decomposition (SVD) is a basic matrix decomposition that is particularly stable under all weather conditions. The definition is given on the next page.

✐

✐

✐

✐

✐

✐

✐

✐

4.4. Singular values 81

Singular Value Decomposition. Given a rectangular, m ×n matrix A, the SVD reads

A = U�V T , where U is an m ×m orthogonal matrix, V is an n × n orthogonal matrix, and � is an m ×n matrix given by

� =



σ1

σ2

. . .

σr

 ,

with the singular values σ1 ≥ σ2 ≥ ·· · ≥ σr > 0. Everything else in � is zero.

Note the generality of the SVD in terms of matrix dimensions: no relationship between the positive integers m and n is imposed, and the rank of A satisfies rank(A) = r ≤ min(m,n). To get a feeling for the dimensions of the matrices involved in such a decomposition, see Figure 4.5.

A = U � V T

Figure 4.5. Singular value decomposition: Am×n = Um×m�m×n V Tn×n.

In MATLAB the relevant command is (you guessed correctly) svd. The instruction [U,Sigma,V]=svd(A) yields the three matrices appearing in the SVD definition. Somewhat confusingly, though usefully, the instruction s=svd(A) gives just the singular values in a one- dimensional array of length min(m,n). The first r positive singular values are padded with zeros if r < min(m,n).

Example 4.15. The SVD of the matrix

A =

 1 2

3 4

5 6



✐

✐

✐

✐

✐

✐

✐

✐

82 Chapter 4. Linear Algebra Background

is given by

U =

 0.229847696400071 0.883461017698525 −0.408248290463864 0.524744818760294 0.240782492132547 0.816496580927726

0.819641941120516 −0.401896033433433 −0.408248290463863

 ;

� =

 9.525518091565107 0

0 0.514300580658642

0 0

 ; V =

0.619629483829340 −0.784894453267053 0.784894453267053 0.619629483829340

 . The singular values of A are extracted from the diagonal of � and hence are σ1 = 9.525518091565107 and σ2 = 0.514300580658642.

For a square n ×n matrix A all three matrices U , V , and � are also n ×n. Then we can write the diagonal matrix � as

� =



σ1

. . .

σr

0 . . .

0

 .

Next, let r = n ≤ m. Then C = AT A is a symmetric positive definite matrix, as we have seen in Section 4.3. Denote its eigenvalues, in decreasing order, by λi , i = 1, . . . ,n. We can write

C = AT A = V�T U T U�V T = V�T �V T . The matrix �T � is diagonal with σ 21 , . . . ,σ

2 n on its main diagonal, and V

T provides a similarity transformation for C . Thus, σi = √λi and

‖A‖2 = σ1. So, the 2-norm of A is simply given by its largest singular value. This applies to any nonsquare matrix.

Note: In many ways singular values are to a general matrix what eigenvalues are to a symmetric positive definite one. On the other hand, make no mistake: for a general square matrix, singular values correspond to eigenvalue magnitudes and as such do not simply replace eigenvalues in terms of information content.

Specific exercises for this section: Exercises 14–16.

✐

✐

✐

✐

✐

✐

✐

✐

4.5. Examples 83

4.5 Examples Linear systems and eigenvalue problems arise in numerous applications and come in many flavors. In this section we present three examples that illustrate the diversity of applications that lead to such linear algebra problems.

The following descriptions are relatively long and may be considered as setting the stage for three case studies, to be returned to repeatedly in later chapters.

Example 4.16 (data fitting). Consider the problem of linear data fitting. Given measurements, or observations, (t1,b1), (t2,b2), . . . , (tm ,bm) = {(ti ,bi )}mi=1, we want to fit to this data a function v(t) which is described as a linear combination of n known, linearly independent functions of t , φ1(t),φ2(t), . . . ,φn(t). The points ti where the observations bi are made are assumed distinct, t1 < t2 < · · · < tm . The functions φi (t) are called basis functions.

Thus, we write

v(t) = n∑

j=1 x jφ j (t)

and seek coefficients x1, . . . , xn such that v(ti ) = bi , i = 1, . . . ,m. Since the sought function v obvi- ously depends on the choice of coefficients, let us denote this explicitly as v(t;x).

We can define ai j = φ j (ti )

and write these interpolation requirements as

a11x1 +a12x2 +·· ·+a1nxn = b1, a21x1 +a22x2 +·· ·+a2nxn = b2,

... = ... am1x1 +am2x2 +·· ·+amnxn = bm .

Let us further assume that m = n (i.e., the same number of data and unknown coefficients) and rewrite the requirements in matrix form Ax = b, where

A =

 a11 a12 · · · a1n a21 a22 · · · a2n

... ...

. . . ...

an1 an2 · · · ann

 , x = 

x1

x2 ...

xn

 , b = 

b1

b2 ...

bn

 .

It can be shown that since the functions φ j (t) are linearly independent, they generate linearly independent columns in A and hence there is a unique solution to the above matrix problem.

For the special case where we choose φ j (t) = t j−1 we obtain that v(t;x) is the interpolating polynomial of degree < n. Our matrix A then becomes

A =

 1 t1 · · · tn−11 1 t2 · · · tn−12 ...

... . . .

...

1 tn · · · tn−1n

 .

✐

✐

✐

✐

✐

✐

✐

✐

84 Chapter 4. Linear Algebra Background

This form of a matrix is called Vandermonde. The polynomial interpolation problem always has a solution, but as it turns out, under certain conditions there are numerical difficulties in solving the Vandermonde system.

As a specific instance suppose we have four observations, (0,1), (0.1,−0.9), (0.8,10), and (1,9). The following MATLAB script generates the corresponding 4×4 matrix A and data vector b, and then it solves the system Ax = b by the “backslash” command. The plotting instructions which follow this produce Figure 4.6.

t = [0,0.1,0.8,1]’; b = [1,-0.9,10,9]’; A = zeros(4,4); % This tells Matlab we have a 4 by 4 matrix in mind powers = 0:3; for j=1:4 A(:,j) = t.^powers(j);

end

x = A \ b; % This solves the system Ax = b

tt = -0.1:.01:1.1; pt = x(1) + x(2).*tt + x(3).*tt.^2 + x(4).*tt.^3; plot(tt,pt) hold on plot(t’,b’,’ro’,’LineWidth’,2) xlabel(’t’) ylabel(’v’)

−0.2 0 0.2 0.4 0.6 0.8 1 1.2 −2

0

2

4

6

8

10

12

t

v

Figure 4.6. Data fitting by a cubic polynomial using the solution of a linear system for the polynomial’s coefficients.

✐

✐

✐

✐

✐

✐

✐

✐

4.5. Examples 85

In real-world applications it is often not advisable or even feasible to have as many basis functions as data points. If laboratory measurements are the source for the data points, then there could very well be some inaccurate measurements, and hence redundancy (in the form of extra data points) would be a blessing. In any case it makes little sense to interpolate noisy measurements. Another instance is where we are merely trying to pick up a trend in the data: we consider a list of average temperatures in each of the past 10 years, say, not to carefully approximate what happened between those years but to obtain an indication of a slope, i.e., whether it has generally been warming up or cooling down.

Thus, we now want to find x ∈ Rn such that v(ti ;x) ≈ bi , i = 1, . . . ,m, where m ≥ n, but usually m > n. When the latter occurs, we say that the problem is overdetermined. It is convenient to imagine values such as n = 1, 2, or 3 and m = 100, say, when you read the following.

An obvious question is in what sense we want to do the data fitting. It is often natural (or at least easy enough) to use the �2-norm, which defines a general least squares problem

min x

m∑ i=1

[bi − v(ti ;x)]2 .

This in turn can be written as determining the parameters x from

min x

m∑ i=1

bi − n∑ j=1

x jφ j (ti )

2 , and further, with the rectangular matrix A having elements ai, j = φ j (ti ) the linear least squares problem can be formulated as

min x

‖b− Ax‖2, where the dimensions of the matrix and vectors are

A ∈ Rm×n , x ∈ Rn , b ∈ Rm , m ≥ n. (Note that squaring ‖b− Ax‖2 yields the same minimizer x.)

We describe several methods for solving the least squares problem in Chapter 6.

Example 4.17 (differential equations). Consider the recovery of a function v(t) from its given sec- ond derivative −g(t) on the interval [0,1]. Such a problem is typical for many applications. An example is the celebrated heat equation in one dimension: if the intensity of a heat source is given and we wish to recover the steady-state temperature of the body that is being heated, then under certain simplifying assumptions about the heat conductivity properties of the material we obtain a problem of the same type.

Obviously the process as described above is not unique: for any v which satisfies −v′′(t) = g(t), also w(t) = v(t)+ α+ βt has the same second derivative for any constants α and β. To fix these constants we need two additional constraints, and we consider the following possibilities:

1. v(0) = v(1) = 0, or 2. v(0) = 0, v′(1) = 0.

This yields a (fortunately simple) differential equation, −v′′ = g, with boundary conditions. To solve it numerically we approximate the differential equation on a discrete mesh, as in Figure 4.7. We subdivide the interval [0,1] into subintervals of size h each (say, h = 0.01), define ti = ih,

✐

✐

✐

✐

✐

✐

✐

✐

86 Chapter 4. Linear Algebra Background

0 1 t

h

0 1 2 N-1 N

Figure 4.7. A discrete mesh for approximating a function and its second derivative in Example 4.17.

i = 0,1, . . . , N , where Nh = 1, and approximate −v′′ = g by17

−vi+1 −2vi + vi−1 h2

= g(ti ), i = 1,2, . . . , N −1.

For the first case of boundary conditions v(0) = 0, v(1) = 0, we write correspondingly v0 = 0, vN = 0, to close the system. Collecting our unknowns into a vector and the difference equations into a matrix, we get Av = g, where

v =



v1

v2 ...

vN−2 vN−1

 , g =



g(t1)

g(t2) ...

g(tN−2) g(tN−1)

 , A = 1

h2



2 −1 −1 2 −1

. . . . . .

. . .

−1 2 −1 −1 2

 .

For instance, if N = 5, then h = .2 and

A = 25

 2 −1 0 0 −1 2 −1 0 0 −1 2 1 0 0 −1 2

 .

The matrix A is symmetric positive definite. (Indeed, this is why we have attached the minus sign to v′′; without it A would have been negative definite.) It is also very sparse when N is large: ai, j = 0 if i > j +1 or j > i +1, so out of (N −1)2 elements of A only fewer than 3(N −1) are nonzero. A typical result is displayed in Figure 4.8.

Turning to the other set of boundary conditions, we still set v0 = 0, but now at the other end we have to discretize the condition v′(1) = 0. We can do this by adding a ghost unknown, vN+1, and

17For any function f (t) ∈ C4[0,1], the second derivative f ′′ has the finite difference approximation

f ′′(ti ) = f (ti+1)−2 f (ti )+ f (ti−1) h2

+O(h2).

This can be readily seen by manipulating Taylor expansions of f (ti+1) and f (ti−1). Thus, for h small, −(v(ti+1)− 2v(ti )+v(ti−1))/h2 ≈ −v′′(ti ) = g(ti ). Solving the specified linear system for the vi ’s we then expect that vi ≈ v(ti ). Indeed, as it turns out, vi = v(ti ) +O(h2), but showing this is well beyond the scope of this chapter. All we are interested in here is the resulting linear system for these vi ’s.

✐

✐

✐

✐

✐

✐

✐

✐

4.5. Examples 87

0 0.2 0.4 0.6 0.8 1 −60

−50

−40

−30

−20

−10

0

10

20

30

40

t

g

(a) The given g(t) = −v′′(t).

0 0.2 0.4 0.6 0.8 1 −1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

t

v (b) The recovered v(t).

Figure 4.8. Recovering a function v satisfying v(0) = v(1) = 0 from its second derivative. Here N = 507.

writing

vN+1 − vN−1 2h

= 0, −vN+1 +2vN − vN−1

h2 = g(tN ),

thus extending the differential equation domain all the way to the boundary t = 1. The “ghost unknown” corresponds to a location beyond the boundary, but you really don’t need to understand this in depth for the present purpose. Thus, the first of these equations yields vN+1 = vN−1, and the second then reads 2

h2 (vN −vN−1) = g(tN ). So we get the linear system Av = g of size N , rather than

N − 1, because now vN qualifies as an unknown, as opposed to the situation previously discussed, where we had v(1) = 0. The system is

v =



v1

v2 ...

vN−1 vN

 , g =



g(t1)

g(t2) ...

g(tN−1) g(tN )

 , A = 1

h2



2 −1 −1 2 −1

. . . . . .

. . .

−1 2 −1 −2 2

 .

This matrix is no longer symmetric. But it is still very sparse and nonsingular. What about the boundary conditions v′(0) = v′(1) = 0? The above derivation yields a singular

matrix A! Its nullspace consists of all constant vectors (i.e., vectors whose components all have one and the same value). Indeed, the differential solution v(t) is also determined only up to a constant in this case.

Example 4.18 (principal component analysis). A veteran technique for data analysis that at its core is nothing but an SVD of a rectangular data matrix is principal component analysis (PCA). Below we first briefly discuss some generalities and then settle into a specific instance of a PCA

✐

✐

✐

✐

✐

✐

✐

✐

88 Chapter 4. Linear Algebra Background

application. We have no intention of reproducing a text on data analysis and statistics here, and if you are not familiar with some of the terminology below, then in our restricted context this should not stop you.

Let A be an m × n matrix, where each column corresponds to a different experiment of the same type and m is the dimension. We can think of each such column as a point (vector) in Rm . Further, we must assume that the empirical mean has already been subtracted from these columns, so that A has zero mean.

Assuming a normal (Gaussian) probability distribution for the data errors leads to considera- tions in Euclidean space; i.e., we are thus justified in using the �2-norm. Hence orthogonal trans- formations naturally arise. The PCA is such a transformation. It is really a coordinate rotation that aligns the transformed axes with the directions of maximum variance. Thus, writing the SVD as

A = U�V T , the largest variance is in the direction of the first column of U (the first principal component), the largest variance on the subspace orthogonal to the first principal component is in the direction of the second column of U , and so on. The matrix

B = U T A = �V T therefore represents a better alignment than the given A in terms of variance differentiation, and this can occasionally be of use.

Note that since the mean is zero, the covariance matrix is given by

C = AAT = U��T U T . This matrix is positive semidefinite and its eigenvectors are the columns of U , namely, the singular vectors which are the principal components here.

A way to make this process more practical is dimensionality reduction. Letting Ur consist of the first r columns of U , r < n, we represent the data by the smaller matrix Br = U Tr A. Then Br = �r V Tr with �r and Vr defined correspondingly. See examples in Section 8.2.

Let us emphasize that the assumptions that make PCA work are not always applicable. For statisticians this is a linear, nonparametric analysis that cannot incorporate prior knowledge and assumes some sublime importance of the variance as the way to differentiate or imply similarity between entities. Nonetheless, on many occasions this approach performs well.

Moving on to show a particular application, consider Figure 4.9(a). Graphics objects in R3

are often represented or approximated by a point cloud, a scattered data set of points that together presumably form a coherent surface. Such representations arise in numerous applications including laser scanning, medical imaging, and visualization. Relevant tasks are “cleaning” the data set from noise and outliers, resampling, downsampling, or upsampling it, and displaying the surface that gave rise to this cloud in a pleasing way (see Figure 11.6 on page 359). For such applications we need the normal directions at the points with respect to the implied surface.

Finding these normals (without forming the surface first!) can be done using PCA. For a fixed point p in the cloud, we define a neighborhood Np of nearby points. A simple choice would be those points whose distance in �2-norm from p is below some value, but there are more sophisticated ways for determining just who is a neighbor of the point p under focus. For our present purposes, let us assume it has been done and there are n = n p such neighbors. Next, we calculate the mean of these neighbors to find the centroid of Np , call it p̄. Our 3×n p data matrix A therefore has pi p − p̄ for its i th column, where pi p is the i th neighbor of p.

Next, we find the three singular vectors of A (also known as the eigenvectors of the covariance matrix). The first two principal vectors span the tangent plane at p. The third one, orthogonal to the first two and hence to the tangent plane, is our desired unsigned normal direction.

✐

✐

✐

✐

✐

✐

✐

✐

4.6. Exercises 89

50 100 150 200

50

100

150

200

250

300

350

400

450

500

550

(a) Point cloud.

50 100 150 200

50

100

150

200

250

300

350

400

450

500

550

(b) With normals.

Figure 4.9. Example 4.18: a point cloud representing (a) a surface in three-dimensional space, and (b) together with its unsigned normals.

An additional, global, pass over the points is then required to determine what sign our normal vectors should have to point outward rather than inward of the implied surface, but this is beyond the present scope. The results of applying this approach to Figure 4.9(a) are depicted in Fig- ure 4.9(b).

Specific exercises for this section: Exercise 17.

4.6 Exercises 0. Review questions

(a) What is a singular matrix?

(b) Suppose Ax = b �= 0 is a linear system and A is a square, singular matrix. How many solutions is it possible for the system to have?

(c) Suppose we know that a matrix is nonsingular. How many solutions are there for the linear system Ax = 0?

(d) What is the spectrum of a matrix?

(e) Define algebraic multiplicity and geometric multiplicity of eigenvalues.

(f) What is a diagonalizable matrix?

✐

✐

✐

✐

✐

✐

✐

✐

90 Chapter 4. Linear Algebra Background

(g) How is the norm of a vector related to the absolute value of a scalar?

(h) Prove that for any given vector, ‖x‖1 ≥ ‖x‖2. (i) What is an induced matrix norm?

(j) What is the spectral radius of a matrix? Is it a norm?

(k) Suppose we define a function of a matrix to be the maximum absolute value among all its entries. Is it a norm? If not, give a counterexample.

(l) Is it true that every symmetric positive definite matrix is necessarily nonsingular?

(m) Give an example of a 3×3 orthogonal matrix not equal to the identity. (n) What is the fundamental difference (from a linear algebra point of view) between a data

fitting problem, where the number of data points is larger than the number of coefficients to be determined, and the case where these quantities are equal?

1. Show that if all row-sums of a square matrix are equal to zero, then the matrix is singular.

2. Determine whether the following is true or false and justify your answer: if Ax = λx and A is nonsingular, then 1

λ is an eigenvalue of A−1.

3. This exercise requires a fresher knowledge of complex arithmetic than what is assumed else- where in this chapter; consult page 71.

(a) Show that the eigenvalues of a symmetric matrix are real.

(b) Show that if the eigenvalues of a real matrix are not real, then the matrix cannot be symmetric.

(c) A real square matrix A is called skew-symmetric if AT = −A. Show that the eigenvalues of a skew-symmetric matrix are either purely imaginary or zero.

4. (a) Consider the �p-norm of a vector, defined on page 74. Show that the first two properties of a norm hold. (The third, triangle inequality property of a norm holds as well but is tougher to show in general.)

(b) Find the 1-norm, 2-norm, 3-norm, and maximum norm of the vectors

x = (1,2,3,4)T and y = (1000,−2000,3000,−4000)T .

5. Determine whether the following statement is true or false and give a justification: If A is nonsingular, then for any induced norm, ‖A−1‖ = ‖A‖−1.

6. For an m ×n matrix A show that

‖A‖1 = max 1≤ j≤n

m∑ i=1

|ai j |.

7. Let A be symmetric positive definite. Show that the so-called energy norm

‖x‖A = √

xT Ax

is indeed a (vector) norm.

[Hint: Show first that it suffices to consider only diagonal matrices A with positive entries on the main diagonal.]

✐

✐

✐

✐

✐

✐

✐

✐

4.6. Exercises 91

8. Which of the following matrices are necessarily orthogonal?

(a) Permutation matrices, which are obtained by permuting rows or columns of the identity matrix, so that in each row and each column we still have precisely one value equal to 1;

(b) symmetric positive definite matrices;

(c) nonsingular matrices;

(d) diagonal matrices.

9. Find all the values of a and b for which the matrix a 1 1+b 1 a 1

1−b2 1 a

 is symmetric positive definite.

10. (a) Show that the matrix  c s −s c

 is orthogonal if c2 + s2 = 1.

(b) Givens rotations are based on rotation operations of the form c s −s c

 a1 a2

=  α

0

 . In other words, given a vector we rotate it (while preserving its length) so that one of its components is zeroed out.

i. Use orthogonality to express α in terms of a1 and a2. ii. Find c and s that do the job.

11. (a) Show that the eigenvalues of an orthogonal matrix are equal to 1 in modulus.

(b) A projector is a square matrix P that satisfies P2 = P . Find the eigenvalues of a projec- tor.

12. The condition number of an eigenvalue λ of a given matrix A is defined as

s(λ) = 1 xT w

,

where x is a (right) eigenvector of the matrix, satisfying Ax = λx, and w is a left eigenvector, satisfying wT A = λwT . Both x and w are assumed to have a unit �2-norm. Loosely speak- ing, the condition number determines the difficulty of computing the eigenvalue in question accurately; the smaller s(λ) is, the more numerically stable the computation is expected to be.

Determine the condition numbers of the eigenvalue 4 for the two matrices discussed in Ex- ample 4.7. Explain the meaning of your results and how they are related to the observations made in the example.

✐

✐

✐

✐

✐

✐

✐

✐

92 Chapter 4. Linear Algebra Background

13. Let A be a real m × n matrix and denote by S the subspace of Rn consisting of all those vectors that are orthogonal to any vector in null(AT ). Show that S = range(A).

14. Let A be real and symmetric, and denote its eigenvalues by λi , i = 1, . . . ,n. Find an expression for the singular values of A in terms of its eigenvalues.

15. Let A be skew-symmetric, and denote its singular values by σ1 ≥ σ2 ≥ ·· ·σn ≥ 0. Show that (a) If n is even, then σ2k = σ2k−1 ≥ 0, k = 1,2, . . . ,n/2. If n is odd, then the same relation-

ship holds up to k = (n −1)/2 and also σn = 0. (b) The eigenvalues λ j of A can be written as

λ j = (−1) j ıσ j , j = 1, . . . ,n.

16. (a) Suppose that A is an orthogonal matrix. What are its singular values?

(b) Is the SVD of a given matrix A unique in general?

17. Obtain the matrix A for Example 4.17 in the case where v′(0) = v′(1) = 0, and show that it is singular.

4.7 Additional notes This chapter is different from all others in our book in that it aims to provide background and as such does not propose or discuss algorithms except in the case studies of Section 4.5.

We have found it necessary to include it (while not including a similar chapter on calculus) because there appears to be more variation in students’ backgrounds when it comes to linear algebra. The more theoretical concepts of linear algebra such as rings and ideals are not necessary for our present purposes, but the notions of norms, orthogonality, and symmetric positive definiteness, to mention a few, are fundamental to numerical computing.

If this book were written without an eye on practical computing issues, a review chapter of this sort would likely have had a bit more on determinants. They are interesting creatures indeed, but their computational usefulness has greatly declined in recent decades.

Needless to say, our chapter is not meant as a replacement for a linear algebra textbook. One of many good books is Strang [64].

The behavior of a vector norm function ‖ · ‖ near zero is interesting. Let xε be a family of vectors depending on a parameter ε ≥ 0 such that ‖xε‖ = ε. Then so long as ε > 0 we have no further information on the components of xε. But for ε = 0 we know that each and every component of x0 is zero.

Example 4.16 is a primer for problems of data fitting, further considered in several later chap- ters; see in particular Sections 6.1 and 10.1. Similarly, Example 4.17 is a primer for boundary value problems in ordinary differential equations; see in particular Section 16.7. Finally, Example 4.18 re- lates directly to Chapter 8 and elsewhere. Figure 4.9 was produced by H. Huang, and further details and references for point cloud treatment can be found in Huang et al. [45].

✐

✐

✐

✐

✐

✐

✐

✐

Chapter 5

Linear Systems: Direct Methods

In this chapter we consider numerical methods for solving a system of linear equations Ax = b. We assume that the given matrix A is real, n × n, and nonsingular and that b is a given real vector in Rn , and we seek a solution x that is necessarily also a vector in Rn . Such problems arise frequently in virtually any branch of science, engineering, economics, or finance.

There is really no single technique that is best for all cases. Nonetheless, the many available numerical methods can generally be divided into two classes: direct methods and iterative methods. The present chapter is devoted to methods of the first type. In the absence of roundoff error, such methods would yield the exact solution within a finite number of steps.

The basic direct method for solving linear systems of equations is Gaussian elimination, and its various aspects and variants occupy the first seven sections of this chapter. Section 5.1 presents the method in simple terms.

The bulk of the algorithm involves only the matrix A and amounts to its decomposition into a product of two matrices that have a simpler form. This is called an LU decomposition, developed in Section 5.2. Such an alternative view is useful, for instance, when there are several right-hand-side vectors b each requiring a solution, as the LU decomposition can then be shared.

The simple algorithm of Section 5.1 is not guaranteed to be stable or even well-defined in general. In Section 5.3 we modify it using pivoting strategies to make the algorithm of Gaussian elimination with pivoting practically stable in general. Then in Section 5.4 we discuss efficient methods for implementing these algorithms.

The following three sections are devoted to variants of the same basic algorithm for special classes of matrices. Important simplifications can be made for symmetric positive definite matri- ces, as discussed in Section 5.5. If the matrix A contains only a few nonzero elements, then it is called sparse. Techniques for storing such matrices and for solving linear systems for the special case where all elements are zero outside a narrow band along the main diagonal are presented in Section 5.6. Direct methods for handling more general sparse matrices are briefly considered in Section 5.7.

In Section 5.8 we consider not the numerical solution of Ax = b but rather its assessment. Since there are roundoff errors, the result of any of our algorithms is approximate, and so the ques- tion of how close the approximate solution is to the exact one arises. This in turn depends on the condition number of the matrix, a concept that is defined and further discussed here.

93

✐

✐

✐

✐

✐

✐

✐

✐

94 Chapter 5. Linear Systems: Direct Methods

5.1 Gaussian elimination and backward substitution In this section we show the following:

• How to solve linear equations when A is in upper triangular form. The algorithm is called backward substitution.

• How to transform a general system of linear equations into an upper triangular form, to which backward substitution can be applied. The algorithm is called Gaussian elimination.

Note: Experience suggests that many of our students have already seen much of the ma- terial in Section 5.1. As the section count in this chapter increases, though, there may be fewer people to whom this is all old news, so pick your spot for jumping in.

Backward substitution

Occasionally, A has a special structure that makes the solution process simple. For instance, if A is diagonal, written as

A =

 a11

a22 . . .

ann

 (by this notation we mean that the off-diagonal elements are all 0), then the linear equations are uncoupled, reading aii xi = bi , and the solution is obviously

xi = bi aii

, i = 1,2, . . . ,n. A more involved instance of a special structure is an upper triangular matrix

A =

 a11 a12 · · · a1n

a22 . . .

... . . .

...

ann

 ,

where all elements below the main diagonal are zero: ai j = 0 for all i > j . In this case each equation is possibly coupled only to those following it but not to those

preceding it. Thus, we can solve an upper triangular system of equations backwards. The last row reads annxn = bn , so xn = bnann . Next, now that we know xn , the row before last can be written as an−1,n−1xn−1 = bn−1 − an−1,nxn , so xn−1 = bn−1−an−1,n xnan−1,n−1 . Next the previous row can be dealt with, yielding xn−2, etc. We obtain the backward substitution algorithm given on the next page. In MATLAB this can be written as

x(n) = b(n) / A(n,n); for k = n-1:-1:1 x(k) = ( b(k) - A(k,k+1:n)*x(k+1:n) ) / A(k,k);

end

✐

✐

✐

✐

✐

✐

✐

✐

5.1. Gaussian elimination and backward substitution 95

Algorithm: Backward Substitution. Given an upper triangular matrix A and a right-hand-side b,

for k = n : −1 : 1 xk =

bk −∑nj=k+1 akj x j akk

end

In Example 4.1 we performed one step of this backward substitution algorithm when obtaining x1 once we knew x2.

It is worth noting that the successful completion of the algorithm is possible only if the diag- onal elements of A, aii , are nonzero. If any of them is zero, then the algorithm experiences what is known as a breakdown. However, for an upper triangular matrix such as A, a zero diagonal term will occur only if the matrix is singular, which violates our assumption from the beginning of this chapter. So we do not need to worry about breakdowns here.

Cost of backward substitution

What is the cost of this algorithm? In a simplistic way we just count each floating point operation (such as + and ∗) as a flop. The number of flops required here is

1+ n−1∑ k=1

((n − k)+ (n − k)+1)= n∑

k=1 (2(n − k)+1)= n2.

In general, this way of evaluating complexity of algorithms considers only part of the picture. It does not take into account data movement between elements of the computer’s memory hierarchy. In fact, concerns of data locality can be crucial to the execution of an algorithm. The situation is even more complex on multiprocessor machines. In MATLAB, the same backward substitution algorithm implemented as

for k = n:-1:1 x(k) = b(k); for j = k+1:n

x(k) = x(k) - A(k,j)*x(j); end x(k) = x(k) / A(k,k);

end

may take much longer to execute compared to the vectorized script presented before, especially for large matrices, because no advantage is taken of the way in which the matrix and vector elements are stored. A general rule for generating efficient MATLAB implementations is to avoid for- and while-loops whenever possible.

Still, noting that the run time of this algorithm is O(n2) and not O(n3) is often meaningful in practice.

✐

✐

✐

✐

✐

✐

✐

✐

96 Chapter 5. Linear Systems: Direct Methods

Forward substitution

Another instance of a special matrix structure is a lower triangular matrix

A =

 a11

a21 a22 ...

. . . . . .

an1 an2 · · · ann

 ,

where all elements above the main diagonal are zero: ai j = 0 for all i < j . The situation is similar to the upper triangular case, except that we apply forward substitution to compute a solution. The algorithm is given on this page.

Algorithm: Forward Substitution. Given a lower triangular matrix A and a right-hand-side b,

for k = 1 : n xk =

bk −∑k−1j=1 akj x j akk

end

Example 5.1. Consider solving the equations

5x1 = 15, x1 +2x2 = 7,

−x1 +3x2 +2x3 = 5. The matrix A is lower triangular, and in our general notation we have

A =

 5 0 0

1 2 0

−1 3 2

 , b = 

15

7

5

 . Applying the forward substitution algorithm we get x1 = 155 = 3, then x2 = 7−32 = 2, then x3 = 5+3−6

2 = 1.

Gaussian elimination

Next, assume that the matrix A has no special zero-structure. Gaussian elimination is a generaliza- tion of the procedure used in Example 4.1. Thus, we use elementary row transformations to reduce the given linear system to an equivalent upper triangular form. Then we solve the resulting system using backward substitution.

The reason this works is that the solution to our problem Ax = b is invariant to • multiplication of a row by a constant, • subtraction of a multiple of one row from another, and • row interchanges.

✐

✐

✐

✐

✐

✐

✐

✐

5.1. Gaussian elimination and backward substitution 97

These claims can be easily verified directly. Simply recall that Ax = b is a concise form for the system of equations

a11x1 +a12x2 +·· ·+a1nxn = b1, a21x1 +a22x2 +·· ·+a2nxn = b2,

... = ... an1x1 +an2x2 +·· ·+annxn = bn ,

and check what each of these operations amounts to. Performing each of these operations can be recorded directly on the augmented matrix

(A|b) =

 a11 a12 · · · a1n b1 a21 a22 · · · a2n b2

... ...

. . . ...

...

an1 an2 · · · ann bn

 .

Eliminating one column at a time

1. Eliminate the first column elements below the main diagonal:

• Subtract a21a11 × the first row from the second row. • Subtract a31a11 × the first row from the third row.

...

• Subtract an1a11 × the first row from the nth row.

This produces

(A(1)|b(1)) =

 a11 a12 · · · a1n b1 0 a(1)22 · · · a(1)2n b(1)2 ...

... . . .

... ...

0 a(1)n2 · · · a(1)nn b(1)n

 .

2. Next, consider the (n −1)×n submatrix of (A(1)|b(1)) obtained by ignoring the first row and column. These are the elements that have been modified by the first stage of the process, described above, but not set to 0. Apply exactly the same step as before to this submatrix; i.e., subtract a32

a(1)22 × the second row from the third row, etc.

The reason we are “allowed” to do this is that the first row remains untouched by these op- erations, and the first column for the other rows is all zeros, so nothing changes in the first column during subsequent operations.

✐

✐

✐

✐

✐

✐

✐

✐

98 Chapter 5. Linear Systems: Direct Methods

After this second stage we have an augmented matrix of the form

(A(2)|b(2)) =



a11 a12 · · · · · · a1n b1 0 a(1)22 a

(1) 23 · · · a(1)2n b(1)2

... 0 a(2)33 · · · a(2)3n b(2)3

... ...

... . . .

... ...

0 0 a(2)n3 · · · a(2)nn b(2)n

 .

The superscripts in the above expression show for each element at which stage it has last been modified.

3. Repeat the process. After n −1 such stages we obtain an upper triangular system

(A(n−1)|b(n−1)) =



a11 a12 · · · · · · a1n b1 0 a(1)22 a

(1) 23 · · · a(1)2n b(1)2

... 0 a(2)33 · · · a(2)3n b(2)3

... ...

. . . . . .

... ...

0 0 · · · 0 a(n−1)nn b(n−1)n

 .

This procedure leads to the Gaussian elimination algorithm in its simplest form given on the next page. It is depicted pictorially for n = 4 in Figure 5.1.

In stating the Gaussian elimination algorithm we have not bothered to zero out the elements below the main diagonal in the modified matrix A: this part of the matrix is not considered in the ensuing backward substitution.

Of course, for this algorithm to work we need to assume that all those pivotal elements a(k−1)kk by which we divide are nonzero and in fact not very close to 0 either. We deal with the question of ensuring this later, in Section 5.3. For now, let’s assume there is simply no such issue.

Example 5.2. For Example 4.1 we have n = 2; hence the loops in the general algorithm collapse into defining just one update for k = 1, i = 2, j = 2:

(A|b) =  1 −4 −10

0.5 −1 −2

 , l21 = a21

a11 = 0.5,

a(1)22 =−1−0.5 · (−4)= 1, b(1)2 =−2−0.5 · (−10)= 3,

(A1)|b(1)) =  1 −4 −10

0 1 3

 . Note that in the algorithm we never actually define a(1)22 or b

(1) 2 . Instead, we simply overwrite the

original value a22 = −1 with the new value a22 = 1 and the original value b2 = −2 with the new value b2 = 3.

✐

✐

✐

✐

✐

✐

✐

✐

5.1. Gaussian elimination and backward substitution 99

Algorithm: Gaussian Elimination. Given a real, nonsingular n×n matrix A and a vector b of size n, first transform into upper triangular form,

for k = 1 : n −1 for i = k +1 : n

lik = aik akk

for j = k +1 : n ai j = ai j − likakj

end

bi = bi − likbk end

end

Next, apply the algorithm of backward substitution.

A b A (1)

b(1)

A(2) b(2) A(3) b(3)

Figure 5.1. Gaussian elimination for the case n = 4. Only areas of potentially nonzero entries are shown.

The cost of the Gaussian elimination algorithm in terms of operation count is approximately

n−1∑ k=1

((n − k)+2(n − k)(n − k+1)) ≈ 2 n−1∑ k=1

(n − k)2

= 2((n −1)2+·· ·+1) = 2 3

n3 +O(n2)

✐

✐

✐

✐

✐

✐

✐

✐

100 Chapter 5. Linear Systems: Direct Methods

flops.18 In particular, as the size of the matrix A increases the cost of Gaussian elimination rises cubically. Comparing this to the cost of backward substitution we see that the cost of the elimination phase dominates for all but very small problems.

Specific exercises for this section: Exercises 1–2.

5.2 LU decomposition In this section we show that the process of Gaussian elimination in fact decomposes the matrix A into a product L ×U of a lower triangular matrix L and an upper triangular matrix U .

Let us continue to assume that the elements a(k−1)kk encountered in the Gaussian elimination process are all bounded safely away from 0. We will remove this assumption in the next section.

Elementary lower triangular matrices

Consider the first stage of Gaussian elimination described above. These are elementary row opera- tions applied to the matrix A to zero out its first column below the (1,1) entry. These operations are also applied to the right-hand-side vector b. Note, however, that the operations on b can actually be done at a later time because they do not affect those done on A.

We can capture the effect of this first stage by defining the elementary n ×n lower triangular matrix

M (1) =



1

−l21 1 −l31 1

... . . .

−ln1 1

 .

(This matrix has zeros everywhere except in the main diagonal and in the first column: when de- scribing matrices, blank does not mean nothing—it means zero.) Then the effect of this first stage is seen to produce

A(1) = M (1) A and b(1) = M (1)b. Likewise, the effect of the second stage of Gaussian elimination, which is to zero out the

second column of A(1) below the main diagonal, can be written as

A(2) = M (2) A(1) = M (2)M (1) A and b(2) = M (2)b(1) = M (2)M (1)b, where

M (2) =



1

1

−l32 . . . ...

. . .

−ln2 1

 .

18For those readers who remember and appreciate the usefulness of Riemann sums, the final result can be approxi- mated by recognizing that for large enough n this is nothing but an approximation of the integral

∫ n 0 x

2dx .

✐

✐

✐

✐

✐

✐

✐

✐

5.2. LU decomposition 101

(This matrix has zeros everywhere except in the main diagonal and in the second column below the main diagonal.)

Obtaining the LU decomposition

This procedure is continued, and after n − 1 such stages it transforms the matrix A into an upper triangular matrix, which we call U , by

U = A(n−1) = M (n−1) · · · M (2)M (1) A. Likewise, b(n−1) = M (n−1) · · · M (2)M (1)b. Multiplying U by [M (n−1)]−1, then by [M (n−2)]−1, etc., we obtain

A = LU , where L = [M (1)]−1 · · · [M (n−2)]−1[M (n−1)]−1.

To see what L looks like, note first that [M (k)]−1 has exactly the same form as M (k), i.e., it’s the identity matrix plus nonzero elements in the kth column below the main diagonal, except that these elements are without the minus sign, viz., lk+1,k , lk+2,k , . . . , ln,k . For instance, you can verify directly that the matrix M (2) defined above, multiplied by

[M (2)]−1 =



1

1

l32 . . .

... . . .

ln2 1

 ,

yields the identity matrix. So, [M (k)]−1 are also elementary lower triangular matrices. Now, you can verify directly that

the product of such matrices yields

L =



1

l21 1

l31 l32 1 ...

... . . .

. . .

ln1 ln2 · · · ln,n−1 1

 .

The elements of L are therefore obtained during the Gaussian elimination process! The LU decomposition is depicted pictorially for n = 10 in Figure 5.2.

Example 5.3. Let us verify the above claims for the matrix

A =

 1 −1 3 1 1 0

3 −2 1

 .

✐

✐

✐

✐

✐

✐

✐

✐

102 Chapter 5. Linear Systems: Direct Methods

A = L U

Figure 5.2. LU decomposition for the case n = 10. Only areas of potentially nonzero entries in the square matrices L and U are shown.

1. The Gaussian elimination process for the first column yields l21 = 11 = 1, l31 = 31 = 3, so

M (1) =

 1 0 0

−1 1 0 −3 0 1

 , A(1) = M (1) A = 

1 −1 3 0 2 −3 0 1 −8

 . 2. The Gaussian elimination process for the second column yields l32 = 12 , so

M (2) =

 1 0 0

0 1 0

0 −0.5 1

 , U = A(2) = M (2) A(1) = 

1 −1 3 0 2 −3 0 0 −6.5

 . 3. Note that

[M (1)]−1 =

 1 0 0

1 1 0

3 0 1

 , [M (2)]−1 = 

1 0 0

0 1 0

0 0.5 1

 ,

[M (1)]−1[M (2)]−1 =

 1 0 0

1 1 0

3 0.5 1

= L, and

LU =

 1 0 0

1 1 0

3 0.5 1

 

1 −1 3 0 2 −3 0 0 −6.5

= 

1 −1 3 1 1 0

3 −2 1

= A. This explicitly specifies the LU decomposition of the given matrix A.

We therefore conclude that the Gaussian elimination procedure of the previous section de- composes A into a product of a unit lower triangular19 matrix L and an upper triangular matrix U .

19A unit lower triangular matrix is a lower triangular matrix with 1’s on the main diagonal.

✐

✐

✐

✐

✐

✐

✐

✐

5.2. LU decomposition 103

This is the famous LU decomposition. Together with the ensuing backward substitution the entire solution algorithm for Ax = b can therefore be described in three steps, as depicted in the algorithm given on this page.

Algorithm: Solving Ax = b by LU Decomposition. Given a real nonsingular matrix A, apply LU decomposition first:

A = LU . Given also a right-hand-side vector b:

1. Forward substitution: solve Ly = b.

2. Backward substitution: solve Ux = y.

Separating the decomposition step

Note that decomposing A and solving Ly = b simultaneously is identical to applying the Gaussian elimination algorithm given on page 99. Note also that the decomposition step costs O(n3) oper- ations, whereas the forward and backward substitutions each cost about n2 operations. Thus, the decomposition step dominates the total cost. Moreover, since we do not need b at all to perform the decomposition step, we may perform the LU decomposition once and then solve different linear systems of equations for different right-hand-sides b at an O(n2) cost for each system solve.

Example 5.4. Continuing Example 5.3, suppose we are to solve Ax = b for a new right-hand-side vector

b =

 2

4

1

 . The LU decomposition phase having already been accomplished, we proceed with the forward sub- stitution phase, solving Ly = b and obtaining y1 = 2, y2 = 4− y1 = 2, y3 = 1−3y1 −0.5y2 =−6. Then solving Ux = y yields x3 = 66.5 = 1213 , x2 = 12 (2+3 1213) = 3113 , x1 = 2+ 3113 −3 1213 = 2113 , so

x = 1 13

 21

31

12

 is the sought solution.

Example 5.5. Let us denote the inverse of a nonsingular matrix A by G = A−1. If we are given G, then the solution of a linear system Ax = b can be calculated as Gb, which takes O(n2) flops to evaluate in general.

✐

✐

✐

✐

✐

✐

✐

✐

104 Chapter 5. Linear Systems: Direct Methods

But first we have to find G! One way, as good as any, is to use the LU decomposition of A. Thus, we calculate L and U first. Now, the identity matrix is composed of the unit vectors

ei =



0 ...

0

1

0 ...

0

 (with the value 1 in the i th location, i = 1, . . . ,n). For each of these unit vectors we then solve Axi = ei by writing L(Uxi ) = ei and performing forward and then backward substitutions. Thus xi is the i th column of G and

G = A−1 = [x1, . . . ,xn]. Using again a simple operation count, each forward or backward substitution costs about n2

flops, so for n right-hand sides we have n · 2n2 flops added to the cost of the LU decomposition. Discounting lower order terms in n this approximately yields

2

3 n3 +2n3 = 8

3 n3

flops.

Forming A−1 explicitly and multiplying by b is generally not a recommended way to solve linear systems of equations for several reasons. For one, it can be wasteful in storage, a point that will become much clearer when we discuss banded matrices in Section 5.6. Moreover, it is more computationally expensive than going by the LU decomposition route, though by less than an order of magnitude. Also, it may give rise to a more pronounced presence of roundoff errors. Finally, it simply has no advantage over the methods presented here to offset the disadvantages mentioned above.

Example 5.6. Another simple, yet general, by-product of LU decomposition is a tool for computing the determinant of a matrix. Recall that the determinant of a product of matrices equals the product of the matrix determinants. Moreover, the determinant of an upper triangular or a lower triangular matrix is the product of the elements on the main diagonal of such a matrix. Thus, for a general square matrix A = LU , we have

det(A) = det(L) ·det(U ) = 1 · · ·1 ·u11u22 · · ·unn = u11u22 · · ·unn = �nk=1ukk .

In particular, A is nonsingular if and only if all ukk are nonzero, k = 1, . . . ,n.

Note that storage can be handled very efficiently for the LU decomposition. Assuming that the input matrix A can be discarded, its storage locations can be reused to store both U and L in the

✐

✐

✐

✐

✐

✐

✐

✐

5.3. Pivoting strategies 105

form 

u11 u12 u13 · · · u1n l21 u22 u23

. . . u2n

l31 l32 u33 . . . u3n

... . . .

. . . . . .

...

ln1 ln2 · · · ln,n−1 unn

 .

The main diagonal of L need not be stored since all its values are equal to 1.

Specific exercises for this section: Exercises 3–4.

5.3 Pivoting strategies In this section we modify the algorithm of Gaussian elimination by pivoting, thus making it generally stable.

The process of Gaussian elimination (or LU decomposition) described in Sections 5.1 and 5.2 cannot be applied unmodified to all nonsingular matrices. For instance, the matrix

A = 0 1

1 0

 is clearly nonsingular, and yet a11 = 0, so the algorithm breaks down immediately. Here is another example, slightly more subtle.

Example 5.7. Consider the system

x1 + x2 + x3 = 1, x1 + x2 +2x3 = 2,

x1 +2x2 +2x3 = 1. The matrix A defined by these equations is nonsingular (indeed, the unique solution is x = (1,−1,1)T ), also all elements of A are nonzero, and yet Gaussian elimination yields after the first stage

1 1 1 1

1 1 2 2

1 2 2 1

⇒ 

1 1 1 1

0 0 1 1

0 1 1 0

 . Next, a(1)22 = 0 and we cannot proceed further.

In this case there is an obvious remedy: simply interchange the second and the third rows! This yields 

1 1 1 1

1 2 2 1

1 1 2 2

⇒ 

1 1 1 1

0 1 1 0

0 0 1 1

 , and backward substitution proceeds as usual.

✐

✐

✐

✐

✐

✐

✐

✐

106 Chapter 5. Linear Systems: Direct Methods

In the above examples the process broke down because a zero pivotal element a(k−1)kk was en- countered. But a(k−1)kk does not have to exactly equal 0 for trouble to arise: undesirable accumulation of roundoff error may arise also when a(k−1)kk is near 0.

Example 5.8. Consider a perturbation of Example 5.7 that reads

x1 + x2 + x3 = 1, x1 +1.0001x2+2x3 = 2,

x1 +2x2 +2x3 = 1. The exact solution, correct to 5 digits, is x ≈ (1,−1.0001,1.0001)T . Now, Gaussian elimination in exact arithmetic can be completed and yields

1 1 1 1

1 1.0001 2 2

1 2 2 1

⇒ 

1 1 1 1

0 0.0001 1 1

0 1 1 0

⇒ 

1 1 1 1

0 0.0001 1 1

0 0 −9999.0 −10000

 . Assume next that we are using floating point arithmetic with base β = 10 and precision t = 3. Then backward substitution gives

x3 = 1, x2 = 0, x1 = 0. On the other hand, if we interchange the second and third rows we obtain

1 1 1 1

1 2 2 1

1 1.0001 2 2

⇒ 

1 1 1 1

0 1 1 0

0 0.0001 1 1

⇒ 

1 1 1 1

0 1 1 0

0 0 0.9999 1

 . Now backward substitution with 3 decimal digits gives

x3 = 1.000, x2 =−1.000, x1 = 1.000, which is correct, in that the difference between this solution and the exact solution is less than the rounding unit.

Partial pivoting

The problem highlighted in these simple examples is real and general. Recall from Chapter 2 that roundoff errors may be unduly magnified if divided by a small value. Here, if a pivotal element a(k−1)kk is small in magnitude, then roundoff errors are amplified, both in subsequent stages of the Gaussian elimination process and (even more apparently) in the backward substitution phase.

These simple examples also suggest a strategy to resolve the difficulty, called partial pivoting: as the elimination proceeds for k = 1, . . . ,n − 1, at each stage k choose q = q(k) as the smallest integer for which

|a(k−1)qk | = maxk≤i≤n |a (k−1) ik |,

and interchange rows k and q . (Recall from page 97 that this operation does not change the solution, provided that the corresponding entries of b(k−1) are exchanged as well.) Then proceed with the elimination process.

✐

✐

✐

✐

✐

✐

✐

✐

5.3. Pivoting strategies 107

This Gaussian elimination with partial pivoting (GEPP) strategy certainly resolves the diffi- culty in all the above examples. In Examples 5.7 and 5.8, in particular, for k = 1 we have q = 1, i.e., no row interchange is necessary, but for k = 2 we get q = 3, so the second and third rows are interchanged.

How does partial pivoting affect the LU decomposition? Note first that for the matrix of Example 5.7 it really is not possible to write A = LU with L unit lower triangular and U nonsingular upper triangular (try it!). However, the operation of row interchange is captured in general by a permutation matrix P. We start with P = I and proceed with GEPP; at each stage k where a row interchange is mandated, we record this fact by interchanging rows k and q of P . For Examples 5.7 and 5.8 this yields

P A = LU , where

P =

 1 0 0

0 0 1

0 1 0

 .

Forming P A = LU But does this really work in general? Recall from Section 5.2 that L is obtained as the inverse of a product of elementary matrices, L−1 = M (n−1) · · · M (2)M (1). Now the process is captured instead by the sequence

B = M (n−1) P(n−1) · · · M (2) P(2) M (1) P(1), where each P(i) is an elementary permutation matrix encoding one row interchange. What is im- plied, then, is that we can pull all the permutations together and write

B = L−1 P , where P = P(n−1) · · · P(2) P(1). Such a claim is no longer elementary, dear Watson! Indeed, it is by far the most subtle point in Sections 5.1–5.3. But it turns out to hold in general, namely, the straightforward process of partial pivoting that results in U = B A being upper triangular can be written as P A = LU for an appropriate unit lower triangular matrix L.

Let us show that this is indeed so. Suppose we have a 4 × 4 matrix, and thus the process terminates after three steps. (Everything we discuss here straightforwardly extends to an n × n matrix, and the only reason we opt for the specific case of n = 4 is for ease of exposition.)

We have U = M (3) P(3) M (2) P(2) M (1) P(1) A.

Define M̃ (3) = M (3), M̃ (2) = P(3)M (2)(P(3))T , and M̃ (1) = P(3) P(2) M (1)(P(2))T (P(3))T . Then, by using the fact that the inverse of an elementary permutation matrix is its transpose, we have

U = M̃ (3)M̃ (2) M̃ (1)︸ ︷︷ ︸ M̃

P̃(3) P̃(2) P̃(1)︸ ︷︷ ︸ P

A.

It is tempting to think that we are done, because we have managed to have the M̃ ( j ) and the P( j )

comfortably bundled with their own specimen. Not so fast! The order is right, and P is certainly a valid permutation matrix, but to declare victory we need to be convinced that M̃ (and hence its inverse) is indeed a unit lower triangular matrix.

In general, a symmetric permutation of a lower triangular matrix does not maintain this struc- ture. The only reason it works in this case is because for a given j , the unit lower triangular matrix M ( j ) is basically an identity matrix with the conveniently limited “correction” of having only the

✐

✐

✐

✐

✐

✐

✐

✐

108 Chapter 5. Linear Systems: Direct Methods

n− j entries in the j th column below the main diagonal possibly nonzero. A symmetric permutation of the form we are discussing corresponds to multiplying on the left by an elementary permutation matrix P (i) with i > j and on the right by its transpose, and amounts to swapping rows and columns in a way that merely affects nonzero off-diagonal entries in the strictly lower triangular part of M ( j ). Indeed, zeros that change places are of no interest, and neither are swapped diagonal entries, which are all equal to 1 to begin with! In Exercises 7 and 8 you are given an opportunity to see all this up close, by working on a 4×4 matrix with actual numbers and by looking into a similar procedure for symmetric indefinite matrices.

In general, note that the permutation matrix P is nonsingular, being just a permutation of the rows of the identity matrix. Moreover, P is orthogonal, i.e., P−1 = PT , so we can write

A = (PT L)U , where the rows and columns of PT are just unit vectors.

For GEPP it follows immediately that the multipliers lik (the elements of the unit lower tri- angular matrix L in the decomposition P A = LU ) are all bounded in magnitude by 1, written explicitly as

|lik | ≤ 1, 1 ≤ i ,k ≤ n. In an actual implementation we will not store a matrix P , most of which consists of zeros, nor

do we have to physically interchange matrix rows in memory (although we may end up doing the latter for reasons of easier memory access). We can simply keep track of what row interchanges are implied by the algorithm in a one-dimensional array p. The elements of p are used for indexing the correct rows, as specified in algorithmic form later on. For example, the array p that corresponds to the matrix P in Examples 5.7 and 5.8 is p = [1,3,2].

GEPP stability

Does the incorporation of this partial pivoting procedure guarantee stability of the Gaussian elimina- tion algorithm, in the sense that roundoff errors do not get amplified by factors that grow unbound- edly as the matrix size n increases? As far as PT L is concerned the answer is affirmative, but we have to check U , too. We need to be assured that

gn(A) = max i, j ,k

|a(k)i, j |

(in words, the maximum magnitude of all elements that arise in the course of the elimination process) does not grow too fast, i.e., does not grow exponentially as a function of n.

Unfortunately, it is not possible in general to guarantee stability of GEPP. Indeed, this algo- rithm is not even invariant to scaling of the rows: in Example 5.8 if the third row were originally multiplied by 10−5, then no row interchanges would have resulted upon applying the GEPP strategy. But the difficulty highlighted in that example would persist.

We could modify the partial pivoting strategy into one called scaled partial pivoting. Thus, define initially for each row i of A a size

si = max 1≤ j≤n |ai j |.

Then, at each stage k of the Gaussian elimination procedure, choose q = q(k) as the smallest integer for which

|a(k−1)qk | sq

= max k≤i≤n

|a(k−1)ik | si

,

and interchange rows k and q .

✐

✐

✐

✐

✐

✐

✐

✐

5.3. Pivoting strategies 109

This latter strategy entails a computational overhead and cannot completely eliminate poten- tial instability, as we see below in Example 5.9. Still, we should not underestimate scaling as an important tool for enhancing the numerical stability of the process in general. At the very least, we should ensure that the linear system to be solved has rows that all have comparable norm. In cer- tain situations it may be worth paying the extra computational cost of ensuring reasonable scaling throughout the whole elimination process. See Example 5.11 for a case in point.

Example 5.9. Let us give a rare instance where Gaussian elimination with scaled partial pivoting is unstable. Applying it to the matrix

A =



1 0 · · · · · · 0 1 −1 1 0 · · · 0 1 −1 −1 1 . . . ... 1

... . . . −1 . . . 0 ...

... . . .

. . . . . .

. . . 1

−1 · · · · · · · · · −1 1

 produces no row interchanges, and yet after the kth step, k = 1,2, . . . ,n −1, the elements in the last column satisfy a(k)i,n = 2k , i = k +1, . . . ,n (please check!), so gn(A) = a(n−1)n,n = 2n−1. Such a growth rate of gn is unacceptable.

A strategy that does guarantee stability is complete pivoting: at each stage choose q and r as the smallest integers for which

|a(k−1)qr | = max k≤i, j≤n |a

(k−1) i j |,

and interchange both row q and column r with row i and column j , respectively. However, this strategy is significantly more expensive than partial pivoting, and instances in which (scaled) partial pivoting really fails in a big way appear to be extremely rare in practice. For this reason, most of the commonly used computer codes for directly solving linear systems use a GEPP approach.

Matrices requiring no pivoting

We end this section by mentioning that there are certain families of matrices for which it is known that pivoting is not required (at least theoretically). One such family is symmetric positive matrices, to the decomposition of which we will devote a whole section soon. Another class are diagonally dominant matrices. An n ×n matrix A is diagonally dominant if

|aii | ≥ n∑

j=1 j �=i

|ai j |.

In words, for each row the diagonal element is at least as large in magnitude as the sum of mag- nitudes of the rest of the elements combined. It is possible to show that if a nonsingular matrix is diagonally dominant, Gaussian elimination does not require pivoting.

Specific exercises for this section: Exercises 5–9.

✐

✐

✐

✐

✐

✐

✐

✐

110 Chapter 5. Linear Systems: Direct Methods

5.4 Efficient implementation In general, the performance of an algorithm depends not only on the number of arithmetic operations that are carried out but also on the frequency of memory access. So, if the algorithms we have seen thus far in this chapter can be arranged to work on chunks of vectors and matrices, then their efficiency increases. In this section we explore this important practical issue. Specifically, we briefly discuss the efficient implementation of LU decomposition and other basic linear algebra operations.

Vectorization

MATLAB, in particular, used to implement if-, for-, and while-loops in a relatively inefficient way, while doing operations on chunks of matrices very efficiently. MathWorks, the commercial company that owns the MATLAB product, has put considerable effort in the last few years into closing the gaps in performance, and these are now much smaller compared to the past. In any case, different variants of the algorithms we have seen in this chapter which have precisely the same operation count may perform very differently, depending on the implementation. We have already had a glimpse of this in Section 5.1, where the two for-loops which the backward substitution algorithm uses were implemented in a MATLAB script using only one for-loop, with the other replaced by an inner product of the form A(k,k+1:n)*x(k+1:n).

Even more important, this also can be done for the LU decomposition algorithm. Let us as- sume no pivoting at first, and consider the first algorithm we saw on page 99, ignoring the operations on the right-hand-side b. Of the three loops involving k, i , and j , the loop on k appears to be par- ticularly sequential (i.e., it is not easy to couple together or perform more than one stage at a time). The results of the altered matrix at stage k are used at stage k +1, and later on. But this is not so for the other two loops. The determination of the multipliers lik , done in the i -loop for a given k, can be done all at once, as a vector

lk =

 lk+1,k lk+2,k

...

ln,k

 . Similarly for the subsequent update (the j -loop) of the rows of the submatrix of A: each row i is updated by a multiple lik of the row ak,k+1:n .

For two vectors y and z of the same length, the inner product is defined to recall as the scalar

yT z = m∑

i=1 yi zi .

This is used in the forward and backward substitution algorithms. The outer product, on the other hand, is defined as

yzT ,

i.e., it is the matrix whose (i , j )th element is yi z j .

Example 5.10. For the two vectors

y = 3

2

 , z = 

1

2

3

 ,

✐

✐

✐

✐

✐

✐

✐

✐

5.4. Efficient implementation 111

there is no inner product because the lengths don’t match, but there are two outer products, given by

yzT = 3 6 9

2 4 6

 and zyT = 

3 2

6 4

9 6

 . MATLAB carries out these operations quietly, when issuing instructions such as y * z’. So you have to be careful about whether the array you have there corresponds to a row or a column vector, as unwelcome surprises may otherwise result.

Now, the i and j for-loops in the LU decomposition algorithm can be expressed in terms of an outer product:

for k = 1 : n −1 lk =

( ak+1,k

akk , . . . ,

an,k akk

)T Ak+1:n,k+1:n = Ak+1:n,k+1:n − lk ∗ak,k+1:n

end

This outer product form allows us to express the bulk of Gaussian elimination as operations on consecutive elements in chunks of matrices. If we now incorporate partial pivoting, then to retain the ability to relate to chunks of A we will actually execute row interchanges if necessary, rather than use indirect indexing.

Our code for solving linear systems of equations

We are now in position to write an efficient linear system solver. It is not as robust, general and efficient as MATLAB’s backslash instruction x= A\b, but on the other hand we can see what is going on. Our code uses the algorithm given on page 103, and you should be able to just read it.

function x = ainvb(A,b) % % function x = ainvb(A,b) % % solve Ax = b

[p,LU] = plu (A); y = forsub (LU,b,p); x = backsub (LU,y);

function [p,A] = plu (A) % % function [p,A] = plu (A) % % Perform LU decomposition with partial pivoting. % Upon return the coefficients of L and U replace those % of the input n by n nonsingular matrix A. The row interchanges % performed are recorded in the 1D array p.

n = size(A,1);

✐

✐

✐

✐

✐

✐

✐

✐

112 Chapter 5. Linear Systems: Direct Methods

% initialize permutation vector p p = 1:n;

% LU decomposition with partial pivoting for k = 1:n-1

% find row index of relative maximum in column k [val,q] = max(abs(A(k:n,k))); q = q + k-1;

% interchange rows k and q and record this in p A([k,q],:)=A([q,k],:); p([k,q])=p([q,k]);

% compute the corresponding column of L J=k+1:n; A(J,k) = A(J,k) / A(k,k);

% update submatrix by outer product A(J,J) = A(J,J) - A(J,k) * A(k,J);

end

function y = forsub (A,b,p) % % function y = forsub (A,b,p) % % Given a unit lower triangular, nonsingular n by n matrix A, % an n-vector b, and a permutation p, % return vector y which solves Ay = Pb

n = length(b);

% permute b according to p b=b(p);

% forward substitution y(1) = b(1); for k = 2:n y(k) = b(k) - A(k,1:k-1) * y(1:k-1);

end

function x = backsub (A,b) % % function x = backsub (A,b) % % Given an upper triangular, nonsingular n by n matrix A and % an n-vector b, return vector x which solves Ax = b

n = length(b); x = b; x(n) = b(n) / A(n,n); for k = n-1:-1:1 x(k) = ( b(k) - A(k,k+1:n)*x(k+1:n) ) / A(k,k);

end

✐

✐

✐

✐

✐

✐

✐

✐

5.4. Efficient implementation 113

In the MATLAB version we have used a function like plu which actually utilizes the three for-loops on i , j , and k would take about three times longer to obtain the LU decomposition of a 100× 100 matrix. Note that these two implementations have the same operation (or flop) count. Other examples can be easily constructed where two algorithms require the same operation count to achieve the same output, differing only by the amount of memory access, with vastly different CPU times. How can they differ so drastically in execution time, then?

Fast memory access and BLAS

Indeed, the simple flop count that was given in Section 5.1 for Gaussian elimination and for forward and backward substitutions is useful for a coarse (“first cut”) indication of which algorithm is ex- pected to be faster. However, these operation counts completely ignore the relative cost of logical operations and the important question of memory access.

Computer memories are built as hierarchies, ranging from very fast, expensive, and small to slow, cheap, and large. In decreasing order of speed, such a hierarchy would typically include

• registers, • cache, • memory, and • disk.

Arithmetic and logical operations are typically done only at the registers, at the top of the hierarchy, so data stored elsewhere have to be moved into the registers first. Such data movement, especially if it does not start at the cache, can be much more expensive than the cost of an arithmetic operation. Moving data in chunks, rather than using many independent data item moves, makes a big difference in overall performance.

Computer manufacturers, especially of high-performance machines, have standardized basic matrix operations such as matrix-matrix and matrix-vector multiplications into basic linear algebra subroutines (BLAS).20

The actual implementation is machine-dependent, but the user can see a library of BLAS which are machine-independent, so one is not required to modify code because the laptop battery has died. The BLAS are organized into levels: the higher the level, the larger the number of flops per memory reference that can be achieved. The first level (BLAS1) includes operations such as multiplying a vector by a scalar and adding another vector, as in

z = a ∗x+y (SAXPY),

as well as inner products. The mysterious word SAXPY is actually a widely used term in numerical linear algebra, standing for “scalar alpha X plus Y”. The second level (BLAS2) are matrix-vector multiplications. These include triangular systems and matrix updates by vector outer products. Thus, the LU decomposition utilizing outer products uses level-2 rather than level-1 BLAS. The third- level routines (BLAS3) are for matrix-matrix multiplications and include triangular systems with many right-hand sides. The details become complex, involving, for instance, the possibility of LU decomposition by blocks. This may be important in special circumstances for high-performance computing, but it goes well beyond the scope of this chapter.

20The S in this acronym is not a plural but rather is a remnant of days when the Fortran language reigned, to the fury of computer scientists.

✐

✐

✐

✐

✐

✐

✐

✐

114 Chapter 5. Linear Systems: Direct Methods

Scaling the rows of the given problem

Now that we have introduced the function plu let us return to the question of scaling. Note that the partial pivoting strategy implemented here does not take scaling into account, so we should ensure ahead of time that the matrix A is not scaled too poorly, i.e., replace the problem Ax = b if necessary by a scaled version S Ax = Sb, with S a diagonal matrix.

Example 5.11. Let us define a linear system to be solved by the script

n = 100; h = 1/(n-1); K = 100; A = zeros(n,n); for i = 2:n-1 A(i,i) = -2/h^2 - K; A(i,i-1) = -1/h^2; A(i,i+1) = -1/h^2;

end A(1,1) = 1; A(n,n) = 1; % end definition of A

xe = ones(n,1); % exact solution of 1’s b = A*xe; % corresponding right hand side

Having defined the exact solution to be x = (1,1, . . . ,1)T , we can calculate the errors that result upon applying our linear system solver introduced above and those for MATLAB’s staple “backslash” command. The obtained errors are 1.92e-11 and 5.68e-11, respectively.

Such errors can often be attributed to the conditioning of the problem; see Section 5.8. How- ever, here the source for the loss of accuracy is different. Dividing each bi and row i of A by si = max1≤ j≤n |ai j | (notice that s1 = sn = 1 and si ≈ 2h−2 otherwise), and solving the scaled sys- tems by the same two codes, the obtained errors are 1.52e-14 and 1.38e-14, respectively. This is an improvement by three orders of magnitude!

The scaling does induce a different pivoting strategy in plu. Without it the first row is used for pivoting only in the 76th step, whereas with scaling no row permutation is found necessary.

Specific exercises for this section: Exercises 10–11.

5.5 The Cholesky decomposition Recall from Chapter 4 that a matrix A is symmetric if AT = A and positive definite if

xT Ax > 0 ∀ x �= 0.

Such matrices extend in important ways the concept of a positive scalar. In this section we see how solving linear systems for such matrices through the LU decomposition correspondingly simplifies, leading to the Cholesky decomposition.

No need for pivoting

It is easy to see that if A is symmetric positive definite, then all principal minors A[ j :k, j :k] are also symmetric positive definite (simply consider all vectors x that have zeros in the first j −1 and the last n − k places). The determinants of these principal minors are all positive as well, and it transpires

✐

✐

✐

✐

✐

✐

✐

✐

5.5. The Cholesky decomposition 115

that the pivots ukk resulting from LU decomposition are also all positive. Indeed, it can be shown that Gaussian elimination without pivoting is a stable algorithm for symmetric positive definite matrices.

Symmetrizing the LU decomposition

Consider the LU decomposition of a symmetric positive definite matrix, written as

A = LU . Recall that L is unit lower triangular, whereas U is upper triangular with diagonal elements ukk > 0. We can write

U =



u11

u22 . . .

. . .

unn





1 u12u11 · · · · · · u1n u11

1 u23u22 · · · u2n u22

. . . ...

. . . ...

1

 = DŨ ,

so the LU decomposition reads A = L DŨ .

But transposing it we have A = AT = Ũ T DLT .

Since this decomposition is unique we must have Ũ T = L, i.e., the decomposition reads A = L DLT ,

where L is unit lower triangular and D is diagonal with positive elements ukk . It is also possible to write D = D1/2 D1/2 with

D1/2 = diag{√u11, . . . ,√unn}, whence the LU decomposition is written as

A = GGT

with G = L D1/2 a lower triangular matrix. This is called the Cholesky decomposition. It is possible to derive an algorithm for its implementation which requires not only roughly half the space (due to symmetry, of course) but also half the flops of the general LU decomposition algorithm, viz., 1 3 n

3 +O(n2). This can be done by imposing the equality A = GGT elementwise. As naive as this may sound at first, it is something that one cannot do for Gaussian elimination algorithms for general matrices without being potentially forced to employ complex arithmetic.

Example 5.12. In the 2×2 case we can require

A = a11 a12

a12 a22

= g11 0

g21 g22

g11 g21 0 g22

 .

✐

✐

✐

✐

✐

✐

✐

✐

116 Chapter 5. Linear Systems: Direct Methods

Now we have precisely three conditions for the three unknown quantities g11, g12, and g22. For a11 we get a11 = g211, from which it follows that g11 =

√ a11. Next, we have for a12 the

condition g21 = a12g11 , where g11 is already known. Finally, using the equation for a22 we get g22 =

√ a22 − g221.

This algorithm, given below, can be straightforwardly applied to larger matrices in the same fashion. In this algorithm the entries of A are overwritten by the entries of G. A key here, and the reason attempting to follow the same strategy for general matrices (that is, not necessarily symmetric positive definite) will fail, is that the arguments of the square root are guaranteed to be positive if A is symmetric positive definite. (This is not entirely trivial to show, but please trust us: it is true.)

Algorithm: Cholesky Decomposition. Given a symmetric positive definite n × n matrix A, this algorithm overwrites its lower part with its Cholesky factor.

for k = 1 : n −1 akk = √akk for i = k +1 : n

aik = aik akk

end

for j = k +1 : n for i = j : n

ai j = ai j −aika jk end

end

end

ann =√ann

In MATLAB, the reliable built-in function chol calculates the so-called Cholesky factor of a matrix. Notice, though, that R=chol(A) gives the matrix R = GT such that A = RT R. It does not make a difference whether R or G is used, so long as it is done consistently.

Example 5.13. Let us create a symmetric positive definite matrix as follows. For a given n, fill an n×n array C by random numbers. With a little luck, the corresponding matrix C will be nonsingular, and thus A = CT C will be symmetric positive definite (check this!).

We now create also a random exact solution x and define the right-hand-side b = Ax. Next we “forget” that we know x and try to find it, once through the Cholesky decomposition (involving no pivoting) and once through the general LU decomposition procedure with pivoting. The following script carries all this out and records relative errors:

C = randn(n,n); A = C’*C; xe = randn(n,1); % the exact solution b = A*xe; % generate right-hand-side data

R = chol(A); % Cholesky factor % the following line is for compatibility with our forsub

✐

✐

✐

✐

✐

✐

✐

✐

5.6. Sparse matrices 117

D = diag(diag(R)); L = D \ R’; bb = D \ b; p = 1:n; y = forsub(L,bb,p); % forward substitution R’y = b x = backsub(R,y); % backward substitution Rx = y rerx = norm(x-xe)/norm(xe) % error by Cholesky

xd = ainvb(A,b); % ignores spd use partial pivoting rerxd = norm(xd-xe)/norm(xe) % error by general routine

Running this script with n = 500, say, the routine ainvb uses very few (and, as it turns out, unnecessary!) row permutations. When using the more efficient Cholesky decomposition, errors of at least a similar quality and occasionally better are produced. Of course, no partial pivoting is utilized with Cholesky.

We return to this script in Example 5.20.

Specific exercises for this section: Exercises 12–14.

5.6 Sparse matrices There are many situations in practice where the matrix is large and sparse. What this means is that most of its elements are equal to zero. For an n × n matrix, it is convenient to think of a sparse matrix as having O(n) nonzero elements.

Frequently, the sparsity of a matrix originates from the nature of the underlying operator. The latter often represents local operations, where only a few components of a given grid or network of nodes “interact” with each other. For instance, the second tridiagonal matrix that we have seen in Example 4.17 in the context of discretization of a one-dimensional second derivative operator is N × N , but of its N2 elements only 3N − 2 are nonzero. For h = 0.01 we have N = 100, so this translates into 298 out of 10,000 elements of A being nonzero. The discretized second derivative is indeed an example of a local operator: its value for a certain point on a grid depends only on the point itself and its two immediate neighbors.

For very large sparse matrices, simply ignoring the sparsity property is not a practical option, because memory requirements may become prohibitive. For instance, think of the above-mentioned tridiagonal matrix, with N = 1,000,000. Storing the approximately 3 million nonzero elements in double precision, and other data such as row and column indices, consumes a fraction of computer memory in today’s terms: a mere few tens of megabytes. But if we try to store this matrix on all of its 1 trillion entries in memory, then we would be seriously testing (or, in fact, wildly exceeding) the limits of our computer’s fast memory resources, and for no good reason at all: there is no need to waste space on storing zero entries—we just need to know where they are.

Storing sparse matrices

So, a problem has been identified, but how do we solve it? When a matrix is sparse it is not nec- essarily best to store it as a two-dimensional array. Rather, we could store it in the form of three nonzero one-dimensional arrays, which contain row indices, column indices, and the corresponding numerical values. Notice that the first two are integer arrays.

The triplet form (or coordinate form) is the simplest format: an n × n matrix that has m nonzero entries is represented by vectors {i, j,v}, which are m-long each. For an integer 1 ≤ k ≤ m, the corresponding kth entries of ik , jk ,vk represent vk = aik , jk . This is the format used in MATLAB’s sparse matrix representation.

Other commonly used formats are the compressed sparse row form and the compressed sparse column form. These forms are in fact slightly more economical to work with (in terms of performing matrix operations) than the triplet form. The idea here is that if we go in an orderly fashion by rows

✐

✐

✐

✐

✐

✐

✐

✐

118 Chapter 5. Linear Systems: Direct Methods

or by columns, one of the integer arrays could simply contain pointers to the start of every row or column in the other integer array, saving some storage along the way. For instance, in sparse row form, the vector i has n +1 entries, whereas j and v have m entries. Column indices of the entries in row k are stored in jik through jik+1−1.

Example 5.14. The matrix

A =

 2.4 −3 0 0 0 1.3

0 2 −6

 is represented in MATLAB’s triplet form as follows:

(1,1) 2.4000 (1,2) -3.0000 (3,2) 2.0000 (2,3) 1.3000 (3,3) -6.0000

Note that the entries are presented column by column. Alternatively, Figure 5.3 illustrates the compressed sparse row form of the matrix.

1 1 2.4

3 2 -3

4 3 1.3

6 2 2

3 -6

eof

Figure 5.3. Compressed row form of the matrix in Example 5.14. Shown are the vectors i, j,v. The arrows and the “eof” node are intended merely for illustration of where the elements of the vector i point to in j and correspondingly in v.

It is impossible to appreciate the merits of sparse storage schemes for such a small matrix, but these concepts become useful for larger sparse matrices.

We also mention data structures that are based on storing matrices by their diagonals. The MATLAB command spdiags does so. This often comes in handy, as many matrices in applica- tions can be characterized in terms of a few nonzero diagonals. The matrices of Example 4.17 are in such a form.

✐

✐

✐

✐

✐

✐

✐

✐

5.6. Sparse matrices 119

Check out the commands sparse and full. The help documentation for these commands provides the names of a few additional MATLAB commands that deal with sparse matrices.

When representing matrices in sparse form, part of the programming effort is related to writ- ing functions for matrix operations. For example, matrix addition typically requires inserting new nonzero entries, unless the matrices share identical nonzero patterns. It is not difficult to program, but is not as trivial and seamless as matrix addition in the dense case. Instances of basic operations that entail writing specialized routines are the transpose operation, matrix-vector product, matrix multiply, and several other operations.

Banded matrices and their LU decomposition

When solving a linear system of equations, Ax = b, the effectiveness of Gaussian elimination and its variants greatly depends on the particular sparsity pattern of the matrix A. In Section 5.7 we briefly mention ways of increasing the efficiency by performing effective permutations. In Chapter 7 we discuss in detail alternatives to direct methods for situations where forming the LU decomposition results in a prohibitive amount of extra floating point operations and extra storage. But there are special cases of sparse matrices for which the LU decomposition works very effectively. One such instance is the class of banded matrices.

The matrix A is banded if all its elements are zero outside a band of diagonals around the main diagonal. Thus, there are positive integers p and q , 1 ≤ p,q ≤ n, such that

ai j = 0 if i ≥ j + p or if j ≥ i +q .

The bandwidth is defined as the number of consecutive diagonals that may contain nonzero ele- ments, i.e., p+q −1. The upper bandwidth is q and the lower bandwidth is p.

A particular banded matrix is depicted pictorially for n = 10, p = 2, and q = 3 in Figure 5.4. Elementwise we have

A =



a11 · · · a1q ...

. . . . . .

. . .

ap1 . . .

. . . . . .

. . .

. . . . . .

. . . . . .

. . .

. . . . . .

. . . . . . an−q+1,n

. . . . . .

. . . ...

an,n−p+1 · · · ann

 .

For a diagonal matrix, p = q = 1; for a full (or dense) matrix, p = q = n. More interestingly, in Example 4.17 we see tridiagonal matrices, where p = q = 2.21

21If p = q, then p−1 = q −1 is often referred to as the semibandwidth of the matrix.

✐

✐

✐

✐

✐

✐

✐

✐

120 Chapter 5. Linear Systems: Direct Methods

A

Figure 5.4. A banded matrix for the case n = 10, p = 2, q = 3. Only areas of potentially nonzero entries are shaded in green.

LU decomposition for banded matrices

Let us consider the LU decomposition without pivoting first. In the loops on i and on j we simply don’t need to zero out elements that are already zero or to update rows with that part of the pivot’s row which is zero. For a banded matrix the algorithm is therefore directly modified and is given on the current page.

Algorithm: LU Decomposition for Banded Matrices. Given a real, banded, n ×n matrix A which requires no pivoting, the following overwrites the upper triangular part of A by U :

for k = 1 : n −1 for i = k +1 : k + p−1

lik = aik akk

for j = k +1 : k +q −1 ai j = ai j − likakj

end

end

end

✐

✐

✐

✐

✐

✐

✐

✐

5.6. Sparse matrices 121

The resulting L and U factors then inherit the band form and can be written as

L =



1 ... 1

l p1 . . . 1 . . .

. . . . . .

. . . . . .

. . .

. . . . . .

. . .

ln,n−p+1 · · · 1

 ,

U =



u11 · · · u1q . . .

. . . . . .

. . . . . .

. . .

. . . . . .

. . .

. . . . . . un−q+1,n . . .

...

unn

 .

(Combine Figures 5.2 and 5.4 to see it in color.) Of course, none of these zero triangles is ever stored. If the band sizes p and q remain small

and fixed as the matrix size n grows, then we obtain an algorithm with O(n) storage and O(n) execution time—a great improvement. Applying higher level BLAS is more complicated, though.

For the case of a tridiagonal matrix, in particular, the i and j loops simplify into single state- ments. The resulting algorithm is occasionally referred to as the Thomas algorithm; see Exercise 17.

Pivoting

Just because a matrix is banded does not mean that it no longer requires pivoting! If the usual row partial pivoting is applied, then the band structure may be altered due to row interchanges. Using such pivoting, the upper bandwidth q of U may increase to q + p − 1. Moreover, although the number of nonzeros in each column of the unit lower triangular L remains as without pivoting, they may no longer be tucked next to the main diagonal in a neat band. (Please check this statement, e.g., by constructing a suitable example and running MATLAB’s routine lu.)

In the above discussion we did not assume anything about the sparsity pattern within the band. In general, those locations in A where an original zero is overwritten by a nonzero element during LU decomposition are referred to as a fill-in. Banded matrices for which no pivoting is required and which do not have zeros within the band yield no fill-in, as we have seen. Row permutations used during partial pivoting do introduce some fill-in into L, but not overwhelmingly so when the band is narrow.

However, significant sparsity within a not-very-narrow band is harder to take care of. Banded matrices that have many zeros also within the band serve as a motivating example for the class of

✐

✐

✐

✐

✐

✐

✐

✐

122 Chapter 5. Linear Systems: Direct Methods

iterative solution methods, discussed in Chapter 7. Techniques for minimizing the damage caused by fill-in when using direct methods, with which the present chapter is concerned, are discussed next.

Specific exercises for this section: Exercises 15–19.

5.7 Permutations and ordering strategies We have already noticed that the effectiveness of Gaussian elimination may strongly depend on the sparsity pattern of the matrix; banded matrices introduced in the previous section provide a convincing example that great advances can be made with LU decomposition if no crippling fill-in results. In this section we discuss ways of making the Gaussian elimination procedure for more general sparse matrices as effective as possible by permutations and reordering strategies.

Note: The topic of Section 5.7 is more advanced and more current than those of other sections in this chapter.

The effect of row and column permutations on fill-in

Let us start with a long but motivating example which shows that in fact reordering may result in a significant performance difference in operation count and storage requirements.

Example 5.15. Consider an n ×n matrix A whose elements satisfy

ai j = { �= 0 if i = j or i = 1 or j = 1, 0 otherwise.

An illustration of A for a 5 × 5 case is given below, where × stands for a possibly nonzero matrix element.22 Consider

A =



× × × × × × × 0 0 0 × 0 × 0 0 × 0 0 × 0 × 0 0 0 ×

 .

Suppose now that C is a matrix obtained by swapping the first and the last rows of A, so

C =



× 0 0 0 × × × 0 0 0 × 0 × 0 0 × 0 0 × 0 × × × × ×

 .

22To fully appreciate the effect of sparsity and fill-in you really should think of a large matrix, say, n = 500 or n = 5000.

✐

✐

✐

✐

✐

✐

✐

✐

5.7. Permutations and ordering strategies 123

If A represents a matrix associated with a linear system Ax = b, then the first and the last elements of b are also exchanged. This corresponds to simply writing down the equations of the system in a slightly different order, just like we did when applying row partial pivoting in Section 5.3.

However, if A is symmetric, which can be important, as we saw in Section 5.5, then we lose this property in C . Therefore, we apply the same swapping also to the columns of A. This corresponds to swapping the first and last unknowns and yields a matrix B whose elements are given by

bi j = { �= 0 if i = j or i = n or j = n, 0 otherwise.

An illustration of the sparsity pattern of B for the 5×5 case is given by

B =



× 0 0 0 × 0 × 0 0 × 0 0 × 0 × 0 0 0 × × × × × × ×

 .

The matrices A and B are often referred to as arrow matrices, for obvious reasons. If A is symmetric, then so is B .

Suppose that no pivoting is required (for example, if A and hence B are diagonally dominant), and let us denote by A = L AUA the LU decomposition of A and by B = L BUB the LU decompo- sition of B . What are the nonzero patterns of L A,UA , L B ,UB and the overall computational costs entailed in solving the system with A and in solving the system with B?

For A the first stage of Gaussian elimination produces a disastrous fill-in: when zeroing out the first column below the (1,1) element, we subtract scalar multiples of the first row from each of the other rows, and this yields

A(1) =



× × × × × 0 × × × × 0 × × × × 0 × × × × 0 × × × ×

 .

At this point the original, attractive sparsity pattern of the matrix is gone! The resulting A(1) looks like what one would get by applying LU decomposition to any generic dense matrix, and subse- quently the factors L A and UA are fully dense.

The situation with the matrix B is much more encouraging. It is straightforward to see that thanks to the fact that the first row of B has only nonzero elements in the first and last columns, we get no fill-in, so

B(1) =



× 0 0 0 × 0 × 0 0 × 0 0 × 0 × 0 0 0 × × 0 × × × ×

 .

✐

✐

✐

✐

✐

✐

✐

✐

124 Chapter 5. Linear Systems: Direct Methods

The same happens in the next stages of the LU decomposition, yielding

L B =



× 0 0 0 0 0 × 0 0 0 0 0 × 0 0 0 0 0 × 0 × × × × ×

 , UB =



× 0 0 0 × 0 × 0 0 × 0 0 × 0 × 0 0 0 × × 0 0 0 0 ×

 .

Thus, the overall storage requirements as well as the overall computational work are very modest for B .

There is nothing special about the size n = 5; everything above is applicable to a general n. It is worth your time doing Exercise 21: the difference in storage and flop count estimates between handling A and B is intriguing, showing that swapping equations and unknowns in the original linear system results in a substantially improved situation.

Permutation strategies

To understand things in a more general way, let us revisit the notion of permutation matrices. We have defined those in the context of partial pivoting in Section 5.3 and have observed that a permu- tation matrix P satisfies PT = P−1. Now, suppose that P is a given permutation matrix, and p is the corresponding one-dimensional array that represents the same permutation. Then Ax = b and P Ax = Pb have the same solution. Using PT P = I , let us write

(P APT )(Px) = Pb. Doing this incorporates mathematically the fact that we are now looking at Px rather than x, as per the performed permutation. Note that if A is symmetric, then so is B = P APT . We can rewrite the linear system as

By = c, where y = Px and c = Pb. In Example 5.15, P is a permutation matrix associated with the array p= [n,2,3,4, . . . ,n −2,n −1,1].

In modern direct solvers much attention is devoted to the question of what permutation matrix should be selected. Based on the discussion so far, we may intuitively think of two possible ways of aiming to reduce the storage and computational work:

• Reduce the bandwidth of the matrix. • Reduce the expected fill-in in the decomposition stage.

The question is how to accomplish the above mentioned goals in practice. One danger to be aware of is the combinatorial nature of the issue at hand. Suffice it to observe that for a linear system with n unknowns there are n! different orderings. We certainly do not want to sort through such a number of possibilities when the solution algorithm itself involves O(n3) operations; the latter all of a sudden seems small in comparison. Therefore, we need to be assured that the amount of work for determining the ordering does not dominate the computation, and hence seeking an “optimal” graph may turn out to be too costly an adventure. The art of finding effective orderings within a reasonable cost must rely on good heuristics and does not always have a solid theory to back it up.

Commonly used approaches for deriving effective ordering strategies utilize the matrix graph, discussed next.

✐

✐

✐

✐

✐

✐

✐

✐

5.7. Permutations and ordering strategies 125

Matrix graph

Suppose a matrix is structurally symmetric. This means that if ai j = 0, then also a j i = 0. For instance, the matrices A and B in Example 5.15 are structurally symmetric, whereas C is not. The graph of our matrix A comprises vertices and edges as follows: a row/column index is represented as a vertex, and if ai j �= 0, then there is an edge that connects vertex i to vertex j . Since we assume structural symmetry we need not worry about distinguishing between ai j and a j i .

The graphs of the matrices A and B of Example 5.15 are depicted in Figure 5.5. The graphs of any matrix A and a symmetric permutation P APT are identical in terms of the edges; the only difference is in the labeling of the vertices. In particular, vertex i is replaced by vertex pi , where p is the permutation array.

1

2

3

45 5

2

3

41

Figure 5.5. Graphs of the matrices A (left) and B (right) of Example 5.15.

In the process of Gaussian elimination, each stage can be described in terms of the matrix graph as follows: upon zeroing out the (i , j ) entry of the matrix with entry ( j , j ) being the current pivot, the vertices that are the neighbors of vertex j will cause the creation of an edge connecting them to vertex i , if such an edge does not already exist. For instance, in Example 5.15, when attempting to zero out entry (5,1) using entry (1,1) as pivot, in the graph of A(1) all vertices j connected to vertex 1 generate an edge (5, j ), and hence new edges (2,5), (3,5), and (4,5) appear. In contrast, for B no new edge is generated because there are no edges connected to vertex 1 other than vertex 5 itself.

The above underlines the importance of the degree of a vertex, i.e., the number of edges emanating from it. Generally speaking, we should postpone dealing with vertices of a high degree as much as possible. For the matrix A in Example 5.15 all the vertices except vertex 1 have degree 1, but vertex 1 has degree 4 and we start off the Gaussian elimination by eliminating it; this results in disastrous fill-in. On the other hand, for the matrix B all vertices except vertex 5 have degree 1, and vertex 5 is the last vertex we deal with. Until we hit vertex 5 there is no fill because the latter is the only vertex that is connected to the other vertices. When we deal with vertex 5 we identify vertices that should hypothetically be generated, but they are already in existence to begin with, so we end up with no fill whatsoever.

Two vertex labeling strategies

Ordering strategies in abundance have been derived throughout the years. As tempting as it is, we will not describe them in detail.

Many of these algorithms rely on heuristics related to the degrees of the vertices of the matrix graphs and use a good amount of auxiliary objects from graph theory. The reverse Cuthill Mc- Kee (RCM) algorithm is one that is effective at reducing bandwidth, and the minimum degree or approximate minimum degree (AMD) algorithm aims at minimizing the expected fill-in.

✐

✐

✐

✐

✐

✐

✐

✐

126 Chapter 5. Linear Systems: Direct Methods

Example 5.16. Let us illustrate how intriguing the nonzero structure of these matrices could be. The leftmost plot in Figure 5.6 shows the sparsity pattern of a certain well-known symmetric

positive definite matrix A. This matrix is derived in Example 7.1 on page 168, but all that concerns us here is its zero pattern.

The middle plot of Figure 5.6 shows the effect of RCM ordering. It can be believed from the picture that the goal was to minimize the bandwidth. The pattern is pretty elegant, at least in the eyes of those who appreciate the contemporary art of matrix sparsity plots (or, alternatively, wind surfing). The rightmost plot depicts the result of applying the AMD algorithm. Here it is harder to identify a unifying theme. But note that, as expected, rows with more elements (corresponding to vertices with a higher degree) appear closer to the bottom of the matrix, i.e., their index or label value is higher.

0 20 40 60

0

10

20

30

40

50

60

nz = 288 0 20 40 60

0

10

20

30

40

50

60

nz = 288 0 20 40 60

0

10

20

30

40

50

60

nz = 288

Figure 5.6. Sparsity pattern of a certain symmetric positive definite matrix A (left), its RCM ordering (middle), and approximate minimum degree ordering (right).

Notice the numbers nz at the bottom of each plot. This stands for “nonzeros” and states how many nonzero entries the matrix has. Since we only permute rows and columns, nz does not change here. But it will change following the procedure in Example 5.17.

The MATLAB code for generating the plots of Figure 5.6 (omitting the generation of A and a couple of output commands) is

p=symrcm(A); spy(A(p,p)); p=symamd(A); spy(A(p,p));

The actual work is encoded in the two built-in functions with the suggestive names (other than spy).

Example 5.17. Figure 5.7 shows the sparsity patterns of the Cholesky factors of the same matrices as in Example 5.16. There is fill in the original matrix as well as for the RCM matrix, but since the latter has a lower bandwidth the overall number of nonzeros is smaller, so we see an improvement. The fill-in is yet smaller for the AMD algorithm, which as expected (by its name, at least) produces the Cholesky factor with approximately the least number of nonzeros.

Try to figure out the corresponding MATLAB instructions for Figure 5.7 that are additional to those displayed in Example 5.16.

Specific exercises for this section: Exercises 20–21.

✐

✐

✐

✐

✐

✐

✐

✐

5.8. Estimating errors and the condition number 127

0 20 40 60

0

10

20

30

40

50

60

nz = 519 0 20 40 60

0

10

20

30

40

50

60

nz = 428 0 20 40 60

0

10

20

30

40

50

60

nz = 354

Figure 5.7. Sparsity pattern of the Cholesky factors of A (left), its RCM ordering (middle), and approximate minimum degree ordering (right).

5.8 Estimating errors and the condition number Thus far in this chapter we have discussed at length solution methods for solving

Ax = b, and Chapter 7 will add several more. In this section we take a short break from methods and ponder the question of how accurate we may expect our numerical solutions to be.

The error in the numerical solution

Suppose that, using some algorithm, we have computed an approximate solution x̂. With x denoting as usual the exact, unknown solution, we would like to be able to evaluate the absolute error ‖x− x̂‖, or the relative error

‖x− x̂‖ ‖x‖ ,

in some norm such as those defined in Section 4.2. Needless to say, an exact expression for these errors is not something we can get our hands on: if the error were known, then the error-free solution x would also be known! So, we seek an upper bound on the error and rely on quantities that we can directly calculate.

One such quantity is the residual

r̂ = b− Ax̂. In the ideal world we have r = b− Ax = 0, but in practice there will always be some nonzero

residual r̂ because of roundoff errors. When using iterative methods in Chapter 7 there is also convergence errors to reckon with, as in Chapter 3, but let’s not worry about that now. A stable Gaussian elimination algorithm variant will deliver a residual with a small norm. The question is, what can be concluded from this about the error in x̂?

Example 5.18. Let

A = 1.2969 .8648

.2161 .1441

 , b = .8642

.1440

 .

✐

✐

✐

✐

✐

✐

✐

✐

128 Chapter 5. Linear Systems: Direct Methods

Suppose that, using some algorithm, the approximate solution

x̂ =  .9911 −.4870

 was calculated. Then

r̂ = b− Ax̂ = −10−8

10−8

 , so ‖r̂‖∞ = 10−8. This is a sensibly small residual for many practical purposes.

However, the exact solution is

x =  2 −2

 , so ‖x− x̂‖∞ = 1.513. Thus, the error is of the same size as the solution itself, roughly 108 times that of the residual!

It is important to understand that we cannot expect our algorithms to deliver much more than a small residual. We return to this in Section 7.4.

Condition number and a relative error estimate

How does the residual r̂ generally relate to the error in x̂? We can write

r̂ = b− Ax̂ = Ax− Ax̂ = A(x− x̂), so

x− x̂ = A−1r̂. Let us choose a vector norm (say, the maximum norm), and its corresponding matrix norm.23 Then

‖x− x̂‖ = ‖A−1r̂‖ ≤ ‖A−1‖‖r̂‖. This gives a bound on the absolute error in x̂ in terms of ‖A−1‖. But usually the relative error is more meaningful. Since ‖b‖ ≤ ‖A‖‖x‖ implies 1‖x‖ ≤ ‖A‖‖b‖ , we have

‖x− x̂‖ ‖x‖ ≤ ‖A

−1‖‖r̂‖ ‖A‖‖b‖ .

We therefore define the condition number of the matrix A as

κ(A) = ‖A‖‖A−1‖ and write the bound obtained on the relative error as

‖x− x̂‖ ‖x‖ ≤ κ(A)

‖r̂‖ ‖b‖ . (5.1)

23Whereas the choice of which vector norm to use is less important for our current purposes, it does fix the choice of the matrix norm as the corresponding induced matrix norm.

✐

✐

✐

✐

✐

✐

✐

✐

5.8. Estimating errors and the condition number 129

In words, the relative error in the solution is bounded by the condition number of the matrix A times the relative error in the residual. Memorizing this estimate would be worth your while.

The notion of conditioning of a problem is fundamental and has been briefly discussed in Section 1.3. Here it can be expressed in terms of a quantity relating to the matrix A alone. Note that κ(A) does depend on the norm in which it is defined. To make sure it is clear what norm is used (when it is important to make the distinction) we will use notation such as κ2(A), κ∞(A), and so on. Regardless of norm, we have

1 = ‖I‖ = ‖A−1 A‖ ≤ κ(A), at one end of the scale, so a matrix is ideally conditioned if its condition number equals 1, and κ(A) = ∞ for a singular matrix at the other end of the scale. In fact, it can be shown that if A is nonsingular and E is the matrix of minimum �2-norm such that A+ E is singular, then

‖E‖2 ‖A‖2 =

1

κ2(A) .

So, the condition number measures how close to singularity in the relative sense a matrix is.

Example 5.19. Continuing with Example 5.18, we have

A−1 = 108  .1441 −.8648 −.2161 1.2969

 . This yields

‖A−1‖∞ = 1.513×108, hence κ∞(A) = 2.1617 ·1.513×108 ≈ 3.27×108. Regarding the actual computed x̂, note that 1.5132 <

3.27 .8642 . So, indeed

‖x− x̂‖∞ ‖x‖∞ < κ∞(A)

‖r̂‖∞ ‖b‖∞ .

However, the mere knowledge that the bound (5.1) holds would not make the computed solution necessarily satisfactory for all purposes.

Finally, if we modify a22 ← .8648·.21611.2969 , then obviously the matrix becomes singular, because the columns will be a scalar multiple of one another. So, the matrix A+ E is singular, where

E = 0 0

0 .8648·.21611.2969 − .1441

≈ 0 0

0 −7.7×10−9

 . Indeed, for this particular perturbation we have

‖E‖ ‖A‖ ≈

1

κ(A)

in both �2- and �∞-norms.

In MATLAB use cond(A,inf), or cond(A,2), etc., to find the condition number in the specified norm. But usually there is really no reason, other than for deductive purposes, why we

✐

✐

✐

✐

✐

✐

✐

✐

130 Chapter 5. Linear Systems: Direct Methods

would want to find κ(A) exactly. Typically, we want just a cheap estimate, especially when the condition number may be large. A cheap estimate of a lower bound for κ1(A) is obtained by the instruction condest(A). Assuming we have such an estimate, let us now tie this to an actual computation.

The error when using a direct method

There are two separate issues. The more practical one is the actual estimation, or bounding, of the relative error in the result, given a calculated residual r̂ = b− Ax̂. This has already been answered in Equation (5.1), regardless of how the residual r̂ arises.24

The other issue is tying the above to our direct methods for solving linear equations. Consider Gaussian elimination with pivoting, and assume no error other than the accumulation of roundoff errors. The accumulated error in the LU decomposition can be viewed as the exact decomposition of a slightly perturbed matrix à = A+ δA. Then the forward and backward substitutions add more small perturbations, each of the order of the rounding unit times n, so the entire algorithm may be considered as producing the exact solution of a perturbed problem

(A+ δA)x̂ = b+ δb. This is called backward error analysis.

A simple substitution shows that

r̂ = b− Ax̂ = (δA)x̂− δb. How large are δA and δb, and thus r̂? An LU decomposition procedure with pivoting can be shown to yield a perturbation bounded by

‖δA‖∞ ≤ ηφ(n)gn(A), where φ(n) is a low order polynomial in n (cubic at most) and η is the rounding unit. The extra perturbations which the forward and backward substitutions yield, in particular the bound on ‖δb‖∞, are significantly smaller. Thus, as long as the pivoting keeps gn(A) growing only moderately in n (like a low order polynomial, but not exponentially), and assuming n not to be gigantic, the overall perturbations δA and δb are not larger than a few orders of magnitude times η.

Plugging the expression for r̂ into Equation (5.1) we finally obtain

‖x− x̂‖ ‖x‖ ≤ κ(A)

‖δb‖+‖δA‖‖x̂‖ ‖b‖ .

This can be further beautified, provided ‖δA‖< 1/‖A−1‖, to read ‖x− x̂‖ ‖x‖ ≤

κ(A)

1−κ(A)(‖δA‖/‖A‖) (‖δb‖

‖b‖ + ‖δA‖ ‖A‖

) .

In summary, a stable algorithm is responsible for producing a small residual. This yields acceptably small error in the solution if the problem is well-conditioned, i.e., if κ(A) is not too large. Just what is “small” and what is “too large” may depend on the particular circumstances, but the qualitative concept is general.

24The residual could actually arise as a result of measurement errors in data, for instance.

✐

✐

✐

✐

✐

✐

✐

✐

5.8. Estimating errors and the condition number 131

Example 5.20. Let us return to Example 5.13. If you invoke its script with n = 500 several times, you’ll get different results, as befits an unseeded random number generator. But a typical run (unless you are incredibly lucky) would yield errors that are well above the rounding unit.

Using the Cholesky decomposition, in one instance tried we had a relative error of 2.6×10−11 and in another the relative error was 8.0× 10−10. The corresponding errors using ainvb (which required two and three theoretically unnecessary row permutations) were fairly close and slightly worse. Recall that the default rounding unit is η ≈ 10−16.

In search for the lost digits of accuracy we computed the condition numbers κ2 (A) of the con- structed random matrices A and these were, respectively, 8.8×105 and 3.9×108. These condition numbers, then, account for the loss of accuracy in the computed solutions.

Indeed, the relative residuals were 8.1×10−16 and 9.2×10−16, respectively, which shows not only that both our variants of Gaussian elimination performed very well but also that the bound (5.1) is respected and indicative of actual error behavior.

More on the condition number

The condition number is of fundamental importance in numerical computing. Note that we cannot use the determinant of A to indicate closeness to singularity, as the following example demonstrates.

Example 5.21. Consider A = 0.1I , explicitly written as

A =

 0.1

0.1 . . .

0.1

 .

This matrix is perfectly conditioned: κ(A) = 1. Yet, det(A) = 0.1n. For n = 16 the determinant is already of the order of our standard rounding unit.

The opposite can also hold, namely, a matrix A can be nearly singular and yet its determinant is not small. Such is the case for

A =

 1 −1 · · · −1

. . . . . .

... . . . −1

1

 ,

whose inverse has the element 2n−2 in the upper right corner. The condition number grows expo- nentially in n while det(A) = 1.

Conditioning of orthogonal matrices

An important class of matrices that are perfectly conditioned in the induced �2-norm are orthogonal matrices, introduced in Section 4.3. Recall that an n ×n, real matrix Q is orthogonal if QT = Q−1. (For instance, a permutation matrix is orthogonal.) Then for any vector x satisfying ‖x‖2 = 1 we have

‖Qx‖22 = xT QT Qx = xT x = 1,

✐

✐

✐

✐

✐

✐

✐

✐

132 Chapter 5. Linear Systems: Direct Methods

so ‖Q‖2 = 1. Likewise, ‖Q−1‖2 = ‖QT ‖2 = 1, hence κ2(Q) = 1. Orthogonal transformations are therefore particularly stable and thus a long-time favorite in numerical linear algebra.

Note: Orthogonal matrices are ideally conditioned. On the other hand, there is nothing in principle to prevent a symmetric positive definite matrix from having a large condition number.

Conditioning of symmetric positive definite matrices

Let us consider another important class of matrices, the symmetric positive definite ones. Recall from Section 4.3 that the eigenvalues of such matrices are all real and positive, so let’s write them as

λ1 ≥ λ2 ≥ ·· · ≥ λn > 0.

Thus, ‖A‖2 = √ ρ(AT A) =

√ λ21 = λ1. Also, the inverse of A is symmetric positive definite with

eigenvalues

0 < 1

λ1 ≤ 1

λ2 ≤ ·· · ≤ 1

λn .

Thus, ‖A−1‖2 = 1λn , and this yields the compact expression

κ2(A) = λ1 λn

.

Example 5.22. Recall from Example 4.9 that the “unit circle”, i.e., all the vectors with unit norm, actually corresponds to our usual notion of a circle in R2 when using the �2-norm. In Figure 5.8 we plot the mapping of this unit circle by the symmetric positive definite matrix

A =  1 .75

.75 1

 . Producing such a plot is very simple in MATLAB:

d = .75; A = [1,d;d,1]; t = 0:.01:10; m = length(t); x(1,1:m) = sin(t); x(2,1:m) = cos(t); y = A*x; plot (y(1,:),y(2,:),’b’, x(1,:),x(2,:),’r:’) xlabel(’x_1’) ylabel(’x_2’)

The eigenvalues of A are .25 and 1.75, so κ(A) = 1.750.25 = 7. These eigenvalues are the lengths of the semi-axes of the ellipse in Figure 5.8. The larger the difference between the eigenvalues (i.e., the larger the condition number), the more “long and skinny” the unit circle becomes under this transformation.

Specific exercises for this section: Exercises 22–24.

✐

✐

✐

✐

✐

✐

✐

✐

5.9. Exercises 133

−1.5 −1 −0.5 0 0.5 1 1.5

−1

−0.5

0

0.5

1

x 1

x 2

Figure 5.8. Stretching the unit circle by a symmetric positive definite transformation.

5.9 Exercises 0. Review questions

(a) What is the overall cost in flops of applying a forward or backward solve?

(b) What is the overall cost in flops of decomposing a matrix using LU?

(c) During the course of Gaussian elimination without pivoting a zero pivot has been en- countered. Is the matrix singular?

(d) State three disadvantages of computing the inverse of a matrix to solve a linear system rather than using the LU decomposition approach.

(e) The complete pivoting strategy is numerically stable, whereas partial pivoting is not always stable. Why is the latter approach preferred in practice in spite of this?

(f) Show that for any symmetric positive definite matrix A there is a nonsingular matrix H such that A = H T H .

(g) Give three examples of data structures for storing a sparse matrix. Why aren’t they used for full (dense) matrices, too?

(h) How are the storage requirements and overall computational work different for banded matrices compared with general dense matrices?

(i) State two possible strategies for reordering the equations and unknowns in a way that improves the efficiency of Gaussian elimination for sparse matrices.

(j) What is the fundamental difference between error and residual in terms of computabil- ity?

(k) Suppose we compute an approximate solution x̃ for Ax = b and get r = b− Ax̃ whose norm is very small. Can we conclude in this case that the error x− x̃ must also be small?

(l) The condition number is an important concept, but it is rarely computed exactly. Why?

✐

✐

✐

✐

✐

✐

✐

✐

134 Chapter 5. Linear Systems: Direct Methods

1. Consider the problem

x1 − x2 +3x3 = 2, x1 + x2 = 4,

3x1 −2x2 + x3 = 1. Carry out Gaussian elimination in its simplest form for this problem. What is the resulting upper triangular matrix?

Proceed to find the solution by backward substitution.

2. The Gauss–Jordan method used to solve the prototype linear system can be described as follows. Augment A by the right-hand-side vector b and proceed as in Gaussian elimination, except use the pivot element a(k−1)kk to eliminate not only a

(k−1) ik for i = k+1, . . . ,n but also the

elements a(k−1)ik for i = 1, . . . ,k − 1, i.e., all elements in the kth column other than the pivot. Upon reducing (A|b) into

a(n−1)11 0 · · · 0 b(n−1)1 0 a(n−1)22

. . . ... b(n−1)2

... . . .

. . . 0 ...

0 · · · 0 a(n−1)nn b(n−1)n

 ,

the solution is obtained by setting

xk = b (n−1) k

a(n−1)kk , k = 1, . . . ,n.

This procedure circumvents the backward substitution part necessary for the Gaussian elimi- nation algorithm.

(a) Write a pseudocode for this Gauss–Jordan procedure using, e.g., the same format as for the one appearing in Section 5.2 for Gaussian elimination. You may assume that no pivoting (i.e., no row interchanging) is required.

(b) Show that the Gauss–Jordan method requires n3 +O(n2) floating point operations for one right-hand-side vector b—roughly 50% more than what’s needed for Gaussian elim- ination.

3. Let A and T be two nonsingular, n ×n real matrices. Furthermore, suppose we are given two matrices L and U such that L is unit lower triangular, U is upper triangular, and

T A = LU . Write an algorithm that will solve the problem

Ax = b for any given vector b in O(n2) complexity. First explain briefly yet clearly why your algo- rithm requires only O(n2) flops (you may assume without proof that solving an upper trian- gular or a lower triangular system requires only O(n2) flops). Then specify your algorithm in detail (including the details for lower and upper triangular systems) using pseudocode or a MATLAB script.

✐

✐

✐

✐

✐

✐

✐

✐

5.9. Exercises 135

4. The classical way to invert a matrix A in a basic linear algebra course augments A by the n×n identity matrix I and applies the Gauss–Jordan algorithm of Exercise 2 to this augmented matrix (including the solution part, i.e., the division by the pivots a(n−1)kk ). Then A−1 shows up where I initially was.

How many floating point operations are required for this method? Compare this to the oper- ation count of 83 n

3 +O(n2) required for the same task using LU decomposition (see Exam- ple 5.5).

5. Let

A =

 5 6 7 8

0 4 3 2

0 0 0 1

0 0 −1 −2

 .

(a) The matrix A can be decomposed using partial pivoting as

P A = LU , where U is upper triangular, L is unit lower triangular, and P is a permutation matrix. Find the 4×4 matrices U , L, and P .

(b) Given the right-hand-side vector b = (26,9,1,−3)T , find x that satisfies Ax = b. (Show your method: do not just guess.)

6. Let B be any real, nonsingular n × n matrix, where n is even, and set A = B − BT . Show that A does not admit an LU decomposition (i.e., some pivoting must be applied, even if A is nonsingular).

7. Let

A =

 0 1 0 0

1 0 3 0

−.5 0 −.2 1 −.5 −.3 1 0

 .

Decompose A using partial pivoting as

P A = LU , where U is upper triangular, L is unit lower triangular and P is a permutation matrix. Record all the elementary matrices, P (i), and M (i) along the way. (You can use MATLAB to help you with these small, annoying calculations.)

Show that L relates to [M (3)]−1[M (2)]−1[M (1)]−1 in the way explained in Section 5.3, and find M̃ (1), M̃ (2), and M̃ (3).

8. Symmetric indefinite matrices require pivoting, and it can be shown that a numerically stable Gaussian elimination process has the form

P APT = L DLT ,

✐

✐

✐

✐

✐

✐

✐

✐

136 Chapter 5. Linear Systems: Direct Methods

where P is a permutation matrix and D is a block diagonal matrix of a certain form. The MATLAB command ldl produces such a decomposition.

For simplicity, suppose that D is diagonal (rather than block diagonal), and suppose we are given a sequence of elementary permutation matrices interleaved with unit lower triangular matrices. Show that, similarly to GEPP in the general case, we can collect all elementary per- mutation matrices together into a single permutation matrix, bringing about the above defined L DLT decomposition.

9. Can you find an example other than the one given on page 109 where scaled partial pivoting is unstable while complete pivoting does well?

[If you have difficulties finding such a example, then the one in the text may look contrived.]

10. The MATLAB code given in Section 5.4 for solving linear systems of equations, using LU decomposition in outer form with partial pivoting, works well if the matrix A is nonsingular to a working precision. But if A is singular, then the exit is not graceful.

Please fix this by modifying the functionsainvb and plu to include checks so that ainvb+plu will always return as follows:

(i) In case of a nonsingular system, return silently with the solution x.

(ii) In case of a singular system, display a message regarding the singularity and return with x assigned the value NaN.

Assume that MATLAB will complain if a division by a number smaller than eps= 2.22× 10−16 is attempted. (You want to avoid precisely this sort of complaint.)

11. Apply the modified solver obtained in Exercise 10 to the following systems. In each case, check the difference between the computed solution x and the result of MATLAB’s built-in solver A \ b.

(a)

x1 + x2 + x4 = 2, 2x1 + x2 − x3 + x4 = 1,

4x1 − x2 −2x3 +2x4 = 0, 3x1 − x2 − x3 + x4 = −3.

(b) Same as the previous system, but with the coefficient of x4 in the last equation set to a4,4 = 2.

(c)

x1 + x2 + x3 = 1, x1 + (1+10−15)x2 +2x3 = 2,

x1 +2x2 +2x3 = 1. (d) Same as the previous system, but with the second equation multiplied by 1020.

(e)

A =

 1 2 3

1 2 3

1 2 3

 , b = 

1

2

1

 .

✐

✐

✐

✐

✐

✐

✐

✐

5.9. Exercises 137

(f)

A =

 1 2 3

0 0 0

3 2 1

 with the same right-hand side as before.

12. The Cholesky algorithm given on page 116 has all those wretched loops as in the Gaussian elimination algorithm in its simplest form. In view of Section 5.4 and the program ainvb we should be able to achieve also the Cholesky decomposition effect more efficiently.

Write a code implementing the Cholesky decomposition with only one loop (on k), utilizing outer products.

13. The script in Example 5.13 will give different answers each time you run it, because the random number generator starts at different spots.

Please fix this, making the program having the option to run on the same input when reinvoked. (Check help randn to see how to do this.)

Run the program for n = 200 and n = 500. What are your observations? 14. This exercise is for lovers of complex arithmetic.

Denote by xH the conjugate transpose of a given complex-valued vector x, and likewise for a matrix. The �2-norm is naturally extended to complex-valued vectors by

‖x‖22 = xH x = n∑

i=1 |xi |2,

which in particular is real, possibly unlike xT x.

A complex-valued n × n matrix A is called Hermitian if AH = A. Further, A is positive definite if

xH Ax > 0

for all complex-valued vectors x �= 0.

(a) Show that the 1×1 matrix A = ı is symmetric but not Hermitian, whereas B = ( 2 ı−ı 2) is Hermitian positive definite but not symmetric. Find the eigenvalues of A and of B . [In general,