Solution Manual needed

alonlywolf

optimal_control_systemssubbaram_naidu.pdf

Home >Engineering homework help >Electrical Engineering homework help >Solution Manual needed

OPTIMAL CONTROL SYSTEMS

~,.~t~.~ -: ".. .u&:iuut ~

....aIMlJ f~t jS ~ J

'1 6 .3 z. : ~ OJ"':'

lrAr If I 1 a : ~ i!J1i

Electrical Engineering Textbook Series Richard C. Dorf, Series Editor

University of California, Davis

Forthcoming and Published Titles

Applied Vector Analysis Matiur Rahman and Isaac Mulolani

Continuous Signals and Systems with MATLAB Taan EIAli and Mohammad A. Karim

Discrete Signals and Systems with MATLAB Taan EIAIi

Electromagnetics Edward J. Rothwell and Michael J. Cloud

Optimal Control Systems Desineni Subbaram Naidu

OPTIMAL CONTROL SYSTEMS

Desineni Subbaram Naidu Idaho State Universitv . Pocatello. Idaho. USA

o CRC PRESS

Boca Raton London New York Washington, D.C.

Cover photo: Terminal phase (using fuel-optimal control) of the lunar landing of the Apollo 11 mission. Courtesy of NASA.

TJ "l13 N1. b'~ <'l ~ot

Library of Congress Cataloging-in-Publication Data

Naidu, D. s. (Desineni S.), 1940- Optimal control systems I by Desineni Subbaram N aidu.

p. cm.- (Electrical engineering textbook series) Includes bibliographical references and index. ISBN 0-8493-0892-5 (alk. paper) 1. Automatic control. 2. Control theory. 3. Mathematical optimization. I. Title II.

Series.

2002067415

This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use.

Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior permission in writing from the publisher.

The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying.

Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe.

Visit the CRC Press Web site at www.crcpress.com

No claim to original u.S. Government works International Standard Book Number 0-8493-0892-5

Library of Congress Card Number 2002067415 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0

Printed on acid-free paper

"Because the shape of the whole universe is most per- fect and, in fact, designed by the wisest Creator, nothing in all of the world will occur in which no maximum or minimum rule is somehow shining forth. "

Leohard Euler, 1144

Dedication

My deceased parents who shaped my life

Desineni Rama Naidu

Desineni Subbamma

and

My teacher who shaped my education

Buggapati A udi Chetty

vii

Preface

Many systems, physical, chemical, and economical, can be modeled by mathematical relations, such as deterministic and/or stochastic dif- ferential and/or difference equations. These systems then change with time or any other independent variable according to the dynamical re- lations. It is possible to steer these systems from one state to another state by the application of some type of external inputs or controls. If this can be done at all, there may be different ways of doing the same task. If there are different ways of doing the same task, then there may be one way of doing it in the "best" way. This best way can be minimum time to go from one state to another state, or maximum thrust developed by a rocket engine. The input given to the system corresponding to this best situation is called "optimal" control. The measure of "best" way or performance is called "performance index" or "cost function." Thus, we have an "optimal control system," when a system is controlled in an optimum way satisfying a given performance index. The theory of optimal control systems has enjoyed a flourishing period for nearly two decades after the dawn of the so-called "modern" control theory around the 1960s. The interest in theoretical and prac- tical aspects of the subject has sustained due to its applications to such diverse fields as electrical power, aerospace, chemical plants, economics, medicine, biology, and ecology.

Aim and Scope

In this book we are concerned with essentially the control of physical systems which are "dynamic" and hence described by ordinary differ- ential or difference equations in contrast to "static" systems, which are characterized by algebraic equations. Further, our focus is on "deter- ministic" systems only.

The development of optimal control theory in the sixties revolved around the "maximum principle" proposed by the Soviet mathemati- cian L. S. Pontryagin and his colleagues whose work was published in English in 1962. Further contributions are due to R. E. Kalman of the United States. Since then, many excellent books on optimal control theory of varying levels of sophistication have been published.

This book is written keeping the "student in mind" and intended to provide the student a simplified treatment of the subject, with an

viii

appropriate dose of mathematics. Another feature of this book is to assemble all the topics which can be covered in a one-semester class. A special feature of this book is the presentation of the procedures in the form of a summary table designed in terms of statement of the prob- lem and a step-by-step solution of the problem. Further, MATLAB©

and SIMULINK© 1 , including Control System and Symbolic Math Toolboxes, have been incorporated into the book. The book is ideally suited for a one-semester, second level, graduate course in control sys- tems and optimization.

Background and Audience

This is a second level graduate text book and as such the background material required for using this book is a first course on control sys- tems, state space analysis, or linear systems theory. It is suggested that the student review the material in Appendices A and B given at the end of the book. This book is aimed at graduate students in Electrical, Mechanical, Chemical, and Aerospace Engineering and Applied Math- ematics. It can also be used by professional scientists and engineers working in a variety of industries and research organizations.

Acknowledgments

This book has grown out of my lecture notes prepared over many years of teaching at the Indian Institute of Technology (IIT), Kharagpur, and Idaho State University (ISU), Pocatello, Idaho. As such, I am indebted to many of my teachers and students. In recent years at ISU, there are many people whom I would like to thank for their encouragement and cooperation. First of all, I would like to thank the late Dean Hary Charyulu for his encouragement to graduate work and research which kept me "live" in the area optimal control. Also, I would like to mention a special person, Kevin Moore, whose encouragement and cooperation made my stay at ISU a very pleasant and scholarly productive one for many years during 1990-98. During the last few years, Dean Kunze and Associate Dean Stuffie have been of great help in providing the right atmosphere for teaching and research work.

IMATLAB and SIMULINK are registered trademarks of The Mathworks, Inc., Natick, MA, USA.

Next, my students over the years were my best critics in providing many helpful suggestions. Among the many, special mention must be made about Martin Murillo, Yoshiko Imura, and Keith Fisher who made several suggestions to my manuscript. In particular, Craig Rieger ( of Idaho National Engineering and Environmental Laboratory (INEEL)) deserves special mention for having infinite patience in writ- ing and testing programs in MATLAB© to obtain analytical solutions to matrix Riccati differential and difference equations.

The camera-ready copy of this book was prepared by the author using H\'IEX of the PCTEX322 Version 4.0. The figures were drawn using CoreiDRAW3 and exported into H\'IEX document.

Several people at the publishing company CRC Press deserve men- tion. Among them, special mention must be made about Nora Konopka, Acquisition Editor, Electrical Engineering for her interest, understand- ing and patience with me to see this book to completion. Also, thanks are due to Michael Buso, Michelle Reyes, Helena Redshaw, and Judith Simon Kamin. I would like to make a special mention of Sean Davey who helped me in many issues regarding H\'IEX. Any corrections and suggestions are welcome via email to naiduds@isu. edu

Finally, it is my pleasant duty to thank my wife, Sita and my daugh- ters, Radhika and Kiranmai who have been a great source of encour- agement and cooperation throughout my academic life.

Desineni Subbaram Naidu Pocatello, Idaho

June 2002

2:rg..'lEX is a registered trademark of Personal 'lEX, Inc., Mill Valley, CA. 3CorelDRAW is a registered trademark of Corel Corporation or Corel Corporation Limited.

ACKNOWLEDGMENTS

The permissions given by

1. Prentice Hall for D. E. Kirk, Optimal Control Theory: An Intro- duction, Prentice Hall, Englewood Cliffs, NJ, 1970,

2. John Wiley for F. L. Lewis, Optimal Control, John Wiley & Sons, Inc., New York, NY, 1986,

3. McGraw-Hill for M. Athans and P. L. Falb, Optimal Control: An Introduction to the Theory and Its Applications, McGraw-Hill Book Company, New York, NY, 1966, and

4. Springer-Verlag for H. H. Goldstine, A History of the Calculus of Variations, Springer-Verlag, New York, NY, 1980,

are hereby acknowledged.

AUTHOR'S BIOGRAPHY

Desineni "Subbaram" Naidu received his B.E. degree in Electrical Engineer- ing from Sri Venkateswara University, Tirupati, India, and M.Tech. and Ph.D. degrees in Control Systems Engineering from the Indian Institute of Technol- ogy (lIT), Kharagpur, India. He held various positions with the Department of Electrical Engineering at lIT. Dr. Naidu was a recipient of a Senior National Research Council (NRC) Associateship of the National Academy of Sciences, Washington, DC, tenable at NASA Langley Research Center, Hampton, Virginia, during 1985-87 and at the U. S. Air Force Research Laboratory (AFRL) at Wright-Patterson Air Force Base (WPAFB), Ohio, during 1998- 99. During 1987-90, he was an adjunct faculty member in the Department of Electrical and Computer Engineering at Old Dominion University, Norfolk, Virginia. Since August 1990, Dr. Naidu has been a professor at Idaho State University. At present he is Director of the Measurement and Control Engi- neering Research Center; Coordinator, Electrical Engineering program; and Associate Dean of Graduate Studies in the College of Engineering, Idaho State University, Pocatello, Idaho.

Dr. Naidu has over 150 publications including a research monograph, Sin- gular Perturbation Analysis of Discrete Control Systems, Lecture Notes in Mathematics, 1985; a book, Singular Perturbation Methodology in Control Systems, lEE Control Engineering Series, 1988; and a research monograph entitled, Aeroassisted Orbital Transfer: Guidance and Control Strategies, Lec- ture Notes in Control and Information Sciences, 1994.

Dr. Naidu is (or has been) a member of the Editorial Boards of the IEEE Transaction on Automatic Control, (1993-99), the International Journal of Robust and Nonlinear Control, (1996-present), the International Journal of Control-Theory and Advanced Technology (C-TAT), (1992-1996), and a mem- ber of the Editorial Advisory Board of Mechatronics: The Science of Intelli- gent Machines, an International Journal, (1992-present).

Professor Naidu is an elected Fellow of The Institute of Electrical and Elec- tronics Engineers (IEEE), a Fellow of World Innovation Foundation (WIF), an Associate Fellow of the American Institute of Aeronautics and Astronautics (AIAA) and a member of several other organizations such as SIAM, ASEE, etc. Dr. Naidu was a recipient of the Idaho State University Outstanding Re- searcher Award for 1993-94 and 1994-95 and the Distinguished Researcher Award for 1994-95. Professor Naidu's biography is listed (multiple years) in Who's Who among America's Teachers, the Silver Anniversary 25th Edition of Who's Who in the West, Who's Who in Technology, and The International Directory of Distinguished Leadership.

Contents 1 Introduction 1

1.1 Classical and Modern Control . . . . . . . . . . . . . . . . . . . . . 1 1.2 Optimization................................. 4 1.3 Optimal Control .............................. 6

1.3.1 Plant ................................. 6 1.3.2 Performance Index ........................ 6 1.3.3 Constraints ............................. 9 1.3.4 Formal Statement of Optimal Control System .... 9

1.4 Historical Tour .............................. 11 1.4.1 Calculus of Variations .................... 11 1.4.2 Optimal Control Theory .................. 13

1.5 About This Book ............................. 15 1.6 Chapter Overview ............................ 16 1.7 Problems ................................... 17

2 Calculus of Variations and Optimal Control 19 2.1 Basic Concepts .............................. 19

2.1.1 Function and Functional .................. 19 2.1.2 Increment ............................. 20 2.1.3 Differential and Variation . . . . . . . . . . . . . . . . . . 22

2.2 Optimum of a Function and a Functional ............ 25 2.3 The Basic Variational Problem ................... 27

2.3.1 Fixed-End Time and Fixed-End State System ... 27 2.3.2 Discussion on Euler-Lagrange Equation ........ 33 2.3.3 Different Cases for Euler-Lagrange Equation .... 35

2.4 The Second Variation . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.5 Extrema of Functions with Conditions .............. 41

2.5.1 Direct Method .......................... 43 2.5.2 Lagrange Multiplier Method ................ 45

2.6 Extrema of Functionals with Conditions ............ 48 2.7 Variational Approach to Optimal Control Systems . . . . . 57

xiii

XIV

2.7.1 Terminal Cost Problem ................... 57 2.7.2 Different Types of Systems ................. 65 2.7.3 Sufficient Condition ...................... 67 2.7.4 Summary of Pontryagin Procedure ........... 68

2.8 Summary of Variational Approach ................. 84 2.8.1 Stage I: Optimization of a Functional . . . . . . . . . 85 2.8.2 Stage II: Optimization of a Functional with

Condition ............................. 86 2.8.3 Stage III: Optimal Control System with

Lagrangian Formalism .................... 87 2.8.4 Stage IV: Optimal Control System with

Hamiltonian Formalism: Pontryagin Principle ... 88 2.8.5 Salient Features ......................... 91

2.9 Problems ................................... 96

3 Linear Quadratic Optimal Control Systems I 101 3.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . .. 101 3.2 Finite-Time Linear Quadratic Regulator ........... 104

3.2.1 Symmetric Property of the Riccati Coefficient Matrix .............................. 109

3.2.2 Optimal Control ....................... 110 3.2.3 Optimal Performance Index . . . . . . . . . . . . . .. 110 3.2.4 Finite-Time Linear Quadratic Regulator:

Time-Varying Case: Summary ............. 112 3.2.5 Salient Features. . . . . . . . . . . . . . . . . . . . . . .. 114 3.2.6 LQR System for General Performance Index ... 118

3.3 Analytical Solution to the Matrix Differential Riccati Equation .................... 119 3.3.1 MATLAB© Implementation of Analytical

Solution to Matrix DRE. . . . . . . . . . . . . . . . .. 122 3.4 Infinite-Time LQR System I . . . . . . . . . . . . . . . . . . .. 125

3.4.1 Infinite-Time Linear Quadratic Regulator: Time-Varying Case: Summary ............. 128

3.5 Infinite-Time LQR System II ................... 129 3.5.1 Meaningful Interpretation of Riccati Coefficient . 132 3.5.2 Analytical Solution of the Algebraic

Riccati Equation . . . . . . . . . . . . . . . . . . . . . .. 133 3.5.3 Infinite-Interval Regulator System:

Time-Invariant Case: Summary ............. 134 3.5.4 Stability Issues of Time-Invariant Regulator. . .. 139

3.5.5 Equivalence of Open-Loop and Closed-Loop Optimal Controls ....................... 141

3.6 Notes and Discussion ......................... 144 3.7 Problems .................................. 147

4 Linear Quadratic Optimal Control Systems II 151 4.1 Linear Quadratic Tracking System: Finite-Time Case 152

4.1.1 Linear Quadratic Tracking System: Summary 157 4.1.2 Salient Features of Tracking System . . . . . . . .. 158

4.2 LQT System: Infinite-Time Case ................. 166 4.3 Fixed-End-Point Regulator System ............... 169 4.4 LQR with a Specified Degree of Stability . . . . . . . . . .. 175

4.4.1 Regulator System with Prescribed Degree of Stability: Summary . . . . . . . . . . . . . . . . . . . .. 177

4.5 Frequency-Domain Interpretation ................ 179 4.5.1 Gain Margin and Phase Margin ............ 181

4.6 Problems.................................. 188

5 Discrete-Time Optimal Control Systems 191 5.1 Variational Calculus for Discrete-Time

Systems .................................. 191 5.1.1 Extremization of a Functional .............. 192 5.1.2 Functional with Terminal Cost ............. 197

5.2 Discrete-Time Optimal Control Systems ........... 199 5.2.1 Fixed-Final State and Open-Loop Optimal

Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 203 5.2.2 Free-Final State and Open-Loop Optimal Control 207

5.3 Discrete-Time Linear State Regulator System ................................... 207 5.3.1 Closed-Loop Optimal Control: Matrix Difference

Riccati Equation . . . . . . . . . . . . . . . . . . . . . . . 209 5.3.2 Optimal Cost Function .................. 213

5.4 Steady-State Regulator System .................. 219 5.4.1 Analytical Solution to the Riccati Equation .... 225

5.5 Discrete-Time Linear Quadratic Tracking System . . . .. 232 5.6 Frequency-Domain Interpretation ................ 239 5.7 Problems .................................. 245

XVI

6 Pontryagin Minimum Principle 249 6.1 Constrained System .......................... 249 6.2 Pontryagin Minimum Principle . . . . . . . . . . . . . . . . .. 252

6.2.1 Summary of Pontryagin Principle . . . . . . . . . . . 256 6.2.2 Additional Necessary Conditions ............ 259

6.3 Dynamic Programming. . . . . . . . . . . . . . . . . . . . . . . . 261 6.3.1 Principle of Optimality .................. 261 6.3.2 Optimal Control Using Dynamic Programming . 266 6.3.3 Optimal Control of Discrete-Time Systems .... 272 6.3.4 Optimal Control of Continuous-Time Systems .. 275

6.4 The Hamilton-Jacobi-Bellman Equation ............ 277 6.5 LQR System Using H-J-B Equation ............. " 283 6.6 Notes and Discussion ......................... 288

7 Constrained Optimal Control Systems 293 7.1 Constrained Optimal Control . . . . . . . . . . . . . . . . . .. 293

7.1.1 Time-Optimal Control of LTI System ........ 295 7.1.2 Problem Formulation and Statement . . . . . . . .. 295 7.1.3 Solution of the TOC System ............... 296 7.1.4 Structure of Time-Optimal Control System .... 303

7.2 TOC of a Double Integral System ................ 305 7.2.1 Problem Formulation and Statement. . . . . . . .. 306 7.2.2 Problem Solution ....................... 307 7.2.3 Engineering Implementation of Control Law ... 314 7.2.4 SIMULINK© Implementation of Control Law .. 315

7.3 Fuel-Optimal Control Systems ................... 315 7.3.1 Fuel-Optimal Control of a Double Integral System 316 7.3.2 Problem Formulation and Statement ......... 319 7.3.3 Problem Solution. . . . . . . . . . . . . . . . . . . . . .. 319

7.4 Minimum-Fuel System: LTI System ............... 328 7.4.1 Problem Statement ..................... 328 7.4.2 Problem Solution. . . . . . . . . . . . . . . . . . . . . .. 329 7.4.3 SIMULINK© Implementation of Control Law. . 333

7.5 Energy-Optimal Control Systems ................ 335 7.5.1 Problem Formulation and Statement ......... 335 7.5.2 Problem Solution. . . . . . . . . . . . . . . . . . . . . .. 339

7.6 Optimal Control Systems with State Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 351 7.6.1 Penalty Function Method. . . . . . . . . . . . . . . .. 352 7.6.2 Slack Variable Method ................... 358

xvii

7.7 Problems .................................. 361

Appeddix A: Vectors and Matrices 365 A.1 Vectors ................................... 365 A.2 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 A.3 Quadratic Forms and Definiteness ................ 376

Appendix B: State Space Analysis 379 B.1 State Space Form for Continuous-Time Systems ...... 379 B.2 Linear Matrix Equations. . . . . . . . . . . . . . . . . . . . . .. 381 B.3 State Space Form for Discrete-Time Systems . . . . . . .. 381 B.4 Controllability and Observability ................. 383 B.5 Stabilizability, Reachability and Detectability ........ 383

Appendix C: MATLAB Files 385 C.1 MATLAB© for Matrix Differential Riccati Equation .. 385

C.l.1 MATLAB File lqrnss.m .................. 386 C.l.2 MATLAB File lqrnssf.m .................. 393

C.2 MATLAB© for Continuous-Time Tracking System ... 394 C.2.1 MATLAB File for Example 4.1(example4_l.m) . 394 C.2.2 MATLAB File for Example 4.1(example4_1p.m). 397 C.2.3 MATLAB File for Example 4.1(example4_1g.m). 397 C.2.4 MAT LAB File for Example 4.1(example4_1x.m). 397 C.2.5 MATLAB File for Example 4.2(example4_l.m) . 398 C.2.6 MATLAB File for Example 4.2( example4_2p.m). 400 C.2.7 MATLAB File for Example 4.2(example4_2g.m). 400 C.2.8 MATLAB File for Example 4.2( example4_2x.m). 401

C.3 MATLAB© for Matrix Difference Riccati Equation ... 401 C.3.1 MAT LAB File lqrdnss.m . . . . . . . . . . . . . . . .. 401

C.4 MATLAB© for Discrete-Time Tracking System ...... 409

References . ..................................... 415

Index . ......................................... 425

List of Figures 1.1 Classical Control Configuration . . . . . . . . . . . . . . . . . . . . 1 1.2 Modern Control Configuration .................... 3 1.3 Components of a Modern Control System ............ 4 1.4 Overview of Optimization . . . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Optimal Control Problem ....................... 10

2.1 Increment ~f, Differential df, and Derivative j of a Function f ( t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.2 Increment ~J and the First Variation 8J of the Func- tional J .................................... 24

2.3 ( a) Minimum and (b) Maximum of a Function f ( t) . . . . . 26 2.4 Fixed-End Time and Fixed-End State System ........ 29 2.5 A Nonzero g(t) and an Arbitrary 8x(t) ............. 32 2.6 Arc Length ................................. 37 2.7 Free-Final Time and Free-Final State System ......... 59 2.8 Final-Point Condition with a Moving Boundary B(t) .... 63 2.9 Different Types of Systems: (a) Fixed-Final Time and

Fixed-Final State System, (b) Free-Final Time and Fixed- Final State System, (c) Fixed-Final Time and Free-Final State System, (d) Free-Final Time and Free-Final State System .................................... 66

2.10 Optimal Controller for Example 2.12 ............... 72 2.11 Optimal Control and States for Example 2.12 ......... 74 2.12 Optimal Control and States for Example 2.13 ......... 77 2.13 Optimal Control and States for Example 2.14 ......... 81 2.14 Optimal Control and States for Example 2.15 ......... 84 2.15 Open-Loop Optimal Control ..................... 94 2.16 Closed-Loop Optimal Control .................... 95

3.1 State and Costate System ...................... 107 3.2 Closed-Loop Optimal Control Implementation ....... 117

X'lX

3.3 Riccati Coefficients for Example 3.1 . . . . . . . . . . . . . .. 125 3.4 Closed-Loop Optimal Control System for Example 3.1 126 3.5 Optimal States for Example 3.1. . . . . . . . . . . . . . . . .. 127 3.6 Optimal Control for Example 3.1 ................ 127 3.7 Interpretation of the Constant Matrix P ........... 133 3.8 Implementation of the Closed-Loop Optimal Control:

Infinite Final Time. . . . . . . . . . . . . . . . . . . . . . . . . .. 135 3.9 Closed-Loop Optimal Control System . . . . . . . . . . . .. 138 3.10 Optimal States for Example 3.2. . . . . . . . . . . . . . . . .. 140 3.11 Optimal Control for Example 3.2 ................ 141 3.12 (a) Open-Loop Optimal Controller (OLOC) and

(b) Closed-Loop Optimal Controller (CLOC) ........ 145

4.1 Implementation of the Optimal Tracking System ..... 157 4.2 Riccati Coefficients for Example 4.1 ............... 163 4.3 Coefficients 91(t) and 92(t) for Example 4.1 ......... 164 4.4 Optimal States for Example 4.1 .................. 164 4.5 Optimal Control for Example 4.1 ................ 165 4.6 Riccati Coefficients for Example 4.2 ............... 167 4.7 Coefficients 91(t) and 92(t) for Example 4.2 ......... 168 4.8 Optimal Control and States for Example 4.2 ........ 168 4.9 Optimal Control and States for Example 4.2 ........ 169 4.10 Optimal Closed-Loop Control in Frequency Domain ... 180 4.11 Closed-Loop Optimal Control System with Unity

Feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 184 4.12 Nyquist Plot of Go(jw) ........................ 185 4.13 Intersection of Unit Circles Centered at Origin

and -1 + jO ............................... 186

5.1 State and Costate System. . . . . . . . . . . . . . . . . . . . .. 205 5.2 Closed-Loop Optimal Controller for Linear Discrete-Time

Regulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 215 5.3 Riccati Coefficients for Example 5.3 ............... 219 5.4 Optimal Control and States for Example 5.3 ........ 220 5.5 Optimal Control and States for Example 5.3 ........ 221 5.6 Closed-Loop Optimal Control for Discrete-Time

Steady-State Regulator System . . . . . . . . . . . . . . . . .. 223 5.7 Implementation of Optimal Control for Example 5.4 . .. 226 5.8 Implementation of Optimal Control for Example 5.4 ... 227 5.9 Riccati Coefficients for Example 5.5. . . . . . . . . . . . . .. 231

XXI

5.10 Optimal States for Example 5.5 ................ " 232 5.11 Optimal Control for Example 5.5 ................ 233 5.12 Implementation of Discrete-Time Optimal Tracker .... 239 5.13 Riccati Coefficients for Example 5.6 . . . . . . . . . . . . . .. 240 5.14 Coefficients 91(t) and 92(t) for Example 5.6 ......... 241 5.15 Optimal States for Example 5.6. . . . . . . . . . . . . . . . .. 241 5.16 Optimal Control for Example 5.6 ................ 242 5.17 Closed-Loop Discrete-Time Optimal Control System. . . 243

6.1 (a) An Optimal Control Function Constrained by a Boundary (b) A Control Variation for Which -8u(t) Is Not Admissible ........................... 254

6.2 Illustration of Constrained (Admissible) Controls ..... 260 6.3 Optimal Path from A to B . . . . . . . . . . . . . . . . . . . .. 261 6.4 A Multistage Decision Process .................. 262 6.5 A Multistage Decision Process: Backward Solution .... 263 6.6 A Multistage Decision Process: Forward Solution ..... 265 6.7 Dynamic Programming Framework of Optimal State

Feedback Control . . . . . . . . . . . . . . . . . . . . . . . . . . .. 271 6.8 Optimal Path from A to B . . . . . . . . . . . . . . . . . . . . . 290

7.1 Signum Function . . . . . . . . . . . . . . . . . . . . . . . . . . .. 299 7.2 Time-Optimal Control ........................ 299 7.3 Normal Time-Optimal Control System ............. 300 7.4 Singular Time-Optimal Control System ............ 301 7.5 Open-Loop Structure for Time-Optimal Control System 304 7.6 Closed-Loop Structure for Time-Optimal Control System 306 7.7 Possible Costates and the Corresponding Controls .... 309 7.8 Phase Plane Trajectories for u = + 1 (dashed lines) and

u = -1 (dotted lines) ......................... 310 7.9 Switch Curve for Double Integral Time-Optimal Control

System ................................... 312 7.10 Various Trajectories Generated by Four Possible Control

Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 313 7.11 Closed-Loop Implementation of Time-Optimal Control

Law ..................................... 315 7.12 SIMULINK@ Implementation of Time-Optimal

Control Law ............................... 316 7.13 Phase-Plane Trajectory for 1'+: Initial State (2,-2) and

Final State (0,0) ............................ 317

xxii

7.14 Phase-Plane Trajectory for 7-: Initial State (-2,2) and Final State (0,0) ............................ 317

7.15 Phase-Plane Trajectory for R+: Initial State (-1,-1) and Final State (0,0) ............................ 318

7.16 Phase-Plane Trajectory for R_: Initial State (1,1) and Final State (0,0) ............................ 318

7.17 Relations Between A2(t) and lu*(t)1 + u*(t)A2(t) ...... 322 7.18 Dead-Zone Function. . . . . . . . . . . . . . . . . . . . . . . . .. 323 7.19 Fuel-Optimal Control ......................... 323 7.20 Switching Curve for a Double Integral Fuel-Optimal

Control System ............................. 324 7.21 Phase-Plane Trajectories for u(t) = 0 .............. 325 7.22 Fuel-Optimal Control Sequences ................. 326 7.23 E-Fuel-Optimal Control. ....................... 327 7.24 Optimal Control as Dead-Zone Function ........... 330 7.25 Normal Fuel-Optimal Control System ............. 331 7.26 Singular Fuel-Optimal Control System ............. 332 7.27 Open-Loop Implementation of Fuel-Optimal Control

System ................................... 333 7.28 Closed-Loop Implementation of Fuel-Optimal Control

System ................................... 334 7.29 SIMULINK@ Implementation of Fuel-Optimal Control

Law ..................................... 334 7.30 Phase-Plane Trajectory for "Y+: Initial State (2,-2) and

Final State (0,0) ............................ 336 7.31 Phase-Plane Trajectory for "Y-: Initial State (-2,2) and

Final State (0,0) ............................ 336 7.32 Phase-Plane Trajectory for R 1 : Initial State (1,1) and

Final State (0,0) ............................ 337 7.33 Phase-Plane Trajectory for R3: Initial State (-1,-1) and

Final State (0,0) ............................ 337 7.34 Phase-Plane Trajectory for R 2 : Initial State (-1.5,1) and

Final State (0,0) ............................ 338 7.35 Phase-Plane Trajectory for R4: Initial State (1.5,-1) and

Final State (0,0) ............................ 338 7.36 Saturation Function .......................... 343 7.37 Energy-Optimal Control ....................... 344 7.38 Open-Loop Implementation of Energy-Optimal Control

System ................................... 345

XXlll

7.39 Closed-Loop Implementation of Energy-Optimal Control System . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 346

7.40 Relation between Optimal Control u*(t) vs (a) q*(t) and (b) 0.5A*(t) ................................ 348

7.41 Possible Solutions of Optimal Costate A*(t) ......... 349 7.42 Implementation of Energy-Optimal Control Law ...... 351 7.43 Relation between Optimal Control u*(t) and Optimal

Costate A2 ( t) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 358

List of Tables 2.1 Procedure Summary of Pontryagin Principle for Bolza

Problem ................................... 69

3.1 Procedure Summary of Finite-Time Linear Quadratic Regulator System: Time-Varying Case. . . . . . . . . . . .. 113

3.2 Procedure Summary of Infinite-Time Linear Quadratic Regulator System: Time-Varying Case. . . . . . . . . . . .. 129

3.3 Procedure Summary of Infinite-Interval Linear Quadratic Regulator System: Time-Invariant Case . . . . . . . . . . .. 136

4.1 Procedure Summary of Linear Quadratic Tracking System159 4.2 Procedure Summary of Regulator System with Prescribed

Degree of Stability . . . . . . . . . . . . . . . . . . . . . . . . . .. 178

5.1 Procedure Summary of Discrete-Time Optimal Control System: Fixed-End Points Condition .............. 204

5.2 Procedure Summary for Discrete-Time Optimal Control System: Free-Final Point Condition ............... 208

5.3 Procedure Summary of Discrete-Time, Linear Quadratic Regulator System ............................ 214

5.4 Procedure Summary of Discrete-Time, Linear Quadratic Regulator System: Steady-State Condition . . . . . . . . .. 222

5.5 Procedure Summary of Discrete-Time Linear Quadratic Tracking System ............................ 238

6.1 Summary of Pontryagin Minimum Principle ......... 257 6.2 Computation of Cost during the Last Stage k = 2 ..... 269 6.3 Computation of Cost during the Stage k = 1,0 ....... 270 6.4 Procedure Summary of Hamilton-Jacobi-Bellman (HJB)

Approach ................................. 280

xxv

XXVI

7.1 Procedure Summary of Optimal Control Systems with State Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . .. 355

Chapter 1

Introduction

In this first chapter, we introduce the ideas behind optimization and optimal control and provide a brief history of calculus of variations and optimal control. Also, a brief summary of chapter contents is presented.

1.1 Classical and Modern Control The classical (conventional) control theory concerned with single input and single output (8180) is mainly based on Laplace transforms the- ory and its use in system representation in block diagram form. From Figure 1.1, we see that

Reference Input

R(s) +

Error Signal

- E(s)

Y(s) R(s)

G(s) 1 + G(s)H(s)

c ompensator ant Control B PI

Input ... G (s) .. Gc(s) U(s) p

Feedback

H(s) ...

(1.1.1)

Output

yes)

Figure 1.1 Classical Control Configuration

2 Chapter 1: Introduction

where s is Laplace variable and we used

(1.1.2)

Note that

1. the input u(t) to the plant is determined by the error e(t) and the compensator, and

2. all the variables are not readily available for feedback. In most cases only one output variable is available for feedback.

The modern control theory concerned with multiple inputs and multi- ple outputs (MIMO) is based on state variable representation in terms of a set of first order differential (or difference) equations. Here, the system (plant) is characterized by state variables, say, in linear, time- invariant form as

x(t) = Ax(t) + Bu(t) y(t) = Cx(t) + Du(t)

(1.1.3)

(1.1.4)

where, dot denotes differentiation with respect to (w.r.t.) t, x(t), u(t), and y( t) are n, r, and m dimensional state, control, and output vectors respectively, and A is nxn state, B is nxr input, Cis mxn output, and D is mxr transfer matrices. Similarly, a nonlinear system is characterized by

x(t) = f(x(t), u(t), t)

y(t) = g(x(t), u(t), t). (1.1.5)

(1.1.6)

The modern theory dictates that all the state variables should be fed back after suitable weighting. We see from Figure 1.2 that in modern control configuration,

1. the input u( t) is determined by the controller (consisting of er- ror detector and compensator) driven by system states x(t) and reference signal r ( t ) ,

2. all or most of the state variables are available for control, and

3. it depends on well-established matrix theory, which is amenable for large scale computer simulation.

1.1 Classical and Modern Control 3

Plant

Control Output

Input .. p .. u(t) y(t)

State

R eference x(t)

Input C ..

'" r(t)

Controller

Figure 1.2 Modern Control Configuration

The fact that the state variable representation uniquely specifies the transfer function while there are a number of state variable representa- tions for a given transfer function, reveals the fact that state variable representation is a more complete description of a system.

Figure 1.3 shows components of a modern control system. It shows three components of modern control and their important contributors. The first stage of any control system theory is to obtain or formulate the dynamics or modeling in terms of dynamical equations such as dif- ferential or difference equations. The system dynamics is largely based on the Lagrangian function. Next, the system is analyzed for its perfor- mance to find out mainly stability of the system and the contributions of Lyapunov to stability theory are well known. Finally, if the system performance is not according to our specifications, we resort to design [25, 109]. In optimal control theory, the design is usually with respect to a performance index. We notice that although the concepts such as Lagrange function [85] and V function of Lyapunov [94] are old, the techniques using those concepts are modern. Again, as the phrase mod- ern usually refers to time and what is modern today becomes ancient after a few years, a more appropriate thing is to label them as optimal control, nonlinear control, adaptive control, robust control and so on.

4 Chapter 1: Introduction

I Modem Control System I ~

~r ~ r

System Dynamics System Analysis System Synthesis (Modeling) (Perfonnance) (Design)

r r ~

State Function of V Function of H Function of Lagrange Lyapunov Pontraygin

(1788) (1892) (1956)

Figure 1.3 Components of a Modern Control System

1.2 Optimization Optimization is a very desirable feature in day-to-day life. We like to work and use our time in an optimum manner, use resources optimally and so on. The subject of optimization is quite general in the sense that it can be viewed in different ways depending on the approach (al- gebraic or geometric), the interest (single or multiple), the nature of the signals (deterministic or stochastic), and the stage (single or multiple) used in optimization. This is shown in Figure 1.4. As we notice that the calculus of variations is one small area of the big picture of the op- timization field, and it forms the basis for our study of optimal control systems. Further, optimization can be classified as static optimization and dynamic optimization.

1. Static Optimization is concerned with controlling a plant under steady state conditions, i.e., the system variables are not chang- ing with respect to time. The plant is then described by algebraic equations. Techniques used are ordinary calculus, Lagrange mul- tipliers, linear and nonlinear programming.

2. Dynamic Optimization concerns with the optimal control of plants under dynamic conditions, i.e., the system variables are changing with respect to time and thus the time is involved in system description. Then the plant is described by differential

1.2 Optimization

Calculus and Lagrange Multipliers

OPTIMIZATION

Algebraic Approach

Multiple Interest

Game Theory

Stochastic

Multiple Stage

Dynamic Programming

Linear and Nonlinear Programming

Functional Analysis

Figure 1.4 Overview of Optimization

6 Chapter 1: Introduction

(or difference) equations. Techniques used are search techniques, dynamic programming, variational calculus (or calculus of varia- tions) and Pontryagin principle.

1.3 Optimal Control

The main objective of optimal control is to determine control signals that will cause a process (plant) to satisfy some physical constraints and at the same time extremize (maximize or minimize) a chosen per- formance criterion (performance index or cost function). Referring to Figure 1.2, we are interested in finding the optimal control u*(t) (* in- dicates optimal condition) that will drive the plant P from initial state to final state with some constraints on controls and states and at the same time extremizing the given performance index J.

The formulation of optimal control problem requires

1. a mathematical description (or model) of the process to be con- trolled (generally in state variable form),

2. a specification of the performance index, and

3. a statement of boundary conditions and the physical constraints on the states and/or controls.

1.3.1 Plant

For the purpose of optimization, we describe a physical plant by a set of linear or nonlinear differential or difference equations. For example, a linear time-invariant system is described by the state and output rela- tions (1.1.3) and (1.1.4) and a nonlinear system by (1.1.5) and (1.1.6).

1.3.2 Performance Index

Classical control design techniques have been successfully applied to lin- ear, time-invariant, single-input, single output (8180) systems. Typical performance criteria are system time response to step or ramp input characterized by rise time, settling time, peak overshoot, and steady state accuracy; and the frequency response of the system characterized by gain and phase margins, and bandwidth.

In modern control theory, the optimal control problem is to find a control which causes the dynamical system to reach a target or fol-

1.3 Optimal Control 7

low a state variable (or trajectory) and at the same time extremize a performance index which may take several forms as described below.

1. Performance Index for Time-Optimal Control System: We try to transfer a system from an arbitrary initial state x(to) to a specified final state x( t f) in minimum time. The corresponding performance index (PI) is

i t!

J = dt = t f - to = t*. to

(1.3.1 )

2. Performance Index for Fuel-Optimal Control System: Con- sider a spacecraft problem. Let u(t) be the thrust of a rocket engine and assume that the magnitude I u( t) I of the thrust is pro- portional to the rate of fuel consumption. In order to minimize the total expenditure of fuel, we may formulate the performance index as

i t!

J = lu(t)ldt. to

(1.3.2)

For several controls, we may write it as

(1.3.3)

where R is a weighting factor.

3. Performance Index for Minimum-Energy Control Sys- tem: Consider Ui (t) as the current in the ith loop of an electric network. Then 2:i!1 u;(t)ri (where, ri is the resistance of the ith loop) is the total power or the total rate of energy expenditure of the network. Then, for minimization of the total expended energy, we have a performance criterion as

(1.3.4)

or in general,

i t!

J = u'(t)Ru(t)dt to

(1.3.5)

8 Chapter 1: Introduction

where, R is a positive definite matrix and prime (') denotes trans- pose here and throughout this book (see Appendix A for more details on definite matrices).

Similarly, we can think of minimization of the integral of the squared error of a tracking system. We then have,

i t!

J = x/(t)Qx(t)dt to

(1.3.6)

where, Xd(t) is the desired value, xa(t) is the actual value, and x(t) = xa(t) - Xd(t), is the error. Here, Q is a weighting matrix, which can be positive semi-definite.

4. Performance Index for Terminal Control System: In a ter- minal target problem, we are interested in minimizing the error between the desired target position Xd (t f) and the actual target position Xa (t f) at the end of the maneuver or at the final time t f. The terminal (final) error is x ( t f) = Xa ( t f) - Xd ( t f ). Taking care of positive and negative values of error and weighting factors, we structure the cost function as

(1.3.7)

which is also called the terminal cost function. Here, F is a positive semi-definite matrix.

5. Performance Index for General Optimal Control System: Combining the above formulations, we have a performance index in general form as

i t!

J = x/(tf)Fx(tf) + [X/(t)QX(t) + u/(t)Ru(t)]dt to

(1.3.8)

or,

i t!

J = S(x(tf),tf) + V(x(t),u(t),t)dt to

(1.3.9)

where, R is a positive definite matrix, and Q and F are positive semidefinite matrices, respectively. Note that the matrices Q and R may be time varying. The particular form of performance index (1.3.8) is called quadratic (in terms of the states and controls) form.

1.3 Optimal Control 9

The problems arising in optimal control are classified based on the structure of the performance index J [67]. If the PI (1.3.9) contains the terminal cost function S(x(t), u(t), t) only, it is called the Mayer problem, if the PI (1.3.9) has only the integral cost term, it is called the Lagrange problem, and the problem is of the Bolza type if the PI contains both the terminal cost term and the integral cost term as in (1.3.9). There are many other forms of cost functions depending on our performance specifications. However, the above mentioned performance indices (with quadratic forms) lead to some very elegant results in optimal control systems.

1.3.3 Constraints

The control u( t) and state x( t) vectors are either unconstrained or constrained depending upon the physical situation. The unconstrained problem is less involved and gives rise to some elegant results. From the physical considerations, often we have the controls and states, such as currents and voltages in an electrical circuit, speed of a motor, thrust of a rocket, constrained as

(1.3.10)

where, +, and - indicate the maximum and minimum values the vari- ables can attain.

1.3.4 Formal Statement of Optimal Control System

Let us now state formally the optimal control problem even risking rep- etition of some of the previous equations. The optimal control problem is to find the optimal control u*(t) (* indicates extremal or optimal value) which causes the linear time-invariant plant (system)

x(t) = Ax(t) + Bu(t) (1.3.11)

to give the trajectory x* (t) that optimizes or extremizes (minimizes or maximizes) a performance index

J = x'(tf)Fx(tf) + J.tJ [x'(t)Qx(t) + u'(t)Ru(t)]dt (1.3.12) to

or which causes the nonlinear system

x(t) = f(x(t), u(t), t) (1.3.13)

10 Chapter 1: Introduction

to give the state x*(t) that optimizes the general performance index

i tf

J = S(x(tj), tf) + V(x(t), u(t), t)dt to

(1.3.14)

with some constraints on the control variables u( t) and/or the state variables x(t) given by (1.3.10). The final time tf may be fixed, or free, and the final (target) state may be fully or partially fixed or free. The entire problem statement is also shown pictorially in Figure 1.5. Thus,

Optimal Control System I + .. •

Plant I Cost Function II Constraints , ,

J J ~

.. ~ J* ... ~

J* ..

u*(t) u(t) r

u*(t) u(t)

(a) Minimum (b) Maximum

Figure 1.5 Optimal Control Problem

we are basically interested in finding the control u*(t) which when applied to the plant described by (1.3.11) or (1.3.13), gives an optimal performance index J* described by (1.3.12) or (1.3.14).

The optimal control systems are studied in three stages.

1. In the first stage, we just consider the performance index of the form (1.3.14) and use the well-known theory of calculus of varia- tions to obtain optimal functions.

2. In the second stage, we bring in the plant (1.3.11) and try to address the problem of finding optimal control u*(t) which will

1.4 Historical Tour 11

drive the plant and at the same time optimize the performance index (1.3.12). Next, the above topics are presented in discrete- time domain.

3. Finally, the topic of constraints on the controls and states (1.3.10) is considered along with the plant and performance index to ob- tain optimal control.

1.4 Historical Tour We basically consider two stages of the tour: first the development of calculus of variations, and secondly, optimal control theory [134, 58, 99, 28]1.

1.4.1 Calculus oj Variations

According to a legend [88], Tyrian princess Dido used a rope made of cowhide in the form of a circular arc to maximize the area to be occupied to found Carthage. Although the story of the founding of Carthage is fictitious, it probably inspired a new mathematical dis- cipline, the calculus of variations and its extensions such as optimal control theory.

The calculus of variations is that branch of mathematics that deals with finding a function which is an extremum (maximum or minimum) of a functional. A functional is loosely defined as a function of a func- tion. The theory of finding maxima and minima of functions is quite old and can be traced back to the isoperimetric problems considered by Greek mathematicians such as Zenodorus (495-435 B.C.) and by Poppus (c. 300 A.D.). But we will start with the works of Bernoulli. In 1699 Johannes Bernoulli (1667-1748) posed the brachistochrone prob- lem: the problem of finding the path of quickest descent between two points not in the same horizontal or vertical line. This problem which was first posed by Galileo (1564-1642) in 1638, was solved by John, his brother Jacob (1654- 1705), by Gottfried Leibniz (1646-1716), and anonymously by Isaac Newton (1642-1727). Leonard Euler (1707-1783) joined John Bernoulli and made some remarkable contributions, which influenced Joseph-Louis Lagrange (1736-1813), who finally gave an el-

IThe permission given by Springer-Verlag for H. H. Goldstine, A History of the Calculus of Variations, Springer-Verlag, New York, NY, 1980, is hereby acknowledged.

12 Chapter 1: Introduction

egant way of solving these types of problems by using the method of (first) variations. This led Euler to coin the phrase calculus of vari- ations. Later this necessary condition for extrema of a functional was called the Euler - the Lagrange equation. Lagrange went on to treat variable end - point problems introducing the multiplier method, which later became one of the most powerful tools-Lagrange (or Euler- Lagrange) multiplier method-in optimization.

The sufficient conditions for finding the extrema of functionals in cal- culus of variations was given by Andrien Marie Legendre (1752-1833) in 1786 by considering additionally the second variation. Carl Gustav Jacob Jacobi (1804-1851) in 1836 came up with a more rigorous anal- ysis of the sufficient conditions. This sufficient condition was later on termed as the Legendre-Jacobi condition. At about the same time Sir William Rowan Hamilton (1788-1856) did some remarkable work on mechanics, by showing that the motion of a particle in space, acted upon by various external forces, could be represented by a single func- tion which satisfies two first-order partial differential equations. In 1838 Jacobi had some objections to this work and showed the need for only one partial differential equation. This equation, called Hamilton-Jacobi equation, later had profound influence on the calculus of variations and dynamic programming, optimal control, and as well as on mechanics.

The distinction between strong and weak extrema was addressed by Karl Weierstrass (1815-1897) who came up with the idea of the field of extremals and gave the Weierstrass condition, and sufficient condi- tions for weak and strong extrema. Rudolph Clebsch (1833-1872) and Adolph Mayer proceeded with establishing conditions for the more gen- eral class of problems. Clebsch formulated a problem in the calculus of variations by adjoining the constraint conditions in the form of differ- ential equations and provided a condition based on second variation. In 1868 Mayer reconsidered Clebsch's work and gave some elegant re- sults for the general problem in the calculus of variations. Later Mayer described in detail the problems: the problem of Lagrange in 1878, and the problem of Mayer in 1895.

In 1898, Adolf Kneser gave a new approach to the calculus of varia- tions by using the result of Karl Gauss (1777-1855) on geodesics. For variable end-point problems, he established the transversality condi- tion which includes orthogonality as a special case. He along with Oskar Bolza (1857-1942) gave sufficiency proofs for these problems. In 1900, David Hilbert (1862-1943) showed the second variation as a

1.4 Historical Tour 13

quadratic functional with eigenvalues and eigenfunctions. Between 1908 and 1910, Gilbert Bliss (1876-1951) [23] and Max Mason looked in depth at the results of Kneser. In 1913, Bolza formulated the problem of Bolza as a generalization of the problems of Lagrange and Mayer. Bliss showed that these three problems are equivalent. Other notable contributions to calculus of variations were made by E. J. McShane (1904-1989) [98], M. R. Hestenes [65], H. H. Goldstine and others. There have been a large number of books on the subject of calculus of variations: Bliss (1946) [23], Cicala (1957) [37], Akhiezer (1962) [1], Elsgolts (1962) [47], Gelfand and Fomin (1963) [55], Dreyfus (1966) [45], Forray (1968) [50], Balakrishnan (1969) [8], Young (1969) [146], Elsgolts (1970) [46], Bolza (1973) [26], Smith (1974) [126], Weinstock (1974) [143], Krasnov et al. (1975) [81], Leitmann (1981) [88], Ew- ing (1985) [48], Kamien and Schwartz (1991) [78], Gregory and Lin (1992) [61], Sagan (1992) [118], Pinch (1993) [108], Wan (1994) [141], Giaquinta and Hildebrandt (1995) [56, 57], Troutman (1996) [136], and Milyutin and Osmolovskii (1998) [103].

1.4.2 Optimal Control Theory

The linear quadratic control problem has its origins in the celebrated work of N. Wiener on mean-square filtering for weapon fire control dur- ing World War II (1940-45) [144, 145]. Wiener solved the problem of designing filters that minimize a mean-square-error criterion (perfor- mance measure) of the form

(1.4.1)

where, e( t) is the error, and E {x} represents the expected value of the random variable x. For a deterministic case, the above error criterion is generalized as an integral quadratic term as

J = 10 00

e'(t)Qe(t)dt (1.4.2)

where, Q is some positive definite matrix. R. Bellman in 1957 [12] introduced the technique of dynamic programming to solve discrete- time optimal control problems. But, the most important contribution to optimal control systems was made in 1956 [25] by L. S. Pontryagin (formerly of the United Soviet Socialistic Republic (USSR)) and his as- sociates, in development of his celebrated maximum principle described

14 Chapter 1: Introduction

in detail in their book [109]. Also, see a very interesting article on the "discovery of the Maximum Principle" by R. V. Gamkrelidze [52], one of the authors of the original book [109]. At this time in the United States, R. E. Kalman in 1960 [70] provided linear quadratic regulator (LQR) and linear quadratic Gaussian (LQG) theory to design optimal feedback controls. He went on to present optimal filtering and estima- tion theory leading to his famous discrete Kalman filter [71] and the continuous Kalman filter with Bucy [76]. Kalman had a profound ef- fect on optimal control theory and the Kalman filter is one of the most widely used technique in applications of control theory to real world problems in a variety of fields.

At this point we have to mention the matrix Riccati equation that appears in all the Kalman filtering techniques and many other fields. C. J. Riccati [114, 22] published his result in 1724 on the solution for some types of nonlinear differential equations, without ever knowing that the Riccati equation would become so famous after more than two centuries!

Thus, optimal control, having its roots in calculus of variations de- veloped during 16th and 17th centuries was really born over 300 years ago [132]. For additional details about the historical perspectives on calculus of variations and optimal control, the reader is referred to some excellent publications [58, 99, 28, 21, 132].

In the so-called linear quadratic control, the term "linear" refers to the plant being linear and the term "quadratic" refers to the perfor- mance index that involves the square or quadratic of an error, and/or control. Originally, this problem was called the mean-square control problem and the term "linear quadratic" did not appear in the litera- ture until the late 1950s.

Basically the classical control theory using frequency domain deals with single input and single output (SIS0) systems, whereas modern control theory works with time domain for SISO and multi-input and multi-output (MIMO) systems. Although modern control and hence optimal control appeared to be very attractive, it lacked a very impor- tant feature of robustness. That is, controllers designed based on LQR theory failed to be robust to measurement noise, external disturbances and unmodeled dynamics. On the other hand, frequency domain tech- niques using the ideas of gain margin and phase margin offer robustness in a natural way. Thus, some researchers [115, 95], especially in the United Kingdom, continued to work on developing frequency domain

1.5 About This Book 15

approaches to MIMO systems. One important and relevant field that has been developed around

the 1980s is the Hoo-optimal control theory. In this framework, the work developed in the 1960s and 1970s is labeled as H2-optimal control theory. The seeds for Hoo-optimal control theory were laid by G. Zames [148], who formulated the optimal Hoo-sensitivity design problem for SISO systems and solved using optimal N evanilina-Pick interpolation theory. An important publication in this field came from a group of four active researchers, Doyle, Glover, Khargonekar, and Francis[44], who won the 1991 W. R. G. Baker Award as the best IEEE Transactions paper. There are many other works in the field of Hoo control ([51, 96, 43, 128, 7, 60, 131, 150, 39, 34]).

1.5 About This Book This book, on the subject of optimal control systems, is based on the author's lecture notes used for teaching a graduate level course on this subject. In particular, this author was most influenced by Athans and Falb [6], Schultz and Melsa [121], Sage [119], Kirk [79], Sage and White [120], Anderson and Moore [3] and Lewis and Syrmos [91], and one finds the footprints of these works in the present book.

There were a good number of books on optimal control published during the era of the "glory of modern control," (Leitmann (1964) [87], Tou (1964) [135], Athans and Falb (1966) [6], Dreyfus (1966) [45], Lee and Markus (1967) [86], Petrov (1968) [106], Sage (1968) [119], Citron (1969) [38], Luenberger (1969) [93], Pierre (1969) [107], Pun (1969) [110], Young (1969) [146], Kirk (1970) [79], Boltyanskii [24], Kwaker- naak and Sivan (1972) [84], Warga (1972) [142], Berkovitz (1974) [17], Bryson and Ho (1975) [30]), Sage and White (1977) [120], Leitmann (1981) [88]), Ryan (1982) [116]). There has been renewed interest with the second wave of books published during the last few years (Lewis (1986) [89], Stengal (1986) [127], Christensen et al. (1987) [36] Ander- son and Moore (1990) [3], Hocking (1991) [66], Teo et al. (1991) [133], Gregory and Lin (1992) [61], Lewis (1992) [90], Pinch (1993) [108], Do- rato et al. (1995) [42], Lewis and Syrmos (1995) [91]), Saberi et al. (1995) [117], Sima (1996) [124], Siouris [125], Troutman (1996) [136] Bardi and Dolcetta (1997) [9], Vincent and Grantham (1997) [139], Milyutin and Osmolovskii (1998) [103], Bryson (1999) [29], Burl [32], Kolosov (1999) [80], Pytlak (1999) [111], Vinter (2000) [140], Zelikin

16 Chapter 1: Introduction

(2000) [149], Betts (2001) [20], and Locatelli (2001) [92]. The optimal control theory continues to have a wide variety of appli-

cations starting from the traditional electrical power [36] to economics and management [16, 122, 78, 123].

1.6 Chapter Overview This book is composed of seven chapters. Chapter 2 presents opti- mal control via calculus of variations. In this chapter, we start with some basic definitions and a simple variational problem of extremizing a functional. We then bring in the plant as a conditional optimization problem and discuss various types of problems based on the bound- ary conditions. We briefly mention both Lagrangian and Hamiltonian formalisms for optimization. Next, Chapter 3 addresses basically the linear quadratic regulator (LQR) system. Here we discuss the closed- loop optimal control system introducing matrix Riccati differential and algebraic equations. We look at the analytical solution to the Riccati equations and development of MATLAB© routine for the analytical solution. Tracking and other problems of linear quadratic optimal con- trol are discussed in Chapter 4. We also discuss the gain and phase margins of the LQR system.

So far the optimal control of continuous-time systems is described. Next, the optimal control of discrete-time systems is presented in Chap- ter 5. Here, we start with the basic calculus of variations and then touch upon all the topics discussed above with respect to the continuous-time systems. The Pontryagin Principle and associated topics of dynamic programming and Hamilton-Jacobi-Bellman results are briefly covered in Chapter 6. The optimal control of systems with control and state constraints is described in Chapter 7. Here, we cover topics of control constraints leading to time-optimal, fuel-optimal and energy-optimal control systems and briefly discuss the state constraints problem.

Finally, the Appendices A and B provide summary of results on ma- trices, vectors, matrix algebra and state space, and Appendix C lists some of the MATLAB© files used in the book.

1.7 Problems 17

1. 7 Problems

Problem 1.1 A D.C. motor speed control system is described by a second order state equation,

:h (t) = 25x2(t) X2(t) = -400Xl(t) - 200X2(t) + 400u(t) ,

where, Xl(t) = the speed of the motor, and X2(t) = the current in the armature circuit and the control input u( t) = the voltage input to an amplifier supplying the motor. Formulate a performance index and optimal control problem to keep the speed constant at a particular value.

Problem 1.2 [83] In a liquid-level control system for a storage tank, the valves connecting a reservoir and the tank are controlled by gear train driven by a D. C. motor and an electronic amplifier. The dynamics is described by a third order system

Xl(t) = -2Xl(t) X2(t) = X3(t) X3(t) = -10X3(t) + 9000u(t)

where, Xl(t) = is the height in the tank, X2(t) = is the angular posi- tion of the electric motor driving the valves controlling the liquid from reservoir to tank, X3(t) = the angular velocity of the motor, and u(t) = is the input to electronic amplifier connected to the input of the motor. Formulate optimal control problem to keep the liquid level constant at a reference value and the system to act only if there is a change in the liquid level.

Problem 1.3 [35] In an inverted pendulum system, it is required to maintain the upright position of the pendulum on a cart. The linearized state equations are

Xl(t) = X2(t)

X2(t) = -X3(t) + O.2u(t) X3(t) = X4(t)

X4(t) = 10x3(t) - O.2u(t)

18 Chapter 1: Introduction

where, Xl (t) = is horizontal linear displacement of the cart, X2(t) = is linear velocity of the cart, X3(t) = is angular position of the pendulum from vertical line, X4(t) = is angular velocity, and u(t) = is the horizon- tal force applied to the cart. Formulate a performance index to keep the pendulum in the vertical position with as little energy as possible.

Problem 1.4 [101J A mechanical system consisting of two masses and two springs, one spring connecting the two masses and the other spring connecting one of the masses to a fixed point. An input is applied to the mass not connected to the fixed point. The displacements (XI(t) and X2 (t)) and the corresponding velocities (X3 (t) and X4 (t)) of the two masses provide a fourth-order system described by

XI(t) = X3(t) X2(t) = X4(t) X3(t) = -4XI(t) + 2X2(t) X4(t) = XI(t) - X2(t) + u(t)

Formulate a performance index to minimize the errors in displacements and velocities and to minimize the control effort.

Problem 1.5 A simplified model of an automobile suspension system is described by

mx(t) + kx(t) = bu(t)

where, x(t) is the position, u(t) is the input to the suspension system (in the form of an upward force), m is the mass of the suspension system, and k is the spring constant. Formulate the optimal control problem for minimum control energy and passenger comfort. Assume suitable values for all the constants.

Problem 1.6 [112J Consider a continuous stirred tank chemical reac- tor described by

XI(t) = -O.lXI(t) - 0.12x2(t) X2(t) = -0.3XI(t) - 0.012x2(t) - 0.07u(t)

where, the normalized deviation state variables of the linearized model are Xl (t) = reaction variable, X2 (t) = temperature and the control variable u(t) = effective cooling rate coefficient. Formulate a suitable performance measure to minimize the deviation errors and to minimize the control effort.

Chapter 2

Calculus of Variations and Optimal Control

Calculus of variations (Co V) or variational calculus deals with finding the optimum (maximum or minimum) value of a functional. Varia- tional calculus that originated around 1696 became an independent mathematical discipline after the fundamental discoveries of L. Euler (1709-1783), whom we can claim with good reason as the founder of calculus of variations.

In this chapter, we start with some basic definitions and a simple variational problem of extremizing a functional. We then incorporate the plant as a conditional optimization problem and discuss various types of problems based on the boundary conditions. We briefly men- tion both the Lagrangian and Hamiltonian formalisms for optimization. It is suggested that the student reviews the material in Appendices A and B given at the end of the book. This chapter is motivated by [47, 79, 46, 143, 81, 48]1.

2.1 Basic Concepts 2.1.1 Function and Functional

We discuss some fundamental concepts associated with functionals along side with those of functions. (a) Function: A variable x is a function of a variable quantity t, (writ-

IThe permission given by Prentice Hall for D. E. Kirk, Optimal Control Theory: An Intro- duction, Prentice Hall, Englewood Cliffs, NJ, 1970, is hereby acknowledged.

20 Chapter 2: Calculus of Variations and Optimal Control

ten as x(t) = !(t)), if to every value of t over a certain range of t there corresponds a value x; i.e., we have a correspondence: to a number t there corresponds a number x. Note that here t need not be always time but any independent variable.

Example 2.1

Consider

x(t) = 2t2 + 1. (2.1.1 )

For t = 1, x = 3, t = 2, x = 9 and so on. Other functions are x(t) = 2t; X(tb t2) = tt + t§.

N ext we consider the definition of a functional based on that of a function. (b) Functional: A variable quantity J is a functional dependent on a function ! (x), written as J = J (f (x) ), if to each function f (x), there corresponds a value J, i.e., we have a correspondence: to the function f (x) there corresponds a number J. Functional depends on several functions.

Example 2.2

Let x(t) = 2t2 + 1. Then

{I (I 2 5 J(x(t)) = io x(t)dt = io (2t2 + l)dt = 3 + 1 = 3 (2.1.2)

is the area under the curve x(t). If v(t) is the velocity of a vehicle, then

l ti

J ( v ( t )) = v ( t ) dt to

(2.1.3)

is the path traversed by the vehicle. Thus, here x(t) and v(t) are functions of t, and J is a functional of x(t) or v(t).

Loosely speaking, a functional can be thought of as a "function of a function."

2.1.2 Increment

We consider here increment of a function and a functional.

2.1 Basic Concepts 21

(a) Increment of a Function: In order to consider optimal values of a function, we need the definition of an increment [47, 46, 79].

DEFINITION 2.1 The increment of the function I, denoted by ~/, is defined as

~/~/(t + ~t) - I(t). (2.1.4)

It is easy to see from the definition that ~I depends on both the independent variable t and the increment of the independent variable ~t, and hence strictly speaking, we need to write the increment of a function as ~/(t, ~t).

Example 2.3

find the increment of the function I ( t) .

Solution: The increment ~I becomes

~I ~ I(t + ~t) - I(t) = (tl + ~iI + t2 + ~t2? - (tl + t2)2 = (tl + ~tl)2 + (t2 + ~t2)2 + 2(iI + ~h)(t2 + ~t2) -

(tI + t§ + 2tlt2) = 2(tl + t2)~tl + 2(tl + t2)~t2 + (~tl)2 + (~t2)2

(2.1.5)

+2~tl~t2. (2.1.6)

(b) Increment of a Functional: Now we are ready to define the increment of a functional.

DEFINITION 2.2 The increment of the functional J, denoted by ~J, is defined as

I ~J~J(x(t) + 8x(t)) - J(x(t))·1 (2.1. 7)

Here 8x(t) is called the variation of the function x(t). Since the in- crement of a functional is dependent upon the function x(t) and its

22 Chapter 2: Calculus of Variations and Optimal Control

variation 8x(t), strictly speaking, we need to write the increment as ilJ(x(t),8x(t)).

Example 2.4

Find the increment of the functional

(2.1.8)

Solution: The increment of J is given by

ilJ ~ J(x(t) + 8x(t)) - J(x(t)),

= it! [2(x(t) + 8x(t))2 + 1] dt _it! [2x2(t) + 1] dt, to to

i t!

= [4x(t)8x(t) + 2(8x(t) )2] dt. (2.1.9) to

2.1.3 Differential and Variation

Here, we consider the differential of a function and the variation of a functional. (a) Differential of a Function: Let us define at a point t* the increment of the function J as

ilf~J(t* + ilt) - J(t*). (2.1.10)

By expanding J (t* + ilt) in a Taylor series about t*, we get

Af = f(t') + (:), At + :, (~n, (At)2 + ... - f(t*). (2.1.11)

Neglecting the higher order terms in ilt,

Af = (:) * At = j(t*)At = df. (2.1.12)

Here, df is called the differential of J at the point t*. j(t*) is the derivative or slope of J at t*. In other words, the differential dJ is the first order approximation to increment ilt. Figure 2.1 shows the relation between increment, differential and derivative.

2.1 Basic Concepts 23

f(t)

f(t* +~t) ....... '. . .. ... . ......... '

[(to) ~:~~~ ~ [ ... :. : ::t ~ .. .1~. . ~t :~ .

o t* t*+~t t

Figure 2.1 Increment fl.j, Differential dj, and Derivative j of a Function j ( t)

Example 2.5

Let j(t) = t 2 + 2t. Find the increment and the derivative of the function j ( t).

Solution: By definition, the increment fl.j is

fl.j £ j(t + fl.t) - j(t), = (t + fl.t)2 + 2(t + fl.t) - (t2 + 2t), = 2tfl.t + 2fl.t + ... + higher order terms, = 2(t + l)fl.t, = j(t)fl.t.

Here, j(t) = 2(t + 1).

(2.1.13)

(b) Variation of a Functional: Consider the increment of a func- tional

fl.J£J(x(t) + 8x(t)) - J(x(t)). (2.1.14)

24 Cbapter 2: Calculus of Variations and Optimal Control

Expanding J(x(t) + 8x(t)) in a Taylor series, we get

{)J 1 {)2 J jj.J = J(x(t)) + -{) 8x(t) + -, {) 2 (8x(t))2 + ... - J(x(t))

x 2. x {)J 1 {)2J 2

= {)x 8x(t) + 2! {)x2 (8x(t)) + ...

= 8 J + 82 J + ... , (2.1.15)

where,

{)J 8J = {)x 8x(t) and (2.1.16)

are called the first variation (or simply the variation) and the second variation of the functional J, respectively. The variation 8 J of a func- tional J is the linear (or first order approximate) part (in 8x(t)) of the increment jj.J. Figure 2.2 shows the relation between increment and the first variation of a functional.

J(x(t»

J(x*(t)+Ox(t» . . . . . . . . . .. ... . ......... ,

. J(x*(t» ... .:. : J ~~ .. .1~. :.. ~ : ox(t):

o x*(t) x*(t)+ Ox(t) x(t)

Figure 2.2 Increment jj.J and the First Variation 8J of the Functional J

2.2 Optimum of a Function and a Functional 25

Example 2.6

Given the functional

i t!

J(x(t)) = [2x2(t) + 3x(t) + 4]dt, to

(2.1.17)

evaluate the variation of the functional.

Solution: First, we form the increment and then extract the vari- ation as the first order approximation. Thus

~J ~ J(x(t) + 8x(t)) - J(x(t)),

i t!

= [2(x(t) + 8x(t))2 + 3(x(t) + 8x(t)) + 4) to

-(2x2(t) + 3x(t) + 4)] dt,

i t!

= [4x(t)8x(t) + 2(8x(t))2 + 38x(t)] dt. to

(2.1.18)

Considering only the first order terms, we get the (first) variation as

i t!

8J(x(t),8x(t)) = (4x(t) + 3)8x(t)dt. to

(2.1.19)

2.2 Optimum of a Function and a Functional We give some definitions for optimum or extremum (maximum or min- imum) of a function and a functional [47, 46, 79]. The variation plays the same role in determining optimal value of a functional as the dif- ferential does in finding extremal or optimal value of a function.

DEFINITION 2.3 Optimum of a Function: A function f (t) is said to have a relative optimum at the point t* if there is a positive parameter E such that for all points t in a domain V that satisfy It - t* I < E, the increment of f(t) has the same sign (positive or negative).

In other words, if

~f = f(t) - f(t*) 2:: 0, (2.2.1)

26 Chapter 2: Calculus of Variations and Optimal Control

f(t f(t)

v 0

t* t

f(t)

0 t

o t* t

(a)

m 0 t* t

f(t)

0 t

t* o~------------. . t

... ~ ;.~~., ~ f(t*)<0

(b)

Figure 2.3 (a) Minimum and (b) Maximum of a Function f (t)

then, f(t*) is a relative local minimum. On the other hand, if

b.f = f(t) - f(t*) ~ 0, (2.2.2)

then, f (t*) is a relative local maximum. If the previous relations are valid for arbitrarily large E, then, f(t*) is said to have a global absolute optimum. Figure 2.3 illustrates the (a) minimum and (b) maximum of a function.

It is well known that the necessary condition for optimum of a func- tion is that the (first) differential vanishes, i.e., df = O. The sufficient condition

2.3 The Basic Variational Problem

1. for minimum is that the second differential is positive, i.e., d2 f > 0, and

2. for maximum is that the second differential is negative, i.e., d2 f < 0.

If d2 f = 0, it corresponds to a stationary (or inflection) point.

DEFINITION 2.4 Optimum of a Functional: A functional J is said to have a relative optimum at x* if there is a positive E such that for all functions x in a domain n which satisfy Ix - x* I < E, the increment of J has the same sign.

In other words, if

!1J = J(x) - J(x*) ~ 0, (2.2.3)

then J(x*) is a relative minimum. On the other hand, if

!1J = J(x) - J(x*) ~ 0, (2.2.4)

then, J(x*) is a relative maximum. If the above relations are satisfied for arbitrarily large E, then, J(x*) is a global absolute optimum.

Analogous to finding extremum or optimal values for functions, in variational problems concerning functionals, the result is that the vari- ation must be zero on, an optimal curve. Let us now state the result in the form of a theorem, known as fundamental theorem of the calculus of variations, the proof of which can be found in any book on calculus of variations [47, 46, 79].

THEOREM 2.1 For x*(t) to be a candidate for an optimum, the (first) variation of J must

be zero on x*(t), i.e., 6J(x*(t), 6x(t)) = ° for all admissible values of 6x(t). This is a necessary condition. As a sufficient condition for minimum, the second variation 62 J > 0, and for maximum 62 J < 0.

2.3 The Basic Variational Problem 2.3.1 Fixed-End Time and Fixed-End State System

We address a fixed-end time and fixed-end state problem, where both the initial time and state and the final time and state are fixed or given

28 Chapter 2: Calculus of Variations and Optimal Control

a priori. Let x(t) be a scalar function with continuous first derivatives and the vector case can be similarly dealt with. The problem is to find the optimal function x* (t) for which the functional

i t!

J(x(t)) = V(x(t), x(t), t)dt to

(2.3.1)

has a relative optimum. It is assumed that the integrand V has con- tinuous first and second partial derivatives w.r.t. all its arguments; to and t f are fixed (or given a priori) and the end points are fixed, i.e.,

x(t = to) = Xo; x(t = tf) = xf' (2.3.2)

We already know from Theorem 2.1 that the necessary condition for an optimum is that the variation of a functional vanishes. Hence, in our attempt to find the optimum of x(t), we first define the increment for J, obtain its variation and finally apply the fundamental theorem of the calculus of variations (Theorem 2.1).

Thus, the various steps involved in finding the optimal solution to the fixed-end time and fixed-end state system are first listed and then discussed in detail.

• Step 1: Assumption of an Optimum

• Step 2: Variations and Increment

• Step 3: First Variation

• Step 4: Fundamental Theorem

• Step 5: Fundamental Lemma

• Step 6: Euler-Lagrange Equation

• Step 1: Assumption of an Optimum: Let us assume that x*(t) is the optimum attained for the function x(t). Take some admissible function xa(t) = x*(t) + 8x(t) close to x*(t), where 8x(t) is the variation of x*(t) as shown in Figure 2.4. The function xa(t) should also satisfy the boundary conditions (2.3.2) and hence it is necessary that

(2.3.3)

2.3 The Basic Variational Problem 29

x(t)

xo ..... .

Figure 2.4 Fixed-End Time and Fixed-End State System

• Step 2: Variations and Increment: Let us first define the incre- ment as

6.J(x*(t), 8x(t)) ~ J(x*(t) + 8x(t), x*(t) + 8x(t), t) -J(x*(t), x*(t), t)

I t!

= V (x*(t) + 8x(t), x*(t) + 8x(t), t) dt to

I t!

- V(x*(t), x*(t), t)dt. to

(2.3.4)

which by combining the integrals can be written as

I t!

6.J(x*(t), 8x(t)) = [V (x*(t) + 8x(t), x*(t) + 8x(t), t) to

- V(x* (t), x*(t), t)] dt. (2.3.5)

where,

x(t) = d:~t) and 8x(t) = :t {8x(t)} (2.3.6) Expanding V in the increment (2.3.5) in a Taylor series about the point x*(t) and x*(t), the increment 6.J becomes (note the

30 Chapter 2: Calculus of Variations and Optimal Control

cancelation of V(x*(t), x*(t), t))

~J = ~J(x*(t), 8x(t))

= l' [8V(X*(~~X*(t), t) 6x(t) + 8V(X*(~~ x*(t), t) 6x(t) ~ {8 2 V( ... ) (8 ())2 82 V( ... ) (8· ( ))2 + 2! 8x2 x t + 8x2 X t +

+ 2~:~~·) 6x (t)6x (t) } + .. -] dt. (2.3.7)

Here, the partial derivatives are w.r.t. x(t) and x(t) at the opti- mal condition (*) and * is omitted for simplicity .

• Step 3: First Variation: Now, we obtain the variation by retain- ing the terms that are linear in 8x(t) and 8x(t) as

8J(x*(t),8x(t)) = it! [8V(X*(t), x*(t), t) 8x(t) to 8x 8V(x*(t), x*(t), t)8· ( )] d

+ 8x x t t. (2.3.8)

To express the relation for the first variation (2.3.8) entirely in terms containing 8x(t) (since 8x(t) is dependent on 8x(t)), we integrate by parts the term involving 8x(t) as (omitting the ar- guments in V for simplicity)

1:' (~~) * 6x(t)dt = 1:' (~~) * ! (6x(t))dt = 1:' (~~) * d(6x(t)), = [( ~~) * 6X(t{:

_it! 8x(t)~ (8~) dt. to dt 8x *

(2.3.9)

In the above, we used the well-known integration formula J udv = uv - J vdu where u = 8V/8X and v = 8x(t)). Using (2.3.9), the

2.3 The Basic Variational Problem 31

relation (2.3.8) for first variation becomes

8J(x*(t),6x(t)) = {' (~:) * 6x(t)dt + [( ~~) * 6X(t)[ _ rtf !i (a~) 8x(t)dt,

lto dt ax * = rtf [(av) _!i (a~) ]8x(t)dt

lto ax * dt ax *

+ [( ~~) * 6x(t)] I:: . (2.3.10) Using the relation (2.3.3) for boundary variations in (2.3.10), we get

8J(x*(t),6x(t)) = 1:' [( ~:) * - :t (~~) .l6X(t)dt. (2.3.11) • Step 4: Fundamental Theorem: We now apply the fundamental

theorem of the calculus of variations (Theorem 2.1), i.e., the vari- ation of J must vanish for an optimum. That is, for the optimum x*(t) to exist, 8J(x*(t),8x(t)) = O. Thus the relation (2.3.11) becomes

rtf [(av) _!i (a~) ]8X(t)dt = O. lto ax * dt ax *

(2.3.12)

Note that the function 8x(t) must be zero at to and tf, but for this, it is completely arbitrary .

• Step 5: Fundamental Lemma: To simplify the condition ob- tained in the equation (2.3.12), let us take advantage of the fol- lowing lemma called the fundamental lemma of the calculus of variations [47, 46, 79].

LEMMA 2.1

If for every function g(t) which is continuous,

l tf

g(t)8x(t)dt = 0 to

(2.3.13)

32 Chapter 2: Calculus of Variations and Optimal Control

where the function 8x(t) is continuous in the interval [to, tf]' then the function 9 ( t) must be zero everywhere throughout the interval [to, t f] . (see Figure 2.5.)

Proof: We prove this by contradiction. Let us assume that g(t) is nonzero (positive or negative) during a short interval [ta, tb]. Next, let us select 8x(t), which is arbitrary, to be positive (or negative) through- out the interval where 9 ( t) has a nonzero value. By this selection of 8x(t), the value of the integral in (2.3.13) will be nonzero. This contradicts our assumption that g( t) is non-zero during the interval. Thus g( t) must be identically zero everywhere during the entire inter- val [to, tf] in (2.3.13). Hence the lemma.

get)

8x(t)

Figure 2.5 A Nonzero g(t) and an Arbitrary 8x(t)

• Step 6: Euler-Lagrange Equation: Applying the previous lemma to (2.3.12), a necessary condition for x*(t) to be an optimal of the functional J given by (2.3.1) is

( av(x*(t),x*(t),t)) _ ~ (av(x*(t),.x*(t),t)) = 0 (2.3.14)

ax * dt ax *

2.3 The Basic Variational Problem 33

or in simplified notation omitting the arguments in V,

(aV) _!i (aV) = 0 ax * dt ax * (2.3.15) for all t E [to, tf]. This equation is called Euler equation, first published in 1741 [126].

A historical note is worthy of mention.

Euler obtained the equation (2.3.14) in 1741 using an elab- orate and cumbersome procedure. Lagrange studied Euler's results and wrote a letter to Euler in 1755 in which he ob- tained the previous equation by a more elegant method of "variations" as described above. Euler recognized the sim- plicity and generality of the method of Lagrange and intro- duced the name calculus of variations. The all important fundamental equation (2.3.14) is now generally known as Euler-Lagrange (E.-L') equation after these two great math- ematicians of the 18th century. Lagrange worked further on optimization and came up with the well-known Lagrange multiplier rule or method.

2.3.2 Discussion on Euler-Lagrange Equation

We provide some comments on the Euler-Lagrange equation [47,46].

1. The Euler-Lagrange equation (2.3.14) can be written in many different forms. Thus (2.3.14) becomes

d V - - (V·) = 0

x dt x (2.3.16)

where,

Vx = ~: = Vx(x*(t), ±*(t), t); Vi; = ~~ = Vi;(X*(t), x*(t), t). (2.3.17)

Since V is a function of three arguments x*(t), x*(t), and t, and

34 Chapter 2: Calculus of Variations and Optimal Control

that x*(t) and x*(t) are in turn functions of t, we get

Combining (2.3.16) and (2.3.18), we get an alternate form for the EL equation as

(2.3.19)

2. The presence of -it and/or x*(t) in the EL equation (2.3.14) means that it is a differential equation.

3. In the EL equation (2.3.14), the term aV(x*(~i:x*(t),t) is in general a function of x*(t), x*(t), and t. Thus when this function is differentiated w.r.t. t, x*(t) may be present. This means that the differential equation (2.3.14) is in general of second order. This is also evident from the alternate form (2.3.19) for the EL equation.

4. There may also be terms involving products or powers of x* (t), x*(t), and x*(t), in which case, the differential equation becomes nonlinear.

5. The explicit presence of t in the arguments indicates that the coefficients may be time-varying.

6. The conditions at initial point t = to and final point t = t f leads to a boundary value problem.

7. Thus, the Euler-Lagrange equation (2.3.14) is, in general, a non- linear, time-varying, two-point boundary value, second order, or- dinary differential equation. Thus, we often have a nonlinear two-point boundary value problem (TPBVP). The solution of the nonlinear TPBVP is quite a formidable task and often done us- ing numerical techniques. This is the price we pay for demanding optimal performance!

2.3 The Basic Variational Problem 35

8. Compliance with the Euler-Lagrange equation is only a necessary condition for the optimum. Optimal may sometimes not yield either a maximum or a minimum; just as inflection points where the derivative vanishes in differential calculus. However, if the Euler-Lagrange equation is not satisfied for any function, this indicates that the optimum does not exist for that functional.

2.3.3 Different Cases for Euler-Lagrange Equation

We now discuss various cases of the EL equation. Case 1: V is dependent of x(t), and t. That is, V = V(x(t), t). Then Vx = O. The Euler-Lagrange equation (2.3.16) becomes

This leads us to

d dt (Vx) = o.

V x ' = oV(x*(t), t) = C

where, C is a constant of integration.

(2.3.20)

(2.3.21)

Case 2: V is dependent of x(t) only. That is, V = V(x(t)). Then Vx = O. The Euler-Lagrange equation (2.3.16) becomes

d dt (Vx) = 0 ~ Vx = C. (2.3.22)

In general, the solution of either (2.3.21) or (2.3.22) becomes

(2.3.23)

This is simply an equation of a straight line. Case 3: V is dependent of x(t) and x(t). That is, V = V(x(t), x(t)). Then vtx = O. Using the other form of the Euler-Lagrange equation (2.3.19), we get

Vx - Vxxx*(t) - Vxxx*(t) = O. (2.3.24)

Multiplying the previous equation by x*(t), we have

x*(t) [Vx - Vxxx*(t) - Vxxx*(t)] = o. (2.3.25)

This can be rewritten as

! (V - x*(t)Vx) = 0 ~ V - x*(t)Vx = C. (2.3.26)

36 Chapter 2: Calculus of Variations and Optimal Control

The previous equation can be solved using any of the techniques such as, separation of variables. Case 4: V is dependent of x(t), and t, i.e., V = V(x(t), t). Then, Vi; = 0 and the Euler-Lagrange equation (2.3.16) becomes

8V(x*(t), t) = 0 ax . (2.3.27)

The solution of this equation does not contain any arbitrary constants and therefore generally speaking does not satisfy the boundary con- ditions x(to) and x(tf). Hence, in general, no solution exists for this variational problem. Only in rare cases, when the function x(t) satisfies the given boundary conditions x(to) and x(tf), it becomes an optimal function.

Let us now illustrate the application of the EL equation with a very simple classic example of finding the shortest distance between two points. Often, we omit the * (which indicates an optimal or extremal value) during the working of a problem and attach the same to the final solution.

Example 2.7

Find the minimum length between any two points.

Solution: It is well known that the solution to this problem is a straight line. However, we like to illustrate the application of Euler- Lagrange equation for this simple case. Consider the arc between two points A and B as shown in Figure 2.6. Let ds be the small arc length, and dx and dt are the small rectangular coordinate values. Note that t is the independent variable representing distance and not time. Then,

(2.3.28)

Rewriting

ds = VI + x 2 (t)dt, where x(t) = ~~. (2.3.29) N ow the total arc length S between two points x (t = to) and x (t = t f) is the performance index J to be minimized. Thus,

S = J = J ds = rtf VI + x2 (t)dt = rtf V(x(t))dt (2.3.30) Jto Jto

2.3 The Basic Variational Problem 37

x(t)

xo ... 'A::

Figure 2.6 Arc Length

where, V(x(t)) = Jl + x2 (t). Note that V is a function of x(t) only. Applying the Euler-Lagrange equation (2.3.22) to the perfor- mance index (2.3.30), we get

x*(t) _ C VI + X*2(t) - .

Solving this equation, we get the optimal solution as

x*(t) = C1t + C2 .

(2.3.31)

(2.3.32)

This is evidently an equation for a straight line and the constants C1 and C2 are evaluated from the given boundary conditions. For example, if x(O) = 1 and x(2) = 5, C 1 = 2 and C2 = 1 the straight line is x*(t) = 2t + 1.

Although the previous example is a simple one,

1. it illustrates the formulation of a performance index from a given simple specification or a statement, and

2. the solution is well known a priori so that we can easily verify the application of the Euler-Lagrange equation.

In the previous example, we notice that the integrand V in the func- tional (2.3.30), is a function of x(t) only. Next, we take an example, where, V is a function of x(t), x(t) and t.

38 Chapter 2: Calculus of Variations and Optimal Control

Example 2.8

Find the optimum of

J = l [:i;2(t) - 2tX(t)] dt that satisfy the boundary (initial and final) conditions

x(O) = 1 and x(2) = 5.

(2.3.33)

(2.3.34)

Solution: In the EL equation (2.3.19), we first identify that V = x 2 (t) - 2tx(t). Then applying the EL equation (2.3.15) to the performance index (2.3.33) we get

av _ ~ (av) = 0 ----+ -2t - ~ (2x(t)) = 0 ax dt ax dt

----+ x(t) = t.

Solving the previous simple differential equation, we have

t 3 x*(t) ="6 + CIt + C2

(2.3.35)

(2.3.36)

where, C1 and C2 are constants of integration. Using the given boundary conditions (2.3.19) in (2.3.36), we have

4 x(O) = 1 ----+ C2 = 1, x(2) = 5 ----+ C1 = 3' (2.3.37)

With these values for the constants, we finally have the optimal function as

t 3 4 x*(t) = "6 + "3 t + 1. (2.3.38)

Another classical example in the calculus of variations is the brachis- tochrone (from brachisto, the shortest, and chrones, time) problem and this problem is dealt with in almost all books on calculus of variations [126].

Further, note that we have considered here only the so-called fixed- end point problem where both (initial and final) ends are fixed or given in advance. Other types of problems such as free-end point problems are not presented here but can be found in most of the books on the calculus of variations [79, 46, 81, 48]. However, these free-end point problems are better considered later in this chapter when we discuss the optimal control problem consisting of a performance index and a physical plant.

2.4 The Second Variation 39

2.4 The Second Variation In the study of extrema of functionals, we have so far considered only the necessary condition for a functional to have a relative or weak ex- tremum, i.e., the condition that the first variation vanish leading to the classic Euler-Lagrange equation. To establish the nature of opti- mum (maximum or minimum), it is required to examine the second variation. In the relation (2.3.7) for the increment consider the terms corresponding to the second variation [120],

8 2 J = f :! [( ~~) . (8x(t))2 + (~:~) • (8X(t))2

+ 2 (::;x) * 8X(t)8X(t)] dt. (2.4.1) Consider the last term in the previous equation and rewrite it in terms of 8x(t) only using integration by parts (f udv = uv - f vdu where, u = :;¥X8x(t) and v = 8x(t)). Then using 8x(to) = 8x(tf) = 0 for fixed-end conditions, we get

82 J = ~ rtf [{ (82V) _!i ( 8 2V.) } (8x(t))2 2 ltD 8x2 dt 8x8x

* *

+ (~:~). (8X(t))2] dt. (2.4.2) According to Theorem 2.1, the fundamental theorem of the calculus of variations, the sufficient condition for a minimum is 82 J > O. This, for arbitrary values of 8x(t) and 8x(t), means that

(8 2V) d (82V)

8x2 * - dt 8x8x * > 0, (2.4.3)

(8 2V)

8x2 * > O. (2.4.4)

For maximum, the signs of the previous conditions are reversed. Al- ternatively, we can rewrite the second variation (2.4.1) in matrix form as

2 1 tf . 8x2 8x8± 8x(t)

[ 82V 82V]

8 J = 210 [8x(t) 8x(t)] ::rx ~:'; * [8X(t) ] dt 1 rtf . [8X(t)]

= "2 ltD [8x(t) 8x(t)]II 8x(t) dt (2.4.5)

40 Chapter 2: Calculus of Variations and Optimal Control

where,

(2.4.6)

If the matrix II in the previous equation is positive (negative) definite, we establish a minimum (maximum). In many cases since 8x(t) is arbitrary, the coefficient of (8x(t))2, i.e., 8 2V /8x 2 determines the sign of 82 J. That is, the sign of second variation agrees with the sign of 8 2V / 8x2. Thus, for minimization requirement

(2.4.7)

For maximization, the sign of the previous equation reverses. In the literature, this condition is called Legendre condition [126].

In 1786, Legendre obtained this result of deciding whether a given optimum is maximum or minimum by examining the second variation. The second variation technique was fur- ther generalized by Jacobi in 1836 and hence this condition is usually called Legendre-Jacobi condition.

Example 2.9

Verify that the straight line represents the minimum distance be- tween two points.

Solution: This is an obvious solution, however, we illustrate the second variation. Earlier in Example 2.7, we have formulated a functional for the distance between two points as

(2.4.8)

and found that the optimum is a straight line x*(t) = Clt + C2. To satisfy the sufficiency condition (2.4.7), we find

x*(t) 1 3/2

. (2.4.9) [1+x*2(t)]

Since x*(t) is a constant (+ve or -ve) , the previous equation satisfies the condition (2.4.7). Hence, the distance between two points as given by x*(t) (straight line) is minimum.

2.5 Extrema of Functions with Conditions 41

Next, we begin the second stage of optimal control. We consider op- timization (or extremization) of a functional with a plant, which is considered as a constraint or a condition along with the functional. In other words, we address the extremization of a functional with some condition, which is in the form of a plant equation. The plant takes the form of state equation leading us to optimal control of dynamic systems. This section is motivated by [6, 79, 120, 108].

2.5 Extrema of Functions with Conditions We begin with an example of finding the extrema of a function under a condition (or constraint). We solve this example with two methods, first by direct method and then by Lagrange multiplier method. Let us note that we consider this simple example only to illustrate some basic concepts associated with conditional extremization [120].

Example 2.10

A manufacturer wants to maximize the volume of the material stored in a circular tank subject to the condition that the mate- rial used for the tank is limited (constant). Thus, for a constant thickness of the material, the manufacturer wants to minimize the volume of the material used and hence part of the cost for the tank.

Solution: If a fixed metal thickness is assumed, this condition im- plies that the cross-sectional area of the tank material is constant. Let d and h be the diameter and the height of the circular tank. Then the volume contained by the tank is

V(d, h) = wd2 h/4 (2.5.1) and the cross-sectional surface area (upper, lower and side) of the tank is

A(d, h) = 2wd2 /4 + wdh = Ao. (2.5.2) Our intent is to maximize V(d, h) keeping A(d, h) = Ao, where Ao is a given constant. We discuss two methods: first one is called the Direct Method using simple calculus and the second one is called Lagrange Multiplier Method using the Lagrange multiplier method.

1 Direct Method: In solving for the optimum value directly, we eliminate one of the variables, say h, from the volume relation (2.5.1) using the area relation (2.5.2). By doing so, the condition is embedded in the original function to be extremized. From (2.5.2),

h = Ao - wd2 /2

7rd . (2.5.3)

42 Chapter 2: Calculus of Variations and Optimal Control

Using the relation (2.5.3) for height in the relation (2.5.1) for vol- ume

(2.5.4)

Now, to find the extrema of this simple calculus problem, we dif- ferentiate (2.5.4) w.r.t. d and set it to zero to get

~o _ ~7rd2 = O. (2.5.5)

Solving, we get the optimal value of d as

d* = J2Ao . 37r

(2.5.6)

By demanding that as per the Definition 2.3 for optimum of a function, the second derivative of V w.r.t. d in (2.5.4) be neg- ative for maximum, we can easily see that the positive value of the square root function corresponds to the maximum value of the function. Substituting the optimal value of the diameter (2.5.6) in the original cross-sectional area given by (2.5.2), and solving for the optimum h *, we get

h* = J2AO • 37r

(2.5.7)

Thus, we see from (2.5.6) and (2.5.7) that the volume stored by a tank is maximized if the height of the tank is made equal to its diameter.

2 Lagrange Multiplier Method: Now we solve the above prob- lem by applying Lagrange multiplier method. We form a new func- tion to be extremized by adjoining a given condition to the original function. The new adjoined function is extremized in the normal way by taking the partial derivatives w.r. t. all its variables, making them equal to zero, and solving for these variables which are ex- tremals. Let the original volume relation (2.5.1) to be extremized be rewritten as

f(d, h) = 7rd2 h/4

and the condition (2.5.2) to be satisfied as

g(d, h) = 27rd2/4 + 7rdh - Ao = O.

(2.5.8)

(2.5.9)

Then a new adjoint function £, (called Lagrangian) is formed as

£'(d, h, -\) = f(d, h) + -\g(d, h) = 7rd2h/4 + -\(27rd2/4 + 7rdh - Ao) (2.5.10)

----------------------------------

2.5 Extrema of Functions with Conditions 43

where, A, a parameter yet to be determined, is called the Lagrange multiplier. Now, since the Lagrangian C is a function of three optimal variables d, h, and A, we take the partial derivatives of £(d, h, A) w.r.t. each of the variables d, h and A and set them to zero. Thus,

ac ad = 7rdh/2 + A(7rd + 7rh) = 0 ac ah = 7rd2 /4 + A(7rd) = 0 ac 2 aA = 27rd /4 + 7rdh - Ao = o.

(2.5.11 )

(2.5.12)

(2.5.13)

Now, solving the previous three relations (2.5.11) to (2.5.13) for the three variables d*, h *, and A *, we get

d* = J2AO; h* = J2AO ; ,X* = -J Ao . 37r 37r 247r

(2.5.14)

Once again, to maximize the volume of a cylindrical tank, we need to have the height (h *) equal to the diameter (d*) of the tank. Note that we need to take the negative value of the square root function for A in (2.5.14) in order to satisfy the physical requirement that the diameter d obtained from (2.5.12) as

d = -4A (2.5.15) is a positive value.

Now, we generalize the previous two methods.

2.5.1 Direct Method

N ow we generalize the preceding method of elimination using differen- tial calculus. Consider the extrema of a function f(XI, X2) with two interdependent variables Xl and X2, subject to the condition

(2.5.16)

As a necessary condition for extrema, we have

af af df = -a dXI + -a dX2 = o.

Xl X2 (2.5.17)

However, since dXI and dX2 are not arbitrary, but related by the con- dition

ag ag dg = -a dXI + -a dX2 = 0,

Xl X2 (2.5.18)

44 Chapter 2: Calculus of Variations and Optimal Control

it is not possible to conclude as in the case of extremization of functions without conditions that

and (2.5.19)

in the necessary condition (2.5.17). This is easily seen, since if the set of extrema conditions (2.5.19) is solved for optimal values xi and x2' there is no guarantee that these optimal values, would, in general satisfy the given condition (2.5.16).

In order to find optimal values that satisfy both the condition (2.5.16) and that of the extrema conditions (2.5.17), we arbitrarily choose one of the variables, say Xl, as the independent variable. Then X2 becomes a dependent variable as per the condition (2.5.16). Now, assuming that 8g/8x2 i- 0, (2.5.18) becomes

{ 8g/ 8XI}

dX2 = - 8g/8x 2 dXI (2.5.20)

and using (2.5.20) in the necessary condition (2.5.17), we have

(2.5.21)

As we have chosen dXI to be the independent, we now can consider it to be arbitrary, and conclude that in order to satisfy (2.5.21), we have the coefficient of dXI to be zero. That is

(%:J (%:J -(%:J (%:J = o. (2.5.22) Now, the relation (2.5.22) and the condition (2.5.16) are solved simul- taneously for the optimal solutions xi and x2' Equation (2.5.22) can be rewritten as

=0. (2.5.23)

This is also, as we know, the Jacobian of f and 9 w.r.t. Xl and X2. This method of elimination of the dependent variables is quite tedious for higher order problems.

2.5 Extrema of Functions with Conditions 45

2.5.2 Lagrange Multiplier Method

We now generalize the second method of solving the same problem of extrema of functions with conditions. Consider again the extrema of the function f(Xl, X2) subject to the condition

(2.5.24)

In this method, we form an augmented Lagrangian function

(2.5.25)

where, A, a parameter (multiplier) yet to be determined, is the La- grange multiplier. Let us note that using the given condition (2.5.24) in the Lagrangian (2.5.25), we have

(2.5.26)

and therefore a necessary condition for extrema is that

df = d£ = O. (2.5.27)

Accepting the idea that the Lagrangian (2.5.25) is a better represen- tation of the entire problem than the equation (2.5.26) in finding the extrema, we have from the Lagrangian relation (2.5.25)

d£ = df + Adg = O. (2.5.28)

Using (2.5.17) and (2.5.18) in (2.5.28), and rearranging

[ 8 f 8g 1 [ 8 f 89 1 8Xl + A 8Xl dXl + 8X2 + A 8X2 dX2 = O. (2.5.29)

Now dXl and dX2 are both not independent and hence cannot immedi- ately conclude that each of the coefficients of dXl and dX2 in (2.5.29) must be zero. Let us choose dXl to be independent differential and then dX2 becomes a dependent differential as per (2.5.18). Further, let us choose the multiplier A, which has been introduced by us and is at our disposal, to make one of the coefficients of dXl or dX2 in (2.5.29) zero. For example, let A take on the value A * that makes the coefficient of the dependent differential dX2 equal zero, that is

(2.5.30)

46 Chapter 2: Calculus of Variations and Optimal Control

With (2.5.30), the equation (2.5.29) reduces to

(2.5.31)

Since, dXl is the independent differential, it can be varied arbitrarily. Hence, for (2.5.31) to be satisfied for all dxl, the coefficient of dXl must be zero. That is

oj +A* og =0. OXl OXl

(2.5.32)

Now from (2.5.25), note that

(2.5.33)

yields the constraint relation (2.5.16). Combining the results from (2.5.32), (2.5.30), and (2.5.33), we have

o£ = oj + A* 8g = 0 OXl OXl OXl

(2.5.34)

o£ = oj + A* og = 0 OX2 OX2 8X2

(2.5.35)

o£ (* *) OA = 9 xl,x2 = o. (2.5.36)

The preceding three equations are to be solved simultaneously to obtain xi, x2' and A*. By eliminating A* between (2.5.34) and (2.5.35)

(::J (::J -(::J (::J = 0 (2.5.37) which is the same condition as (2.5.22) obtained by the direct method, thus indicating that we have the same result by Lagrange multiplier method.

Let us note that the necessary conditions (2.5.34) and (2.5.35) are just the same equations which would have been obtained from consid- ering the differentials dXl and dX2 as though they were independent in (2.5.29). Introduction of the multiplier A has allowed us to treat all the variables in the augmented function £(Xl' X2, A) as though each variable is independent. Thus, the multiplier A has acted like a catalyst, appearing in the intermediate stage only.

2.5 Extrema of Functions with Conditions 47

Summarizing, the extrema of a function f(xl, X2) subject to the con- dition (or constraint) g(XI' X2) = 0 is equivalent to extrema of a single augmented function £(XI, X2, A) = f(xl, X2) + Ag(xl, X2) as though Xl, X2 and A are independent. We now generalize this result.

THEOREM 2.2 Consider the extrema of a continuous, real-valued function f(x) f(xl, X2, ... ,xn ) subject to the conditions

gl(X) = gl(XI,X2,··· ,xn ) = 0 g2(X) = g2(XI, X2, ... ,xn ) = 0

(2.5.38)

where, f and g have continuous partial derivatives, and m < n. Let AI, A2, ... ,Am be the Lagrange multipliers corresponding to m conditions, such that the augmented Lagrangian function is formed as

£(x, A) = f(x) + A'g(X), (2.5.39)

where, A' is the transpose of A. Then, the optimal values x* and A * are the solutions of the following n + m equations

a £ = a f + A' ag = 0 ax ax ax (2.5.40) a£ aA = g(x) = o. (2.5.41 )

Features of Lagrange Multiplier: The Lagrange multiplier method is a powerful one in finding the extrema of functions subject to condi- tions. It has the following attractive features:

1. The importance of the Lagrange multiplier technique lies on the fact that the problem of determining the extrema of the function f(x) subject to the conditions g(x) = 0 is embedded within the simple problem of determining the extrema of the simple aug- mented function £(x, A) = f(x) + A'g(X).

2. Introduction of Lagrange multiplier allows us to treat all the vari- ables x and A in the augmented function £(x, A) as though each were independent.

48 Chapter 2: Calculus of Variations and Optimal Control

3. The multiplier A acts like a catalyst in the sense that it is intro- duced to perform a certain duty as given by item 2.

4. The increased dimensionality (n + m) which is characteristic of the Lagrange multiplier method, is generally more than compen- sated by the relative simplicity and systematic procedure of the technique.

The multiplier method was given by Lagrange in 1788.

2.6 Extrema of Functionals with Conditions In this section, we extend our ideas to functionals based on those de- veloped in the last section for functions. First, we consider a functional with two variables, use the results of the previous section on the Co V, derive the necessary conditions and then extend the same for a general nth order vector case. Consider the extremization of the performance index in the form of a functional

subject to the condition (plant or system equation)

with fixed-end-point conditions

XI(tO)=XlO;

XI(tj) = Xlj; X2(tO) = X20 X2(tj) = X2j'

Now we address this problem under the following steps.

• Step 1: Lagrangian

• Step 2: Variations and Increment

• Step 3: First Variation

• Step 4: Fundamental Theorem

• Step 5: Fundamental Lemma

• Step 6: Euler-Lagrange Equation

(2.6.2)

(2.6.3)

2.6 Extrema of Punctionals with Conditions 49

• Step 1: Lagrangian: We form an augmented functional

(2.6.4)

where, A(t) is the Lagrange multiplier, and the Lagrangian £, is defined as

£, = £'(Xl(t),X2(t),Xl(t),X2(t),A(t),t)

= V(Xl(t), X2(t), Xl(t), X2(t), t) +A(t)g(Xl(t), X2(t), Xl(t), X2(t)) (2.6.5)

Note from the performance index (2.6.1) and the augmented per- formance index (2.6.4) that Ja = J if the condition (2.6.2) is satisfied for any A(t).

• Step 2: Variations and Increment: Next, assume optimal values and then consider the variations and increment as

Xi(t) = xi(t) + 8Xi(t), Xi(t) = xi(t) + 8Xi(t), i = 1,2 ~Ja = Ja(xi(t) + 8Xi(t), xi(t) + 8Xi(t), t) - Ja(xi(t), xi(t), t),

(2.6.6)

for i = 1,2.

• Step 3: First Variation: Then using the Taylor series expansion and retaining linear terms only, the first variation of the func- tional J a becomes

(2.6.7)

As before in the section on Co V, we rewrite the terms containing 8Xl(t) and 8X2(t) in terms of those containing 8Xl(t) and 8X2(t)

50 Chapter 2: Calculus of Variations and Optimal Control

only (using integration by parts, J udv = uv - J vdu). Thus

Using the above, we have the first variation (2.6.7) as

Since this is a fixed-final time and fixed-final state problem as given by (2.6.3), no variations are allowed at the final point. This means

(2.6.10)

Using the boundary variations (2.6.10) in the augmented first variation (2.6.9), we have

Ma= 1:1 [(:~). -! (:~)J8Xl(t)dt + {I [(:~). _ ! (:~) J 8X2(t)dt. (2.6.11)

• Step 4: Fundamental Theorem: Now, we proceed as follows.

1. We invoke the fundamental theorem of the calculus of vari- ations (Theorem 2.1) and make the first variation (2.6.11) equal to zero.

2.6 Extrema of Functionals with Conditions 51

2. Remembering that both 8Xl(t) and 8X2(t) are not indepen- dent, because Xl(t) and X2(t) are related by the condition (2.6.2), we choose 8X2(t) as the independent variation and 8Xl(t) as the dependent variation.

3. Let us choose the multiplier A*(t) which is arbitrarily intro- duced and is at our disposal, in such a way that the coef- ficient of the dependent variation 8Xl(t) in (2.6.11) vanish. That is

( 8C) _ ~ ( 8C) = 0 8Xl * dt 8Xl * . (2.6.12) With these choices, the first variation (2.6.11) becomes

f [(:~). -! (:~) J 8X2(t)dt = O. (2.6.13) • Step 5: Fundamental Lemma: Using the fundamental lemma of

CoY (Lemma 2.1) and noting that since 8X2(t) has been chosen to be independent variation and hence arbitrary, the only way (2.6.13) can be satisfied, in general, is that the coefficient of 8Xl(t) also vanish. That is

( 8C) _ ~ ( 8C) = 0 8X2 * dt 8X2 * . (2.6.14) Also, from the Lagrangian(2.6.5) note that

(8C) = 0 8A *

(2.6.15)

yields the constraint relation (2.6.2) .

• Step 6: Euler-Lagrange Equation: Combining the various rela- tions (2.6.12), (2.6.14), and (2.6.15), the necessary conditions for extremization of the functional (2.6.1) subject to the condition (2.6.2) (according to Euler-Lagrange equation) are

( 8C) _ ~ ( 8C) = 0 8Xl * dt 8Xl * (2.6.16) ( 8C) _ ~ ( 8C) = 0 8X2 * dt 8X2 * (2.6.17) (8C) _ ~ (8C) = 0 8A * dt 8.\ * . (2.6.18)

52 Chapter 2: Calculus of Variations and Optimal Control

Let us note that these conditions are just the ones that would have been obtained from the Lagrangian (2.6.5), as if both 8Xl(t) and 8X2(t) had been independent. Also, in (2.6.18), the Lagrangian £ is independent of ~(t) and hence the condition (2.6.18) is really the given plant equation (2.6.2).

Thus, the introduction of the Lagrange multiplier A( t) has en- abled us to treat the variables Xl(t) and X2(t) as though they were independent, in spite of the fact that they are related by the condition (2.6.2). The solution of the two, second-order differ- ential equations (2.6.16) and (2.6.17) and the condition relation (2.6.2) or (2.6.18) along with the boundary conditions (2.6.3) give the optimal solutions xi(t), x2(t), and A*(t).

Now, we generalize the preceding procedure for an nth order system. Consider the extremization of a functional

i t!

J = V(x(t), x(t), t)dt to

(2.6.19)

where, x(t) is an nth order state vector, subject to the plant equation ( condition)

gi(X(t), x(t), t) = 0; i = 1,2, ... ,m (2.6.20)

and boundary conditions, x(O) and x(tf). We form an augmented functional

i t!

Ja = £(x(t), x(t), A(t), t)dt to

(2.6.21)

where, the Lagrangian £ is given by

I£(x(t), x(t), A(t), t) = V(x(t), x(t), t) + A'(t)gi(X(t), x(t), t) I (2.6.22)

and the Lagrange multiplier A(t) = [Al(t), A2(t), ... , Am(t)]'. We now apply the Euler-Lagrange equation on Ja to yield

(8£) _ ~ (8£) = 0 8x * dt 8x * ' (2.6.23)

( 8£) d (8£) . ~ * - di 8I * = 0 ~ 9i(X(t), x(t), t) = o. (2.6.24)

2.6 Extrema of Functionals with Conditions 53

Note that from (2.6.22), the Lagrangian £ is independent of ~(t) and hence the Euler-Lagrange equation (2.6.24) is nothing but the given relation regarding the plant or the system (2.6.20). Thus, we solve the Euler-Lagrange equation (2.6.23) along with the given boundary conditions. Let us now illustrate the preceding method by a simple example.

Example 2.11

Minimize the performance index

(2.6.25)

with boundary conditions

x(O) = 1; x(l) = 0 (2.6.26)

subject to the condition (plant equation)

x(t) = -x(t) + u(t). (2.6.27)

Solution: Let us solve this problem by the two methods, i.e., the direct method and the Lagrange multiplier method.

1 Direct Method: Here, we eliminate u(t) between the perfor- mance index (2.6.25) and the plant (2.6.27) to get the functional as

J = t [x2(t) + (x(t) + x(t))2]dt = t [2x2(t) + x2(t) + 2x(t)x(t)]dt. (2.6.28)

Now, we notice that the functional (2.6.28) absorbed the condition (2.6.27) within itself, and we need to consider it as a straight for- ward extremization of a functional as given earlier. Thus, applying the Euler-Lagrange equation

(OV) _ ~ (OV) = 0 ax * dt ax * (2.6.29) to the functional (2.6.28), where,

(2.6.30)

54 Chapter 2: Calculus of Variations and Optimal Control

we get

4x*(t) + 2x*(t) - :t (2x*(t) + 2x*(t)) = O. (2.6.31)

Simplifying the above

x*(t) - 2x*(t) = 0 (2.6.32)

the solution (see later for use of MATLAB©) of which gives the optimal as

(2.6.33)

where, the constants C1 and C2, evaluated using the given bound- ary conditions (2.6.26), are found to be

(2.6.34)

Finally, knowing the optimal x* ( t ), the optimal control u * ( t ) is found from the plant (2.6.27) to be

u*(t) = x*(t) + x*(t) = C1 (1 - V2)e- V2t + C2 (1 + V2)e V2t . (2.6.35)

Although the method appears to be simple, let us note that it is not always possible to eliminate u(t) from (2.6.25) and (2.6.27) especially for higher-order systems.

2 Lagrange Multiplier Method: Here, we use the ideas devel- oped in the previous section on the extremization of functions with conditions. Consider the optimization of the functional (2.6.25) with the boundary conditions (2.6.26) under the condition describ- ing the plant (2.6.27). First we rewrite the condition (2.6.27) as

g(x(t), x(t), u(t)) = x(t) + x(t) - u(t) = o. (2.6.36) Now, we form an augmented functional as

J = l [x2(t) + u2(t) + A(t){X(t) + x(t) - u(t)} 1 dt = l C(x(t), x(t), u(t), A(t))dt (2.6.37)

where, A(t) is the Lagrange multiplier, and

C(x(t), x(t), u(t), A(t)) = x2 (t) + u2 (t) +A(t) {x(t) + x(t) - u(t)} (2.6.38)

2.6 Extrema of Functionals with Conditions 55

is the Lagrangian. Now, we apply the Euler-Lagrange equation to the previous Lagrangian to get

(~~) * - :t (~~) * = 0 -----> 2x*(t) + ). *(t) - ~ *(t) = 0 (2.6.39) (~~) * - :t (~) * = 0 -----> 2u*(t) - ).*(t) = 0 (2.6.40) (~~) * - :t (!f) * = 0 -----> x*(t) + x*(t) - u*(t) = 0 (2.6.41)

and solve for optimal x*(t), u*(t), and A*(t). We get first from (2.6.40) and (2.6.41)

A*(t) = 2u*(t) = 2(x*(t) + x*(t)). (2.6.42)

Using the equation (2.6.42) in (2.6.39)

2x*(t) + 2(x*(t) + x*(t)) - 2(x*(t) + x*(t)) = O. (2.6.43) Solving the previous equation, we get

x*(t) - 2x*(t) = 0 -----+ x*(t) = C1e-v'2t + C2ev'2t. (2.6.44)

Once we know x*(t), we get A*(t) and hence u*(t) from (2.6.42) as

u* (t) = i;* (t) + x* (t) = C1 (1 - V2)e-v'2t + C2(1 + V2)ev'2t. (2.6.45)

Thus, we get the same results as in direct method. The constants C1 and C2, evaluated using the boundary conditions (2.6.26) are the same as given in (2.6.34).

The solution for the set of differential equations {2.6.32} with the boundary conditions (g. 6. 26) for Example 2.11 using Symbolic Toolbox of the MATLAIi9, Version 6, is shown below.

56 Chapter 2: Calculus of Variations and Optimal Control

******************************************************** x=dsolve('D2x-2*x=O','x(O)=1,x(1)=O')

x =

-(exp(2 A(1/2))A2+1)/(exp(2 A(1/2))A2-1)*sinh(2 A(1/2)*t)+ cosh(2A(1/2)*t)

1/2 2 (exp(2 )

1/2 + 1) sinh(2 t) 1/2

- ----------------------------- + cosh(2 t)

u =

1/2 2 exp(2 ) - 1

-(exp(2A(1/2))A2+1)/(exp(2A(1/2))A2-1)*cosh(2A(1/2)*t)*2 A(1/2)+ sinh(2 A(1/2)*t)*2 A(1/2)-(exp(2 A(1/2))A2+1)/(exp(2 A(1/2))A2_ 1)*sinh(2A(1/2)*t)+cosh(2A(1/2)*t)

1/2 2 (exp(2 )

1/2 1/2 + 1) cosh(2 t) 2

1/2 2 exp(2 ) - 1

1/2

1/2 1/2 + sinh(2 t) 2

1/2 2 (exp(2 ) + 1) sinh(2 t) 1/2

- ----------------------------- + cosh(2 t) 1/2 2

exp(2 ) - 1 *********************************************************

It is easy to see that the previous solution for optimal x*(t) is the same as given in (2.6.33) and (2.6.34).

Let us note once again that the Lagrange multiplier ..\(t) helped us to treat the augmented functional (2.6.38) as if it contained independent variables x(t) and u(t), although they are dependent as per the plant equation (2.6.36).

2.7 Variational Approach to Optimal Control Systems 57

2.7 Variational Approach to Optimal Control Systems

In this section, we approach the optimal control system by variational techniques, and in the process introduce the Hamiltonian function, which was used by Pontryagin and his associates to develop the fa- mous Minimum Principle [109].

2.7.1 Terminal Cost Problem

Here we consider the optimal control system where the performance index is of general form containing a final (terminal) cost function in addition to the integral cost function. Such an optimal control problem is called the Bolza problem. Consider the plant as

x(t) = f(x(t), u(t), t), (2.7.1)

the performance index as

i tf

J(u(t)) = S(x(tf), tf) + V(x(t), u(t), t)dt to

(2.7.2)

and given boundary conditions as

x(to) = xo; x( t f) is free and t f is free (2.7.3) where, x(t) and u(t) are n- and r- dimensional state and control vectors respectively. This problem of Bolza is the one with the most general form of the performance index.

The Lagrange problem was first discussed in 1762, Mayer considered his problem in 1878, and the problem of Bolza was formulated in 1913.

Before we begin illustrating the Pontryagin procedure for this problem, let us note that

rtf dS(:(t) , t) dt = S(x(t), t)I~~ = S(x(tf), tf) - S(x(to), to). (2.7.4) lto t

Using the equation (2.7.4) in the original performance index (2.7.2), we get

J2(U(t» = 1:' [V(x(t), u(t), t) + ~~] dt = rf V(x(t), u(t), t)dt + S(x(tf), tf) - S(x(to), to). (2.7.5)

lto

58 Chapter 2: Calculus of Variations and Optimal Control

Since S(x(to), to) is a fixed quantity, the optimization of the original performance index J in (2.7.2) is equivalent to that of the performance index J2 in (2.7.5). However, the optimal cost given by (2.7.2) is dif- ferent from the optimal cost (2.7.5). Here, we are interested in finding the optimal control only. Once the optimal control is determined, the optimal cost is found using the original performance index J in (2.7.2) and not J2 in (2.7.5). Also note that

d[S(x(t), t)] = (as)' . () as dt ax x t + at' (2.7.6)

We now illustrate the procedure in the following steps. Also, we first introduce the Lagrangian and then, a little later, introduce the Hamil- tonian. Let us first list the various steps and then describe the same in detail.

• Step 1: Assumption of Optimal Conditions

• Step 2: Variations of Control and State Vectors

• Step 3: Lagrange Multiplier

• Step 4: Lagrangian

• Step 5: First Variation

• Step 6: Condition for Extrema

• Step 7 : Hamiltonian

• Step 1: Assumptions of Optimal Conditions: We assume opti- mum values x*(t) and u*(t) for state and control, respectively. Then

J(u*(t)) = 1:1 [V(x*(t), u*(t), t) + dS(x;?), t)] dt ic*(t) = f(x*(t), u*(t), t). (2.7.7)

• Step 2: Variations of Controls and States: We consider the variations (perturbations) in control and state vectors as (see Figure 2.7)

x(t) = x*(t) + 8x(t); u(t) = u*(t) + 8u(t). (2.7.8)

2.7 Variational Approach to Optimal Control Systems 59

x(t)

x*(t)+ ox(t)

~ __ .......... : ...... : ... ~ ...... T ..

Figure 2.7 Free-Final Time and Free-Final State System

Then, the state equation (2.7.1) and the performance index (2.7.5) ': become

x*(t) + 6x(t) = f(x*(t) + 6x(t), u*(t) + 6u(t), t)

[tf+ 6t

f [ dS] J(u(t)) = ito V(x*(t) + 6x(t), u*(t) + 6u(t), t) + dt dt

(2.7.9)

• Step 3: Lagrange Multiplier: Introducing the Lagrange multi- plier vector .x(t) (also called costate vector) and using (2.7.6), we introduce the augmented performance index at the optimal condition as

[4 (8S)' (8S) Ja(u*(t)) = lto [V(x*(t), u*(t), t) + 8x * x*(t) + 8t * +.x'(t) {f(x*(t), u*(t), t) - x*(t)}]dt (2.7.10)

60 Chapter 2: Calculus of Variations and Optimal Control

and at any other (perturbed) condition as

l tf+8tf

Ja(u(t)) = [V(x*(t) + 8x(t), u*(t) + 8u(t), t) to

(8S)' (8S) + ax * [x*(t) + 8x(t)] + at * +A'(t) [f(x*(t) + 8x(t), u*(t) + 8u(t), t) - {x*(t) + 8x(t)}]]dt. (2.7.11)

• Step 4: Lagrangian: Let us define the Lagrangian function at optimal condition as

£ = £(x*(t), x*(t), u*(t), A(t), t)

= V(x*(t), u*(t),t) + (~:)~ x*(t) + c;: +A'(t) {f(x*(t), u*(t), t) - x*(t)} (2.7.12)

and at any other condition as

£8 = £8(x*(t) +8x(t),x*(t) +8x(t),u*(t) +8U(t),A(t),t) = V(x*(t) + 8x(t), u*(t) + 8u(t), t)

+ (:) ~ [x*(t) + 8x(t)] + (c;:) * +A'(t) [f(x*(t) + 8x(t), u*(t) + 8u(t), t) - {x*(t) + 8x(t)}] . (2.7.13)

With these, the augmented performance index at the optimal and any other condition becomes

2.7 Variational Approach to Optimal Control Systems 61

Using mean-value theorem and Taylor series, and retaining the linear terms only, we have

(2.7.15)

• Step 5: First Variation: Defining increment fl.J, using Taylor series expansion, extracting the first variation 8 J by retaining only the first order terms, we get the first variation as

~J = Ja(u(t)) - Ja(u*(t))

it' = (£6 - £)dt + £It 8tf to I {t I { (a £ ) I (a £) I (a £ ) I }

8J = ito ax * 8x(t) + ax * 8x(t) + au * 8u(t) dt + £It, 8tf. (2.7.16)

Considering the 8x(t) term in the first variation (2.7.16) and in- tegrating by parts (using J udv = uv - J vdu),

f (~): 6x(t)dt = {' (~;): :t (6x(t» dt = [(~»X(t)JC

- {' [:t (~) J 6x(t)dt. (2.7.17)

62 Chapter 2: Calculus of Variations and Optimal Control

Also note that since x(to) is specified, 8x(to) = O. Thus, using (2.7.17) the first variation 8J in (2.7.16) becomes

oj = 1:' [(~). -:t (~) J ox(t)dt + f (~)~ ou(t)dt + Cit! ot, + [(~»X(t)] L (2.7.18)

• Step 6: Condition for Extrema: For extrema of the functional J, the first variation 8 J should vanish according to the fundamental theorem (Theorem 2.1) of the CoV. Also, in a typical control system such as (2.7.1), we note that 8u(t) is the independent control variation and 8x(t) is the dependent state variation. First, we choose A(t) = A*(t) which is at our disposal and hence £* such that the coefficient of the dependent variation 8x(t) in (2.7.18) be zero. Then, we have the Euler-Lagrange equation

(8£) _!i (8£) = 0 8x * dt 8x * (2.7.19) where the partials are evaluated at the optimal (*) condition. Next, since the independent control variation 8u(t) is arbitrary, the coefficient of the control variation 8u(t) in (2.7.18) should be set to zero. That is

(8£) = O. 8u * (2.7.20) Finally, the first variation (2.7.18) reduces to

(2.7.21)

Let us note that the condition (or plant) equation (2.7.1) can be written in terms of the Lagrangian (2.7.12) as

(8£) = O. 8A *

(2.7.22)

2.7 Variational Approach to Optimal Control Systems 63

X(t)

tangent at P . . . : .............. . .........•...... . ~.

X f

........... .

. ... : .. 9(t)

. .: <>t .... . f

o t

Figure 2.8 Final-Point Condition with a Moving Boundary B(t)

In order to convert the expression containing 8x(t) in (2.7.21) into an expression containing 8xf (see Figure 2.8), we note that the slope of x*(t) + 8x(t) at tf is aI?proximated as

(2.7.23)

which is rewritten as

8Xf = 8x(tf) + {x*(t) + 8x(t)} 8tf (2.7.24) and retaining only the linear (in 8) terms in the relation (2.7.24), we have

(2.7.25)

Using (2.7.25) in the boundary condition (2.7.21), we have the general boundary condition in terms of the Lagrangian as

(2.7.26)

• Step 7: Hamiltonian: We define the Hamiltonian 1l* (also called the Pontryagin 1l function) at the optimal condition as

11l* = V(x*(t), u*(t), t) + A*' (t)f(x*(t), u*(t), t), I (2.7.27)

64 Chapter 2: Calculus of Variations and Optimal Control

where,

'H* = 'H* (x* (t), u* (t), A * (t), t).

Then from (2.7.12) the Lagrangian £* in terms of the Hamiltonian 'H* becomes

£* = £*(x*(t), x*(t), u*(t), A*(t), t) = 'H*(x*(t), u*(t), A*(t), t)

+ (~:) ~ x*(t) + (~~) * - >.*' (t)x*(t). (2.7.28)

Using (2.7.28) in (2.7.20), (2.7.19), and (2.7.22) and noting that the terminal cost function S = S (x( t), t), we have the control, state and costate equations, respectively expressed in terms of the Hamiltonian. Thus, for the optimal control u*(t), the relation (2.7.20) becomes

(2.7.29)

for the optimal state x* ( t), the relation (2.7.19) becomes

(8£) _ ~ (8£*) = 0 ~ 8x* dt 8x* (81t) (828)' (8

28) d {(88)' } - + - x*(t)+ - -- - -A*(t) =O~ 8x * &2 8x8t dt 8x * * *

(81t) (8 28)' ( 828) [(828)' ( 828) . * 1 8x * + 8x2 * x*(t) + 8x8t * - 8x2 * x*(t) + 8x8t * - A (t) = 0

leading to

( 8'H) = -'\*(t) ax * (2.7.30)

and for the costate A * ( t),

(8£) = 0 -----+ (8'H) = X*(t). 8A * 8A *

(2.7.31)

Looking at the similar structure of the relation (2.7.30) for the optimal costate A*(t) and (2.7.31) for the optimal state x*(t) it

2.7 Variational Approach to Optimal Control Systems 65

is clear why .x(t) is called the costate vector. Finally, using the relation (2.7.28), the boundary condition (2.7.26) at the optimal condition reduces to

(2.7.32)

This is the general boundary condition for free-end point system in terms of the Hamiltonian.

2. 7.2 Different Types of Systems

We now obtain different cases depending on the statement of the prob- lem regarding the final time tf and the final state x(tf) (see Figure 2.9).

• Type (a): Fixed-Final Time and Fixed-Final State System: Here, since tf and x(tf) are fixed or specified (Figure 2.9(a», both 8tf and 8xf are zero in the general boundary condition (2.7.32), and there is no extra boundary condition to be used other than those given in the problem formulation.

• Type (b): Free-Final Time and Fix ed-Final State System: Since tf is free or not specified in advance, 8tf is arbitrary, and since X(tf) is fixed or specified, 8xf is zero as shown in Figure 2.9(b). Then, the coefficient of the arbitrary 8t f in the general boundary condition (2.7.32) is zero resulting in

(2.7.33)

• Type (c): Fixed-Final Time and Free-Final State System: Here tf is specified and x(tf) is free (see Figure 2.9(c». Then 8tf is zero and 8x f is arbitrary, which in turn means that the coefficient of oXf in the general boundary condition (2.7.32) is zero. That is

( as _ -x*(t») = 0 ----+ -X*(tf) = (as) ax *tfax *tf

(2.7.34)

66 Chapter 2: Calculus of Variations and Optimal Control

x(t) x*(t)+ox(t)

xo ..

(a)

x(t)

xo ..

o to t (c)

x(t)

Xo ...

o to

(b)

it (d)

Figure 2.9 Different Types of Systems: (a) Fixed-Final Time and Fixed-Final State System, (b) Free-Final Time and Fixed-Final State

System, (c) Fixed-Final Time and Free-Final State System, (d) Free-Final Time and Free-Final State System

2.7 Variational Approach to Optimal Control Systems 67

• Type (d): Free-Final Time and Dependent Free-Final State Sys- tem: If t f and x( t f) are related such that x( t f) lies on a moving curve 8(t) as shown in Figure 2.8, then

(2.7.35)

Using (2.7.35), the boundary condition (2.7.32) for the optimal condition becomes

Since t f is free, 8t f is arbitrary and hence the coefficient of 8t f in (2.7.36) is zero. That is

• Type (e): Free-Final Time and Independent Free-Final State: If t f and x( t f) are not related, then 8t f and 8x f are unrelated, and the boundary condition (2.7.32) at the optimal condition becomes

(2.7.38)

(2.7.39)

2.7.3 Sufficient Condition

In order to determine the nature of optimization, i.e., whether it is minimum or maximum, we need to consider the second variation and examine its sign. In other words, we have to find a sufficient condition for extremum. Using (2.7.14), (2.7.28) and (2.7.37), we have the second

68 Chapter 2: Calculus of Variations and Optimal Control

variation in (2.7.16) and using the relation (2.7.28), we get

For the minimum, the second variation 82 J must be positive. This means that the matrix II in (2.7.40)

(2.7.41)

must be positive definite. But the important condition is that the second partial derivative of 1t* w.r.t. u(t) must be positive. That is

(2.7.42)

and for the maximum, the sign of (2.7.42) is reversed.

2. 7.4 Summary of Pontryagin Procedure Consider a free-final time and free-final state problem with general cost function (Bolza problem), where we want to minimize the performance index

i t!

J = S(x(tf), tf) + V(x(t), u(t), t)dt to

(2.7.43)

for the plant described by

x(t) = f(x(t), u(t), t) (2.7.44)

with the boundary conditions as

x(t = to) = Xo; t = tf is free and x(tf) is free. (2.7.45)

2.7 Variational Approach to Optimal Control Systems

Table 2.1 Procedure Summary of Pontryagin Principle for Bolza Problem

A. Statement of the Problem Given the plant as x(t) = f(x(t), u(t), t), the performance index as

J = S(x(tf), tf) + Jt~ V(x(t), u(t), t)dt, and the boundary conditions as x(to) = Xo and tf and x(tf) = xf are free, find the optimal control.

B. Solution of the Problem Step 1 Form the Pontryagin 1i function

1i(x(t), u(t), A(t), t) = V(x(t), u(t), t) + A' (t)f(x(t), u(t), t). Step 2 Minimize 1i w.r.t. u(t)

(~~t = 0 and obtain u*(t) = h(x*(t), A*(t), t).

Step 3 U sing the results of Step 2 in Step 1, find the optimal 1i* 1i*(x*(t), h(x*(t), A*(t), t), A*(t), t) = 1i*(x*(t), A*(t), t).

Step 4 Solve the set of 2n differential equations

x* (t) = + (~~) * and'\ * (t) = - (~":.) * with initial conditions Xo and the final conditions

[1i* + ~~]tf 8tf + [(~~t - A*(t)]:f 8xf = O. Step 5 Substitute the solutions of x* (t), A" (t) from Step 4

into the expression for the optimal control u*(t) of Step 2. C. Types of Systems

a . Fixed-final time and fixed-final state system, Fig. 2.9(a) b). Free-final time and fixed-final state system, Fig. 2.9(b) c . Fixed-final time and free-final state system, Fig. 2.9(c) d). Free-final time and dependent free-final state system, Fig. 2.9(d). e . Free-final time and independent free-final state system Type Substitutions Boundary Conditions

(a) 8tf = 0,8xf = 0 x(to) = xo, x(tf) = xf

(b) 8tf =1= 0, 8xf = 0 x( to) = Xo, x( t f) = x f' l1i* + Ft J t f = 0 (c) 8tf = 0, 8xf =1= 0 x( to) = xo, A * (t f) = (~tt f

(d) 8xf = 6(tf )8tf x(to) = Xo, x(tf) = 6(tf)

[H* + ~~ + { ( ~~) * - A * (t)} I 6( t) L f = 0 (e) 8tf =1= 0 8x(to) = Xo

8xf =1= 0 [1i*+as] =0 [(as) -A*(t)] =0 at tf 'ax * tf

70 Chapter 2: Calculus of Variations and Optimal Control

Here, x(t) and u(t) are n- and r- dimensional state and control vec- tors respectively. Let us note that u( t) is unconstrained. The entire pro- cedure (called Pontryagin Principle) is now summarized in Table 2.1.

Note: From Table 2.1 we note that the only difference in the proce- dure between the free-final point system without the final cost function (Lagrange problem) and free-final point system with final cost function (Bolza problem) is in the application of the general boundary condition.

To illustrate the Pontryagin method described previously, consider the following simple examples describing a second order system. Specif- ically, we selected a double integral plant whose analytical solutions for the optimal condition can be obtained and the same verified by using MATLAB©.

First we consider the fixed-final time and fixed-final state problem (Figure 2.9(a), Table 2.1, Type (a)).

Example 2.12

Given a second order (double integrator) system as

Xl(t) = X2(t) X2(t) = u(t)

and the performance index as

(2.7.46)

(2.7.47)

find the optimal control and optimal state, given the boundary (initial and final) conditions as

x(O) = [1 2]'; x(2) = [1 0]'. (2.7.48)

Assume that the control and state are unconstrained.

Solution: We follow the step-by-step procedure given in Table 2.1. First, by comparing the present plant (2.7.46) and the PI (2.7.47) with the general formulation of the plant (2.7.1) and the PI (2.7.2), we identify

1 V(x(t), u(t), t) = V(u(t)) = "2u2(t)

f(x(t), u(t), t) = [fl, f2]',

where, fl = X2(t), f2 = u(t).

(2.7.49)

2.7 Variational Approach to Optimal Control Systems

• Step 1: Form the Hamiltonian function as

H = H(XI(t), X2(t), u(t), AI(t), A2(t)) = V(u(t)) + -X'(t)f(x(t) , u(t))

1 = 2u2(t) + Al (t)X2(t) + A2(t)U(t).

(2.7.50)

• Step 2: Find u*(t) from

aH au = 0 ~ u*(t) + A2(t) = 0 ~ u*(t) = -A2(t). (2.7.51)

• Step 3: Using the results of Step 2 in Step 1, find the optimal H*as

H* (xi(t) , X2(t), Ai(t) , A2(t)) = ~A22 (t) + Ai (t)X2(t) - A22 (t)

= Ai(t)x2(t) - ~A22(t). (2.7.52)

• Step 4: Obtain the state and costate equations from

xi(t) = + (:) * = x2(t)

x:;(t) = + (:) * = -A;(t)

. (aH) Ai(t) = - aXI * = 0

A;(t) = - (:) * = -Ai(t). (2.7.53)

Solving the previous equations, we have the optimal state and costate as

* C3 3 C4 2 Xl (t) = tit - 2t + C2t + C1

X2(t) = ~3t2 - C4t + C2 Ai(t) = C3 A2(t) = -C3t + C4. (2.7.54)

• Step 5: Obtain the optimal control from

u*(t) = -A2(t) = C3 t - C4 (2.7.55)

72 Chapter 2: Calculus of Variations and Optimal Control

Optimal u*(t) J x2*(t) J x1*(Q Controller Figure 2.10 Optimal Controller for Example 2.12

where, 01, O2 , 0 3 , and 0 4 are constants evaluated using the given boundary conditions (2.7.48). These are found to be

01 = 1, O2 = 2, 03 = 3, and 04 = 4. (2.7.56)

Finally, we have the optimal states, costates and control as

xi(t) = 0.5t3 - 2t2 + 2t + 1, X2(t) = 1.5t2 - 4t + 2, Ai(t) = 3, A2(t) = -3t + 4, u*(t) = 3t - 4. (2.7.57)

The system with the optimal controller is shown in Figure 2.10.

The solution for the set of differential equations (2.7.53) with the boundary conditions C2.7.48} for Example 2.12 using Symbolic Toolbox of the MATLA~, Version 6, is shown below.

************************************************************** %% Solution Using Symbolic Toolbox (STB) in %% MATLAB Version 6.0 %% S=dsolve('Dxl=x2,Dx2=-lambda2,Dlambdal=0,Dlambda2=-lambdal, ... xl(0)=1,x2(0)=2,xl(2)=1,x2(2)=0') S.xl S.x2 S.lambdal S.lambda2

S =

S.xl

lambdal: [lxl symJ lambda2: [lxl symJ

xl: [lxl symJ x2: [lxl symJ

ans=

2.7 Variational Approach to Optimal Control Systems

S.x2

ans= 2-4*t+3/2*t-2

S.lambdal

ans= 3 S.lambda2

ans=

Plot command is used for which we need to %% convert the symbolic values to numerical values. j=l; for tp=O: .02: 2 t=sym(tp); xlp(j)=double(subs(S.xl)); %% subs substitutes S.xl to xlp x2p(j)=double(subs(S.x2)); %% double converts symbolic to numeric up(j)=-double(subs(S.lambda2)); %% optimal control u = -lambda_2 tl(j)=tp; j=j+l; end plot(tl,xlp, 'k' ,tl,x2p, 'k' ,tl,up, 'k:') xlabel ('t') gtext('x_l(t) ') gtext (' x_2(t) ') gtext ('u(t) ') *********************************************************

It is easy to see that the previous solutions for xi(t) , x1(t) , Ai(t) , A2(t), and u*(t) = -A2(t) obtained by using MATLAB© are the same as those given by the analytical solutions (2.7.571 The op- timal control and state are plotted (using MATLAB©) in Fig- ure 2.11.

Next, we consider the fixed-final time and free-final state case (Fig-

74 Chapter 2: Calculus of Variations and Optimal Control

-2

" ·3 ,,." , .. ,

.,'

,. ,. ..

u(t), .. ,. .-

.,' .,

'" ,

" -4~~~~--~--~--~--~--~--~--~--~ o 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 2

Figure 2.11 Optimal Control and States for Example 2.12

ure 2.9(b), Table 2.1, Type (c)) of the same system.

Example 2.13

Consider the same Example 2.12 with changed boundary conditions as

x(O) = [1 2]'; xl(2) = 0; x2(2) is free. (2.7.58) Find the optimal control and optimal states.

Solution: Following the procedure illustrated in Table 2.1 (Type ( c) ), we get the same optimal states, costates, and control as given in (2.7.54) and (2.7.55) which are repeated here for convenience.

* 0 3 3 04 2 X1(t) = (ft - 2:t +02 t + Cl,

X2(t) = ~3t2 - 04t + C2, Ai(t) = 0 3 , A2(t) = -03t + 0 4, u*(t) = -A2(t) = C3t - 0 4. (2.7.59)

The only difference is in solving for the constants C1 to C4. First of all, note that the performance index (2.7.47) does not contain the terminal cost function S. From the given boundary conditions

2.7 Variational Approach to Optimal Control Systems 75

(2.7.58), we have tf specified to be 2 and hence 8tf is zero in the general boundary condition (2.7.32).

Also, since x2(2) is free, 8X2j is arbitrary and hence the corre- sponding final condition on the costate becomes

(2.7.60)

(since S = 0). Thus we have the four boundary conditions as

With these boundary conditions substituted in (2.7.59), the con- stants are found to be

01 = 1; 02 = 2; 03 = 15/8; 04 = 15/4. (2.7.62)

Finally the optimal states, costates and control are given from (2.7.59) and (2.7.62) as

* 5 3 15 2 xl (t) = 16 t - 8 t + 2t + 1,

* 15 2 15 x2 (t) = -t - -t + 2

16 4 '

*( ) 15 Al t = 8' * 15 15

A2(t) = -8t + 4' * 15 15

u (t) = 8 t - 4· (2.7.63)

The solution for the set of differential equations (2.7.53) with the boundary conditions {2. 7. 58} for Example 2.13 using Symbolic Toolbox of the MATLAdS) , Version 6, is shown below.

*************************************************************** %% Solution Using Symbolic Toolbox (STB) in %% MATLAB Version 6.0 %% S=dsolve('Dxl=x2,Dx2=-lambda2,Dlambdal=0,Dlambda2=-lambdal, xl(0)=1,x2(0)=2,xl(2)=0,lambda2(2)=0')

S =

76 Chapter 2: Calculus of Variations and Optimal Control

lambdal: [lxl sym] lambda2: [lxl sym]

xl: [lxl sym] x2: [lxl sym]

S.xl

ans=

S.x2

ans=

S.lambdal

ans=

15/8

S.lambda2

ans=

-15/8*t+15/4

%% Plot command is used for which we need to %% convert the symbolic values to numerical values. j=l; for tp=O:.02:2 t=sym(tp); xlp(j)=double(subs(S.xl)); %% subs substitutes S.xl to xlp x2p(j)=double(subs(S.x2)); %% double converts symbolic to numeric up(j)=-double(subs(S.lambda2)); %% optimal control u = -lambda_2 tl(j)=tp; j=j+l; end plot(tl,xlp, 'k' ,tl,x2p, 'k' ,tl,up, 'k: ') xlabel ( , t ' ) gtext ('x_l (t) ')

2.7 Variational Approach to Optimal Control Systems 77

gtext ('x_2(t)')

gtext ('u(t) ')

*******************************************************

It is easy to see that the previous solutions for xi (tj, x2 (t), Ai (t), A2 (t), and u*(t) = -A2(t) obtained by using MATLAB© are the same as those given by (2.7.63) obtained analytically. The optimal control and states for Example 2.13 are plotted in Figure 2.12.

0 "".-....

-1

-2 .-'

.. .. ,"

-3 .. " " ..

-4 0 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 2

Figure 2.12 Optimal Control and States for Example 2.13

Next, we consider the free-final time and independent free-final state case (Figure 2.9(e), Table 2.1, Type (e)) of the same system.

Example 2.14

Consider the same Example 2.12 with changed boundary conditions as

Find the optimal control and optimal state.

Solution: Following the procedure illustrated in Table 2.1 (Type (e)), we get the same optimal control, states and costates as given

78 Chapter 2: Calculus of Variations and Optimal Control

in (2.7.54) and (2.7.55) which are repeated here for convenience.

* C3 3 C4 2 X1(t) = 6 t - 2t + C2t + C1,

X2(t) = ~3t2 - C4t + C2, Ai(t) = C3, A2(t) = -C3t + C4, U*(t) = -A2(t) = C3t - C4. (2.7.65)

The only difference is in solving for the constants C1 to C4 and the unknown t f. First of all, note that the performance index (2.7.47) does not contain the terminal cost function S, that is, S = O. From the given boundary conditions (2.7.64), we have tf unspecified and hence otf is free in the general boundary condition (2.7.32) leading to the specific final condition

(2.7.66)

Also, since X2 (t f) is free, OX2 f is arbitrary and hence the general boundary condition (2.7.32) becomes

>'2(tf) = (:!) = 0 (2.7.67) where ~ is given by (2.7.52). Combining (2.7.64), (2.7.66) and (2.7.67), we have the following 5 boundary conditions for the 5 unknowns (4 constants of integration C1 to C4 and 1 unknown t f) as

x1(0)=I; X2(0) =2; X1(tf) =3;

A2(tf) = 0; A1(tf)X2(tf) - 0.5A~(tf) = o. (2.7.68)

Using these boundary conditions along with (2.7.65), the constants are found to be

C 1 = 1; C 2 = 2; C3 = 4/9; C4 = 4/3; tf = 3. (2.7.69)

Finally, the optimal states, costates, and control are given from

2.7 Variational Approach to Optimal Control Systems 79

(2.7.65) and (2.7.69) as

(2.7.70)

The solution for the set of differential equations (2.7.53) with the boundary conditions C2. 7.68} for Example 2.14 using Symbolic Toolbox of the MATLA~, Version 6 is shown below.

******************************************************** %% Solution Using Symbolic Toolbox (STB) in %% of MATLAB Version 6 %% clear all S=dsolve('Dx1=x2,Dx2=-lam2,Dlaml=O,Dlam2=-lam1,xl(0)=l,

x2(0)=2,x1(tf)=3,lam2(tf)=0') t='tf' ; eq1=subs(S.x1)-'x1tf'; eq2=subs(S.x2)-'x2tf'; eq3=S.lam1-'lam1tf'; eq4=subs(S.lam2)-'lam2tf'; eq5='lamltf*x2tf-0.5*lam2tf A 2'; S2=solve(eq1,eq2,eq3,eq4,eq5,'tf,x1tf,x2tf,lam1tf,

lam2tf','lam1tf<>0') %% lam1tf<>0 means lam1tf is not equal to 0; %% This is a condition derived from eq5. %% Otherwise, without this condition in the above %% SOLVE routine, we get two values for tf (1 and 3 in this case) %% tf=S2.tf xltf=S2.xltf; x2tf=S2.x2tf; clear t x1=subs(S.xl) x2=subs(S.x2) lam1=subs(S.lam1)

80 Chapter 2: Calculus of Variations and Optimal Control

lam2=subs(S.lam2) %% Convert the symbolic values to %% numerical values as shown below. j=l; tf=double(subs(S2.tf)) %% coverts tf from symbolic to numerical for tp=O:O.05:tf t=sym(tp); %% coverts tp from numerical to symbolic xlp(j)=double(subs(S.xl)); %% subs substitutes S.xl to xlp x2p(j)=double(subs(S.x2)); %% double converts symbolic to numeric up(j)=-double(subs(S.lam2)); %% optimal control u = -lambda_2 tl(j)=tp; j=j+l ; end plot(tl,xlp, 'k' ,tl,x2p, 'k' ,tl,up, 'k:') xlabel('t' ) gtext (' x_l (t) ') gtext ( , x_2 ( t) , ) gtext ('u(t) ')

*******************************************************

The optimal control and states for Example 2.14 are plotted in Figure 2.13.

Finally, we consider the fixed-final time and free-final state system with a terminal cost (Figure 2.9 (b), Table 2.1, Type (b)).

Example 2.15

We consider the same Example 2.12 with changed performance index

1 2 1 2 1102 2 J = -[xl(2) - 4] + -[x2(2) - 2] + -2 u dt 220

(2.7.71)

and boundary conditions as

x(O) = [1 2]; x(2) = is free. (2.7.72)

Following the procedure illustrated in Table 2.1 (Type (b)), we get the same optimal control, states and costates as given in (2.7.54) and (2.7.55), which are reproduced here for ready reference. Thus

2.7 Variational Approach to Optimal Control Systems 81

2.5

1.5

0 _~o#--

",-~

.. -'" ,-- -0.5 u(t)

-,-",,- .. --,,-

.4flJ~

-1 -,-, .. - -,,-'"

-1.5 0 0.5 1.5 2 2.5 3

Figure 2.13 Optimal Control and States for Example 2.14

we have

* C3 3 C4 2 X1(t) = (it - 2t + C2t + C1,

X2(t) = ~3t2 - C4t + C2, Ai(t) = C3, A2(t) = -C3t + C4, U*(t) = -A2(t) = C3t - C4. (2.7.73)

The only difference is in solving for the constants C1 to C4 using the given and obtained boundary conditions. Since t f is specified as 2, 6t f is zero and since x(2) unspecified, 6xf is free in the boundary condition (2.7.32), which now reduces to

(2.7.74)

where,

(2.7.75)

82 Chapter 2: Calculus of Variations and Optimal Control

Thus, (2.7.74) becomes

AHtf) = (a as ) ~ AH2) = xl(2) - 4 Xl tf

A2(tf) = (a as ) ~ A2(2) = x2(2) - 2. X2 tf

(2.7.76)

Now, we have two initial conditions from (2.7.72) and two final conditions from (2.7.76) to solve for the four constants as

(2.7.77)

Finally, we have the optimal states, costates and control given as

* 1 3 2 2 Xl (t) = 14 t -"7 t + 2t + 1,

*( ) 3 2 4 x2 t = -t - -t + 2 14 7 '

Ai(t) = ~,

A2(t) = -~t + ~, u*(t) = ~t - i.

7 7 (2.7.78)

The previous results c~ also obtained using Symbolic Math Toolbox of the MATLAB\9, Version 6, as shown below.

*************************************************************** %% Solution Using Symbolic Math Toolbox (STB) in %% MATLAB Version 6 %% S=dsolve('Dxl=x2,Dx2=-lambda2,Dlambdal=O,Dlambda2=-lambdal, xl(O)=1,x2(O)=2,lambdal(2)=x12-4,lambda2(2)=x22-2') t='2' ; S2=solve(subs(S.xl)-'x12',subs(S.x2)-'x22','x12,x22'); %% solves for xl(t=2) and x2(t=2) x12=S2.x12; x22=S2.x22; clear t

S =

2.7 Variational Approach to Optimal Control Systems

lambdal: [lxl sym] lambda2: [lxl sym]

xl: [lxl sym] x2: [lxl sym]

xl=subs(S.xl)

xl =

x2=subs(S.x2)

x2 =

lambdal=subs(S.lambdal)

lambdal

3/7

lambda2=subs(S.lambda2)

lambda2 =

4/7-3/7*t

%% Plot command is used for which we need to %% convert the symbolic values to numerical values. j=l; for tp=O: .02:2 t=sym(tp); xlp(j)=double(subs(S.xl»; %% subs substitutes S.xl to xlp x2p(j)=double(subs(S.x2»; %% double converts symbolic to numeric up(j)=-double(subs(S.lambda2»; %% optimal control u = -lambda_2 tl(j)=tp; j=j+l; end

84 Chapter 2: Calculus of Variations and Optimal Control

plot(t1,x1p, 'k' ,t1,x2p, 'k' ,t1,up, 'k: ') xlabel ('t') gtext (' x_1 (t) ') gtext (' x_2 (t) ') gt ext ( 'u ( t) , )

***************************************************************

It is easy to see that the previous solutions for xi(t)~2(t), Ai (t), A2(t), and u*(t) = -A2(t) obtained by using MATLAB\0 are the same as those given by (2.7.78) obtained analytically.

The optimal control and states for Example 2.15 are plotted in Figure 2.14.

-1~--~--~--~--~--~--~--~--~~--~~ o 0.2 0.4 0.6 0.8 1.2 1.4 1.6 1.8 2

Figure 2.14 Optimal Control and States for Example 2.15

2.8 Summary of Variational Approach In this section, we summarize the development of the topics covered so far in obtaining optimal conditions using the calculus of variations. The development is carried out in different stages as follows. Also shown is the systematic link between various stages of development.

2.8 Summary of Variational Approach

2.8.1 Stage I: Optimization of a Functional

Consider the optimal of

i t!

J = V(x(t), x(t), t)dt to

with the given boundary conditions

x(to) fixed and x(tj) free.

(2.8.1)

(2.8.2)

Then, the optimal function x* (t) should satisfy the Euler-Lagrange equation

(8V) _ ~ (8V) _ 0 8x * dt 8x * - . (2.8.3) The general boundary condition to be satisfied at the free-final point is given by [79]

(2.8.4)

This boundary condition is to be modified depending on the nature of the given t j and x( t j ). Although the previous general boundary condition is not derived in this book, it can be easily seen to be similar to the general boundary condition (2.7.26) in terms of the Lagrangian which embeds a performance index and a dynamical plant into a single augmented performance index with integrand C.

The sufficient condition for optimum is the Legendre condition given by

(82V) 8x2 * > 0

for minimum (2.8.5)

and

(82V) 8x2 * < 0

for maximum. (2.8.6)

86 Chapter 2: Calculus of Variations and Optimal Control

2.8.2 Stage II: Optimization of a Functional with Condition

Consider the optimization of a functional

l tf

J = V(x(t), x(t), t)dt to

(2.8.7)

with given boundary values as

x(to) fixed and x(t,) free, (2.8.8)

and the condition relation

g(x(t), x(t), t) = O. (2.8.9)

Here, the condition (2.8.9) is absorbed by forming the augmented func- tional

l tf

Ja = £(x(t), x(t), .x(t), t)dt to

(2.8.10)

where, .x(t) is the Lagrange multiplier (also called the costate function), and £ is the Lagrangian given by

I £(x(t), x(t), .x(t), t) = V(x(t), x(t), t) + .x'(t)g(x(t), x(t), t)., (2.8.11)

Now, we just use the results of the previous Stage I for the augmented functional (2.8.10) except its integrand is £ instead of V. For optimal condition, we have the Euler-Lagrange equation (2.8.3) for the aug- mented functional (2.8.10) given in terms of x(t) and .x(t) as

(8£) _ ~ (8£) = 0 state equation and 8x * dt 8x * (8£) _ ~ (8£) _ 0 costate equation. 8.x * dt 8>.. *-

(2.8.12)

(2.8.13)

Let us note from (2.8.11) that the Lagrangian £ is independent of >.. * (t) and that the Euler-Lagrange equation (2.8.13) for the costate .x(t) is nothing but the constraint relation (2.8.9). The general boundary

2.8 Summary of Variational Approach 87

condition (2.8.4) to be satisfied at the free-final point (in terms of £) is given by

(2.8.14)

This boundary condition is to be modified depending on the nature of the given tf and x(tf)·

2.8.3 Stage III: Optimal Control System with Lagrangian Formalism

Here, we consider the standard control system with a plant described by [56]

x(t) = f(x(t), u(t), t), (2.8.15)

the given boundary conditions as

x(to) is fixed and x(tf) is free, (2.8.16)

and the performance index as

i t!

J(u(t)) = V(x(t), u(t), t)dt. to

(2.8.17)

Now, we rewrite the plant equation (2.8.15) as the condition relation (2.8.9) as

g(x(t), x(t), u(t), t) = f(x(t), u(t), t) - x(t) = o. (2.8.18)

Then we form the augmented functional out of the performance index (2.8.17) and the condition relation (2.8.18) as

i t!

Ja(u(t)) = £(x(t), x(t), u(t), A(t), t)dt to

(2.8.19)

where, the Lagrangian £ is given as

£ = £(x(t), x(t), u(t), A(t), t)

= V(x(t), u(t), t) + A'(t) {f(x(t), u(t), t) - x(t)}. (2.8.20)

88 Chapter 2: Calculus of Variations and Optimal Control

Now we just use the previous results of Stage II. For optimal condition, we have the set of Euler-Lagrange equations (2.8.12) and (2.8.13) for the augmented functional (2.8.19) given in terms of x(t), '\(t), and u(t) as

( 0£) _ ~ (0£) = 0 state equation, (2.8.21) ox * dt Ox *

( 0£) _ ~ (0£) _ 0 costate equation, and (2.8.22) 0'\ * dt {)~ *-

( 0£) _ ~ (8£) = 0 control equation. (2.8.23) ou * dt ou * Note from (2.8.20) that the Lagrangian £ is independent of ~ * (t) and u*(t) and that the Euler-Lagrange equation (2.8.22) is the same as the constraint relation (2.8.18). The general boundary condition (2.8.14) to be satisfied at the free-final point becomes

[c - x'(t) (~) L Otf + (~)~t OX! = O. f f

(2.8.24)

This boundary condition is to be modified depending on the nature of the given tf and x(tf).

2.8.4 Stage IV: Optimal Control System with Hamiltonian Formalism: Pontryagin Principle

Here, we just transform the previous Lagrangian formalism to Hamil- tonian formalism by defining the Hamiltonian as [57]

H(x(t), u(t), '\(t), t) = V(x(t), u(t), t) + ,\'(t)f(x(t), u(t), t) (2.8.25)

which in terms of the Lagrangian (2.8.20) becomes

£(x(t), x(t), u(t), '\(t), t) = H(x(t), u(t), '\(t), t) - '\'(t)x(t). (2.8.26)

Now using (2.8.26), the set of Euler-Lagrange equations (2.8.21) to (2.8.23) which are in terms of the Lagrangian, are rewritten in terms

2.8 Summary of Variational Approach 89

of the Hamiltonian as

(aH) _ ~(-A*) = 0 ax * dt (2.8.27)

(aH) _ x*(t) - ~(O) = 0 aA * dt (2.8.28)

(aH) _ ~(O) = 0 8u * dt (2.8.29) which in turn are rewritten as

:ic*(t) = + (~) * state equation, (2.8.30)

,,*(t) ___ (8H) " costate equation, and ax *

(2.8.31)

10 -_+(88 H U)*1 . . control equation. (2.8.32)

Similarly using (2.8.26), the boundary condition (2.8.24) is rewritten in terms of the Hamiltonian as

[H - A'(t)X(t) - X'(t)(-A(t))] I*tj 8tf + [-A'(t)] I*tj 8xf = 0 (2.8.33)

which becomes

(2.8.34)

The sufficient condition is

(~) * > 0 for minimum and (2.8.35)

( ~:~) * < 0 for maximum. (2.8.36)

The state, costate, and control equations (2.8.30) to (2.8.32) are solved along with the given initial condition (2.8.16) and the final condition (2.8.34) leading us to a two-point boundary value problem (TPBVP).

90 Chapter 2: Calculus of Variations and Optimal Control

Free-Final Point System with Final Cost Function

This problem is an extension of the problem in Stage IV, with the addition of final cost function. We summarize the result risking the repetition of some of the equations. Let the plant be described as

x(t) = f(x(t), u(t), t) (2.8.37)

and the performance index be

l tf

J(u(t)) = S(x(tf), tf) + V(x(t), u(t), t)dt to

(2.8.38)

along with the boundary conditions

x(to) is fixed and x(tf) is free. (2.8.39)

Now, if we rewrite the performance index (2.8.38) to absorb the final cost function S within the integrand, then the results of Stage III can be used to get the optimal conditions. Thus we rewrite (2.8.38) as

rtf [ (8S)' 8S] J1 (u(t)) = lto V(x(t), u(t), t) + 8x x(t) + at dt. (2.8.40) N ow we repeat the results of Stage III except for the modification of the final condition equation (2.8.34). Thus the state, costate and control equations are

x*(t) = + (~~)* state equation (2.8.41 ) . * (81-l) A (t) = - -

8x * costate equation (2.8.42)

control equation (2.8.43)

and the final boundary condition is

(2.8.44)

2.8 Summary of Variational Approach

The sufficient condition for optimum is

(~) * > 0 for minimum and

(~) * < 0 for maximum.

(2.8.45)

(2.8.46)

The state, costate, and control equations (2.8.41) to (2.8.43) are solved along with the given initial condition (2.8.39) and the final condition (2.8.44), thus this formulation leads us to a TPBVP.

2.8.5 Salient Features

We now discuss the various features of the methodology used so far in obtaining the optimal conditions through the use of the calculus of variations [6, 79, 120, 108]. Also, we need to consider the problems discussed above under the various stages of development. So we refer to the appropriate relations of, say Stage III or Stage IV during our discussion.

1. Significance of Lagrange Multiplier: The Lagrange multiplier A(t) is also called the costate (or adjoint) function.

(a) The Lagrange multiplier A(t) is introduced to "take care of" the constraint relation imposed by the plant equation (2.8.15).

(b) The costate variable A(t) enables us to use the Euler-Lagrange equation for each of the variables x(t) and u(t) separately as if they were independent of each other although they are dependent of each other as per the plant equation.

2. Lagrangian and Hamiltonian: We defined the Lagrangian and Hamiltonian as

£ = £(x(t), x(t), A(t), u(t), t)

= V(x(t), u(t), t) +A'(t) {f(x(t), u(t), t) - x(t)}

1t = 1t(x(t) , u(t), A(t), t)

= V(x(t), u(t), t)

+A'(t)f(x(t), u(t), t).

(2.8.47)

(2.8.48)

92 Chapter 2: Calculus of Variations and Optimal Control

In defining the Lagrangian and Hamiltonian we use extensively the vector notation, still it should be noted that these £ and 1t are scalar functions only.

3. Optimization of Hamiltonian

(a) The control equation (2.8.32) indicates the optimization of the Hamiltonian w.r.t. the control u(t). That is, the opti- mization of the original performance index (2.8.17), which is a functional subject to the plant equation (2.8.15), is equiv- alent to the optimization of the Hamiltonian function w.r.t. u( t). Thus, we "reduced" our original functional optimiza- tion problem to an ordinary function optimization problem.

(b) We note that we assumed unconstrained or unbounded con- trol u(t) and obtained the control relation 81t/8u = O.

( c) If u( t) is constrained or bounded as being a member of the set U, i.e., u(t) E U, we can no longer take 81t/8u = 0, since u( t), so calculated, may lie outside the range of the permissible region U. In practice, the control u(t) is always limited by such things as saturation of amplifiers, speed of a motor, or thrust of a rocket. The constrained optimal control systems are discussed in Chapter 7.

(d) Regardless of any constraints on u(t), Pontryagin had shown that u(t) must still be chosen to minimize the Hamiltonian. A rigorous proof of the fact that u( t) must be chosen to op- timize 1t function is Pontryagin's most notable contribution to optimal control theory. For this reason, the approach is often called Pontryagin Principle. So in the case of con- strained control, it is shown that

min 1t(x*(t), .x*(t), u(t), t) = 1t(x*(t), .x*(t), u*(t), t) uEU

(2.8.49)

or equivalently

11t(x*(t), .x*(t), u*(t), t) :::; 1t(x*(t), .x*(t), u(t), t).1

(2.8.50)

4. Pontryagin Maximum Principle: Originally, Pontryagin used a slightly different performance index which is maximized rather

2.8 Summary of Variational Approach 93

than minimized and hence it is called Pontryagin Maximum Prin- ciple. For this reason, the Hamiltonian is also sometimes called Pontryagin H-function. Let us note that minimization of the per- formance index J is equivalent to the maximization of -J. Then, if we define the Hamiltonian as

H(x(t), u(t), A(t), t) = - V(x(t), u(t), t) + 5..' (t)f(x(t), u(t), t) (2.8.51)

we have Maximum Principle. Further discussion on Pontryagin Principle is given in Chapter 6.

5. Hamiltonian at the Optimal Condition: At the optimal condition the Hamiltonian can be written as

H* = H* (x* (t), u* (t), A * (t), t) dH* dH*

dt dt

= (~~) ~ x*(t) + (:)~ i*(t) + (~~)~ u*(t) + (a;:) * . (2.8.52)

Using the state, costate and control equations (2.8.30) to (2.8.32) in the previous equation, we get

(2.8.53)

We observe that along the optimal trajectory, the total derivative of H w.r.t. time is the same as the partial derivative of H w.r.t. time. If H does not depend on t explicitly, then

~I=o ~

(2.8.54)

and 1i is constant w.r. t. the time t along the optimal trajectory.

6. Two-Point Boundary Value Problem (TPBVP): As seen earlier, the optimal control problem of a dynamical system leads to a TPBVP.

94 Chapter 2: Calculus of Variations and Optimal Control

Open-Loop u*(t) Plant

x*(t) Optimal Controller

.. ..

Figure 2.15 Open-Loop Optimal Control

(a) The state and costate equations (2.8.30) and (2.8.32) are solved using the initial and final conditions. In general, these are nonlinear, time varying and we may have to resort to numerical methods for their solutions.

(b) We note that the state and costate equations are the same for any kind of boundary conditions.

( c) For the optimal control system, although obtaining the state and costate equations is very easy, the computation of their solutions is quite tedious. This is the unfortunate feature of optimal control theory. It is the price one must pay for demanding the best performance from a system. One has to weigh the optimization of the system against the computa- tional burden.

7. Open-Loop Optimal Control: In solving the TPBVP arising due to the state and costate equations, and then substituting in the control equation, we get only the open-loop optimal control as shown in Figure 2.15. Here, one has to construct or realize an open-loop optimal controller (OLOC) and in many cases it is very tedious. Also, changes in plant parameters are not taken into account by the OLOC. This prompts us to think in terms of a closed-loop optimal control (CLOC), i.e., to obtain optimal control u*(t) in terms of the state x*(t) as shown in Figure 2.16. This CLOC will have many advantages such as sensitive to plant parameter variations and simplified construction of the controller. The closed-loop optimal control systems are discussed in Chap- ter 7.

2.8 Summary of Variational Approach 95

u*(t) .. Plant x*(t) r/ • ~ -

Closed-Loop +- Optimal Controller

Figure 2.16 Closed-Loop Optimal Control

96 Chapter 2: Calculus of Variations and Optimal Control

2.9 Problems

1. Make reasonable assumptions wherever necessary.

2. Use MATLAB© wherever possible to solve the problems and plot all the optimal controls and states for all problems. Provide the relevant MATLAB© m files.

Problem 2.1 Find the extremal of the following functional

with the initial condition as x(O) = 0 and the final condition as x(2) = 5.

Problem 2.2 Find the extremal of the functional

to satisfy the boundary conditions x( -2) = 3, and x(O) = O.

Problem 2.3 Find the extremal for the following functional

with x(l) = 1 and x(2) = 10.

Problem 2.4 Consider the extremization of a functional which is de- pendent on derivatives higher than the first derivative x(t) such as

i t!

J(x( t), t) = V(x( t), x( t), i(t), t)dt. to

with fixed-end point conditions. Show that the corresponding Euler- Lagrange equation is given by

2.9 Problems 97

Similarly, show that, in general, for extremization of

i t!

J = V ( x ( t ), ± ( t) , x ( t), ... , x (r) ( t), t) dt to

with fixed-end point conditions, the Euler-Lagrange equation becomes

Problem 2.5 A first order system is given by

±(t) = ax(t) + bu(t)

and the performance index is

lint! J = - (qx 2 (t) + ru 2 (t))dt 2 0

where, x(to) = Xo and x(tf) is free and tf being fixed. Show that the optimal state x* (t) is given by

*( ) _ sinh(3(tf - t) x t - Xo . h(3 ,

S'ln tf

Problem 2.6 A mechanical system is described by

x(t) = u(t)

find the optimal control and the states by minimizing

J = ~ r5 u 2 (t)dt 2 10

such that the boundary conditions are

x(t = 0) = 2; x(t = 5) = 0; ±(t = 0) = 2; ±(t = 5) = O.

Problem 2.7 For the first order system

dx dt = -x(t) + u(t)

find the optimal control u* (t) to minimize

J = If [x 2 (t) + u2 (t)]dt where, tf is unspecified, and x(O) = 5 and x(tf) = o. Also find tf·

98 Chapter 2: Calculus of Variations and Optimal Control

Problem 2.8 Find the optimal control u* (t) of the plant

Xl(t)=X2(t); Xl(O) =3, xl(2)=0

X2(t) = -2Xl(t) + 5u(t); X2(0) = 5, x2(2) = 0

which minimizes the performance index

J = ~ l [xM + u2 (t)] dt. Problem 2.9 A second order plant is described by

Xl(t) =X2(t)

X2(t) = -2Xl(t) - 3X2(t) + 5u(t)

and the cost function is

J = f" [xi(t) + u2 (t)Jdt. Find the optimal control, when Xl(O) = 3 and X2(0) = 2.

Problem 2.10 For a second order system

Xl(t) = X2(t)

X2(t) = -2Xl(t) + 3u(t)

with performance index

(7r/2 J = 0.5xi(1T/2) + io 0.5u2(t)dt

and boundary conditions x(O) = [0 1]' and x(t j) is free, find the opti- mal control.

Problem 2.11 Find the optimal control for the plant

Xl(t)=X2(t)

X2(t) = -2Xl(t) + 3u(t) with performance criterion

121 2 J = "2Fll [Xl(tj) - 4] + "2F22 [X2(tj) - 2]

1 rtf [ ] +"2 io xi(t) + 2x~(t) + 4u2(t) dt and initial conditions as x(O) = [1 2]'. The additional conditions are given below.

2.9 Problems 99

1. Fixed-final conditions Fn = 0, F22 = 0, t f = 2, x(2) = [4 6]'.

2. Free-final time conditions Fn = 3, F22 = 5, x(tf) = [4 6]' and tf is free.

3. Free-final state conditions, Fn x2(2) = 6.

0, Xl (2) is free and

4. Free-final time and free-final state conditions, Fn = 3, F22 = 5 and the final state to have xI(tf) = 4 and x2(tf) to lie on 8(t) = -5t + 15.

Problem 2.12 For the D.C. motor speed control system described in Problem 1.1, find the open-loop optimal control to keep the speed constant at a particular value and the system to respond for any dis- turbances from the regulated value.

Problem 2.13 For the liquid-level control system described in Prob- lem 1.2, find the open-loop optimal control to keep the liquid level constant at a reference value and the system to act only if there is a change in the liquid level.

Problem 2.14 For the inverted pendulum control system described in Problem 1.3, find the open-loop, optimal control to keep the pendulum in a vertical position.

Problem 2.15 For the mechanical control system described in Prob- lem 1.4, find the open-loop, optimal control to keep the system at equilibrium condition and act only if there is a disturbance.

Problem 2.16 For the automobile suspension control system described in Problem 1.5, find the open-loop, optimal control to provide minimum control energy and passenger comfort.

Problem 2.17 For the chemical control system described in Prob- lem 1.6, find the open-loop, optimal control to keep the system at equilibrium condition and act only if there is a disturbance.

@@@@@@@@@@@@@@@

Chapter 3

Linear Quadratic Optimal Control Systems I

In this chapter, we present the closed-loop optimal control of linear plants or systems with quadratic performance index or measure. This leads to the linear quadratic regulator (LQR) system dealing with state regulation, output regulation, and tracking. Broadly speaking, we are interested in the design of optimal linear systems with quadratic per- formance indices. It is suggested that the student reviews the material in Appendices A and B given at the end of the book. This chapter is inspired by [6, 3, 89]1.

3.1 Problem Formulation We discuss the plant and the quadratic performance index with par- ticular reference to physical significance. This helps us to obtain some elegant mathematical conditions on the choice of various matrices in the quadratic cost functional. Thus, we will be dealing with an opti- mization problem from the engineering perspective.

Consider a linear, time-varying (LTV) system

x(t) = A(t)x(t) + B(t)u(t) y(t) = C(t)x(t)

(3.1.1)

(3.1.2)

IThe permissions given by John Wiley for F. L. Lewis, Optimal Control, John Wiley & Sons, Inc., New York, NY, 1986, and McGraw-Hill for M. Athans and P. L. Falb, Optimal Control: An Introduction to the Theory and Its Applications, McGraw-Hill Book Company, New York, NY, 1966, are hereby acknowledged.

101

102 Chapter 3: Linear Quadratic Optimal Control Systems I

with a cost functional (CF) or performance index (PI)

J(u(t)) = J(x(to), u(t), to)

1 I ] ="2 [z(tf) - y(tf)] F(tf) [z(tf) - y(tf)

11tf +- [[z(t) - y(t)]' Q(t) [z(t) - y(t)] + u'(t)R(t)u(t)] dt 2 to

(3.1.3)

where, x(t) is nth state vector, y(t) is mth output vector, z(t) is mth ref- erence or desired output vector (or nth desired state vector, if the state x(t) is available), u(t) is rth control vector, and e(t) = z(t) - y(t) (or e(t) = z(t) - x(t), if the state x(t) is directly available) is the mth error vector. A(t) is nxn state matrix, B(t) is nxr control matrix, and C(t) is mxn output matrix. We assume that the control u(t) is unconstrained, 0 < m :::; r :::; n, and all the states and/or outputs are completely measurable. The preceding cost functional (3.1.3) contains quadratic terms in error e(t) and control u(t) and hence called the quadratic cost functional 2 . We also make certain assumptions to be de- scribed below on the various matrices in the quadratic cost functional (3.1.3). Under these assumptions, we will find that the optimal control is closed-loop in nature, that is, the optimal control u(t) is a function of the state x(t) or the output y(t). Also, depending on the final time tf being finite (infinite), the system is called finite- (infinite-) time horizon system. Further, we have the following categories of systems.

1. If our objective is to keep the state x(t) near zero (i.e., z(t) o and C = I), then we call it state regulator system. In other words, the objective is to obtain a control u(t) which takes the plant described by (3.1.1) and (3.1.2) from a nonzero state to zero state. This situation may arise when a plant is subjected to unwanted disturbances that perturb the state (for example, sudden load changes in an electrical voltage regulator system, sudden wind gust in a radar antenna positional control system).

2. If our interest is to keep the output y(t) near zero (i.e., z(t) = 0), then it is termed the output regulator system.

2See Appendix A for more details on Quadratic Forms and Definiteness and other related topics.

3.1 Problem Formulation 103

3. If we try to keep the output or state near a desired state or output, then we are dealing with a tracking system. We see that in both state and output regulator systems, the desired or reference state is zero and in tracking system the error is to be made zero. For example, consider again the antenna control system to track an aircraft.

Let us consider the various matrices in the cost functional (3.1.3) and their implications.

1. The Error Weighted Matrix Q(t): In order to keep the error e(t) small and error squared non-negative, the integral of the expres- sion ~e/(t)Q(t)e(t) should be nonnegative and small. Thus, the matrix Q (t) must be positive semidefinite. Due to the quadratic nature of the weightage, we have to pay more attention to large errors than small errors.

2. The Control Weighted Matrix R(t): The quadratic nature of the control cost expression ~u/(t)R(t)u(t) indicates that one has to pay higher cost for larger control effort. Since the cost of the control has to be a positive quantity, the matrix R( t) should be positive definite.

3. The Control Signal u(t): The assumption that there are no con- straints on the control u(t) is very important in obtaining the closed loop optimal configuration.

Combining all the previous assumptions, we would like on one hand, to keep the error small, but on the other hand, we must not pay higher cost to large controls.

4. The Terminal Cost Weighted Matrix F(tf): The main purpose of this term is to ensure that the error e(t) at the final time tf is as small as possible. To guarantee this, the corresponding matrix F( t f) should be positive semidefinite.

Further, without loss of generality, we assume that the weighted matrices Q(t), R(t), and F(t) are symmetric. The quadratic cost functional described previously has some attractive features:

(a) It provides an elegant procedure for the design of closed-loop optimal controller.

104 Chapter 3: Linear Quadratic Optimal Control Systems I

(b) It results in the optimal feed-back control that is linear in state function.

That is why we often say that the "quadratic performance index fits like a glove" [6].

5. Infinite Final Time: When the final time t f is infinity, the termi- nal cost term involving F(tf) does not provide any realistic sense since we are always interested in the solutions over finite time. Hence, F(tf) must be zero.

3.2 Finite-Time Linear Quadratic Regulator Now we proceed with the linear quadratic regulator (LQR) system, that is, to keep the state near zero during the interval of interest. For the sake of completeness we shall repeat the plant and performance index equations described in the earlier section. Consider a linear, time-varying plant described by

x(t) = A(t)x(t) + B(t)u(t)

with a cost functional

J(u) = J(x(to), u(t), to)

= ~ x' (t f ) F ( t f )x ( t f ) 11tf +- [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt 2 to

= ~ x' ( t f ) F ( t f ) x( t f )

+~ rtf [X'(t) u'(t)] [Q(t) 0 ] [X(t)] dt 2 lto 0 R(t) u(t)

(3.2.1)

(3.2.2)

where, the various vectors and matrices are defined in the last section. Let us note that here, the reference or desired state z(t) = 0 and hence the error e(t) = 0 - x(t) itself is the state, thereby implying a state regulator system. We summarize again various assumptions as follows.

1. The control u( t) is unconstrained. However, in many physical situations, there are limitations on the control and state and the case of unconstrained control is discussed in a later chapter.

3.2 Finite-Time Linear Quadratic Regulator 105

2. The initial condition x(t = to) = Xo is given. The terminal time t f is specified, and the final state x( t f) is not specified.

3. The terminal cost matrix F(tf) and the error weighted matrix Q(t) are nxn positive semidefinite matrices, respectively; and the control weighted matrix R( t) is an rxr positive definite matrix.

4. Finally, the fraction ~ in the cost functional (3.2.2) is associated mainly to cancel a 2 that would have otherwise been carried on throughout the result, as seen later.

We follow the Pontryagin procedure described in Chapter 2 (Table 2.1) to obtain optimal solution and then propose the closed-loop configu- ration. First, let us list the various steps under which we present the method.

• Step 1: Hamiltonian

• Step 2: Optimal Control

• Step 3: State and Costate System

• Step 4: Closed-Loop Optimal Control

• Step 5: Matrix Differential Riccati Equation

Now let us discuss the preceding steps in detail.

• Step 1: Hamiltonian: Using the definition of the Hamiltonian given by (2.7.27) in Chapter 2 along with the performance index (3.2.2), formulate the Hamiltonian as

1 1 H(x(t), u(t), A(t)) = "2x'(t)Q(t)x(t) + "2u'(t)R(t)u(t)

+A' (t) [A(t)x(t) + B(t)u(t)] (3.2.3)

where, A(t) is the costate vector of nth order.

• Step 2: Optimal Control: Obtain the optimal control u*(t) using the control relation (2.7.29) as

~~ = 0 ----+ R(t)u*(t) + B'(t)A*(t) = 0 (3.2.4)

leading to

u*(t) = _R-l(t)B'(t)A*(t) (3.2.5)

106 Chapter 3: Linear Quadratic Optimal Control Systems I

where, we used

:u Gul(t)R(t)U(t)} = R(t)u(t) and

:u {A' (t)B(t)u(t)} = B' (t)A(t). Similar expressions are used throughout the rest of the book. Further details on such relations are found in Appendix A. We immediately notice from (3.2.5) the need for R(t) to be positive definite and not positive semidefinite so that the inverse R-l(t) exists .

• Step 3: State and Costate System: Obtain the state and costate equations according to (2.7.30) and (2.7.31) as

x*(t) = + (~~) * ---> x*(t) = A(t)x*(t) + B(t)u*(t) (3.2.6)

j,*(t) = - (~~) * ---> j, *(t) = -Q(t)x*(t) - A'(t)A*(t). (3.2.7)

Substitute the control relation (3.2.5) in the state equation (3.2.6) to obtain the (state and costate) canonical system (also called Hamiltonian system) of equations

[ x*(t)] [A(t) -E(t)] [x*(t)] .,\ * (t) = - Q (t) - A' (t) A * (t)

(3.2.8)

where E(t) = B(t)R-l(t)B'(t). The general boundary condition given by the relation (2.7.32) is reproduced here as

[1i* + ~~Lf i5tf + [(~~) * - A*(t{ i5xf = 0 (3.2.9) where, S equals the entire terminal cost term in the cost func- tional (3.2.2). Here, for our present system tf is specified which makes 8tf equal to zero in (3.2.9), and x(tf) is not specified which makes 8xf arbitrary in (3.2.9). Hence, the coefficient of 8xf in (3.2.9) becomes zero, that is,

A*(tf) = (:~f)) * a [~x' (t f )F( t f )x( t f)] *

= 8x(tf) = F(tf)x (tf)· (3.2.10)

3.2 Finite-Time Linear Quadratic Regulator 107

This final condition on the costate ,x * (t f) together with the initial condition on the state Xo and the canonical system of equations (3.2.8) form a two-point, boundary value problem (TPBVP). The state-space representation of the set of relations for the state and costate system (3.2.8) and the control (3.2.5) is shown in Fig- ure 3.1.

u*(t) J x*(t)

State A(t)

-1 R (t) -Q(t)

B'(t) ').,*(1)

Costate

Figure 3.1 State and Costate System

• Step 4: Closed-Loop Optimal Control: The state space repre- sentation shown in Figure 3.1 prompts us to think that we can obtain the optimal control u*(t) as a function (negative feedback) of the optimal state x*(t). Now to formulate a closed-loop opti- mal control, that is, to obtain the optimal control u*(t) which is a function of the costate ,x*(t) as seen from (3.2.5), as a function of the state x*(t), let us examine the final condition on ,x*(t) given

108 Chapter 3: Linear Quadratic Optimal Control Systems I

by (3.2.10). This in fact relates the costate in terms of the state at the final time t f. Similarly, we may like to connect the costate with the state for the complete interval of time [to, t f]. Thus, let us assume a transformation [113, 102]

A*(t) = P(t)x*(t) (3.2.11)

where, P(t) is yet to be determined. Then, we can easily see that with (3.2.11), the optimal control (3.2.5) becomes

u*(t) = -R-1 (t)B'(t)P(t)x*(t) (3.2.12)

which is now a negative feedback of the state x*(t). Note that this negative feedback resulted from our "theoretical development" or "mathematics" of optimal control procedure and not introduced intentionally [6].

Differentiating (3.2.11) w.r.t. time t, we get

.x*(t) = P(t)x*(t) + P(t)x*(t). (3.2.13)

Using the transformation (3.2.11) in the control, state and costate system of equations (3.2.5), (3.2.6) and (3.2.7), respectively, we get

x*(t) = A(t)x*(t) - B(t)R-l(t)B'(t)P(t)x*(t),

.x*(t) = -Q(t)x*(t) - A'(t)P(t)x*(t).

(3.2.14)

(3.2.15)

Now, substituting state and costate relations (3.2.14) and (3.2.15) in (3.2.13), we have

-Q(t)x*(t) - A'(t)P(t)x*(t) = P(t)x*(t) + P(t) [A(t)x*(t) - B(t)R- 1 (t)B'(t)P(t)x*(t)] ~

[P(t) + P(t)A(t) + A'(t)P(t) + Q(t)- P(t)B(t)R- 1 (t)B'(t)P(t)] x*(t) = 0 (3.2.16)

Essentially, we eliminated the costate function A * (t) from the control (3.2.5), the state (3.2.6) and the costate (3.2.7) equations by introducing the transformation (3.2.11) .

• Step 5: Matrix Differential Riccati Equation: Now the relation (3.2.16) should be satisfied for all t E [to, t f 1 and for any choice of

3.2 Finite-Time Linear Quadratic Regulator 109

the initial state x*(to). Also, P(t) is not dependent on the initial state. It follows that the equation (3.2.16) should hold good for any value of x*(t). This clearly means that the function P(t) should satisfy the matrix differential equation

P(t) + P(t)A(t) + A' (t)P(t) + Q(t) - P(t)B(t)R- 1 (t)B'(t)P(t) = O. (3.2.17)

This is the matrix differential equation of the Riccati type, and often called the matrix differential Riccati equation (DRE). Also, the transformation (3.2.11) is called the Riccati transformation, P(t) is called the Riccati coefficient matrix or simply Riccati ma- trix or Riccati coefficient, and (3.2.12) is the optimal control (feed- back) law. The matrix DRE (3.2.17) can also be written in a compact form as

P(t) = -P(t)A(t) - A'(t)P(t) - Q(t) + P(t)E(t)P(t) (3.2.18)

where E(t) = B(t)R-l(t)B'(t).

Comparing the boundary condition(3.2.10) and the Riccati trans- formation (3.2.11), we have the final condition on P(t) as

>..*(tf) = P(tf)x*(tf) = F(tf)x*(tf) ---+

\ P(tf) = F(tf)'\ (3.2.19)

Thus, the matrix DRE (3.2.17) or (3.2.18) is to be solved backward in time using the final condition (3.2.19) to obtain the solution P ( t) for the entire interval [to, t f ] .

3.2.1 Symmetric Property of the Riccati Coefficient Ma- trix

Here, we first show an important property of the Riccati matrix P(t). The fact that the nxn matrix P(t) is symmetric for all t E [to, tf]' i.e., P (t) = P' (t) can be easily shown as follows. First of all, let us note that from the formulation of the problem itself, the matrices F(tf), Q(t), and R(t) are symmetric and therefore, the matrix B(t)R-l(t)B'(t) is also symmetric. Now transposing both sides of the matrix DRE (3.2.18), we notice that both P(t) and P'(t) are solutions of the same differential equation and that both satisfy the same final condition (3.2.19).

110 Chapter 3: Linear Quadratic Optimal Control Systems I

3.2.2 Optimal Control

Is the optimal control u*(t) a minimum? This can be answered by considering the second partials of the Hamiltonian (3.2.3). Let us recall from Chapter 2 that this is done by examining the second variation of the cost functional. Thus, the condition (2.7.41) (reproduce~ here for convenience) for examining the nature of optimal control is that the matrix

(3.2.20)

must be positive definite (negative definite) for minimum (maximum). In most of the cases this reduces to the condition that

( 8

21i) ou2

(3.2.21 )

must be positive definite (negative definite) for minimum (maximum). Now using the Hamiltonian (3.2.3) and calculating the various partials,

(3.2.22)

Substituting the previous partials in the condition (3.2.20), we have

[ Q(t) 0 ]

II= 0 R(t) . (3.2.23)

Since R(t) is positive definite, and Q(t) is positive semidefinite, it fol- lows that the preceding matrix (3.2.23) is only positive semidefinite. However, the condition that the second partial of 1i w.r.t. u*(t), which is R( t), is positive definite, is enough to guarantee that the control u*(t) is minimum.

3.2.3 Optimal Performance Index

Here, we show how to obtain an expression for the optimal value of the performance index.

3.2 Finite- Time Linear Quadratic Regulator 111

THEOREM 3.1 The optimal value of the PI (3.2.2) is given by

J*(x*(t), t) = ~x*'(t)P(t)x*(t). (3.2.24)

Proof: First let us note that

i t! d 1

- (x*'(t)P(t)x*(t)) dt = --x*'(to)P(to)x*(to) ~ & 2

+~x*'(tj)P(tj)x*(tj). (3.2.25)

Substituting for ~x*'(tj )P(tj )x*(tj) from (3.2.25) into the PI (3.2.2), and noting that P(tj) = F(tj) from (3.2.19) we get

J* (x* (to), to) = ~x*' (to)P( to)x* (to) lit! +- [x*'(t)Q(t)x*(t) + u*'(t)R(t)u*(t) 2 to

+ :t (x*'(t)P(t)x*(t)) 1 dt = ~x*'(to)P(to)x(to)

lit! +- [x*'(t)Q(t)x*(t) + u*'(t)R(t)u*(t) 2 to

+ x*'(t)P(t)x*(t) + x*'(t)P(t)x*(t) + x*'(t)P(t)x*(t)] dt. (3.2.26)

Now, using the state equation (3.2.14) for x*(t), we get

J* (x* (to), to) = ~x*' (to)P( to)x* (to) lit! +- x*'(t) [Q(t) + A'(t)P(t) + P(t)A(t) 2 to

- P(t)B(t)R- 1(t)B'(t)P(t) + P(t)] x*(t)dt. (3.2.27)

Finally, using the matrix DRE (3.2.18) in the previous relations, the integral

part becomes zero. Thus,

J*(x(to),to) = ~x*'(to)P(to)x*(to). (3.2.28)

112 Chapter 3: Linear Quadratic Optimal Control Systems I

Now, the previous relation is also valid for any x* (t). Thus,

1 J* (x* (t), t) = "2x*' (t)P( t)x* (t). (3.2.29)

In terms of the final time t f' the previous optimal cost becomes

(3.2.30)

Since we are normally given the initial state x( to) and the Riccati coefficient P(t) is solved for all time t, it is more convenient to use the relation (3.2.28).

3.2.4 Finite-Time Linear Quadratic Regulator: Time- Varying Case: Summary

Given a linear, time-varying plant

x(t) = A(t)x(t) + B(t)u(t)

and a quadratic performance index

J = ~ x' ( t f) F ( t f ) x( t f ) 11t! +"2 to [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt

(3.2.31)

(3.2.32)

where, u( t) is not constrained, t f is specified, and x( t f) is not specified, F(tf) and Q(t) are nxn symmetric, positive semidefinite matrices, and R(t) is TXT symmetric, positive definite matrix, the optimal control is given by

I u*(t) = -R- 1 (t)B'(t)P(t)x*(t) = -K(t)x*(t) I (3.2.33)

where K(t) = R- 1 (t)B'(t)P(t) is called Kalman gain and P(t), the nxn symmetric, positive definite matrix (for all t E [to, tfD, is the solution of the matrix differential Riccati equation (DRE)

I P(t) = -P(t)A(t) - A'(t)P(t) - Q(t) + P(t)B(t)R- 1 (t)B'(t)P(t) I (3.2.34)

satisfying the final condition

(3.2.35)

3.2 Finite-Time Linear Quadratic Regulator 113

Table 3.1 Procedure Summary of Finite-Time Linear Quadratic Regulator System: Time-Varying Case

A. Statement of the Problem Given the plant as x(t) = A(t)x(t) + B(t)u(t), the performance index as

J = ~x'(tf )F(tf )x(tf) + ~ It; [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt, and the boundary conditions as x(to) = xo, t f is fixed, and x( t f) is free, find the optimal control, state and performance index.

B. Solution of the Problem Step 1 Solve the matrix differential Riccati equation

P(t) = -P(t)A(t) - A'(t)P(t) - Q(t) + P(t)B(t)R- 1 (t)B'(t)P(t) with final condition P(t = tf) = F(tf).

Step 2 Solve the optimal state x* (t) from x*(t) = [A(t) - B(t)R- 1 (t)B'(t)P(t)] x*(t) with initial condition x(to) = Xo.

Step 3 Obtain the optimal control u*(t) as u*(t) = -K(t)x*(t) where, K(t) = R- 1 (t)B'(t)P(t).

Step 4 Obtain the optimal performance index from J* = ~x*'(t)P(t)x*(t).

the optimal state is the solution of

I x*(t) = [A(t) - B(t)R- 1 (t)B'(t)P(t)] x*(t) I (3.2.36)

and the optimal cost is

1 J* = -x*'(t)P(t)x*(t) 2 . (3.2.37)

The optimal control u*(t), given by (3.2.33), is linear in the optimal state x*(t). The entire procedure is now summarized in Table 3.l. Note: It is simple to see that one can absorb the! that is associated with J by redefining a performance measure as

J2 = 2J = x'(tf )F(tf )x(tf)

l. tJ

+ [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt, to

(3.2.38)

114 Chapter 3: Linear Quadratic Optimal Control Systems I

get the corresponding matrix differential Riccati equation for J2 as

P2(t) = _ P 2(t) A(t) _ A'(t) P 2(t) _ Q(t) 222

+ P2(t)B(t)R- 1 (t)B'(t)P 2(t) 2 2

with final condition

(3.2.39)

(3.2.40)

Comparing the previous DRE for J 2 with the corresponding DRE (3.2.34) for J, we can easily see that P2(t) = 2P(t) and hence the optimal control becomes

u*(t) = -R- 1 (t)B'(t)P 2(t)x*(t) = _ K 2(t)x*(t) 2 2

= -R-1 (t)B'(t)P(t)x*(t) = -K(t)x*(t). (3.2.41)

Thus, using J 2 without the ~ in the performance index, we get the same optimal control (3.2.41) for the original plant (3.2.31), but the only difference being that the Riccati coefficient matrix P 2 (t) is twice that of P(t) and J2 is twice that of J(for example, see [3, 42]).

However, we will retain the ~ in J throughout the book due to the ob- vious simplifications in obtaining the optimal control, state and costate equations (3.2.4), (3.2.6) and (3.2.7), respectively. Precisely, the factor ~ in the PI (3.2.2) and hence in the Hamiltonian (3.2.3) gets eliminated while taking partial derivatives of the Hamiltonian w.r.t. the control, state and costate functions.

3.2.5 Salient Features

We next discuss the various salient features of the state regulator sys- tem and the matrix differential Riccati equation.

1. Riccati Coefficient: The Riccati coefficient matrix P(t) is a time- varying matrix which depends upon the system matrices A(t) and B (t), the performance index (design) matrices Q (t), R( t) and F(tf), and the terminal time tf, but P(t) does not depend upon the initial state x( to) of the system.

2. P(t) is symmetric and hence it follows that the nxn order ma- trix DRE (3.2.18) represents a system of n(n + 1)/2 first order, nonlinear, time-varying, ordinary differential equations.

3.2 Finite-Time Linear Quadratic Regulator 115

3. Optimal Control: From (3.2.21), we see that the optimal control u*(t) is minimum (maximum) if the control weighted matrix R(t) is positive definite (negative definite).

4. Optimal State: Using the optimal control (3.2.12) in the state equation (3.2.1), we have

I x*(t) = [A(t) - B(t)R- 1 (t)B'(t)P(t)] x*(t) = G(t)x*(t) I

(3.2.42)

where

G(t) = A(t) - B(t)R-1 (t)B'(t)P(t). (3.2.43)

The solution of this state differential equation along with the ini- tial condition x(to) gives the optimal state x*(t). Let us note that there is no condition on the closed-loop matrix G(t) regard- ing stability as long as we are considering the finite final time (t f) system.

5. Optimal Cost: It is shown in (3.2.29) that the minimum cost J* is given by

J* = ~x*'(t)P(t)x*(t) for all t E [to,tf] (3.2.44)

where, P(t) is the solution of the matrix DRE (3.2.18), and x*(t) is the solution of the closed-loop optimal system (3.2.42).

6. Definiteness of the Matrix P(t): Since F(tf) is positive semidefi- nite, and P(tf) = F(tf), we can easily say that P(tf) is positive semidefinite. We can argue that P (t) is positive definite for all t E [to, t f). Suppose that P (t) is not positive definite for some t = ts < tf, then there exists the corresponding state x*(ts ) such that the cost function ~x*'(ts)P(ts)x*(ts) ::; 0, which clearly vi- olates that fact that minimum cost has to be a positive quantity. Hence, P(t) is positive definite for all t E [to, tf). Since we al- ready know that P(t) is symmetric, we now have that P(t) is positive definite, symmetric matrix.

7. Computation of Matrix DRE: Under some conditions we can get analytical solution for the nonlinear matrix DRE as shown later. But in general, we may try to solve the matrix DRE (3.2.18) by integrating backwards from its known final condition (3.2.19).

116 Chapter 3: Linear Quadratic Optimal Control Systems I

8. Independence of the Riccati Coefficient Matrix P(t): The matrix P (t) is independent of the optimal state x* (t), so that once the system and the cost are specified, that is, once we are given the system/plant matrices A(t) and B(t), and the performance index matrices F(tf), Q(t), and R(t), we can independently compute the matrix P (t) before the optimal system operates in the forward direction from its initial condition. Typically, we compute (off- line) the matrix P(t) backward in the interval t E [tf, to] and store them separately, and feed these stored values when the system is operating in the forward direction in the interval t E [to, t f] .

9. Implementation of the Optimal Control: The block diagram im- plementing the closed-loop optimal controller (CLOC) is shown in Figure 3.2. The figure shows clearly that the CLOC gets its values of P(t) externally, after solving the matrix DRE backward in time from t = t f to t = to and hence there is no way that we can implement the closed-loop optimal control configuration on-line.

It is to be noted that the optimal control u*(t) can be solved and implemented in open-loop configuration by using the Pontryagin procedure given in Chapter 2. In that case, the open-loop optimal controller (OLOC) is quite cumbersome compared to the equiv- alent closed-loop optimal controller as will be illustrated later in this chapter.

10. Linear Optimal Control: The optimal feedback control u*(t) given by (3.2.12) is written as

I u*(t) = -K(t)x*(t) I (3.2.45)

where, the Kalman gain K(t) = R- 1 (t)B'(t)P(t). Or alterna- tively, we can write

u*(t) = -K~(t)x*(t) (3.2.46)

where, Ka(t) = P(t)B(t)R-l(t). The previous optimal control is linear in state x* (t). This is one of the nice features of the optimal control of linear systems with quadratic cost functionals. Also, note that the negative feedback in the optimal control relation (3.2.46) emerged from the theory of optimal control and was not introduced intentionally in our development.

3.2 Finite-Time Linear Quadratic Regulator 117 .- - .... - .... - .... - - .. - ...................... - - .... - - .... - - .... - ...................... - ...... - ...... ..

u*(t)B: B(t) +

J x*(t)

Plant ...... - - .. .... .. ........ - - ....................... - - ...... - ...................... - - .... - - .... - _.. .. •• 1

........................ - ..................... -- - ............ - - .... - ...... -- .. .. .. .... _ .... .

Closed-Loop Optimal Controller

-1 R (t)B'(t)

P(t)x*(t)

Off-Line Simulation of pet)

Figure 3.2 Closed-Loop Optimal Control Implementation

11. Controllability: Do we need the controllability condition on the system for implementing the optimal feedback control? No, as long as we are dealing with a finite time (t f) system, because the contribution of those uncontrollable states (which are also unstable) to the cost function is still a finite quantity only. How- ever, if we consider an infinite time interval, we certainly need the controllability condition, as we will see in the next section.

A historical note is very appropriate on the Riccati equation [22, 132].

The matrix Riccati equation has its origin in the scalar ver- sion of the equation

x(t) = ax2 (t) + bx(t) + c (3.2.47)

with time varying coefficients, proposed by Jacopo Franceso Riccati around 1715. Riccati (1676-1754) gave the methods of solutions to the Riccati equation. However, the original paper by Riccati was not published immediately because he

118 Chapter 3: Linear Quadratic Optimal Control Systems I

had the "suspicion" that the work was already known to people such as the B ernoullis.

The importance of the Riccati equation, which has been studied in the last two centuries by an extensive number of scientists and engineers, need not be overstressed. The matrix Riccati equation, which is a generalization in matrix form of the original scalar equation, plays a very important role in a range of control and systems theory areas such as linear quadratic optimal control, stability, stochastic filter- ing and control, synthesis of passive networks, differential games and more recently, Hoo-control and robust stabiliza- tion and control.

Did Riccati ever imagine that his equation, proposed more than a quarter millennium ago, would play such an impor- tant and ubiquitous role in modern control engineering and other related fields?

3.2.6 LQR System for General Performance Index

In this subsection, we address the state regulator system with a more general performance index than given by (3.2.2). Consider a linear, time-varying plant described by

x(t) = A(t)x(t) + B(t)u(t), (3.2.48)

with a cost functional

1 J(u) = 2X' (t f )F(t f )x(t f)

1 rtf +2 ito [x/(t)Q(t)x(t) + 2X/(t)SU(t) + u/(t)R(t)u(t)] dt

= ~ X' ( t f ) F ( t f ) x ( t f )

1 rtf [, ,] [Q(t) S(t)] [X(t)] +2 ito x (t) u (t) S/(t) R(t) u(t) dt, (3.2.49)

where, the various vectors and matrices are defined in earlier sections and the nxr matrix S(t) is only a positive definite matrix.

3.3 Anal. Sol. to the Matrix Differential Riccati Equation 119

Using the identical procedure as for the LQR system, we get the matrix differential Riccati equation as

P(t) = -P(t)A(t) - A' (t)P(t) - Q(t)

+ [P ( t) B ( t) + S ( t)] R -1 ( t ) [B I ( t ) P ( t) + S I ( t) ] (3.2.50)

with the final condition on P(t) as

(3.2.51 )

The optimal control is then given by

u(t) = _R-1(t)B/(t) [S/(t) + P(t)] x(t). (3.2.52)

Obviously, when S(t) is made zero in the previous analysis, we get the previous results shown in Table 3.1.

3.3 Analytical Solution to the Matrix Differential Riccati Equation

In this section, we explore an analytical solution for the matric dif- ferential Riccati equation (DRE). This material is based on [138, 89]. Let us rewrite the Hamiltonian system (3.2.8) of the state and costate equations for the time-invariant case as (omitting * for the sake of simplicity)

[ X(t)] [A -E] [X(t)] A(t) = -Q -A' A(t)

(3.3.1)

where, E = BR-1B/. Let

[ A -E] ~ = -Q -A' . (3.3.2) Let us also recall that by the transformation .\(t) = P(t)x(t), we get the differential matrix Riccati equation (3.2.18), rewritten for (time- invariant matrices A, B, Q and R) as

P(t) = -P(t)A - A/p(t) - Q + P(t)BR -lB /p(t), (3.3.3)

with the final condition

(3.3.4)

120 Chapter 3: Linear Quadratic Optimal Control Systems I

The solution P(t) can be obtained analytically (in contrast to numer- ical integration) in terms of the eigenvalues and eigenvectors of the Hamiltonian matrix ~. In order to find analytical solution to the dif- ferential Riccati equation (3.3.3), it is necessary to show that if /1 is an eigenvalue of the Hamiltonian matrix ~ in (3.3.2), then it implies that -/1 is also the eigenvalue of ~ [89, 3]. For this, let us define

r = [ 0 I] -10

(3.3.5)

so that r- 1 = -r. Then by a simple pre- and post-multiplication with r we get

~ = r~/r = -r~/r-l. (3.3.6)

Now, if /1 is an eigenvalue of ~ with corresponding eigenvector v,

~v = /1V (3.3.7)

then

f ~/rv = 1.J,v, ~/rv = - /1fv (3.3.8)

where, we used f- 1 = -f. Rearranging

(3.3.9)

Next, rearranging the eigenvalues of ~ as

(3.3.10)

where, M( -M) is a diagonal matrix with right-half-plane (left-half plane) eigenvalues. Let W, the modal matrix of eigenvectors corre- sponding to D, be defined as

W = [Wn W12] , W21 W22

(3.3.11)

where, [Wn W 21 ]' are the n eigenvectors of the left-half-plane (stable) eigenvalues of ~. Also,

(3.3.12)

3.3 Anal. Sol. to the Matrix Differential Riccati Equation 121

Let us now define a state transformation

[ X(t)] = w [W(t)] = [Wl1 W12] [W(t)] . A(t) z(t) W 21 W22 z(t)

(3.3.13)

Then, using (3.3.12) and (3.3.13), the Hamiltonian system (3.3.1) be- comes

[~~~)] = W- 1 [~i!n = W-l~ [~i!n = W-l~ W [~g)]

= D [ :i!\ ] . (3.3.14) Solving (3.3.14) in terms of the known final conditions, we have

[~~~)] = [e-M~-tf) eM(?-t/)] [~~~)] . Rewriting (3.3.15)

Next, from (3.3.13) and using the final condition (3.3.4)

A(tf) = W 21 W(tf) + W22 Z(tf) = Fx(tf)

= F [Wl1w(tf) + W12Z(tf)] .

Solving the previous relation for z( t f) in terms of w( t f)

z(tf) = T(tf )w(tf), where

(3.3.15)

(3.3.16)

(3.3.17)

T(tf) = - [W22 - FW12]-1 [W21 - FWl1 ]. (3.3.18)

Again, from (3.3.16)

z(t) = e-M(tf-t)z(tf)

= e-M(tf-t)T( t f )w( t f) = e-M(tf-t)T(tf )e-M(tf-t)w(t). (3.3.19)

Rewriting the previous relation as

z(t) = T(t)w(t), where, T(t) = e-M(tf-t)T(tf)e-M(tf-t). (3.3.20)

122 Chapter 3: Linear Quadratic Optimal Control Systems I

Finally, to relate P(t) in (3.3.3) to the relation (3.3.20) for T(t), let us use (3.3.13) to write

..\(t) = W21W(t) + W22Z(t) = P(t)x(t)

= P(t) [WllW(t) + W 12 Z(t)]

and by (3.3.20), the previous relation can be written as

(3.3.21)

[W21 + W22T(t)] w(t) = P(t) [Wll + W12T(t)] w(t). (3.3.22)

Since the previous relation should hold good for all x(to) and hence for all states w(t), it implies that the analytical expression to the solution of P(t) is given by

(3.3.23)

The solution of the matrix DRE (3.2.34) is not readily available with MATLAB and hence a MATLAB-based program was developed for solving the matrix DRE based on the analytical solution of matrix DRE [138] described earlier. The MATLAB solution is illustrated by the following example.

Example 3.1

Let us illustrate the previous procedure with a simple second order example. Given a double integral system

Xl(t)=X2(t), xl(0)=2 X2(t) = -2Xl(t) + X2(t) + u(t), X2(0) = -3, (3.3.24)

and the performance index (PI)

J = ~ [xi(5) + xl(5)X2(5) + 2x~(5)] 1 (5

+2 io [2xi(t) + 6Xl(t)X2(t) + 5x~(t) + 0.25u2(t)] dt, (3.3.25)

obtain the feedback control law.

3.3 Anal. Sol. to the Matrix Differential Riccati Equation 123

Solution: Comparing the present plant (3.3.24) and the PI (3.3.25) of the problem with the corresponding general formulations of the plant (3.2.31) and the PI (3.2.32), respectively, let us first identify the various quantities as

A(t) = [ ~2 ~ ] ;

Q(t) = [~~] ;

B(t) = [~]; F(tf) = [0~5 025] 1

R(t) = r(t) = 4; to = 0; tf = 5.

It is easy to check that the system (3.3.24) is unstable. Let P(t) be the 2x2 symmetric matrix

P(t) = [Pl1(t) P12(t)] . P12(t) P22(t)

(3.3.26)

Then, the optimal control (3.2.33) is given by

u*(t) = -4 [0 1] [Pl1(t) P12(t)] [Xi(t)] P12(t) P22(t) X2(t)

= -4[P12(t)xi(t) + P22(t)X;(t)] (3.3.27)

where, P(t), the 2x2 symmetric, positive definite matrix, is the solution of the matrix DRE (3.2.34)

[ Pl1(t) P12(t)] [Pl1(t) P12(t)] [ ° 1] P12(t) P22(t) = - P12(t) P22(t) -2 1

_ [0 -2] [Pl1(t) P12(t)] 1 1 P12(t) P22(t)

+ [Pl1 (t) P12(t)] [0] 4 [0 1] [Pl1 (t) P12(t)] P12(t) P22(t) 1 P12(t) P22(t)

- [~~] (3.3.28) satisfying the final condition (3.2.35)

[ Pl1 (5) P12 (5)] _ [ 1 0.5] P12(5) P22(5) - 0.5 2 .

(3.3.29)

124 Chapter 3: Linear Quadratic Optimal Control Systems I

Simplifying the matrix DRE (3.3.28), we get

pn(5) = 1, P12(t) = -pn(t) - P12(t) + 2p22(t) + 4P12 (t)p22(t) - 3,

P12(5) = 0.5

P22(t) = -2p12(t) - 2p22(t) + 4P~2(t) - 5, P22(5) = 2.

(3.3.30)

Solving the previous set of nonlinear, differential equations back- ward in time with the given final conditions, one can obtain the numerical solutions for the Riccati coefficient matrix P(t). How- ever, here the solutions are obtained using the analytical solution as given earlier in this section. The solutions for the Riccati coef- ficients are plotted in Figure 3.3. Using these Riccati coefficients, the closed-loop optimal control system is shown in Figure 3.4. Us- ing the optimal control u*(t) given by (3.3.27), the plant equations (3.3.24) are solved forward in time to obtain the optimal states xi(t) and x2(t) as shown in Figure 3.5 for the initial conditions [2 - 3]'. Finally, the optimal control u*(t) is shown in Figure 3.6. The previous resJlJ.ts are obtained using Control System Toolbox of the MATLAB\9, Version 6 as shown below.

The following MATLAB© m file for Example 3.1 requires two additional MATLAB© files lqrnss. m which itself requires lqrnssf. m given in Appendix C.

**************************************************** %% Solution using Control System Toolbox of %% the MATLAB. Version 6 %% The following file example.m requires %% two other files lqrnss.m and lqrnssf.m %% which are given in Appendix clear all A= [0. ,1. ; -2. ,1.] ; B= [0. ; 1.] ; Q= [2. ,3. ; 3. ,5.] ; F=[1. ,0.5;0.5,2.]; R= [. 25] ; tspan=[05]; xO=[2. ,-3.]; [x,u,K]=lqrnss(A,B,F,Q,R,xO,tspan); ******************************************************

3.4 Infinite- Time LQR System I 125

Plot of Riccati Coefficients 2r---~---r----r---~--~----~--~--~----r---~

I I I I I I I I I I

I I I I _ (t" , I I I 1.8 - - - - ~ - - - - -:- - - - - ~ - - - - -:- Pt~~~ - - - - ~ - - - - - ~ - - - - ~ - - - - -:- - - - -

1.6

2 1.4 c: Q)

~ 1.2 Q)

o (...)

'-;; u u

C2 0.8

0.6

0.4

I I I I I

I I I I I I I I I

----,-----~----l----~-----i----'-----r----'--- -~---- I I I I I I I I I

I ____ J _____ ~ ____ ! ____ ~ _____ L ____ ~ _____ ~ ____ J _____ ~ __ _ I I I I I I I I I

I I I I I I

I I I I I I I I I ____ ~ _____ ~ ____ ~ ____ ~ _____ L ____ ~ _____ ~ ____ ~ _____ ~ __ _

I I I , , I I I I

----~-----~----.----~-----+----~-----~----1-----~---- I I I I I

I I ,

I I I I I I I I I I I I I I ----'-----r----'----'-----r----'-----r----'-----r---- I I I I , , I I I , , I , I , I , I

____ ~ _____ : _____ ~ _____ :_ 24 ~~X ____ ~ _____ ~ ____ ~ _____ : ____ _

0.5 1.5 2 2.5 3 3.5 4 4.5

Figure 3.3 Riccati Coefficients for Example 3.1

3.4 Infinite-Time LQR System I

In this section, let us make the terminal (final) time t f to be infinite in the previous linear, time-varying, quadratic regulator system. Then, this is called the infinite-time (or infinite horizon) linear quadratic reg- ulator system [6, 3].

Consider a linear, time-varying plant

x(t) = A(t)x(t) + B(t)u(t), (3.4.1)

and a quadratic performance index

J = ~ tx) [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt, 2 ito

(3.4.2)

where, u(t) is not constrained. Also, Q(t) is nxn symmetric, positive semidefinite matrix, and R(t) is an rxr symmetric, positive definite matrix. Note, it makes no engineering sense to have a terminal cost term with terminal time being infinite.

This problem cannot always be solved without some special condi- tions. For example, if anyone of the states is uncontrollable and/or un- stable, the corresponding performance measure J will become infinite

126 Chapter 3: Linear Quadratic Optimal Control Systems I .- .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ...................................................... -.

I 1 2

u*(t) +0-. J X2*(t~ J Xl *(t) : ... Plant .. .. .. .. .. .. .. .. .. .... ............................................................................................ .. .............. ...

r .. .. .. .. .. .. .. .. .. ..... .................................................... ...................................... .. ............ -.

~-------------+~~ ~.+~------~

Figure 3.4 Closed-Loop Optimal Control System for Example 3.1

and makes no physical sense. On the other hand, with the finite-time system, the performance measure is always finite. Thus, we need to impose the condition that the system (3.4.1) is completely controllable.

U sing results similar to the previous case of finite final time t f (see Table 3.1), the optimal control for the infinite-horizon linear regulator system is obtained as

u*(t) = _R-l(t)B'(t)P(t)x*(t), (3.4.3)

where,

P(t) = lim {P(t)} , tf--+oo

(3.4.4)

the nxn symmetric, positive definite matrix (for all t E [to, tf]) is the solution of the matrix differential Riccati equation (DRE)

P(t) = -P(t)A(t) - A'(t)P(t) - Q(t) + P(t)B(t)R- 1 (t)B'(t)P(t), (3.4.5)

3.4 Infinite-Time LQR System I

(/) Q)

CG ti5 (; E 'E,. 0

g c: o U (; E :a o

Plot of Optimal States 2~--~--~----~--~----~--~----~--~--~--~

1.5

0.5 ~----~-----~----~-----~----~----~-----1 1 1 1 1 1 1 1 1 1

I I I I I f I I

0 ____ J _____ ~ ____ l ____ ~ _____ L ____ J __________________ _

I I I I I I

1 1 1 1 1 1 1 1 1

-0.5 I I I I I I I I

- - - - ~ -x 2 "(t) -1- - - ~ - - - - -:- - - - - ~ - - - - i - - - - -:- - - - - -: - - - - -:- - - - -

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

-1 ----~----.----~-----.----~-----~----.-----~----1 1

I I I I I I I I I

-1.5 _J _____ ~ ____ J ____ J _____ L ____ ~ _____ ~ ____ J ____ ~ ____ _

1 1 1 1 1 1 1 1 1 1 1 1 1 I I I I

-2 1 1 I I I I I I I - --,-----~----i----~-----i----'-----r----'-----~---- 1 I I 1 I 1 I 1 1 I I I I

-2.5 I , I I I I I I I

----~-----~----~----'-----T----~-----~----~-----~---- 1 1 I I 1 1 I I I

I I I 1

_3L---~--~----~--~----~--~--~~--~--~--~ o 0.5 1.5 2 2.5 3 3.5 4 4.5 5

Figure 3.5 Optimal States for Example 3.1

Plot of Optimal Control 15~----~----~----~~----~----~----~--------~----~----~----~

0.5

1 I I I I I I I

----~----- ----T----'-----r----'---------- I

1 1 1 1

----1----------.----~-----~----1-----1

1.5 2.5 3

I I I I

3.5 4.5

Figure 3.6 Optimal Control for Example 3.1

127

128 Chapter 3: Linear Quadratic Optimal Control Systems I

satisfying the final condition

(3.4.6)

The optimal cost is given by

1 A J* = -x*'(t)P(t)x*(t) 2 . (3.4.7)

The proofs for the previous results are found in optimal control text specializing in quadratic methods [3]. Example 3.1 can be easily solved for t f -+ 00 and F = o.

3.4.1 Infinite-Time Linear Quadratic Regulator: Time- Varying Case: Summary

Consider a linear, time-varying plant

x(t) = A(t)x(t) + B(t)u(t), (3.4.8)

and a quadratic performance index

1100 J = - [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt, 2 to

(3.4.9)

where, u(t) is not constrained and x(tf), tf -+ 00 is not specified. Also, Q(t) is nxn symmetric, positive semidefinite matrix, and R(t) is rxr symmetric, positive definite matrix. Then, the optimal control is given by

I u*(t) = -R-1 (t)B'(t)P(t)x*(t) I (3.4.10)

where, P(t), the nxn symmetric, positive definite matrix (for all t E [to, t f], is the solution of the matrix differential Riccati equation (DRE)

I P(t) = -P(t)A(t) - A'(t)P(t) - Q(t) + P(t)B(t)R- 1 (t)B'(t)P(t) I (3.4.11)

satisfying the final condition

P(t = tf -+ (0) = O. (3.4.12)

3.5 Infinite-Time LQR System II

Table 3.2 Procedure Summary of Infinite-Time Linear Quadratic Regulator System: Time-Varying Case

A. Statement of the Problem Given the plant as x(t) = A(t)x(t) + B(t)u(t), the performance index as

J = ~ ftC; [x'(t)Q(t)x(t) + u'(t)R(t)u(t)] dt, and the boundary conditions as x(to) = Xo; x( (0) is free, find the optimal control, state and performance index.

B. Solution of the Problem Step 1 ~olve the matrix differential Riccati equation (DRE)

129

P(t) = -P(t)A(t) - A'(t)P(t) - Q(t) + P(t)B(t)R- 1 (t)B'(t)P(t) with final condition P(t = t f) = O.

Step 2 Solve the optimal state x*(t) from

x*(t) = [A(t) - B(t)R- 1 (t)B'(t)P(t)] x*(t)

with initial condition x(to) = Xo. Step 3 Obtain the optimal control u*(t) from

u*(t) = -R- 1 (t)B'(t)P(t)x*(t). Step 4 Obtain the optimal performance index from

J* = ~x*'(t)P(t)x*(t).

The optimal state is the solution of

I x*(t) = [A(t) - B(t)R- 1 (t)B'(t)P(t)] x*(t) I (3.4.13)

and the optimal cost is

1 A J* = "2x*'(t)P(t)x*(t). (3.4.14)

The optimal control u*(t) given by (3.4.10) is linear in the optimal state x*(t). The entire procedure is now summarized in Table 3.2.

3.5 Infinite-Time LQR System II In this section, we examine the state regulator system with infinite time interval for a linear time-invariant (LTI) system. Let us consider the

130 Chapter 3: Linear Quadratic Optimal Control Systems I

plant as

x(t) = Ax(t) + Bu(t) (3.5.1)

and the cost functional as

11000 J = - [x'(t)Qx(t) + u'(t)Ru(t)] dt 2 0

(3.5.2)

where, x(t) is nth order state vector; u(t) is rth order control vec- tor; A is nxn-order state matrix; B is rxr-order control matrix; Q is nxn-order, symmetric, positive semidefinite matrix; R is rxr-order, symmetric, positive definite matrix. First of all, let us discuss some of the implications of the time-invariance and the infinite final-time.

1. The infinite time interval case is considered for the following rea- sons:

(a) We wish to make sure that the state-regulator stays near zero state after the initial transient.

(b) We want to include any special case of large final time.

2. With infinite final-time interval, to include the final cost function does not make any practical sense. Hence, the final cost term involving F(tf) does not exist in the cost functional (3.5.2).

3. With infinite final-time interval, the system (3.5.1) has to be com- pletely controllable. Let us recall that this controllability condi- tion of the plant (3.5.1) requires that the controllability matrix (see Appendix B)

(3.5.3)

must be nonsingular or contain n linearly independent column vectors. The controllability requirement guarantees that the op- timal cost is finite. On the other hand, if the system is not control- lable and some or all of those uncontrollable states are unstable, then the cost functional would be infinite since the control inter- val is infinite. In such situations, we cannot distinguish optimal control from the other controls. Alternatively, we can assume that the system (3.5.1) is completely stabilizable.

3.5 Infinite-Time LQR System II 131

As before in the case of finite final-time interval, we can proceed and obtain the closed-loop optimal control and the associated Riccati equa- tion. Still P (t) must be the solution of the matrix differential Riccati equation (3.2.34) with boundary condition P(tf) = O. It was shown that the assumptions of [70]

1. controllability and

imply that

(3.5.4)

where, P is the nxn positive definite, symmetric, constant matrix. If P is constant, then P is the solution of the nonlinear, matrix, algebraic Riccati equation (ARE),

dP = 0 = -P A - A'P + PBR-1B'P - Q. dt

Alternatively, we can write (3.5.5) as

PA + A'P + Q - PBR-1B'P = o.

(3.5.5)

(3.5.6)

Note, for a time-varying system with finite-time interval, we have the differential Riccati equation (3.2.34), whereas for a linear time-invariant system with infinite-time horizon, we have the algebraic Riccati equa- tion (3.5.6).

A historical note on R.E. Kalman is appropriate (from SIAM News, 6/94 - article about R.E. Kalman).

Rudolph E. Kalman is best known for the linear filtering technique that he and Richard Bucy [31] developed in 1960- 1961 to strip unwanted noise out of a stream of data [71, 74, 76]. The Kalman filter, which is based on the use of state-space techniques and recursive algorithms, revolution- ized the field of estimation. The Kalman filter is widely used in navigational and guidance systems, radar tracking, sonar ranging, and satellite orbit determination (for the Ranger, Apollo, and Mariner missions, for instance), as well as in fields as diverse as seismic data processing, nuclear power

132 Chapter 3: Linear Quadratic Optimal Control Systems I

plant instrumentation, and econometrics. Among Kalman's many outstanding contributions were the formulation and study of most fundamental state-space notions [72, 73, 77j including controllability, observability, minimality, realiz- ability from input and output data, matrix Riccati equa- tions, linear-quadratic control [70, 75, 75j, and the sepa- ration principle that are today ubiquitous in control. While some of these concepts were also encountered in other con- texts, such as optimal control theory, it was Kalman who recognized the central role that they play in systems analy- sis.

Born in Hungary, Kalman received BS and MS degrees from the Massachusetts Institute of Technology (MIT) and a DSci in engineering from Columbia University in 1957. In the early years of his career he held research positions at In- ternational Business Machines (IBM) and at the Research Institute for Advanced Studies (RIAS) in Baltimore. From 1962 to 1971, he was at Stanford University. In 1971, he became a graduate research professor and director of the Center for Mathematical System Theory at the University of Florida, Gainesville, USA, and later retired with emeri- tus status. Kalman's contributions to control theory and to applied mathematics and engineering in general have been widely recognized with several honors and awards.

3.5.1 Meaningful Interpretation of Riccati Coefficient

Consider the matrix differential Riccati equation (3.2.34) with final condition P(tf) = O. Now consider a simple time transformation T = t f - t. Then, in T scale we can think of the final time t f as the "starting time," P(tf) as the "initial condition," and P as the "steady-state solution" of the matrix DRE. As the time t f ---+ 00, the "transient solution" is pushed to near t f which is at infinity. Then for most of the practical time interval the matrix P (t) becomes a steady state, i.e., a constant matrix P, as shown in Figure 3.7 [6]. Then the optimal control is given by

u*(t) = -R-1B'Px*(t) = -Kx*(t), (3.5.7)

3.5 Infinite-Time LQR System II 133

P(t)= P

1---------------------4) ,

O~.-.steady-state interval-----·~ ~--- transient interval---~tf --+ t i :' 0 +-1 f

Figure 3.7 Interpretation of the Constant Matrix P

where, K = R -1 B'P is called the Kalman gain. Alternatively, we can write

u*(t) = -K~x*(t) (3.5.8)

where, Ka = PBR -1. The optimal state is the solution of the system obtained by using the control (3.5.8) in the plant (3.5.1)

x*(t) = [A - BR- 1B'P] x*(t) = Gx*(t), (3.5.9)

where, the matrix G = A - BR -1 B'P must have stable eigenvalues so that the closed-loop optimal system (3.5.9) is stable. This is required since any unstable states with infinite time interval would lead to an in- finite cost functional J*. Let us note that we have no constraint on the stability of the original system (3.5.1). This means that although the original system may be unstable, the optimal system must be definitely stable.

Finally, the minimum cost (3.2.29) is given by

1 , - J* - -x* (t)Px*(t) -2 . (3.5.10)

3.5.2 Analytical Solution of the Algebraic Riccati Equation

The next step is to find the analytical expression to the steady-state (limiting) solution of the differential Riccati equation (3.3.3). Thus, we are interested in finding the analytical solution to the algebraic Riccati equation (3.5.5). Obviously, one can let the terminal time t f tend to

134 Chapter 3: Linear Quadratic Optimal Control Systems I

00 in the solution (3.3.23) for P(tf). As tf ---t 00, e-M(tf- t ) goes to zero, which in turn makes T(t) tend to zero. Thus, under the usual conditions of (A, B) being stabilizable and (A, v'Q) being reachable, and T = 0, we have from (3.3.23)

I tJ~oo P(t, tt) = P = W 21 W 11·1 (3.5.11) Thus, the solution to the ARE is constructed by using the stable eigen- vectors of the Hamiltonian matrix. For further treatment on this topic, consult [3] and the references therein.

3.5.3 Infinite-Interval Regulator System: Time-Invariant Case: Summary

For a controllable, linear, time-invariant plant

x(t) = Ax(t) + Bu(t), (3.5.12)

and the infinite interval cost functional

1 tx) J = "210 [x'(t)Qx(t) + u'(t)Ru(t)] dt, (3.5.13)

the optimal control is given by

I u*(t) = -R-1B'Px*(t) I (3.5.14)

where, P, the nxn constant, positive definite, symmetric matrix, is the solution of the nonlinear, matrix algebraic Riccati equation (ARE)

/-PA - A'P + PBR-1B'P - Q = 0 I (3.5.15)

the optimal trajectory is the solution of

(3.5.16)

and the optimal cost is given by

1 - J* = -x*'(t)Px*(t) 2 . (3.5.17)

The entire procedure is now summarized in Table 3.3 and the imple- mentation of the closed-loop optimal control (CLOC) is shown in Fig- ure 3.8

3.5 Innnite- Time LQR System II 135

u*(t) J B x*(t) : .:

- A

Plant

,.---- ------_ .. _------------------_ ... _----- ... -------------

...

Closed-Loop Optimal Controller t ...................................................................................................................... 1

Figure 3.8 Implementation of the Closed-Loop Optimal Control: Infinite Final Time

Next, an example is given to illustrate the infinite interval regulator system and the associated matrix algebraic Riccati equation. Let us reconsider the same Example 3.1 with final time t f ----* 00 and F = O.

Example 3.2

Given a second order plant

~h(t)=X2(t), Xl(O) =2 X2(t) = -2Xl(t) + X2(t) + u(t), X2(O) = -3 (3.5.18)

and the performance index

obtain the feedback optimal control law.

Solution: Comparing the plant (3.5.18) and PI (3.5.19) of the present system with the corresponding general formulation of plant (3.5.12) and PI (3.5.13), respectively, let us first identify the various

136 Chapter 3: Linear Quadratic Optimal Control Systems I

Table 3.3 Procedure Summary of Infinite-Interval Linear Quadratic Regulator System: Time-Invariant Case

A. Statement of the Problem Given the plant as x(t) = Ax(t) + Bu(t), the performance index as

J = ~ Iooo [x'(t)Qx(t) + u'(t)Ru(t)] dt, and the boundary conditions as x(to) = Xo; x(oo) = 0, find the optimal control, state and index.

B. Solution of the Problem Step 1 Solve the matrix algebraic Riccati equation (ARE)

-PA - A'P - Q + PBR-1B'P = 0 .. Step 2 Solve the optimal state x* (t) from

x*(t) = [A - BR-1B'P] x*(t) with initial condition x(to) = Xo.

Step 3 Obtain the optimal control u * ( t) from u*(t) = -R-1B'Px*(t).

Step 4 Obtain the optimal performance index from J* = ~x*'(t)Px*(t).

matrices as

A = [~2 ~] ; Q= [;;];

1 R=r =-'

4' to = 0;

Let P be the 2x2 symmetric matrix

P = [~11 ~12] . P12 P22

Then, the optimal control ( 3.5.14) is given by

u*(t) = -4 [0 1] [~1l ~12] [XI(t)] , P12 P22 x2(t)

= -4(jh2xi(t) + P22 x ;(t)],

(3.5.20)

(3.5.21 )

(3.5.22)

(3.5.23)

3.5 InEnite- Time LQR System II 137

where, f>, the 2x2 symmetric, positive definite matrix, is the solu- tion of the matrix algebraic Riccati equation (3.5.15)

[ 0 0] = _ [~11 ~12] [ ° 1] _ [0 -2] [~11 ~12] + ° ° P12 P22 - 2 1 1 1 P12 P22

[ ~11 ~12] [0] 4 [0 1] [~11 ~12] _ [23] . P12 P22 1 P12 P22 3 5 Simplifying the equation (3.5.24), we get

4PI2 + 4P12 - 2 = ° -P11 - P12 + 2P22 + 4P12P22 - 3 = °

-2P12 - 2P22 + 4P~2 - 5 = 0.

(3.5.24)

(3.5.25)

Solving the previous equations for positive definiteness of f> is easy in this particular case. Thus, solve the first equation in (3.5.25) for P12, using this value of P12 in the third equation solve for P22 and finally using the values of P12 and P22 in the second equation, solve for P11. In general, we have to solve the nonlinear algebraic equations and pick up the positive definite values for f>. Hence, we get

f> = [1.7363 0.3660] 0.3660 1.4729 .

(3.5.26)

Using these Riccati coefficients (gains), the closed-loop optimal control (3.5.23) is given by

u*(t) = -4[0.366xi(t) + 1.4729x2(t)] = -[1.464xi(t) + 5.8916x2(t)]. (3.5.27)

Using the closed-loop optimal control u*(t) from (3.5.27) in the original open-loop system (3.5.18), the closed-loop optimal system becomes

xi(t) = X2(t) X2(t) = -2xi(t) + x;(t) - 4[0.366xi(t) + 1.4729x;(t)] (3.5.28)

and the implementation of the closed-loop optimal control is shown in Figure 3.9.

U sing the initial conditions and the Riccati coefficient matrix (3.5.26), the optimal cost (3.5.17) is obtained as

J* = ~ '(O)f> (0) = ~ [2 -3] [1.73630.3660] [ 2 ] 2

x x 2 0.3660 1.4729 -3'

= 7.9047. (3.5.29)

138 Chapter 3: Linear Quadratic Optimal Control Systems I -- .. _----------_ .. _------ .. _--

1 2

u*(t) + + J J Plant

4 1.4729 0.366

Closed-Loop Optimal Controller

Figure 3.9 Closed-Loop Optimal Control System

The previous results can also easily obtained using Control System Toolbox of the MATLABC9, Version 6 as shown below.

********************************************* %% Solution using Control System Toolbox in %% The MATLAB. Version 6 %% For Example:4-3 %%

x10=2; %% initial condition on state x1 x20=-3; %% initial condition on state x2 XO=[x10;x20]; A=[O 1;-2 1]; %% system matrix A B=[O;1]; %% system matrix B Q=[2 3;3 5]; %% performance index weighted matrix R=[O.25]; %% performance index weighted matrix [K,P,EV]=lqr(A,B,Q,R) %% K = feedback matrix;

%% P = Riccati matrix; %% EV = eigenvalues of closed loop system A - B*K K =

3.5 Infinite-Time LQR System II

1.4641 p

1.7363 0.3660

EV = -4.0326 -0.8590

5.8916

0.3660 1.4729

BIN=[O;O]; % dummy BIN for "initial" command C= [1 1]; D= [1] ; tfinal=10; t=0:0.05:10; [Y,X,t]=initial(A-B*K,BIN,C,D,XO,tfinal); x1t=[1 O]*X'; %% extracting xi from vector X x2t=[0 1]*X'; %% extracting x2 from vector X ut=-K*X' ; plot(t,x1t, 'k' ,t,x2t, 'k') xlabel ( , t ' ) gtext ('x_1 (t)') gtext ('x_2(t)') plot (t , ut , ' k ' ) xlabel ('t') gt ext ( , u ( t) , )

**************************************************

139

Using the optimal control u*(t) given by (3.5.23), the plant equations (3.5.18) are solved using MATLAB© to obtain the optimal states xi(t) and x2(t) and the optimal control u*(t) as shown in Figure 3.10 and Figure 3.11. Note that

1. the values of P obtained in the example, are exactly the steady- state values of Example 3.1 and

2. the original plant (3.5.18) is unstable (eigenvalues at 2 ± j1) whereas the optimal dosed-loop system (3.5.28) is stable (eigen- values at -4.0326, -0.8590).

3.5.4 Stability Issues of Time-Invariant Regulator

Let us consider the previous result for linear time-invariant system with infinite-time horizon from relations (3.5.12) to (3.5.17) and Table 3.3. We address briefly some stability remarks of the infinite-time regulator system [3, 89].

140 Chapter 3: Linear Quadratic Optimal Control Systems I

2~--~--~--~--~--~--~--~--~--~--~

1.5

0.5

-1

-3~--~--~--~--~--~--~--~--~--~--~ o 2 3 4 5 6 7 8 9 10

Figure 3.10 Optimal States for Example 3.2

1. The closed-loop optimal system (3.5.16) is not always stable espe- cially when the original plant is unstable and these unstable states are not weighted in the PI (3.5.13). In order to prevent such a sit- uation, we need the assumption that the pair [A, C] is detectable, where C is any matrix such that C'C = Q, which guarantees the stability of closed-loop optimal system. This assumption essen- tially ensures that all the potentially unstable states will show up in the x'(t)Qx(t) part of the performance measure.

2. The Riccati coefficient matrix P is positive definite if and only if [A, C] is completely observable.

3. The detectability condition is necessary for stability of the closed- loop optimal system.

4. Thus both detectabilityand stabilizability conditions are necessary for the existence of a stable closed-loop system.

3.5 Infinite- Time LQR System II

15r---r---r---r---r---r---r---r---~--~--,

2 3 4 5 6 7 8 9 10

Figure 3.11 Optimal Control for Example 3.2

3.5.5 Equivalence of Open-Loop and Closed-Loop Optimal Controls

141

Next, we present a simple example to show an interesting property that an optimal control system can be solved and implemented as an open- loop optimal control (OLOC) configuration or a closed-loop optimal control (CLOC) configuration. We will also demonstrate the simplicity of the CLOC.

Example 3.3

Consider a simple first order system

x(t) = -3x(t) + u(t) (3.5.30)

and the cost function (CF) as

(3.5.31 )

where, x(O) = 1 and the final state x( (0) = O. Find the open-loop and closed-loop optimal controllers.

142 Chapter 3: Linear Quadratic Optimal Control Systems I

Solution: (a) Open-Loop Optimal Control: We use the Pon- tryagin procedure given in Chapter 2 (see Table 2.1). First of all, comparing the given plant (3.5.30) and the CF (3.5.31) with the general formulations (see Table 2.1), identify that

V(x(t), u(t)) = x2(t) + u2(t), f(x(t), u(t)) = -3x(t) + u(t). (3.5.32)

Now, we use the step-by-step procedure given in Table 2.1.

• Step 1: Formulate the Hamiltonian as

1t = V(x(t), u(t)) + A(t)f(x(t), u(t)) = x2(t) + u2(t) + A(t)[-3x(t) + u(t)]. (3.5.33)

• Step 2: The optimal control u*(t) is obtained by minimizing the previous Hamiltonian w.r.t. u as

~1t = 0 ---+ 2u*(t) + A*(t) = 0 ---+ u*(t) = -~A*(t). (3.5.34) uu 2

• Step 3: Using optimal control (3.5.34) in the Hamiltonian func- tion (3.5.33), find the optimal Hamiltonian function as

2 1 2 1t* = x* (t) - 4A* (t) - 3A*(t)x*(t). (3.5.35)

• Step 4: Using the previous optimal 1t*, obtain the set of state and costate equations

x*(t) = a:;;* ---+ x*(t) = -~A*(t) - 3x*(t), (3.5.36) . a1t*· A*(t) = - ax ---+ A*(t) = -2x*(t) + 3A*(t), (3.5.37)

yielding

x*(t) - 10x*(t) = 0, (3.5.38)

the solution of which becomes

x*(t) = CleViOt + C2e-ViOt. (3.5.39)

Using the optimal state (3.5.39) in (3.5.36), we have the costate as

A*(t) = 2 [-x*(t) - 3x*(t)]

= -2C1 (ViO + 3)eViOt + 2C2 (ViO - 3)e-ViOt. (3.5.40)

3.5 Infinite-Time LQR System II 143

Using the initial condition x(O) = 1 in the optimal state (3.5.39), and the final condition (for 8 x f being free) ).. ( t f = 00) = 0 in the optimal costate (3.5.40), we get

x(O) = 1 ----* C 1 + C 2 = 1 )..(00) = 0 ----* C1 = O. (3.5.41 )

Then, the previous optimal state and costate are given as

x*(t) = e-v'IOt; )..*(t) = 2(v'lO - 3)e-v'IOt. (3.5.42)

• Step 5: Using the previous costate solution (3.5.34) of Step 2, we get the open-loop optimal control as

(3.5.43)

(b) Closed-Loop Optimal Control: Here, we use the matrix algebraic Riccati equation (ARE) to find the closed-loop optimal control, as summarized in Table 3.3. First of all, comparing the present plant (3.5.30) and the PI (3.5.31) with the general formu- lation of the plant (3.5.12) and the PI (3.5.13), respectively, we identify the various coefficients and matrices as

A = a = -3; B = b = 1; Q = q = 2; R = r = 2; f> = p. (3.5.44)

Note the PI (3.5.31) does not contain the factor 1/2 as in the general PI (3.5.13) and accordingly, we have Q = q = 2 and R = r = 2.

• Step 1: With the previous values, the ARE (3.5.15) becomes

1 p(-3) + (-3)p - p(l)("2)(l)p+ 2 = 0----*

p2 + 12p - 4 = 0, (3.5.45)

the solution of which is

p = -6 ± 2v'lO. (3.5.46)

• Step 2: Using the positive value of the Riccati coefficient (3.5.46), the closed-loop optimal control (3.5.14) becomes

1 u*(t) = -r- 1bpx*(t) = -"2(-6 + 2v'lO)x*(t)

= -( -3 + v'lO)x*(t). (3.5.47)

144 Chapter 3: Linear Quadratic Optimal Control Systems I

• Step 3: Using the optimal control (3.5.47), the optimal state is solved from (3.5.16) as

x(t) = -3x*(t) - (-3 + v'lO)x*(t) = -v'lOx*(t). (3.5.48) Solving the previous along with the initial condition x(O) = 1, we get the optimal state as

x*(t) = e-v'IOt (3.5.49)

with which the optimal control (3.5.47) becomes

(3.5.50)

Thus, we note that the optimal control (3.5.50) and optimal state (3.5.49) obtained from using the closed-loop optimal control are identical to those of (3.5.43) and (3.5.42), respectively. We can easily extend this analysis for the general case. Intuitively, this equivalence should exist as the optimal control being unique should be same by any method.

The implementation of this open-loop optimal controller (OLOC) is shown in Figure 3.12(a), and that of the closed-loop optimal controller (CLOC) is shown in Figure 3.12(b).

From the previous example, it is clear that

1. from the implementation point of view, the closed-loop optimal controller (v'IO - 3) is much simpler than the open-loop optimal controller (( v'IO-3)e-v'IOt) which is an exponential time function and

2. with a closed-loop configuration, all the advantages of conven- tional feedback are incorporated.

3.6 Notes and Discussion We know that linear quadratic optimal control is concerned with lin- ear plants, performance measures quadratic in controls and states, and regulation and tracking errors. In particular, the resulting optimal controller is closed-loop and linear in state. Note that linear quadratic optimal control is a special class of the general optimal control which includes nonlinear systems and nonlinear performance measures. There are many useful advantages and attractive features of linear quadratic optimal control systems which are enumerated below [3].

3.6 Notes and Discussion 145 .... -----_ .............. --_ .. ..---------_ .... _---------------------_ ..... _-

x*(t)

+ J

OLOC -3

Plant: ~ ......................................................................... ........ I

(a)

u*(t) +

+ J x*(t)

Plant -3

•............. " ........................ ............, ....... ------------ ---- .. _ .. __ ...... _-_ .... _-----_ .......... -----_ ......... _-----_ ...... _-_ ....... -_ ....... , ,

'------------~/I0-3~----~

, , ----_ .... _ ....................... _------_ .... __ .. -- ... -_ ........ _---_ .... _---_ .... _ .. CLOC (b)

Figure 3.12 (a) Open-Loop Optimal Controller (OLOC) and (b) Closed-Loop Optimal Controller (CLOC)

1. Many engineering and physical systems operate in linear range during normal operations.

2. There is a wealth of theoretical results available for linear systems which can be useful for linear quadratic methods.

3. The resulting optimal controller is linear in state and thus easy and simple for implementation purposes in real application of the LQ results.

4. Many (nonlinear) optimal control systems do not have solutions which can be easily computed. On the other hand, LQ optimal control systems have easily computable solutions.

5. As is well known, nonlinear systems can be examined for small variations from their normal operating conditions. For example,

146 Chapter 3: Linear Quadratic Optimal Control Systems I

assume that after a heavy computational effort, we obtained an optimal solution for a nonlinear plant and there is a small change from an operating point. Then, one can easily use to a first approximation a linear model and obtain linear optimal control to drive the original nonlinear system to its operating point.

6. Many of the concepts, techniques and computational procedures that are developed for linear optimal control systems in many cases many be carried on to nonlinear optimal control systems.

7. Linear optimal control designs for plants whose states are mea- surable possess a number of desirable robustness properties (such as good gain and phase margins and a good tolerance to nonlin- earities) of classical control designs.

3.7 Problems 147

3.7 Problems

1. Make reasonable assumptions wherever necessary.

2. Use MATLAB© wherever possible to solve the problems and plot all the optimal controls and states for all problems. Provide the relevant MATLAB© m files.

Problem 3.1 A first order system is given by

±(t) = x(t) + u(t).

(a) Find the unconstrained optimal control law which minimizes the performance index

such that the final time t f is fixed and the final state x(t f) is free. (b) Find the optimal control law as t f -* 00.

Problem 3.2 A system is described by

x(t) + x(t) = u(t)

with initial conditions x(O) = 0 and ±(O) = 1 and the performance index

Find the closed-loop optimal control in terms of x and x and the optimal cost function.

Problem 3.3 A first order system is described by

±(t) = ax(t) + bu(t)

with performance index

148 Chapter 3: Linear Quadratic Optimal Control Systems I

and with a fixed initial state x(O) and final state x(tf) = 0, where tf is fixed. Show that the solution of the Riccati equation is given by

r p(t) = b2 [a - ;3coth{;3(t - tf)}]

and the solution of the optimal state x* (t) is given by

*( ) _ ()sinh{(3(tf - t)} x t - x 0 . h(3

S'ln

where, (3 = Ja 2 + b2q/r.

Problem 3.4 Find the optimal feedback control for the plant

XI(t) = X2(t)

X2(t) = -2XI(t) + 4X2(t) + 5u(t) with performance criterion

121 2 J = 2f11 [xI(tf) - 4] + 2f22 [x2(tf) - 2] +

~ If [5xi(t) + 2x~(t) + 4u2 (t)] dt and initial conditions as x(O) = [1 2]' and the final state x(tf) is free, where t f is specified.

Problem 3.5 Find the closed-loop, unconstrained, optimal control for the system

Xl (t) = X2(t)

X2(t) = -2XI (t) - 3X2 (t) + u(t)

and the performance index

J = 10 00

[xi(t) + x~(t) + u2(t)]dt.

Problem 3.6 Find the optimal feedback control law for the plant

;.h(t) = X2(t) + u(t) X2(t) = -XI(t) - X2(t) + u(t)

and the cost function

J = 10 00

[2xi(t) + 4x~(t) + O.5u2(t)]dt.

3.7 Problems 149

Problem 3.7 Consider a second order system

x(t) + bx(t) + cx(t) = u(t)

and the performance index to be minimized as

Determine the closed-loop optimal control in terms of the state x(t) and its derivative x(t).

Problem 3.8 Given a third order plant,

Xl(t) = X2(t)

X2(t) = X3(t)

X3(t) = -5Xl(t) + -7X2(t) -10X3(t) + 4u(t)

and the performance index

for

1. ql1 = q22 = q33 = 1, r = 1,

2. ql1 = 10, q22 = 1, q33 = 1, r = 1, and

3. ql1 = q22 = q33 = 1, r = 10,

find the positive definite solution for Riccati coefficient matrix f>, op- timal feedback gain matrix K and the eigenvalues of the closed-loop system matrix A - BK.

Problem 3.9 Determine the optimal feedback coefficients and the op- timal control law for the multi-input, multi-output (MIMO) system

. [01] [11] x(t) = 11 x(t) + 0 1 u(t)

and the cost function

J = foCXJ [2xr(t) + 4x~(t) + 0.5ur(t) + 0.25u~(t)]dt.

150 Chapter 3: Linear Quadratic Optimal Control Systems I

Problem 3.10 For the D.C. motor speed control system described in Problem 1.1, find the closed-loop optimal control to keep the speed constant at a particular value and the system to respond for any dis- turbances from the regulated value.

Problem 3.11 For the liquid-level control system described in Prob- lem 1.2, find the closed-loop optimal control to keep the liquid level constant at a reference value and the system to act only if there is a change in the liquid level.

Problem 3.12 [35] For the inverted pendulum control system described in Problem 1.3, find- the closed-loop optimal control to keep the pen- dulum in a vertical position.

Problem 3.13 For the mechanical control system described in Prob- lem 1.4, find the closed-loop optimal control to keep the system at equilibrium condition and act only if there is a disturbance.

Problem 3.14 For the automobile suspension system described in Problem 1.5, find the closed-loop optimal control to keep the system at equilibrium condition and act only if there is a disturbance.

Problem 3.15 For the chemical control system described in Prob- lem 1.6, find the closed-loop optimal control to keep the system at equilibrium condition and act only if there is a disturbance.

@@@@@@@@@@@@@

Chapter 4

Linear Quadratic Optimal Control Systems II

In the previous chapter, we addressed the linear quadratic regulator system, where the aim was to obtain the optimal control to regulate (or keep) the state around zero. In this chapter, we discuss linear quadratic tracking (LQT) systems, and some related topics in linear quadratic regulator theory. It is suggested that the student reviews the material in Appendices A and B given at the end of the book. This chapter is based on [6, 89, 3] 1.

Trajectory Following Systems In tracking (trajectory following) systems, we require that the output of a system track or follow a desired trajectory in some optimal sense. Thus, we see that this is a generalization of regulator system in the sense that the desired trajectory for the regulator is simply the zero state.

IThe permissions given by John Wiley for F. L. Lewis, Optimal Control, John Wiley & Sons, Inc., New York, NY, 1986 and McGraw-Hill for M. Athans and P. L. Falb, Optimal Control: An Introduction to the Theory and Its Applications, McGraw-Hill Book Company, New York, NY, 1966, are hereby acknowledged.

151

152 Chapter 4: Linear Quadratic Optimal Control Systems II

4.1 Linear Quadratic Tracking System: Finite- Time Case

In this section, we discuss the linear quadratic tracking (LQT)system to maintain the output as close as possible to the desired output with minimum control energy [6]. We are given a linear, observable system

x(t) = A(t)x(t) + B(t)u(t) y(t) = C(t)x(t) (4.1.1)

where, x(t) is the nth order state vector, u(t) is the rth order control vector, and y(t) is the mth order output vector. Let z(t) be the mth order desired output and the various matrices A(t), B(t) and C(t) be of appropriate dimensionality. Our objective is to control the system (4.1.1) in such a way that the output y(t) tracks the desired output z(t) as close as possible during the interval [to, tf] with minimum ex- penditure of control effort. For this, let us define the error vector as

e(t) = z(t) - y(t)

and choose the performance index as

1 , J = 2e (tf)F(tf)e(tf)

lit! +-2 [e'(t)Q(t)e(t) + u'(t)R(t)u(t)] dt

(4.1.2)

(4.1.3)

with t f specified and x( t f) not specified. In this way we are dealing with free-final state system. Also, we assume that F(tf) and Q(t) are mxm symmetric, positive semidefinite matrices, and R( t) is rxr sym- metric, positive definite matrix. We now use the Pontryagin Minimum Principle in the following order.

• Step 1 : Hamiltonian

• Step 2: Open-Loop Optimal Control

• Step 3: State and Costate System

• Step 4: Riccati and Vector Equations

• Step 5: Closed-Loop Optimal Control

• Step 6: Optimal State

4.1 Linear Quadratic Tracking System: Finite-Time Case 153

• Step 7: Optimal Cost

Now we discuss these steps in detail. Also, note that we heavily draw upon the results of the previous Chapters 2 and 3. First of all, let us note from (4.1.1) and (4.1.2) that the error e(t) can be expressed as a function of z(t) and x(t) as

e(t) = z(t) - C(t)x(t). (4.1.4)

• Step 1: Hamiltonian: Formulate the Hamiltonian as (see Ta- ble 3.1)

1 1t(x(t), u(t), ,X(t)) = "2 [z(t) - C(t)x(t)]' Q(t) [z(t) - C(t)x(t)]

1 +"2 u' ( t ) R ( t ) u ( t) +,X' (t) [A(t)x(t) + B(t)u(t)] . (4.1.5)

• Step 2: Open-Loop Optimal Control: Using the Hamiltonian (4.1.5), obtain the control equation from

~~ = 0 --> R(t)u(t) + B'(t),\(t) = 0 (4.1.6)

from which we have the optimal control as

I u*(t) = -R- 1 (t)B'(t),X*(t)·1 (4.1.7)

Since the second partial of 1t in (4.1.5) w.r.t. u*(t) is just R(t), and we chose R( t) to be positive definite, we are dealing with a control which minimizes the cost functional (4.1.3).

• Step 3: State and Costate System: The state is given in terms of the Hamiltonian (4.1.5) as

x(t) = ~~ = A(t)x(t) + B(t)u(t) (4.1.8)

and with the optimal control (4.1.7), the optimal state equation (4.1.8) becomes

x*(t) = A(t)x*(t) - B(t)R-l(t)B'(t),X*(t). (4.1.9)

154 Chapter 4: Linear Quadratic Optimal Control Systems II

Using the Hamiltonian (4.1.5), the optimal costate equation be- comes

. * 81t A (t) = - ax

= -C'(t)Q(t)C(t)x*(t) - A'(t)A*(t)

+C'(t)Q(t)z(t). (4.1.10)

For the sake of simplicity, let us define

E(t) = B(t)R-l(t)B'(t), W(t) = C'(t)Q(t).

V(t) = C' (t)Q(t)C(t),

(4.1.11)

Using the relation (4.1.11) and combining the state (4.1.8) and costate (4.1.10) equations, we obtain the Hamiltonian canonical system as

[ x*(t)] [A(t) -E(t)] [X*(t)] [0] '-\*(t) = -V(t) -A'(t) A*(t) + W(t) z(t). (4.1.12)

This canonical system of 2n differential equations is linear, time- varying, but nonhomogeneous with W(t)z(t) as forcing function. The boundary conditions for this state and costate equations are given by the initial condition on the state as

x(t = to) = x(to) (4.1.13)

and the final condition on the costate (for the final time t f spec- ified and state x(tf) being free) given by (3.2.10), which along with (4.1.4) become

A(tf l = &X~t f l [~e'(tf )F(tf le(tf l] = ax~tfl [~[z(tfl - C(tflx(tfl]'F(tfl[z(tfl - C(tflX(tflJ] = C'(tf)F(tf)C(tf)x(tf) - C'(tf)F(tf)z(tf). (4.1.14)

• Step 4: Riccati and Vector Equations: The boundary condition (4.1.14) and the solution of the system (4.1.12) indicate that the state and costate are linearly related as

A*(t) = P(t)x*(t) - g(t) (4.1.15)

4.1 Linear Quadratic Tracking System: Finite-Time Case 155

where, the nxn matrix P(t) and n vector g(t) are yet to be de- termined so as to satisfy the canonical system (4.1.12). This is done by substituting the linear (Riccati) transformation (4.1.15) in the Hamiltonian system (4.1.12) and eliminating the costate function A * (t). Thus, we first differentiate (4.1.15) to get

'\*(t) = P(t)x*(t) + P(t)x*(t) - g(t). (4.1.16)

Now, substituting for x*(t) and '\*(t) from (4.1.12) and eliminat- ing A*(t) with (4.1.15), we get

-V(t)x*(t) - A'(t) [P(t)x*(t) - g(t)] + W(t)z(t) = P(t)x*(t) +P(t) [A(t)x(t) - E(t) {P(t)x*(t) - g(t)}] - g(t). (4.1.17)

Rearranging the above, we get

[P(t) + P(t)A(t) + A'(t)P(t) - P(t)E(t)P(t) + V(t)] x*(t) - [g(t) + A'(t)g(t) - P(t)E(t)g(t) + W(t)z(t)] = O.

(4.1.18)

Now, this relation (4.1.18) must satisfy for all x* (t), z (t) and t, which leads us to the nxn matrix P (t) to satisfy the matrix differential Riccati equation (DRE)

I P(t) = -P(t)A(t) - A'(t)P(t) + P(t)E(t)P(t) - V(t) I ( 4.1.19)

and the n vector g( t) to satisfy the vector differential equation

I g(t) = [P(t)E(t) - A'(t)] g(t) - W(t)z(t).1 (4.1.20)

Since P(t) is nxn symmetric matrix, and g(t) is of n vector, the equations (4.1.19) and (4.1.20) are a set of n(n + 1)/2 + n first-order differential equations. The boundary conditions are obtained from (4.1.15) as

(4.1.21)

which compared with the boundary condition (4.1.14) gives us for all x(tf) and z(tf),

1 P ( t f) = C' ( t f ) F ( t f) C ( t f ), 1

1 g ( t f) = C' ( t f ) F ( t f ) Z ( t f ) ·1

( 4.1.22)

(4.1.23)

156 Chapter 4: Linear Quadratic Optimal Control Systems II

Thus, the matrix DRE (4.1.19) and the vector equation (4.1.20) are to be solved backward using the boundary conditions (4.1.22) and (4.1.23).

• Step 5: Closed-Loop Optimal Control: The optimal control (4.1.7) is now given in terms of the state using the linear transformation ( 4.1.15)

u*(t) = -R -l(t)B' (t) [P(t)x* (t) - g(t)] = -K(t)x*(t) + R-l(t)B'(t)g(t)

where, K(t) = R-1 (t)B'(t)P(t), is the Kalman gain.

( 4.1.24)

• Step 6: Optimal State: Using this optimal control u*(t) from (4.1.24) in the original plant (4.1.1), we have the optimal state obtained from

x*(t) = [A(t) - B(t)R- 1 (t)B'(t)P(t)] x*(t)

+B(t)R -l(t)B' (t)g(t)

= [A(t) - E(t)P(t)]x*(t) + E(t)g(t). (4.1.25)

• Step 7: Optimal Cost: The optimal cost J*(t) for any time t can be obtained as (see [6])

1 , I J*(t) = 2x* (t)P(t)x*(t) - x* (t)g(t) + h(t) (4.1.26)

where, the new function h(t) satisfies [3, 6]

.1 1 1 h(t) = -2 g'(t)B(t)R- (t)B'(t)g(t) - 2z'(t)Q(t)z(t)

1 1 = -2 g'(t)E(t)g(t) - 2z'(t)Q(t)z(t) (4.1.27)

with final condition

(4.1.28)

For further details on this, see [3, 6, 89, 90]. We now summarize the tracking system.

4.1 Linear Quadratic Tracking System: Finite-Time Case 157

f x*(t)

C(t) y* t)

A(t)

Plant

Closed-Loop Optimal Controller

P(t)x*(t)

g(t)

Off-Line Simulation of P(t) and g(t)

Desired z( t) I ............................................................................................................................................................ I

Figure 4.1 Implementation of the Optimal Tracking System

4.1.1 Linear Quadratic Tracking System: Summary

Given the linear, observable system (see Figure 4.1)

x(t) = A(t)x(t) + B(t)u(t) y(t) = C(t)x(t) ( 4.1.29)

the desired output z(t), the error e(t) = z(t) - y(t), and the perfor- mance index

1 lit! J = -e'(tf)F(tf)e(tf) + - [e'(t)Q(t)e(t) + u'(t)R(t)u(t)] dt

2 2 to (4.1.30)

158 Chapter 4: Linear Quadratic Optimal Control Systems II

then the optimal control u*(t) is given by

u*(t) = -R-1 (t)B'(t) [P(t)x*(t) - g(t)]

= -K(t)x*(t) + R- 1 (t)B'(t)g(t) (4.1.31)

where, the nxn symmetric, positive definite matrix P(t) is the solution of the nonlinear, matrix differential Riccati equation (DRE)

P(t) = -P(t)A(t) - A'(t)P(t) + P(t)E(t)P(t) - V(t) (4.1.32)

with final condition

(4.1.33)

and the nth order g(t) is the solution of the linear, nonhomogeneous vector differential equation

g(t) = - [A(t) - E(t)P(t)]' g(t) - W(t)z(t) (4.1.34)

with final condition

(4.1.35)

where, E(t), V(t) and W(t) are defined in (4.1.11), the optimal state (trajectory) is the solution of the linear state equation

x*(t) = [A(t) - E'(t)P(t)] x*(t) + E(t)g(t), ( 4.1.36)

and the optimal cost J*

J*(to) = ~x*' (to)P(to)x*(to) - x*(to)g(to) + h(to). (4.1.37)

The implementation of the tracking system is shown in Figure 4.1. The entire procedure is now summarized in Table 4.1.

4.1.2 Salient Features of Tracking System

1. Riccati Coefficient Matrix P(t): We note that the desired output z(t) has no influence on the matrix differential Riccati equation (4.1.32) and its boundary condition (4.1.33). This means that once the problem is specified in terms of the final time t f' the plant matrices A(t), B(t), and C(t), and the cost functional ma- trices F(tf), Q(t), and R(t), the matrix function P(t) is com- pletely determined.

4.1 Linear Quadratic Tracking System: Finite- Time Case 159

Table 4.1 Procedure Summary of Linear Quadratic Tracking System

A. Statement of the Problem Given the plant as x(t) = A(t)x(t) + B(t)u(t), y(t) = C(t)x(t), e(t) = z(t) - y(t), the performance index as

J = ~e'(tf )F(tf )e(tf) + ~ ftd [e'(t)Q(t)e(t) + u'(t)R(t)u(t)] dt, and the boundary conditions as x(to) = Xo, x(tj) is free, find the optimal control, state and performance index.

B. Solution of the Problem Step 1 Solve the matrix differential Riccati equation

P(t) = -P(t)A(t) - A'(t)P(t) + P(t)E(t)P(t) - V(t), with final condition P(tj) = C'(tj )F(tj )C(tj), and the non-homogeneous vector differential equation g(t) = - [A(t) - E(t)P(t)]' g(t) - W(t)z(t), with final condition g( t j) = C' (t j )F( t j )z( t j) where E(t) = B(t)R-1(t)B'(t), V(t) = C'(t)Q(t)C(t), W(t) = C'(t)Q(t).

Step 2 Solve the optimal state x* (t) from x*(t) = [A(t) - E(t)P(t)] x*(t) + E(t)g(t) with initial condition x(to) = Xo.

Step 3 Obtain optimal control u*(t) from u*(t) = -K(t)x*(t) + R -1 (t)B' (t)g(t), where, K(t) = R- 1(t)B'(t)P(t).

Step 4 The optimal cost J* (to) is J*(to) = ~x*'(to)P(to)x*(to) - x*(to)g(to) + h(to) where h(t) is the solution of b.(t) = - ~g' (t)E(t)g( t) - ~z' (t)Q( t)z(t) with final condition h(tj) = -z'(tj )P(tj )z(tj).

160 Chapter 4: Linear Quadratic Optimal Control Systems II

2. Closed Loop Eigenvalues: From the costate relation (4.1.36), we see the closed-loop system matrix [A(t) - B(t)R-I(t)B'(t)P(t)] is again independent of the desired output z(t). This means the eigenvalues of the closed-loop, optimal tracking system are inde- pendent of the desired output z(t).

3. Tracking and Regulator Systems: The main difference between the optimal output tracking system and the optimal state regulator system is in the vector g(t). As shown in Figure 4.1, one can think of the desired output z(t) as the forcing function of the closed-loop optimal system which generates the signal g(t).

4. Also, note that if we make C(t) = I(t), then in (4.1.11), V(t) = Q(t). Thus, the matrix DRE (4.1.19) becomes the same matrix DRE (3.2.34) that was obtained in LQR system in Chapter 3.

Let us consider a second order example to illustrate the linear quadratic tracking system.

Example 4.1

A second order plant

::h (t) = X2(t), X2(t) = -2XI(t) - 3X2(t) + u(t) y(t) = x(t)

is to be controlled to minimize the performance index

(4.1.38)

(4.1.39)

The final time t f is specified at 20, the final state x( tJ) is free and the admissible controls and states are unbounded. It is required to keep the state XI(t) close to 1. Obtain the feedback control law and plot all the time histories of Riccati coefficients, g vector components, optimal states and control.

Solution: The performance index indicates that the state Xl (t) is to be kept close to the reference input Zl(t) = 1 and since there is no condition on state X2(t), one can choose arbitrarily as Z2(t) = o. Now, in our present case, with e(t) = z(t) - Cx(t), we have el(t) = Zl(t) - XI(t), e2(t) = Z2(t) - X2(t) and C = I.

4.1 Linear Quadratic Tracking System: Finite-Time Case 161

Next, let us identify the various matrices in the present tracking system by comparing the state (4.1.38) and the PI (4.1.39) of the present system (note the absence of the factor 1/2 in PI) with the corresponding state (4.1.29) and the PI (4.1.30), respectively, of the general formulation of the problem, we get

A = [ ~2 _~ ]; B = [~]; C = I; z(t} = [~] ;

Q = [~~] = F(tf}; R = r = 0.004. (4.1.40) Let P(t) be the 2x2 symmetric matrix and g(t) be the 2x1 vector as

P(t) - [Pl1 (t) P12 (t)]. - P12(t) P22(t) ,

g ( t) = [gl ( t) ]. g2(t)

Then, the optimal control given by (4.1.31) becomes

u*(t) = -250 [P12xi(t) + P22X2(t) - g2(t)]

(4.1.41)

(4.1.42)

where, P(t), the positive definite matrix, is the solution of the matrix differential Riccati equation (4.1.32)

[ Pll(t) P12(t)] [Pll(t) P12(t)] [0 1] P12(t) P22(t) = - P12(t) P22(t) -2 -3

_ [0 -2] [Pl1 ( t ) P12 ( t ) ] 1 -3 P12(t) P22(t)

+ [Pll (t) P12(t)] [0] 1 [0 1] P12(t) P22(t) 1 0.004 x

[ Pll(t) P12(t)] _ [20] P12(t) P22(t) 00 (4.1.43)

and g(t), is the solution of the nonhomogeneous vector differential equation obtained from (4.1.34) as

162 Chapter 4: Linear Quadratic Optimal Control Systems II

Simplifying the equations (4.1.43) and (4.1.44), we get

Pn (t) = 250PI2(t) + 4pI2(t) - 2 PI2(t) = 250PI2(t)P22(t) - pn(t) + 3PI2(t) + 2p22(t) P22(t) = 250p~2(t) - 2pI2(t) + 6p22(t) (4.1.45)

with final condition (4.1.33) as

( 4.1.46)

and

[/1(t) = [250PI2(t) + 2] g2(t) - 2 92(t) = -gl(t) + [3 + 250P22 (t)] g2(t) (4.1.47)

with final condition

( 4.1.48)

Note: One has to try various values of the matrix R in order to get a better tracking of the states. Solutions for the functions Pn(t),PI2(t), andp22(t) (Figure 4.2), functions gl(t) and g2(t) (Fig- ure 4.3), optimal states (Figures 4.4) and control (Figure 4.5) for initial conditions x(O) = f-=:-0.5 0] and the final time t f = 20 are obtained using MATLAB© routines given in Appendix C under continuous-time tracking system.

Example 4.2

Consider the same Example 4.1 with a different PI as

(4.1.49)

where, t f is specified and x( t f) is free. Find the optimal control in order that the state Xl (t) track a ramp function Zl (t) = 2t and without much expenditure of control energy. Plot all the variables (Riccati coefficients, optimal states and control) for initial condi- tions x(O) = [-1 0]'.

Solution: The performance index (4.1.49) indicates that the state Xl (t) is to be kept close to the reference input Zl (t) = 2t and since there is no condition on state X2 (t), one can choose arbitrarily as Z2(t) = O. Now, in our present case, with e(t) = z(t) ~ Cx(t), we have el(t) = Zl(t) - XI(t), e2(t) = Z2(t) - X2(t) and C = I.

4.1 Linear Quadratic Tracking System: Finite-Time Case 163

Plot ofP 1.4 r----.----..----..l---.----.----r-; ---r-I: ---r-:I --r-I: -.....,

I ••• I • I I I I I I • I

1 2 ------! .. ----- ~ --- _ .. - -:---- -- -~ -- - ...... ! .. ----- ~ .. -- -- - ~------ -~ .. -- - --~ -- ---- . : : : : (t) : : : : :

:: : P11 : :: • I • I • • I • I

1 ----- -1 --- ---1---- ---r-- ----; -- -- --1 --- ---1- -- -- -y- -- ---;------r- -- -- E ::::::::: Q) :::::::::

:~ O~8 .. --- ... t .... ----of-- -_ .... -:- ----- -~ -- ........ f .... ----; .. -- -_ .. -:-- -_ .. --:- -- .... --~ -- -_ .. ~ ::::::::: o :::::::::

<..> ::::::::: .~ 0.6 ----- -;''' --- --i -- -_ .... -:----- .. -~ -_ ........ ~ .... ---- i- ---- .. ~-- -----r -- .... -- r---"" u ::::::::: .~ • I I I It. • I

Cl: :::::::::

0.4 --- -- -t ------i -- -----1---- -- -~ ------t -- -- -- i- -- ---i----- --~ ------r- -- -- : : : : P12(t): : : : : t I • I I • • • I

02 ----- -. -- -- --~ -- -- -- -1---- -- -.. ------. -- -- -- ~ --- -- -.. ---- --- .. ---- --. ---~ . : : : : . : : : : : : : : P22(tl : : : : • I • I • , I • I

Figure 4.2 Riccati Coefficients for Example 4.1

Next, let us identify the various matrices in the present tracking system by comparing the state (4.1.38) and the PI (4.1.39) of the present problem (note the absence of the factor 1/2 in PI) with the corresponding state (4.1.29) and the PI (4.1.30), respectively, of the general formulation of the problem, we get

A = [ ~2 _!]; B = [~] ; C = I; z(t) = [~] ; Q = [~~] R = r = 0.04. (4.1.50)

Let P(t) be the 2x2 symmetric matrix and g(t) be the 2x1 vector as

P(t) = [Pl1(t) P12(t)] . P12(t) P22(t) ,

g ( t) = [91 ( t )] . 92(t)

Then, the optimal control given by (4.1.31) becomes

(4.1.51)

(4.1.52)

where, P(t), the positive definite matrix, is the solution of the

164

.... o U 0)

> C)

Chapter 4: Linear Quadratic Optimal Control Systems II

2.5~----TII----~:------TII----~:------r~I----~:----~:~----~,I----~l~--~

• I t I · ... I I · ... I •• 1= ,I I I , • • • • I •

2 -- --- -r --- ---1--- ----r--- --T- -----! -- ----1- --- --r-----T ------r- ----- : : : : : ; : : : : : : : : : : : :

1 5 I , , • , J t • • . r- -----r -- --- -1- ----- T- -----r ------ r------1--- -- -T ------r -- --- -r----- I

I I I I I I I

• I • • • I • I I · . . , . . . . . _ .... ---: .. _ ........ : ... -- ...... -:--- .. ---:- ........ --: .. -_ .. --1----- --:--- --- -: ............ : --- .. .. • • I • • · ... . . · . I. . . : : 91 (t):: : : J • • • • I I • • I

0.5 • I I • • t • t I ----- -I ------1---- --T -----T- --- --I ---- --1-- ----r-- -- -T ----- -r ---- -- J

I I I I I I 1 I 2 4 6 8 10 12 14 16 18 20

Figure 4.3 Coefficients 91 (t) and 92 (t) for Example 4.1

3.5.-----~!:-----,:~----.!:-----,I:------:r-----TI:-----,:------rl:-----TI:-----, I I

• I • I I I • • • • • • I I • • • •

3 -----I ----- -:- -- ----r--- ---r --- ---1--- ---: -- ----r------:---- --r ------ • • I I • • • • • I I • • • •

2.5 --- --j --- ---r---r--- -r -----r----1------r ---- -r---- r- ----- 00 2 ....... --- f .. -----i- -- ...... -:- .......... -:- .... ----f-- ........ ; -_ .. -- .. -:- .... ----:- ........ --i------ ~ ::::::::: til :::::::::

1.5 - -- -- t .. -----~- -- ...... -:- --- .... -r"" -- -- t-- .... -- ~ ...... -_ .. ~- -- ----~ .... ---- ~ .. ----- • • • I I • • • • • • • I I • • • •

: : x1 (t): : : : : : : 1 -v-~--~----¥---~--~--~~--¥---~--~--~

0.5

I I

: : : : : : : : : -----1- -----1- -----T ------i ------! ------1- -----T ------i ------1- ----- o __ \,: : .,(t~ : : : : : : _,

iii iii iii

2 4 6 8 10 12 14 16 18 20

Figure 4.4 Optimal States for Example 4.1

4.1 Linear Quadratic Tracking System: Finite-Time Case 165

30~~--~--~--~~--~--~--~~--~ I

I • • I I I I I •

20 ---- r --r----r----r----r---r----r----r----r---- i 10 -- -- --r-- -r---r ----or ---- -r----r- -- --r---r ---r--- I 0 --- -r---r----r---T ----r --or ----r---T -----r ----

-10 -- ---1----- -r- -- -r-----r---- --! ---- -"/"- ---or ---r----r----- -20 ---- -r--- r-- --r--- -r-----:----or ---r-----r- --- --r----- -30 L--..L. __ --L..-__ -L-__ L--.l.._-..I.-_-L-_.1.--.l._--I 024 6 8 10 12 14 16 18 20

Figure 4.5 Optimal Control for Example 4.1

matrix differential Riccati equation (4.1.32)

[ Pll(t) P12(t)] [Pll(t) P12(t)] [0 1] P12(t) P22(t) = - P12(t) P22(t) -2 -3

_ [0 -2] [Pll(t) P12(t)] 1 -3 P12(t) P22(t)

+ [Pll(t) P12(t)] [0]_1 [01] P12(t) P22(t) 1 0.04 .

[ Pll (t) P12(t)]_ [2 0] P12(t) P22(t) 00 (4.1.53)

and g(t), is the solution of the nonhomogeneous vector differential equation obtained from (4.1.34) as

[ !~ i ~ n = - { [ ~ 2 - ~ ] - [~] O_~4 [ 0 1] [:~~m :~~m] r [:~i~i] -[~ n [~~][ ~]- (4_1.54)

166 Chapter 4: Linear Quadratic Optimal Control Systems II

Simplifying the equations (4.1.53) and (4.1.54), we get

PH(t) = 25Pi2(t) + 4p12(t) - 2 P12(t) = 25p12(t)P22(t) - PH (t) + 3p12(t) + 2p22(t) P22(t) = 25p~2(t) - 2p12(t) + 6p22(t) (4.1.55)

with final condition (4.1.33) as

and

[ PH (t f) P12 (t f) ] = [0 0] P12 ( t f) P22 (t f) 0 0

iJ1(t) = [25p12(t) + 2] g2(t) - 4t i}2(t) = -gl(t) + [3 + 25p22(t)] g2(t)

with final condition

[~~i:~ n = [~] .

(4.1.56)

(4.1.57)

(4.1.58)

See the plots of the Riccati coefficients PH(t),P12(t) and P22(t) in Figure 4.6 and coefficients gl(t) and g2(t) in Figure 4.7. Also see the plots of the optimal control u*(t) in Figure 4.9 and optimal states xi(t) and x2(t) in Figure 4.8.

4.2 LQT System: Infinite-Time Case In Chapter 3, in the case of linear quadratic regulator system, we ex- tended the results of finite-time case to infinite-time (limiting or steady- state) case. Similarly, we now extend the results of finite-time case of the linear quadratic tracking system to the case of infinite time [3]. Thus, we restrict our treatment to time-invariant matrices in the plant and the performance index. Consider a linear time-invariant plant as

The error is

x(t) = Ax(t) + Bu(t) y(t) = Cx(t).

e(t) = z(t) - y(t),

and choose the performance index as

lim J = lim ~ roo [e'(t)Qe(t) + u'(t)Ru(t)] dt tr-+ oo tj->oo 2 io

(4.2.1) (4.2.2)

(4.2.3)

( 4.2.4)

4.2 LQT System: Infinite-Time Case 167

1.4 ...----r-" --T""" -~:-~:-----":-~----.:--:...----r-----. , , ,

• • • t. ,. • I I I. I • • • I. ••

1.2 ..... -- --: -- ----~---- ...... -:---- ...... -~ -- ........... : ........... -- ...... ----~--- ----~ ............ -- ------ I I • I' ,.

: : : : P11 (t~ :: • • I I I • I • I

1 -- -- -- t -_ .. -_ .. ~----- .. -:- ...... -- -~ --- .. -- t ...... ---1- ----- ~- -- -_ .. -~ .. -_ .. -- ~--- -- I , • I I • • • • I I • I I • • I I

-E ::: : Q) •••••••••

:Q 0.8 r--'" ---- f- ... -- .... i-------:- ...... -- -:- .... -- -- t ......... --i- -- ----:- -- ----:- ............. r-----I- ~ ::::::::: o :::::::::

{} ::::::::: .~ 0.6 -- -- --;. -- -_ .. ; .. --.---:---- ..... -~ -- -- --;. --- .. -i .. -- -_ .. ~-- ---- -~ -- -- .... ~ .......... (.) ::::::::: .~ ...., I I • I

ex: ::::::::: • • • I t I • , I

0.4 ..... -- --!- -- -_ ... ~ .. ----- -:---- ..... -~ ..... -- --f----- -~ -- -- .... ~- .... -- --:- -- -- --:- -- .... • • • • 'I I' I • • • •• ••

: P12(t): :

Figure 4.6 Riccati Coefficients for Example 4.2

to track the desired signal z(t). Also, we assume that Q is an nxn symmetric, positive semidefinite matrix, and R is a rxr symmetric, positive definite matrix. Note that there is no terminal cost function in the PI (4.2.4) and hence F = O.

An obvious way of getting results for infinite-time (steady-state) case is to write down the results of finite-time case and then simply let tf ---+ 00. Thus, as tf ---+ 00, the matrix function P(t) in (4.1.19) tends to the steady-state value P as the solution of

J-PA - A'P + PBR- 1B'P - C'QC = 0.1 (4.2.5)

Also, the vector function g(t) in (4.1.20) tends to a finite function g(t) as the solution of

g(t) = [PE - A'] g(t) - Wz(t) (4.2.6)

where, E becomes

BR-1B' and W = C'Q. The optimal control (4.1.31)

I u(t) = -R -1 B' [Px(t) - g(t)] .1 (4.2.7)

Further details on this are available in Anderson and Moore [3].

168

.... 0 t) Q)

> Cl

(J) Q)

«i Ci5 (ij

E °a 0

Chapter 4: Linear Quadratic Optimal Control Systems II

30 I I • I I •

25 • I I , ... .. ........... , ................... - ............ -r .............. r ......... .. • I I I I • I I I • I • I I •

• I I I •

20 ............ t .............. ~ ................. -:- ............ -~ .............. ~ ......... .. • I • I • • I I I I , I I I ,

0 0 2 4 6 8 10 12 14 16 18 20

Figure 4.7 Coefficients gl (t) and g2(t) for Example 4.2

50~--~--~--~----~--~--~--~~--~--~--~

-10

I I

• I " I , · . , . . , . ___ ................ __ .. ~ ___ ____ 1 __ ...... __ I. __ .................... __ ..1 ______ ... _____ _ • I I I • I I • I • • • I • • I I • • I • • I I I • I

I I I I I I

• I I • I I I • • .............. t ............... ~ ............ -:- ............ -:- ............. t .............. I ............. -:- ............ -~ .............. ~ ........ ... I • I • I I I • I I I I I I I I I I I I ------t --- ---1- -----t ---- -r-- -- -- t-- ----1- -----1- ----- -~-- -----r -----

f' I I I I , I •• • I I I I I •• • I I • • I •• • I I I • •

• ••• I I • • ............ !.......... • ............ -:- .......... -r ............ ~ ............. ~ .... ...... -1- .......... -r ............ r ........ .. • • I • I I I • •

: : : x (t) : : : : : , • • • I I I , •

.......... ,. ............ , ............ -,- .......... -r ............ T - _ ..... - -, - - _ .... -..,- - - - _ .... -r"'''' - - - - r -- • I I I I I • • • • • I I I I I • I • • • • • I , • t I • • • I I • • I I. t I I • ,. I I. • , • I • I I I • ,

.... - .... _ ....... - .............. of; - _ _ ......... -t- ....... _ .. - ..... - .................... - .......... - ... - .. "1-""""''' _t- .. - ........ to ... - .. ... I • • • • I • • t

, " . . ., . , .. . • I I • I I I •

-20~--~--~----~--~--~----~--~--~----~--~ o 2 4 6 8 10 12 14 16 18 20

Figure 4.8 Optimal Control and States for Example 4.2

4.3 Fixed-End-Point Regulator System 169

O~~~~--~--~--~--~--~---r---r--~

I , • I • I I • I I ••

-100 ------t--- --i-------;-------~------t------ ------1-------~------ ----- · . I. .. I ••• • I · ... .. I ••• I I

• I •• I.

-200 ---- .... :----- -~ ----- .. -:- .. - .. --~ ...... -- -f ...... --- ------~----- .. -~ -- -- -- ----- • • I " • I • • I I I I , • I I I I

• ., I I I I • I I I I

o :::: I :: ~ -300 ---- -- ~ --- ---~ -------:- ------~ -- ---- ~ - --- ------ ~-- -----~ -- ---- ---- (U I... I I I

E "" I I I ".;:; ::::: :: c.. I. I I I • I O " I I. I. I • • I. • I

-400 ------f- .. ----~ .. ------:- ........ .. -~------f ------ .. -- ---~- .... -- -~------ ---- .. I • I I I I. I • • I I I. I r I I I , I , • I I • I • • • I • I , • • I I I • • • I ••

-500 ------t------i-------:- .. -.. ---~------t- .. ---- ------1-------r------. -- -- I I • I I ,. I I • • I. I I • I • • It. • I I I • • I ••• • • , • I •••

• • • • I I. I

-600 L..----II....-----'_---'_--L_--L_--'-_--L.._--L.._--'-_---I o 2 4 6 8 10 12 14 16 18 20

Figure 4.9 Optimal Control and States for Example 4.2

4.3 Fixed-End-Point Regulator System In this section, we discuss the fixed-end-point state regulator system, where the final state x( t f) is zero and the final time t f is fixed [5]. This is different from the conventional free-end-point state regulator system with the final time t j being free, leading to the matrix Riccati differen- tial equation that was discussed in Chapter 3. Following the procedure similar to the free-end- point system, we will arrive at the same matrix differential Riccati equation (3.2.18). But, if we use the earlier trans- formation (3.2.11) to find the corresponding boundary condition for (3.2.18), we see that

(4.3.1)

and for the fixed final condition x( t f) = 0, and for arbitrary >..( t f), we have

(4.3.2)

This means that for the fixed-end-point regulator system, we solve the matrix DRE (3.2.18) using the final condition (4.3.2). In practice, we may start with a very large value of P(tj) instead of 00.

Alternatively, we present a different procedure to find closed-loop optimal control for the fixed-end-point system [5]. In fact, we will use

170 Chapter 4: Linear Quadratic Optimal Control Systems II

what is known as inverse Riccati transformation between the state and costate variables and arrive at matrix inverse Riccati equation.

As before, consider a linear, time-varying system

x(t) = A(t)x(t) + B(t)u(t) (4.3.3) with a co