Driver Attention in Automatic Transmission Cars

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 21, NO. 11, NOVEMBER 2020 4895

Ecological Adaptive Cruise Control for Vehicles With Step-Gear Transmission Based on

Reinforcement Learning Guoqiang Li and Daniel Görges , Member, IEEE

Abstract— In this paper an ecological adaptive cruise controller to reduce the fuel consumption and ensure the safe inter-vehicle distance for vehicles with step-gear transmissions is presented. An optimal control strategy using reinforcement learning with a novel actor-gear-critic architecture is proposed to obtain the continuous traction force trajectory and the discrete gear shift schedule. The traction force is determined from an actor network to maintain a desired inter-vehicle distance which improves the driving safety in a car-following process. The gear shift schedule is derived from a gear network to reduce the fuel consumption. The control strategy is model-free and allows continuous online learning for different driving situations with- out look-ahead velocity information. Particularly the nonlinear vehicle dynamics, the nonlinear transmission efficiency map for different gear ratios, and the nonlinear fuel consumption map are learned for fuel consumption reduction. The proposed controller is evaluated for different driving scenarios to demonstrate its robustness. Furthermore simulation comparisons for different gear shift schedules and velocity trajectories are given underling the advantages in terms of fuel economy and driving safety.

Index Terms— Adaptive cruise control, gear shift control, velocity optimization, reinforcement learning, nonlinearity.

I. INTRODUCTION

DEPLETING energy sources and growing environmen-tal problems necessitate the reduction of the fuel con- sumption of road vehicles. This requirement has strongly promoted novel technologies in the automotive industry like electrified powertrain systems and intelligent transportation systems during the last years. Recently automated driving have become in focus for improving both the fuel consumption and driving safety [1]. Studies on ecological driving which aims at operating the vehicle with fuel-optimized velocity trajectories while keeping a desired average driving speed and traveling time have shown a high potential for fuel economy improvement [2], [3].

A. Literature Review

Several concepts for an ecological assisted and automated driving has been proposed in recent years.

Manuscript received June 13, 2018; revised November 5, 2018; accepted September 30, 2019. Date of publication October 23, 2019; date of current version October 30, 2020. The work of G. Li was supported by the China Scholarship Council. The Associate Editor for this article was J. M. Alvarez. (Corresponding author: Daniel Görges.)

The authors are with the Electromobility Research Group, Depart- ment of Electrical and Computer Engineering, University of Kaiser- slautern, 67663 Kaiserslautern, Germany (e-mail: [email protected]; goerges@ eit.uni-kl.de).

Digital Object Identifier 10.1109/TITS.2019.2947756

Model predictive control (MPC) has been applied to optimize the velocity and gear ratio in vehicles with continuously variable transmission (CVT) for fuel economy in [4]. An accurate look-ahead knowledge for the preceding vehicle is required. An acceleration-based connected cruise control is proposed in [5] to increase the traffic efficiency by vehicular communication. In [6] a distributed adaptive sliding mode control method for car-following scenarios considering acceleration uncertainties has been presented. A robust eco-cooperative adaptive cruise control strategy is formulated to optimize the vehicle acceleration in [7]. A Bayes network model is introduced in [8] to predict the movement of the preceding vehicle. These studies mainly focus on the driving safety using approximated vehicle models while the nonlinearities from the engine and step-gear transmission efficiency characteristics as well as the fuel optimality are not considered. A reinforcement learning-based cooperative adaptive cruise control method is proposed in [9] to maintain a safe distance in a stop-and-go scenario. In [10] a supervised adaptive dynamic programming algorithm for full-range adaptive cruise control is introduced. Both methods utilize a discrete control input and cost during learning. It may deteriorate the performance in a real continuous driving environment. Ecological adaptive cruise controllers have been presented for hybrid electric vehicles with MPC and heuristic dynamic programming to improve the fuel economy in [11], [12] where the nonlinear powertrain characteristics have not been addressed.

With the vehicle kinematic energy, a pulse-and-glide control strategy allows the engine to operate in a highly efficient region. It is an effective tool to reduce the fuel consumption as shown in [13]. In [14] the mechanism of the pulse-and-glide operation is applied for vehicles with a step-gear transmis- sion. A practical rule for controlling the gear position and the engine torque is proposed for real-time implementation. Further control algorithms which incorporate the road grade conditions are presented in [15], [16] to optimize the velocity trajectory for fuel consumption reduction within limited safe ranges. A distance-based ecological driving framework is proposed in [17] to derive the optimal speed profile. A cloud- based velocity optimization method based on a space-domain dynamic programming is developed to improve the fuel econ- omy in [18]. A two-way communication system is designed between the vehicle and the cloud for online application. A power-based vehicle longitudinal controller using the traffic

1524-9050 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

4896 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 21, NO. 11, NOVEMBER 2020

condition ahead and the traffic signal status is proposed in [19] to optimize the velocity. These strategies are appropriate for free-flow driving. The driving safety in terms of the required inter-vehicle space is, however, not ensured, particularly in congested urban areas.

The connectivity between vehicles realizes the cooperative adaptive cruise control using vehicle-to-vehicle communica- tions [20]. Vehicles in a platoon can share driving information with each other for the longitudinal dynamics control to improve the fuel economy and the driving safety. The system delay and the braking capacity of each vehicle in a platoon is studied in [21] to guarantee the minimum safety distance for driving safety. Several factors in connected driving control, such as stochastic communication delays, the string stability, and unknown system dynamics are discussed in [22]–[24]. These studies about the cooperative adaptive cruise control mostly focus on the driving safety while the fuel economy is neglected. The optimization for the gear ratio in the powertrain system is rarely discussed.

For vehicles with CVT, the engine working points can be adjusted to the region with best fuel efficiency using the continuous gear ratios. The optimization is regarded as a continuous problem in [4], [13]. For step-gear transmissions, the discrete control of the gear shift leads to a mixed-integer nonlinear optimization problem with a considerable calcula- tion burden which is challenging for an online calculation [14], [25], [26]. Therefore an online gear shift control within adap- tive cruise control is still an open issue but highly desirable.

B. Motivation and Contributions

Adaptive cruise control has been widely studied to both improve the fuel economy and the driving safety. The existing concepts, however, have limitations. First, a precise longitu- dinal vehicle dynamics model including the nonlinear engine efficiency characteristics, the integer gear positions and the nonlinear transmission efficiency characteristics is generally required for the controller design but not covered. Further- more, the velocity control and the online gear shift control are commonly designed separately for the eco-driving system, lacking a joint optimization for the driving safety and thenfuel economy. Finally, velocity prediction or communication with a preceding vehicle is usually necessary for the velocity control of the host vehicle.

The objective of this paper is to close these gaps. A learning- based ecological adaptive cruise controller to jointly improve the fuel economy and the driving safety in terms of the desired inter-vehicle space in a car-following process is presented. Reinforcement learning (RL) based on a novel actor-gear-critic architecture is utilized to optimize the traction force and the gear shift schedule for fuel economy. The gear shift control is obtained with a gear shift network to minimize the fuel consumption. The traction force is determined by an actor network to follow the velocity of the preceding vehicle within a suitable inter-vehicle distance.

Three original contributions are made in this paper. (i) The gear shift control and the traction force control are realized with a novel learning-based controller to improve the fuel

Fig. 1. Car-following scenario.

economy and driving safety. The method is computationally efficient and thus enables possible online application. (ii) The nonlinear powertrain characteristics including the transmission efficiency for different gear ratios in step-gear transmissions and the fuel consumption map are regarded by the controller. (iii) The controller is model-free and can adapt online to various driving behaviors. A look-ahead velocity prediction is not required, which is different from most other controllers proposed in the literature.

The rest of the paper is organized as follows. In Section II the nonlinear dynamic system and the control problem are presented. Reinforcement learning is introduced in Section III. In Section IV the controller design for the ecological driving based on the actor-gear-critic architecture is discussed. In Section V the proposed controller is investigated for differ- ent driving cycles. Conclusions are finally given in Section VI.

Throughout the paper non-bold symbols indicate scalars and bold symbols indicate vectors.

II. PROBLEM STATEMENT

In this paper a car-following scenario is studied consisting of a host vehicle and a preceding vehicle as shown in Fig. 1. The host vehicle is equipped with an internal combustion engine and an automated manual transmission (AMT). vp and vh are the velocities of the preceding and host vehicles. The inter-vehicle distance is denoted as L. In normal traffic condi- tions particularly congested urban driving situations, vehicles should keep some distance to ensure the safe driving and improve the traffic flow. Therefore it is necessary to control the velocity of the host vehicle to keep a desired inter-vehicle space.

A. Longitudinal Dynamics Model

The longitudinal dynamics model of the host vehicle is formulated as

ẋh = vh v̇h = ah ah =

Ftrac m

− ρ Acdv 2 h

2m −g f cos α−g sin α (1)

where Ftrac is the traction force (When Ftrac < 0, it is denoted as the braking force), ρ is the air density, A is the equivalent area of the vehicle body, cd is the aerodynamic resistance coefficient, m is the vehicle mass, f is the rolling resistance coefficient, and α is the road angle.

The inter-vehicle distance deviation �L is calculated from �L = xp − xh − L des where L des denotes the driver’s desired inter-vehicle space. The relative velocity deviation is

LI AND GÖRGES: ECOLOGICAL ADAPTIVE CRUISE CONTROL FOR VEHICLES WITH STEP-GEAR TRANSMISSION 4897

Fig. 2. Fuel consumption map.

determined from �v = vp − vh. To ensure a safe driving, the constant time headway policy for the inter-vehicle distance

L des = τhvh + d0 (2) is employed where τh is the nominal time headway and d0 is the standstill distance. It can be seen that L des changes with vh, indicating that a longer relative distance is required to guarantee the driving safety at a higher velocity of the host vehicle. For highway driving a long inter-vehicle space ensures sufficient reaction and braking time for the host vehicle, while in low-speed driving a short relative distance is helpful to improve the traffic flow and the road capacity.

The dynamic models of the inter-vehicle distance deviation �L and the relative velocity deviation �v are given by

�̇L = vp − vh − τhah �̇v = ap − ah (3)

where ap is the acceleration of the preceding vehicle.

B. Engine Model

The engine is described by the fuel rate ṁf (g/s). It is a function of the engine torque Te and engine speed ωe as

ṁf = f (Te, ωe) (4) and usually represented by a fuel consumption map. A static engine fuel consumption map which has been obtained by experiments is shown in Fig. 2. It can be seen that for the same engine output power the fuel consumption is lowest when engine works in a region with low speed and high torque. Thus, a higher gear position is helpful to improve the fuel economy.

C. AMT Model

The AMT can adjust the engine working points for dynamic or economic performance through changing the gear ratio. The engine speed and engine torque then result from

ωe = vh

rw ig(g)

Te = Ftracrw ig(g)ηg

for Ftrac > 0 (5)

Fig. 3. Transmission efficiency map of an AMT for the first gear.

where ig is the transmission ratio for the gear position g, rw is the wheel radius, and ηg is the transmission efficiency described by a three-dimensional look-up table. It depends on the input torque T , the input speed ω, and the gear position g as illustrated in Fig. 3.

D. Control Objective

The optimal control problem for ecological driving in the car-following process consists in following the preceding vehi- cle within the safe distance and minimizing the fuel consump- tion during the whole trip. The tracking performance can be evaluated by the inter-vehicle distance. Through tracking the velocity of the preceding vehicle frequent vehicle cut-ins from adjacent lanes can be avoided and in this way the probability of traffic accidents can be reduced. Therefore �L and �v should be minimized for driving safety. Here the driving safety is understood as keeping a desired relative distance to the preceding vehicle in normal driving situations while emergency braking conditions are not considered. Emergency braking conditions must, as in conventional ACC, be handled by an assistive emergency braking system. To improve the fuel economy and tracking performance, the objective of the ecological driving is to control the traction force and the gear shift in order to minimize the cost function by

J = ∫ Tcyc

0

( �L 2 + �v 2 + γ ṁf

) d t (6)

where Tcyc is the total length of the driving trip and γ is a weighting factor. Additionally the physical and safety constraints

amin ≤ ah ≤ amax �L min ≤ �L ≤ �L max �vmin ≤ �v ≤ �vmax Te,min ≤ Te ≤ Te,max ωe,min ≤ ωe ≤ ωe,max ig ∈ {ig1, ig2, ig3, ig4, ig5} (7)

must be satisfied.

4898 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 21, NO. 11, NOVEMBER 2020

Fig. 4. Basic concept of reinforcement learning.

III. REINFORCEMENT LEARNING

In this section the RL algorithm, the formulation of the opti- mal control problem, and techniques to improve the learning efficiency are introduced.

A. Introduction

Nonlinear systems widely exist in transportation sys- tems which generally contain multi-variable couplings and non-affine input-output relationships. The systems are fre- quently modeled by differential-algebraic equations or curve fittings. Generating accurate physical models of these processes for controller design is very elaborate. Recently learning-based control methods are attracting great attention since the system dynamics can be learned from measured data. RL, also referred to as adaptive or approximate dynamic programming (ADP) in the control system community, is a learning-based control method. It makes decisions based on interactions with the environment [27], [28]. The basic concept of RL is shown in Fig. 4 [29]. An agent receives the state of the environment and generates an action based on the state. The action is evaluated from a reward or instantaneous cost indicating the control performance and then adapted by the agent to maximize the reward in the future or minimize the value function.

B. Problem Formulation

Consider a general nonlinear discrete-time dynamic system

xk+1 = f (xk , uk ), k = 0, 1, 2, ... (8) where x ∈ Rn is the state vector, u ∈ Rm is the input vector and f (xk , uk ) is the system dynamics function. A value function for the system (8) can be defined as

J (xk ) = ∞∑

i=k βi−k r (xi , ui )

= r (xk , uk ) + β ∞∑

i=k+1 βi−(k+1)r (xi , ui )

= r (xk , uk ) + β J (xk+1), (9) where β is a discount factor with 0 < β ≤ 1 and r (xi , ui ) is the instantaneous cost (reward). The objective is to find an optimal control policy u∗k = h(xk ) which minimizes the cost function (9), i.e.

u∗k = arg min h(·)

[ r (xk , uk ) + β J (xk+1)

] . (10)

The optimal value function can be determined from the Bellman optimality equation

J ∗(xk ) = r (xk , u∗k ) + β J ∗(xk+1). (11) It is clear that to get the optimal control u∗k at time k in (10),

the optimal value function at time k + 1 should be obtained in advance. This yields a backward procedure and requires the knowledge of the system model to update the state variables. This calculation is usually performed offline. To avoid the offline calculation and overcome the necessity of the system model, ADP with function approximation can be used to make decisions forward in time through learning by trial-and-error interactions with the dynamic environment to solve the optimal control problem.

C. Iterative Learning Algorithm

Given a control policy h(xk ), the value function can be updated iteratively using

J (i+1)(xk ) = r (xk , h(i)(xk )) + β J (i)(xk+1), (12) where i is the iteration step. During the iteration a function approximation

Ĵ (xk ) = wT θ (xk ) (13) is utilized where Ĵ (xk ) is an approximation of the value function J (xk ), w is a parameter vector, and θ (xk ) is a basis function vector. Widely applied basis functions are neural networks, linear functions, and piecewise linear functions. Based on the function approximation (13), a value iteration can be performed, i.e.

Ĵ (i+1)(xk ) = r (xk , h(i)(xk )) + β Ĵ (i)(xk+1). (14) After the value iteration (14), the control policy h(xk ) can be improved by

h(i+1)(xk ) = arg min(r (xk , h(i)(xk )) + β Ĵ (i+1)(xk+1)) (15)

The control policy can be represented in two ways: (i) If the number of control variables is limited and the control space is discrete, the calculation of the cost r (xk , h

(i)(xk )) can be performed by enumeration and the control policy can be evaluated based on (15). (ii) When the control space is continuous, the control policy can be described by a function approximation

ûk = h(xk ) = K T σ (xk ) (16) where K are weighting factors and σ (xk ) is the basis function. The weighting factors are adapted to minimize the difference between u∗k and ûk .

D. Techniques for Improving Learning Efficiency

RL solves the optimal control problem iteratively based on data while the process runs. The amount of the data influences the learning performance and therefore determines the control policy. However, the performance of RL cannot be guaranteed at each time since the data usage efficiency can be low and the

LI AND GÖRGES: ECOLOGICAL ADAPTIVE CRUISE CONTROL FOR VEHICLES WITH STEP-GEAR TRANSMISSION 4899

Fig. 5. Actor-gear-critic architecture.

learning may get stuck locally. Meanwhile the system behavior during learning can lead to unacceptable risks [30]. Supervised learning, on the other hand, can provide input-output mappings and allows incorporating professional experience. Therefore the combination of RL and supervisor information can improve the learning efficiency and safety.

There are usually three major pathways to incorporate supervisor information into RL as noted in [30]. (i) Through a shaping pathway, the supervisor supplies an additional source of evaluative feedback or reward which simplifies the task faced by the learning system. (ii) Through a nominal control pathway, the supervisor sends control signals directly to the system which can override critical commands from the actor to ensure safety. (iii) Through an exploration pathway, the super- visor provides the actor with hints about which actions may or may not be promising. In this paper, the second pathway is used for the gear shift control with a gear network. It will be presented in the following section.

IV. CONTROLLER FOR THE ECO-DRIVING SYSTEM

In this section the traction force and the gear shift are designed based on RL with a novel actor-gear-critic archi- tecture for the eco-driving to improve the driving safety and the fuel economy.

A. Problem Formulation

The actor-gear-critic architecture shown in Fig. 5 consisting of an actor network, a gear network, a critic network, and a supervisor is proposed to solve the optimal control prob- lem (6). The state variables are xk = [�L , �v]T and the input variables are uk = [Ftrac/m, g]T. The actor network generates the traction force per unit u1k = Ftrac/m for velocity tracking while the gear network provides the gear position u2k = g for fuel economy. The supervisor supports the gear network to speed up the learning process and maintain the constraints of the engine torque as well as the engine speed. The critic network approximates the cost function and the reinforcement signal evaluates the instantaneous control performance. In the following subsections, the approximation by neural networks and the learning process for the critic, actor, and gear components will be discussed.

Fig. 6. Structure of the critic network.

B. Critic Network

A critic network with Nch neurons in the hidden layer is used to approximate the value function Ĵ (xk ). The input contains the state vector xk and the control vector uk . The structure of the critic network is shown in Fig. 6. w

(1) c denotes

the weighting vector from the input to the hidden layer and w

(2) c is the weighting vector from the hidden layer to the

output neuron. φc(.) represents the hyperbolic tangent transfer function applied in each neuron z of the hidden layer with φc(zs ) = 1−e−zs1+e−zs , s = 1, 2, . . . , Nch . The output of the critic network can then be derived from

Ĵ (xk ) = w(2)c (k)φc(z(k)) (17) where z(k) = ∑2i=1 w(1)ci (k)xik + ∑2j =1 w(1)c j (k)u j k .

The error function of the critic network can be defined as

ec(k) = β Ĵ (xk ) − [

Ĵ (xk−1) − r (xk , uk ) ]

(18)

where r (xk , uk ) is the instantaneous cost acting as the rein- forcement signal. A shifted value function is considered here to simplify the implementation. In order to satisfy (11), the error function ec(k) should converge to zero.

The learning objective for the critic network is to minimize ec(k) by updating the weights wc. For this purpose, an objec- tive function for ec(k) is introduced as

Ec(k) = 1

2 e2c (k). (19)

A gradient descent adaptation algorithm with a chain deriva- tion rule is used to update the weights iteratively [28]. The adaptation of weights can then be given as

wq+1c (k) = wqc (k) + �wqc (k) �wqc (k) = ηc

[ − ∂ E

q c (k)

∂w q c (k)

] ∂ E

q c (k)

∂w q c (k)

= ∂ E q c (k)

∂e q c (k)

∂e q c (k)

∂ Ĵ q (xk )

∂ Ĵ q (xk )

∂w q c (k)

(20)

where ηc is the learning rate of the critic network and q is the iteration index.

C. Actor Network

A three-layer network shown in Fig. 7 is used for the actor whose input is the state vector xk and whose output is the

4900 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 21, NO. 11, NOVEMBER 2020

Fig. 7. Structure of the actor network.

traction force per unit u1 = Ftrac/m. The hidden layer contains Nah neurons with the hyperbolic tangent transfer function φa(s j ) = 1−e

−s j 1+e−s j , j = 1, 2, . . . , Nah . w

(1) a is the weighting

matrix from the input neurons to the hidden neurons and w(2)a is the weighting vector from the hidden neurons to the output neurons. The actor network can then be described by

u1k = w(2)a (k)φa(s(k)) (21) where s(k) = w(1)a xk . The actor network generates the optimal policy by minimizing Ĵ (xk ), i.e.

u∗1k = arg min Ĵ (xk ). (22) The learning objective of the actor network is to minimize

the function

Ea(k) = Ĵ (xk ) (23) through updating wa .

The learning of the actor network is similar to the one used for the critic network. By defining a gradient descent rule, the weights can be adapted with

wq+1a (k) = wqa (k) + �wqa (k) �wqa (k) = ηa

[ − ∂ E

q a (k)

∂w q a (k)

] ∂ E

q a (k)

∂w q a (k)

= ∂ E q a (k)

∂ Ĵ q (xk )

∂ Ĵ q (xk )

∂u q 1k

∂u q 1k

∂w q a (k)

(24)

where ηa is the learning rate of the actor network and q is the iteration index.

D. Gear Network

In order to avoid shift jumps for the driving comfort, only a sequential gear shift is allowed, i.e. g(k) = g(k − 1) + ug(k). The gear network determines the fuel-optimal gear ratio ig with the gear position g(k) based on the shift command ug(k) and the previous gear position g(k − 1). ug(k) belongs to the set G = {−1, 0, 1} where -1 means downshift, 1 represents upshift, and 0 indicates sustainment. The gear network output space contains three discrete actions.

The structure of the gear network is presented in Fig. 8. The control policy is obtained from the gear network, which is described by three layers with the state vector xk as input.

Fig. 8. Structure of the gear network.

w (1) g is the weighting matrix from the input neurons to the

hidden neurons and w (2) g is the weighting vector from the

hidden neurons to the output neurons. The output of the control policy layer is formulated as

up(k) = w(2)g φg(l ) (25) where l = w(1)g xk , φg(l j ) = 1−e

−l j 1+e−l j is the hyperbolic tangent

transfer function in the hidden layer with j = 1, 2, . . . , Ngh , and Ngh is the number of neurons.

An additional layer with the softmax function

s(upi ) = eupi∑3

k=1 e upk

, i ∈ {1, 2, 3}. (26)

is used to calculate the probability distribution of the shift command. The shift command ug is then determined from

i = arg max i∈{1,2,3}

s(upi ),

ug = G{i } =

⎧⎪⎨ ⎪⎩

−1 if i = 1 0 if i = 2 1 if i = 3

(27)

The learning objective of the gear network aims to minimize Ĵ (xk ) through updating wg.

The weights of the gear network are updated with the gradient descent rule by

w q+1 g (k) = wqg (k) + �wqg (k)

�wqg (k) = ηg [ − ∂ Ĵ

q (xk )

∂w q g (k)

]

= ηg [ − ∂ Ĵ

q (xk )

∂u q g (k)

∂u q g (k)

∂ u q p (k)

∂ u q p (k)

∂w q g (k)

] (28)

where ηg is the learning rate of the gear network and q is the iteration index.

E. Supervisor

During the learning process, the supervisor will assess whether the shift command is feasible or not through checking the engine speed and engine torque. For example, when the engine torque Te > Te,max, then a downshift command is necessary to adjust the engine working point with ug = ug −1.

LI AND GÖRGES: ECOLOGICAL ADAPTIVE CRUISE CONTROL FOR VEHICLES WITH STEP-GEAR TRANSMISSION 4901

Fig. 9. Online control strategy.

TABLE I

MAI N PARAMETERS OF THE HOS T VEHI CLE

On the other hand, if the engine speed ωe > ωe,max, then the upshift command is required to reduce the engine speed by ug = ug + 1. Furthermore, in order to ensure that the engine is working above the idle speed, the downshift command is generated when ωe < ωe,idle.

V. SIMULATIONS AND COMPARISON

The online strategy for the ecological adaptive cruise control is presented in Fig. 9. At each time step, the velocities of the preceding and host vehicles as well as the distance between two vehicles are measured by sensors. From that the inter-vehicle distance deviation �L(k) and the relative velocity deviation �v(k) are calculated and sent to the RL controller. The traction force Ftrac(k) and the gear position g(k) are determined from the actor and gear network respectively. This makes the host vehicle to follow the preceding vehicle. During the learning process, the fuel rate ṁf (k) is measured and the instantaneous cost r is computed from r = �L(k)2+�v(k)2+ γ ṁf (k). Note that the system model is not required for the controller design. The system model (1) to (5) is only used for the simulation. The online learning algorithm of the actor- gear-critic design is customly programmed in MATLAB. In the following subsections simulation analysis for the eco-driving under urban and mixed driving scenarios is provided. The main parameters of the host vehicle are listed in Table I. The road angle is set to zero and the temperature influence on the engine fuel consumption map as well as the transmission map are neglected since this is not in focus of this paper.

A. Learning Algorithm

The initial weighting parameters of the critic, actor, and gear networks are pre-trained with a city driving cycle to get an acceptable control policy. After pre-training, the weighting parameters are adapted online for different driving cycles. The learning rates for the networks are defined as ηa = 6 · 10−6,

TABLE II

PARAMETERS F OR ONLI NE LEARNI NG

ηc = 1 · 10−2, and ηg = 5 · 10−5. The numbers of neurons in the hidden layer are Nch = Nah = Ngh = 40. In each time step the control policy is improved by adapting the weights iteratively using (20), (24), (28). The adaptation of the weighting parameters for the critic and actor networks are performed with the maximal iteration numbers nc = na = 20 and the approximation error tolerances Ta = 1 · 10−8, Tc = 1 · 10−6. If the iteration reaches any of the two limits, the weight adaptation is stopped and the optimal traction force is derived from the actor network. The learning of the gear network is continued when the fuel consumption resulting from the updated gear shift command is higher than that for the gear position at the previous time step. The weight adaptation is finished if ṁf (g(k)) < ṁf (g(k − 1)). Then the optimal gear ratio is applied to the system. The maximal iteration limit ng = 20 is implemented to avoid an infinite loop. The parameters used for online learning are summarized in Table II. The online learning algorithm based on actor-gear- critic is presented in the Appendix.

B. Urban Driving Scenario

The proposed control method is first evaluated for the Urban Dynamometer Driving Schedule (UDDS) cycle from [31]. It is based on measurements taken in Los Angeles during the morning rush hour and is part of the EPA Federal Test Procedure (FTP-75) used in the US. Simulation for the velocity and the distance deviation is illustrated in Fig. 10. The velocity profile of the host vehicle is quite close to that of the preceding vehicle. Meanwhile the magnitude of the distance deviation is below 2 m which indicates a good velocity tracking performance and safe driving due to keeping a safe inter-vehicle range. The acceleration, engine torque, and gear position are shown in Fig. 11. The range of the acceleration is kept with a small value between −2 m/s2 and 2 m/s2. A shift schedule with higher gear position is obtained, yielding an engine operation with high fuel economy.

The developed RL-based controller is model-free and the nonlinearity from the vehicle longitudinal dynamics, engine fuel consumption map and transmission efficiency characteris- tics is respected during the training process through the instan- taneous cost containing the fuel rate ṁf , �L and �v. A linear quadratic (LQ) controller is used as a benchmark, which is also applied in [13], [32]. The LQ controller optimizes the

4902 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 21, NO. 11, NOVEMBER 2020

Fig. 10. Velocity and distance deviation for UDDS cycle.

Fig. 11. Acceleration, engine torque and gear position for UDDS cycle.

vehicle acceleration ah to improve the fuel economy based on a linear model. The cost function for the LQ controller depends on �L, �v and ah. Therefore the control objective is to minimize ah, �L and �v, while the nonlinearities from the powertrain system, such as the engine fuel rate map, the transmission efficiency map, and the gear ratio are neglected. A fuel consumption comparison between RL and LQ is given in Table III. Here the same nonlinear model and gear shift schedule are used for simulation. With RL the fuel consumption is reduced by 2.4%. The average efficiency of the engine and AMT are improved by 1.9% and 1.5% compared to the ones resulting from the LQ controller. It indicates that the derived controller respecting the nonlinearity can improve the fuel economy.

To assess the real-time capability of the RL controller, the computation time is calculated. The calculation is executed on a desktop PC with MATLAB® R2014b. The computer is equipped with Intel® Core™ i7-4790 3.60 GHz CPU and 16 GB RAM. The sampling time is set to 100 ms to realize a fast reaction for driving safety. Table IV presents the aver- age and maximum execution time with 30.7 ms and 63.1 ms

TABLE III

COMPARI S ON OF THE FUEL CONS UMP TI ON F OR UDDS CYCLE

TABLE IV

COMP UTATI ON TI ME WI TH RL F OR UDDS CYCLE

Fig. 12. Velocity and distance deviation for WLTC cycle.

respectively, which are both below the sampling time. A major benefit from the online learning of the RL controller is that the weight adaptation for the control policy is based on a gradient descent backpropagation rule. It leads to low computation burden and efficient calculation. Therefore the proposed controller allows a real-time implementation.

C. Mixed Driving Scenario

To validate the robustness of the proposed controller, sim- ulation results for the Worldwide harmonized Light vehicles Test Cycle (WLTC) which contains low, medium, high and extra high speed and is used in the EU since 2017 are shown in Figs. 12 and 13. The velocity trajectory of the host vehicle can follow the one of the preceding vehicle quite well on different speed. The maximum distance deviation is restricted to 1 m, indicating a safe driving. The acceleration of the host vehicle is kept within −2 m/s2 to 2 m/s2 for driving smoothness. A high gear position is chosen along the trip to improve the engine efficiency and reduce the fuel consump- tion. Table V contains a comparison of the fuel consumption for the RL and LQ controllers. The total fuel consumption with the RL controller is reduced by 3.1%, meanwhile the average efficiency of the engine and AMT are improved by 1.3% and 0.3% compared with that resulting from the LQ controller.

D. Analysis of the Gear Shift Strategy for Fuel Economy

The gear shift strategy based on RL can adjust the engine working points to reduce the fuel consumption. Comparisons with a rule-based gear shift schedule taken from [31] are

LI AND GÖRGES: ECOLOGICAL ADAPTIVE CRUISE CONTROL FOR VEHICLES WITH STEP-GEAR TRANSMISSION 4903

Fig. 13. Acceleration, engine torque and gear position for WLTC cycle.

TABLE V

COMPARI S ON OF THE FUEL COS T F OR WLTC CYCLE

Fig. 14. Comparison of different gear shift strategies for UDDS cycle.

presented in Figs. 14 and 15 for the Urban Dynamometer Driving Schedule (UDDS) and Worldwide harmonized Light vehicles Test Cycle (WLTC). It can be seen that a higher gear position is chosen by RL. This brings the engine working points to the low-speed-high-load region which reduces the fuel consumption. The total fuel consumption is decreased by 12.4% and 6.1% for the UDDS and WLTC as shown in Table VI. Therefore the derived gear shift schedule with RL can realize fuel economy.

E. Analysis of the Velocity Trajectories for Fuel Economy

The proposed control method forces the host vehicle to follow the preceding vehicle and keep a safe distance. The fuel

Fig. 15. Comparison of different gear shift strategies for WLTC cycle.

TABLE VI

COMPARI S ON OF THE FUEL CONS UMP TI ON F OR DI FFERENT GEAR SHI F T STRATEGI ES

Fig. 16. Comparison of the trajectories of the preceding and host vehicle.

consumption usually increases during the vehicle acceleration. An unnecessary acceleration will cause subsequent braking to maintain a safe inter-vehicle space which will lead to higher fuel consumption. Therefore smoothing the acceleration and velocity profile can reduce the fuel consumption. Zoomed-in velocity and acceleration trajectories over 270 seconds for the UDDS are presented in Fig. 16. The velocity profile of the host vehicle is much smoother than that of the preceding vehicle with smaller acceleration and deceleration value. A fuel con- sumption comparison for the same gear shift schedule between the host and preceding vehicles is given in Table VII. The host vehicle has a fuel consumption which is 2.9% and 3.2% lower than that of the preceding vehicle for the UDDS and

4904 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 21, NO. 11, NOVEMBER 2020

TABLE VII

FUEL CONS UMP TI ON COMPARI S ON F OR THE PRECEDI NG AND HOS T VEHI CLE

Algorithm 1 Learning Algorithm With Actor-Gear-Critic

1: State variable: xk = [�L(k), �v(k)]T 2: Control variable: uk = [Ftrac(k), g(k)]T 3: Critic network: Ĵ (k) = fc(xk , uk , r (k), wc) 4: Actor network: Ftrac(k)/m = fa(xk , wa) 5: Gear network: ug(k) = fg(xk , wg) 6: for k = 1 to Tcyc do

7: Calculate Ec(k) = 12 ( β Ĵ (xk ) + r − Ĵ (xk−1)

)2 8: Set q = 0 9: while Ec(k) > Tc & q < nc do

10: Update weights wc(k) = wc(k) + �wc(k) 11: Update value function Ĵ (k) = fc(xk , uk , r, wc) 12: Calcualte Ec(k) 13: set q = q + 1 14: end while→ online learning of the critic network 15: Calculate the value function Ĵ (k) 16: Set �a(k) = Ĵ (k), q = 0 and δa = 0 17: while |�a(k) − δa| > Ta & q < na do 18: δa = �a(k) 19: Update weights wa(k) = wa(k) + �wa(k) 20: Update control variable up(k) = fa(xk , wa) 21: Update value function Ĵ (k) = fc(xk , uk , r, wc) 22: Set �a(k) = Ĵ (k), q = q + 1 23: end while→ online learning of the actor network 24: Calculate ṁf (g(k − 1)) = f (Te(g(k − 1)), we(g(k − 1))) 25: Calculate ṁf (g(k)) = f (Te(g(k)), we(g(k))) 26: Set q = 0 27: while ṁf (g(k)) > ṁf (g(k − 1)) & q < ng do 28: Update weights wg(k) = wg(k) + �wg(k); 29: Update control variable ug(k) = fg (xk , wg) 30: Update gear position g(k) = g(k − 1) + ug(k) 31: Check engine speed and engine torque constraints 32: Update torque Te(g(k)) and speed we(g(k)) 33: Calculate fuel rate ṁf (g(k)), set q = q + 1 34: end while→ online learning of the gear network 35: Update system state xk+1 = f (xk , uk ); 36: end for

WLTC respectively. The fuel economy can be improved by the RL-based controller.

VI. CONCLUSION AND FUTURE WORK

In this paper, a model-free ecological adaptive cruise con- troller for vehicles with step-gear transmission has been pro- posed to jointly improve the fuel economy and the driving safety in terms of a desired inter-vehicle distance. Reinforce- ment learning based on a novel actor-gear-critic architecture

has been designed to optimize the traction force and the gear shift schedule. The nonlinear transmission efficiency and engine fuel consumption maps are considered by the car-following system. The developed control method can real- ize the learning continuously based on the current velocity deviation and relative distance deviation without look-ahead velocity prediction. Simulations for urban and mixed drivings have been performed to validate the robustness and the fuel economy. With RL the overall fuel consumption can be decreased by 12.4% and 6.1% for the UDDS and WLTC in comparison to a rule-based gear shift schedule. The host vehicle has a fuel consumption which is 2.9% and 3.2% lower than the one of the preceding vehicle for UDDS and WLTC due to smoothing the velocity profile when using the same gear shift schedule. The proposed method is computationally efficient and makes an online application possible.

Our future work will focus on the velocity optimization in an ecological driving system for free-flow urban driving with RL. Driving information related to the distance, such as the speed limit, the stop signs, the traffic lights, etc., will be considered in a space domain.

APPENDIX

See Algorithm 1.

REFERENCES

[1] U. Montanaro et al., “Towards connected autonomous driving: Review of use-cases,” Vehicle Syst. Dyn., vol. 57, no. 6, pp. 779–814, 2018.

[2] K. Boriboonsomsin, A. Vu, and M. Barth, “Eco-driving: Pilot evaluation of driving behavior changes among U.S. drivers,” Univ. California Transp. Center, Oakland, CA, USA, Tech. Rep. UCTC-FR-2010-20, 2010.

[3] M. Sivak and B. Schoettle, “Eco-driving: Strategic, tactical, and oper- ational decisions of the driver that influence vehicle fuel economy,” Transp. Policy, vol. 22, pp. 96–99, Jul. 2012.

[4] X. Hu, H. Wang, and X. Tang, “Cyber-physical control for energy-saving vehicle following with connectivity,” IEEE Trans. Ind. Electron., vol. 64, no. 11, pp. 8578–8587, Nov. 2017.

[5] J. I. Ge and G. Orosz, “Dynamics of connected vehicle systems with delayed acceleration feedback,” Transp. Res. C, Emerg. Technol., vol. 46, pp. 46–64, Sep. 2014.

[6] X. Guo, J. Wang, F. Liao, R. Teo, and H. Teo, “Distributed adaptive sliding mode control strategy for vehicle-following systems with non- linear acceleration uncertainties,” IEEE Trans. Veh. Technol., vol. 66, no. 2, pp. 981–991, Feb. 2017.

[7] Y. Shao and Z. Sun, “Robust eco-cooperative adaptive cruise control with gear shifting,” in Proc. IEEE Amer. Control Conf. (ACC), May 2017, pp. 4958–4963.

[8] S. Zhang, Y. Luo, J. Wang, X. Wang, and K. Li, “Predictive energy management strategy for fully electric vehicles based on preceding vehicle movement,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 11, pp. 3049–3060, Nov. 2017.

[9] C. Desjardins and B. Chaib-Draa, “Cooperative adaptive cruise control: A reinforcement learning approach,” IEEE Trans. Intell. Transp. Syst., vol. 12, no. 4, pp. 1248–1260, Dec. 2011.

[10] D. Zhao, Z. Hu, Z. Xia, C. Alippi, Y. Zhu, and D. Wang, “Full- range adaptive cruise control based on supervised adaptive dynamic programming,” Neurocomputing, vol. 125, pp. 57–67, Feb. 2014.

[11] M. Vajedi and N. L. Azad, “Ecological adaptive cruise controller for plug-in hybrid electric vehicles using nonlinear model predictive control,” IEEE Trans. Intell. Transp. Syst., vol. 17, no. 1, pp. 113–122, Jan. 2016.

[12] G. Li and D. Görges, “Ecological adaptive cruise control and energy management strategy for hybrid electric vehicles based on heuristic dynamic programming,” IEEE Trans. Intell. Transp. Syst., vol. 20, no. 9, pp. 3526–3535, Sep. 2019.

LI AND GÖRGES: ECOLOGICAL ADAPTIVE CRUISE CONTROL FOR VEHICLES WITH STEP-GEAR TRANSMISSION 4905

[13] S. E. Li, H. Peng, K. Li, and J. Wang, “Minimum fuel control strategy in automated car-following scenarios,” IEEE Trans. Veh. Technol., vol. 61, no. 3, pp. 998–1007, Mar. 2012.

[14] S. Xu, S. E. Li, X. Zhang, B. Cheng, and H. Peng, “Fuel-optimal cruising strategy for road vehicles with step-gear mechanical transmission,” IEEE Trans. Intell. Transp. Syst., vol. 16, no. 6, pp. 3496–3507, Dec. 2015.

[15] M. A. S. Kamal, M. Mukai, J. Murata, and T. Kawabe, “Ecological vehicle control on roads with up-down slopes,” IEEE Trans. Intell. Transp. Syst., vol. 12, no. 3, pp. 783–794, Sep. 2011.

[16] S. Xu, S. E. Li, B. Cheng, and K. Li, “Instantaneous feedback control for a fuel-prioritized vehicle cruising system on highways with a varying slope,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 5, pp. 1210–1220, May 2017.

[17] H. Lim, W. Su, and C. C. Mi, “Distance-based ecological driving scheme using a two-stage hierarchy for long-term optimization and short-term adaptation,” IEEE Trans. Veh. Technol., vol. 66, no. 3, pp. 1940–1949, Mar. 2017.

[18] E. Ozatay et al., “Cloud-based velocity profile optimization for everyday driving: A dynamic-programming-based solution,” IEEE Trans. Intell. Transp. Syst., vol. 15, no. 6, pp. 2491–2505, Dec. 2014.

[19] Q. Jin, G. Wu, K. Boriboonsomsin, and M. J. Barth, “Power-based optimal longitudinal control for a connected eco-driving system,” IEEE Trans. Intell. Transp. Syst., vol. 17, no. 10, pp. 2900–2910, Oct. 2016.

[20] K. C. Dey et al., “A review of communication, driver characteristics, and controls aspects of cooperative adaptive cruise control (CACC),” IEEE Trans. Intell. Transp. Syst., vol. 17, no. 2, pp. 491–509, Feb. 2016.

[21] A. M. Filho, M. H. Terre, and D. F. Wolf, “Safe optimization of highway traffic with robust model predictive control-based cooperative adaptive cruise control,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 11, pp. 3193–3203, Nov. 2017.

[22] W. B. Qin, M. M. Gomez, and G. Orosz, “Stability and frequency response under stochastic communication delays with applications to connected cruise control design,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 2, pp. 388–403, Feb. 2017.

[23] W. Gao, Z.-P. Jiang, and K. Ozbay, “Data-driven adaptive optimal control of connected vehicles,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 5, pp. 1122–1133, May 2017.

[24] L. Xiao and F. Gao, “Practical string stability of platoon of adaptive cruise control vehicles,” IEEE Trans. Intell. Transp. Syst., vol. 12, no. 4, pp. 1184–1194, Dec. 2011.

[25] A. Bemporad and M. Morari, “Control of systems integrating logic, dynamics, and constraints,” Automatica, vol. 35, no. 3, pp. 407–427, Mar. 1999.

[26] C. G. Cassandras, D. L. Pepyne, and Y. Wardi, “Optimal control of a class of hybrid systems,” IEEE Trans. Autom. Control, vol. 46, no. 3, pp. 398–415, Mar. 2001.

[27] F. L. Lewis and D. Vrabie, “Reinforcement learning and adaptive dynamic programming for feedback control,” IEEE Circuits Syst. Mag., vol. 9, no. 3, pp. 32–50, Aug. 2009.

[28] J. Si and Y.-T. Wang, “Online learning control by association and reinforcement,” IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264–276, Mar. 2001.

[29] R. S. Sutton and A. G. Barto, Reinforcement Learning—An Introduction. Cambridge, MA, USA: MIT Press, 1998.

[30] A. G. Barto, “Supervised actor-critic reinforcement learning,” in Hand- book of Learning and Approximate Dynamic Programming, J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, II, Eds. Hoboken, NJ, USA: Wiley, 2004, pp. 359–380.

[31] T. Markel et al., “Advisor: A systems analysis tool for advanced vehicle modeling,” J. Power Sour., vol. 110, no. 2, pp. 255–266, 2002.

[32] S. E. Li, Q. Guo, L. Xin, B. Cheng, and K. Li, “Fuel-saving servo- loop control for an adaptive cruise control system of road vehicles with step-gear transmission,” IEEE Trans. Veh. Technol., vol. 66, no. 3, pp. 2033–2043, Mar. 2017.

Guoqiang Li received the B.S. and the M.S. degrees from the School of Mechanical Engineering, Beijing Institute of Technology, Beijing, China, in 2011 and 2014, respectively, the Dr.-Ing. degree from the Department of Electrical and Computer Engineering, University of Kaiserslautern, Kaiserslautern, Ger- many, in 2019. His research interests include design and control of transmission systems, optimal control of hybrid electric vehicles, and learning control of transportation systems.

Daniel Görges (S’07–M’10) received the Dipl.- Ing. and Dr.-Ing. degrees from the Department of Electrical and Computer Engineering, University of Kaiserslautern, Kaiserslautern, Germany, in 2005 and 2011, respectively. Since 2019, he has been an apl. Professor and the Head of the Electromobil- ity Research Group, Department of Electrical and Computer Engineering, University of Kaiserslautern, where he was a Juniorprofessor for Electronmobility with the Department of Electrical and Computer Engineering, from 2013 to 2019. His research inter-

ests include methods for model predictive control, distributed control, and learning control and their application in vehicular systems, transportation systems, mechatronic systems, and power systems.