Annotated Bibliography for below attached aricles

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 30, NO. 5, MAY 2019 1565

A Deep Learning Approach to Competing Risks Representation in Peer-to-Peer Lending

Fei Tan , Student Member, IEEE, Xiurui Hou, Jie Zhang, Student Member, IEEE,

Zhi Wei , Senior Member, IEEE, and Zhenyu Yan

Abstract— Online peer-to-peer (P2P) lending is expected to benefit both investors and borrowers due to their low transaction cost and the elimination of expensive intermediaries. From the lenders’ perspective, maximizing their return on investment is an ultimate goal during their decision-making procedure. In this paper, we explore and address a fundamental problem underlying such a goal: how to represent the two competing risks, charge-off and prepayment, in funded loans. We propose to model both potential risks simultaneously, which remains largely unexplored until now. We first develop a hierarchical grading framework to integrate two risks of loans both qualitatively and quantitatively. Afterward, we introduce an end-to-end deep learning approach to solve this problem by breaking it down into multiple binary classification subproblems that are amenable to both feature representation and risks learning. Particularly, we leverage deep neural networks to jointly solve these subtasks, which leads to the in-depth exploration of the interaction involved in these tasks. To the best of our knowledge, this is the first attempt to characterize competing risks for loans in P2P lending via deep neural networks. The comprehensive experiments on real-world loan data show that our methodology is able to achieve an appealing investment performance by modeling the competition within and between risks explicitly and properly. The feature analysis based on saliency maps provides useful insights into payment dynamics of loans for potential investors intuitively.

Index Terms— Competing risks, deep neural networks, peer-to-peer (P2P) lending, return on investment (ROI).

I. INTRODUCTION

PEER-TO-PEER (P2P) lending has become a fast-growingnew channel of financing over the past decade. Quite a few P2P platforms have been developed, includ- ing Lending Club (LC) (www.lendingclub.com), Prosper (www.prosper.com), Yirendai (https://www.yirendai.com), and Zopa (www.zopa.com). Connecting borrowers with investors directly using technology, those P2P platforms claim to operate at a lower cost than traditional bank loan programs, passing the savings on to borrowers in the form of lower rates and to investors in the form of solid returns. Such credit marketplaces

Manuscript received September 28, 2017; revised April 6, 2018 and August 15, 2018; accepted September 2, 2018. Date of publication October 10, 2018; date of current version April 16, 2019. (Corresponding author: Zhi Wei.)

F. Tan, X. Hou, and Z. Wei are with the Department of Computer Science, New Jersey Institute of Technology, Newark, NJ 07102 USA (e-mail: [email protected]; [email protected]; [email protected]).

J. Zhang and Z. Yan are with Adobe Systems, San Jose, CA 95110 USA (e-mail: [email protected]; [email protected]).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TNNLS.2018.2870573

have thus attracted a lot of lenders (investors) and borrowers and result into a large amount of investments. For example, as of June 30, 2018, the total loan issued by LC (the world’s largest P2P lending platform) has exceeded U.S. $38 billion.1

For P2P lending, three major participants are involved in the transaction procedure: the lending platform, lenders, and borrowers. Lenders and borrowers interact with each other directly on the lending platform. Using LC for example, we briefly introduce the working mechanism of P2P lending. Other P2P lending platforms are somewhat similar. In LC, a borrower (sometimes with coborrowers) is supposed to provide his or her detailed profile (e.g., annual income and housing status) and loan information while creating a listing to solicit investments from lenders. After receiving the listing, the platform verifies borrowers’ profile (optional), evaluates their credit, and then assigns a certain grade or subgrade to the listed loan for lenders’ reference. If the listed loan gets fully funded by the expiration date, it will be issued by the platform or otherwise revoked. Afterward, the investors can secure interests and the platform charges service fees from borrowers’ monthly payments. Like in most conventional bank loan programs, borrowers may prepay their loans at any time, in whole or in part, without penalty; lenders will then receive pro-rata share of the payment. A loan can also become “charged off” when there is no longer a reasonable expectation of further payments.

Several P2P lending platforms release their load data to the public, which has received much attention from acad- emia [1]–[7]. The existing works mainly involve a simple binary classification between types of charge-off and full payment [1], [2], loan recommendation based on charge-off risk [6], and multiobjective portfolio optimization [5]. With regard to loan risk modeling, the common focus of previous works is on the overall charge-off risk.

However, the risk of prepayment (the settlement of the entire balance of a loan before its official due date) is often ignored in online P2P lending study although prepayment has been well-studied in the classical literature in other financial industries such as mortgage risks [8]–[11]. As with charge- off, prepayment would also terminate the repayment schedule. In this case, charge-off and prepayment are two competing risks, as they coexist in the same loan over the course of loan repayment. A classical survival analysis with competing risks

1 https://www.lendingclub.com/info/statistics.action.

2162-237X © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

1566 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 30, NO. 5, MAY 2019

can be naturally utilized to model the risks of charge-off and prepayment for time-to-event loan data [12]. However, their focus is on how survival time as a dependent variable is shaped by multiple risks jointly. This works fine for many applications such as clinical trials and insurances in which the time to event is the major concern but not the underlying causes/risks. Different risks are just modeled as covariates simultaneously for a better estimate of the survival time. For P2P lending, however, investors are concerned with both survival time and the underlying causing events. There are three points beyond the focus of classical survival analysis with competing risks.

1) Under many circumstances, the latter might play a dominant role in investment performance. Consider, for instance, a loan with a survival time of 5 months for which the event of charged-off causes loss to investors, whereas the state of prepayment leads to positive returns. It is thus necessary to distinguish different events prop- erly. This also largely explains why the previous research efforts focus on coarse-grained binary statuses.

2) Meanwhile, the earlier the prepayment (charge-off) occurs, the less preferable a loan would be. To put it another way, the discrete survival time of a loan is inherently ordinal for the same risk.

3) What is more, the eventual events cannot cooccur, and the same event cannot occur multiple times for the same loan either. They are actually exclusive with each other irrespective of events or the survival time. Therefore, we propose to model the coarse-grained rivalry between different events, fine-grained competition of survival time within the same event, and underlying ordinal constraints explicitly and simultaneously.

To this end, we first propose a grading rule for each risk independently on the basis of the survival time. We then transform the hierarchical ordinal regression problem to multiple binary classification subproblems [13], [14]. Under the newly formulated framework, we further integrate the censored loans without definite observed events into the repre- sentation. A hierarchical fine-grained risk category with ordi- nal constraints can be generated accordingly. Simultaneously, we fuse the loans of multiple scheduled terms by introducing a masking layer. An architecture of deep neural networks with multiple risk category outputs of multiple terms is finally proposed.

Our main contributions are highlighted as follows.

1) In observance of investors’ concerns in P2P lending, we propose a framework to capture inherent competi- tion within and between multiple risks and underlying ordinal constraints behind time-to-event loan data from the perspective of deep learning.

2) This paper illustrates the general procedure of construct- ing classifiers for time-to-event loan data in P2P lending, which opens a door toward an alternative to modeling competing risks involved in many problems of practical relevance.

II. RELATED WORK

Some related studies are discussed as follows.

A. Peer-to-Peer Lending

In practice, lenders expect that their invested loans get fully funded and issued successfully. Thus, the prediction of fully funded loans was explored [15]. They are able to aid in loan evaluation in terms of investment efficiency. Another important research direction in P2P lending is the risk assessment, which can help lenders reduce the potential risk of investment. The common strategy is to group loans into two categories based on the charge-off risk. Afterward, large numbers of classifiers are leveraged to conduct classification learning [1], [2], [6]. For example, some easy-to-interpret methods, such as logistic regression (LR), were proposed to model the credits of loans and borrowers from an economic perspective [2], [6]. Byan- jankar et al. [1] also proposed a credit scoring model based on artificial neural networks to detect potential charged-off loan applications. More recently, work [5] considered fully funded probability, charge-off risk, and winning-bid probability simul- taneously to propose a multiobjective portfolio optimization approach. These works concentrated on the overall charge-off risk yet ignored its fine-grained survival time in terms of risk modeling. More importantly, the risk of prepayment is in lack of exploration in the above-mentioned studies.

B. Competing Risks Analysis

The popular methods are a cause-specific competing risks model [16] and a proportional hazards model proposed by Fine and Gray [17]. Regarding competing risks of charge-off and prepayment in loan data, the above-mentioned methods can be naturally used for learning payment dynamics. Survival analysis, however, does not emphasize the competition within and between multiple risks. Our proposed framework can integrate such consideration into the modeling effectively. Sirignano et al. [8] proposed to capture the evolution of mortgage statuses by modeling the transition probability using deep learning. Our model would be augmented by the con- sideration of intermediate state transition context, provided that intermediate statuses are available. Unfortunately, in P2P lending, the trajectory of the loan state process is not publicly accessible. The final status or a snapshot status of loans is only available at the released time point. In [9], a cause-specific survival model [18] has been utilized to capture competing nature of prepayment and default for mortgage risks by introducing two position intensities. The focus of work [9] is on obtaining a better understanding of economic factors associated with mortgage risks via modeling survival time as discussed before. Actually, a cause-specific survival analysis is also used as a baseline in this paper where lognormal and exponential intensities are applied. Our preliminary work is simply assumed that prepaid loans are better than charged-off ones and considered only closed loans with the single repay- ment term [19]. In this paper, we propose the hierarchical grading and further consider censored loans with multiple repayment terms comprehensively.

C. Ordinal Regression

Ordinal regression tries to solve an intermediate problem between regression and classification [20]–[22]. The target

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

TAN et al.: DEEP LEARNING APPROACH TO COMPETING RISKS REPRESENTATION IN P2P LENDING 1567

Fig. 1. Schematic overview of P2P loan statuses including charge-off, full payment, and censor.

variable is ordinal, and the relative order between different values is emphasized in model learning.2 Li and Lin [14] proposed a general framework to systematically reduce the ordinal regression to a series of binary classification. It enables the well-tuned binary classification approaches to be readily transformed into appealing ordinal regression algo- rithms with sound theoretical and empirical support. Under this framework, different ordinal regression algorithms, such as perceptron-based and SVM-based methods [23], [24], are proposed to solve the related problems. Different from the classical ordinal regression problem, this paper is the first attempt to assess two competing risks of time-to-event loan data hierarchically. Besides, we introduce masking layers to integrate censored data and multiterm loans into the learning of deep neural networks.

III. METHODOLOGY

In this section, we describe the procedure of competing risks grading in P2P lending and then present the representation learning approach of deep neural networks.

A. Hierarchical Grading of Competing Risks

Let D represent the original data set of loans including N loans {(xi , si , ti )}Ni=1, where xi ∈ X = Rd is the input feature vector, and si ∈ S = {1, 2, 0} is the status of loans with 1, 2, and 0 being charged-off, fully paid, and censored, respectively. ti ∈ T = {0, 1, . . . , T − 1, T } is the total received payment count, where T is the official scheduled term of a loan. Charged-off and fully paid loans are also called closed loans here. In practice, a few loans cannot receive any payment, and we thus have T = {1, . . . , T − 1, T } for simplicity. It is noted that fully paid loans involve two types of payments, i.e., prepayment and scheduled payment, as shown in Fig. 1. The prepayment is thought of as that loans are paid off before the official due date, whereas loans paid off strictly based on the schedule are the type of scheduled payment. As with the risk of charge-off, prepayment is another risk existing

2https://en.wikipedia.org/wiki/Ordinal_regression.

in investments, since less interest can be secured compared with scheduled payment. Besides, the loans without a definite status of charge-off or full payment are referred to as censored loans. As shown in Fig. 1, the maximum of the total received payment counts for charged-off and censored loans is T − 1, since T monthly payments are equivalent to the status of scheduled full payment.

To model both status and payment count of loans, we pro- pose a risk grading rule g : S × T → Y as follows:

yi =

⎧ ⎪⎨

⎪⎩

dti , si = 1 pti , si = 2 cti , si = 0

(1)

where d, p, and c stand for default (i.e., charge-off), full pay- ment, and censor, respectively. si and ti are the status and sur- vival time as mentioned before. Consequently, we have the cor- responding converted data set O = {(xi , yi )}Ni=1, where yi ∈ Y = {c1, c2, . . . , cT −1, d1, d2, . . . , dT −1, p1, p2, . . . , pT }. For closed loans, we have definite final statuses. As per usual, we adopt ≺ as the grading relation. For closed loans, risk grades follow d1 ≺ d2 ≺ · · · ≺ dT −2 ≺ dT −1, and p1 ≺ p2 ≺ · · · ≺ pT −1 ≺ pT . The philosophy behind such a grading strategy is straightforward: in regard to loans of the same status, the more monthly payments are received, the more desirable loans are. Therefore, the identical loans receiving more payment times are assigned higher grades accordingly. For censored loans, their categories are highly related to the observation time point, which leads to the uncertainty of final payment status and payment count. We thus do not place grading on different categories here. In Section III-C, we will revisit this problem for embedding censored loans into the unified representation of closed loans.

B. Methodology Framework Overview

Fig. 2 is given to facilitate the understanding of our modeling pipeline. The framework mainly consists of four components.

1) Generation of risk grades based on loan status, survival time, and loan term.

2) Conversion from the hierarchical ordinal grades to mul- tiple binary outputs of multiple terms. The developed network architecture is reported in Fig. 4.

3) Competing risks prediction and evaluation for held-out loan data.

4) Model interpretation with respect to feature importance quantification.

C. Deep Learning Approach

Mathematically speaking, the risk grading can be defined to search for a mapping rule from input features to the grading category h(·) : X → Y such that

arg min h

1

N

N∑

i=1 Cyi ,h(xi ) (2)

where C is a defined K × K cost matrix with Cyi ,h(xi ) that a sample (xi , yi ) is predicted as category h(xi ). It is

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

1568 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 30, NO. 5, MAY 2019

Fig. 2. Framework overview of the proposed methodology.

Fig. 3. Illustrative overview of the final representation of charge-off, full payment, and censor for loans with 36-month and 60-month terms. Cross (×) indicates the masking position.

reasonably assumed that Cyi ,h(xi ) = 0 if yi = h(xi ); otherwise, Cyi ,h(xi ) > 0 [14]. Naturally, the cost mechanisms should give more penalties to erroneous prediction of the grading category. That is to say, Cyi ,h(xi )−1 ≥ Cyi ,h(xi ) if h(xi ) ≤ yi and Cyi ,h(xi ) ≤ Cyi ,h(xi )+1 if h(xi ) ≥ yi . A popular choice is an absolute cost matrix defined by Cyi ,h(xi ) = |yi − h(xi )|.

For the purpose of competing risks learning and predictive inference, we detail five procedures as follows.

1) Conversion From Closed Grading Categories to Multiple Binary Outputs: The loan data containing K grading cate- gories can be converted to K binary classification problems for each type of statuses. Concretely speaking, given hierarchical grading data set of closed loans Oclosed = {(xi , yi )}Nclosedi=1 ⊂ O = {(xi , yi )}Ni=1, where Nclosed is the number of closed loans, the specific training data for binary classifier k is Bk = {(xi , yki , wki )}Nclosedi=1 , where yki indicates whether the category of sample (xi , yi ) is larger than or equal to category k, and wki is the corresponding weight. Formally, y

k i is defined

as follows:

yki = {

1, yi ≥ k 0, otherwise.

(3)

To ensure that the original cost of risk grading is bounded by weighted zero-one loss on converted examples [14], it is specified that wki = |Cyi ,h(xi ) − Cyi ,h(xi )+1|. Since the absolute cost matrix is adopted here, wki = 1. Another equivalent interpretation is that if the category of a sample is yi = k, it is grouped into lower grading categories {1, 2, . . . , k −1} as well. In this case, the final target vector is z = (1, . . . , 1, 0, 0, 0), where zi (1 ≤ i ≤ k) is set to be 1 and the remaining elements are zeros. It also implies that the loan has gone through the previous monthly payments. In this manner, the final predicted probability vector is thus expected to share the following property: ẑi (i ≤ k) approaches 1 and ẑi (i > k) is close to 0. For types of charge-off and full payment, we have K = T − 1 and T , respectively, as derived by 1. We then further propose to fuse multiple binary outputs of two statuses to generate a unified representation (a vector of 2T − 1 elements) with hierarchical grading constraints (e.g., charge-off and prepayment of the loan with 60-month term as shown in Fig. 3). In this case, yi = dk and yi = pk can be represented as (1d1∼dk , 0dk+1∼dT −1 , 0 p1∼ pT ) and (0d1∼dT −1 , 1 p1∼ pk , 0 pk+1 ∼ pT , ), respectively.

2) Representation of Censored Loans: For the data set of censored loans Ocensored = {(xi , yi )}Ncensoredi=1 ⊂ O = {(xi , yi )}Ni=1, where Ncensored is the number of cen- sored loans, we embed yi = ck (1 ≤ k ≤ T − 1) into the representation scheme of closed loans with z = (1d1∼dk , 0dk+1∼dT −1 , 1 p1∼ pk , 0 pk+1 ∼ pT ). Here, 1d1∼dk and 1 p1∼ pk indicate that the status of charge-off or full payment is possible and k monthly payments have been received until the observation time point. However, the final payment times remain unknown yet. Thus, 0dk+1∼dT −1 and 0 pk+1∼ pT here lead to a biased representation. To correct this, we further introduce a masking vector m = (0d1∼dk , 1dk+1∼dT −1 , 0 p1∼ pk , 1 pk+1∼ pT ), where mi = 1 if element i is masked, mi = 0 otherwise. It can be utilized to exclude biased elements from the procedure of model learning accordingly (e.g., censored loans of 60-month term as shown in Fig. 3).

3) Fusing Multiple Terms With Padding: In practice, loans in P2P lending usually involve multiple scheduled terms. For example, two types of 36-month and 60-month are observed in LC. To integrate the loans of multiple terms for a unified representation, we propose to perform zero padding on representation vector of short terms. Specifically, given L different scheduled terms, Tl ∈ {T1, . . . , TL }, we have the maximum term Tmax = max({T1, . . . , TL }). yi = dk for loan i of term Tl can be zero-padded from (1d1∼dk , 0dk+1∼dTl −1 , 0 p1∼ pTl ) to (1d1∼dk , 0dk+1∼dTl −1 , 0dTl ∼dTmax−1 , 0 p1∼ pTl , 0 pTl +1∼ pTmax ). As 0dTl ∼dTmax−1 and 0 pTl +1∼ pTmax are in no sense, we also introduce a masking vector (0d1∼dk , 0dk+1∼dTl −1 , 1dTl ∼dTmax−1 , 0 p1∼ pTl , 1 pTl +1∼ pTmax ) in the same manner, as mentioned in Section III-C2. Fig. 3 exemplifies this point by loans of 36-month term.

4) Deep Neural Networks: Armed with the above-mentioned analysis, we leverage deep neural networks to conduct a series of binary classification. As shown in Fig. 4, the designed networks consist of an input layer of d -dimensional input feature nodes, fully connected shared

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

TAN et al.: DEEP LEARNING APPROACH TO COMPETING RISKS REPRESENTATION IN P2P LENDING 1569

Fig. 4. Schematic architecture of deep neural networks.

and parallel individual layers with masking outputs, and L parallel output layers of 2Tmax − 1 nodes. Shared layers learn common feature representation across multiple terms, whereas individual layers target specific patterns for the loans of the corresponding term. Different from traditional multiclass classification neural networks with the softmax activation function of the output layer, the sigmoid function is adopted for the underlying architecture. The main idea is to enable output probability of different classifiers to be estimated independently without constraints with each other. The corresponding loss is the widely used binary cross entropy function for loans of term l

Jl = − 1

Nl

Nl∑

i=1

∑

v∈V

( 1 − mvli

)

× [yvli log f vl (xli ) + ( 1 − yvli

) log

( 1 − f vl (xli )

)] (4)

where V = {d1, . . . , dTmax−1, p1, . . . , pTmax }. The probability of output node v ∈ {d1, . . . , dTmax−1, p1, . . . , pTmax } is denoted as f vl (·) ∈ [0, 1]. Since deep neural networks with competing risks representation are developed here, we call it crDNN for brevity. Finally, we can obtain the total loss function over different terms:

J = L∑

l=1 Jl . (5)

5) Predictive Inference: Suppose a new loan xl j is given, and we aim to estimate three metrics: charge-off or default probability p(xl j ), multiclass probability p

v (xl j ), and the survival time s(xl j ).

The estimation formula of p(xl j ) can be naturally derived as follows:

p(xl j ) = f d1l (xl j )

f d1l (xl j ) + f p1

l (xl j ) . (6)

When the binary classifiers f vl (·) are consistent, i.e., f d1l (xl j ) ≥ · · · ≥ f

dTl −1 l (xl j ) and f

p1 l (xl j ) ≥ · · · ≥

f pTl

l (xl j ), multiclass probability p v (xl j ) can be estimated by

pv (xl j ) = {

f vl (xl j ) − f v+1l (xl j ), v ∈ Vl , v /∈ {dTl −1, pTl } f vl (xl j ), v ∈ {dTl −1, pTl }

(7)

where Vl = {d1, . . . , dTl −1, p1, . . . , pTl }. Normalization on pv (xl j ) is introduced to ensure that

∑ v∈Vl p

v (xl j ) = 1.

The survival time probability distribution sk (xl j ) can be estimated by

sk (xl j ) = {

f pTl

l (xl j ), k = Tl f dkl (xl j ) + f

pk l (xl j ), k ∈ {1, 2, . . . , Tl − 1}.

(8)

s(xl j ) can be estimated by

s(xl j ) = min{k : sk (xl j ) < 0.5}. (9) The current modeling scheme, however, cannot explicitly ensure the strict consistency among different binary clas- sifiers theoretically. As in [14] and [25], we apply the above-mentioned formula to predict directly since the con- sistency can be observed well in practical experiments. Fur- thermore, the theoretical consistency will bring about the significant modeling complexity.

IV. EXPERIMENT

In this section, we describe our experimental procedure and report the empirical evaluation results of the proposed framework on the data set from LC. The whole evaluation procedure is composed of two aspects: 10-fold cross-validation and moving-time window experiments.

A. Experimental Data and Preprocessing

We downloaded the loan data as of Quarter 4, 20163

from LC. There are a total of 1 321 864 loan records that involve types of 36-month and 60-month regarding the sched- uled term. Large numbers of loans were issued after the year of 2015. Particularly, more than 60% of loans are still in progress (Current in LC) among all issued loans, which are often called censored data in the classical survival analysis. In addition to censored loans, we also focus on closed loans, which mainly include the statuses of charge-off (Default and Charged Off in LC) and full payment. There are still a tiny proportion of other statuses such as in grace period, late, and so on, which are filtered out for simplicity.

Since released data sets are the statistics of historical loans, they involve many features unavailable in the profiles of loans when they are listed for investment. In order to simulate the real-world investment scenario to the maximum extent, we filter out features of this kind for facilitating the payment dynamics prediction of loans based on the learned model.

Then, we group the related features of loans into numerical and categorical clusters. Regarding numerical features (e.g., loan amount and FICO score), we conduct standardization for training data to transform features to have zero mean and unit variance. Such preprocessing is amenable to the acceleration of the optimization procedure. With the aid of original center and scale of training features, we standardize the numerical features of both validation and test data accordingly. In regard to categorical features (e.g., grade and purpose), we utilize one-hot encoding for training, validation, and testing data to represent different categories for the same feature. Afterward, we fuse numerical and categorical features together and gen- erate the final input feature set.

3 https://www.lendingclub.com/info/download-data.action.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

1570 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 30, NO. 5, MAY 2019

TABLE I

STATI S TI CS OF LOANS OF INTERES T

TABLE II

STATI S TI CS OF LOANS IS S UED AF TER THE CUTOFF TI ME POI NT

After data cleaning, we finally have a total of 1 269 019 loans for study, as detailed in Table I. To tune hyperparameters for the proposed framework and proceed with the modeling evaluation, we conduct stratified 10-fold cross validation. What is more, we split the whole data set with moving time cutoffs, as detailed in Table II.

B. Experimental Results

1) Evaluation Metrics: We consider threefold comparisons.

1) Regarding the binary classification of charged-off and fully paid loans, the classical area under receiv- ing operating curves (AUC@ROC) [26]–[28] and precision–recall curves (AUC@PR) [29] are adopted.

2) The concordance index (C-index), which quantifies the quality of rankings with censored data being considered, is a standard performance measure for model assessment in the survival analysis [30].

3) In Section IV-B5, the return on investment (ROI) is analyzed comparatively in terms of the proper modeling of competing risks.

2) Baseline Algorithms: We compare our proposed frame- work with the following schemes. First, LC has its own grading and evaluation system. Each issued loan is assigned a grade from A to G (five subgrades per grade) with a matched interest rate. Generally speaking, the higher the interest rate is, the riskier the corresponding loan is. Second, LR is frequently utilized for risk evaluation in general purpose credit scoring and P2P lending study [2], [6]. In order to facilitate the training with the large-scale data set, we construct a simple neural network with one input layer and sigmoid activation function with GPU acceleration. Stochastic gradient descent [31] with the Nesterov momentum of 0.9, the learning rate of 0.01, and the decay rate of 10−6 and modern antiover- fitting technique dropout [32] are adopted. Third, to explore the role of hierarchical grading regularization, we also design a multiclass deep neural network without grading constraints (mcDNN), which has the same architecture and hyperpara- meter settings (detailed in Section IV-B3) with crDNN. The multinomial cross entropy is adopted as a loss function for model learning. The activation function of the output layer is the softmax function. Fourth, competing risk-based sur- vival analysis (CRSA) [16], [33] is recently applied to credit scoring [12], [34]. There are cause-specific model [9], [18]

TABLE III

OVERALL PERF ORMANCE F OR THE PROP OS ED METHODOLOGY AND BAS ELI NES: MEAN (STANDARD DEVI ATI ON)

TABLE IV

MATCHED ONE-TAI LED T-TES T I F THE MEAN OF METRI CS F OR CRDNN IS GREATER THAN BAS ELI NES

and alternative proportional hazards model proposed by Fine and Gray [17]. The lognormal and exponential distributions are observed for risks of charge-off and prepayment, respec- tively. In the observance of specific well-studied probability distributions, the parametric modeling of risks (e.g., cause- specific) is more powerful to capture the payment dynam- ics than nonparametric or semiparametric (e.g., proportional hazards model) modeling strategies are. Thus, we adopt the former cause-specific model. The basic R routine package “CFC” [35] is employed to implement the learning procedure. The maximum number of subdivisions Nmax = 400 and the threshold for relative integration error rel.tol = 1e−04.

3) Experimental Setting and Hyperparameter Tuning: We utilize python libraries TensorFlow 1.0.0 and Keras 2.0.9 to build the architecture of deep neural networks. NVIDIA Tesla K80 GPU with the memory of 12 GB is used for model training. There are a total of 4 fully connected hidden shared layers with 200 nodes in each one. Leaky rectified linear unit with gradient α = 0.001 for negative inputs [36] is adopted as an activation function of hidden layers. The dropout rates of four hidden layers are 0.5, 0.5, 0.4, and 0.4, respectively. Individual layers share the same architecture across different terms, which are composed of 2 layers with 200 nodes. The batch size is 128. The maximum of epochs is 500 with the early stopping of patience of 50 epochs.

4) Performance and Analysis: We randomly split the whole data set into 10 partitions and then do 10-fold cross-validation experiments. This data splitting strategy can at least help to evaluate the extent to which the model captures the underlying payment patterns behind historical loan data. For closed loans, the comparison results are shown in Tables III and IV. Overall, our model can present a sound performance gain on baseline algorithms. For AUC@ROC, crDNN seems to be slightly better than others in terms of discriminating charged-off loans from fully paid ones. In this case, the comparison on AUC@PR is also reported, which is more amenable to the modeling evaluation for class imbalance [29]. The observed AUC@PR of around 0.4 for the proposed method is appealing given the class imbalance ratio of 1:4 between charge-off and full payment. It is demonstrated that such superiority of the proposed model is more evident against baseline methods.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

TAN et al.: DEEP LEARNING APPROACH TO COMPETING RISKS REPRESENTATION IN P2P LENDING 1571

Either AUC@ROC or AUC@PR, however, focuses on the binary class decision, which cannot assess the fine-grained payment dynamics. Therefore, C-index is further introduced to quantify the ranking quality of the proposed method. It is noted that neither LR nor LC is capable of rendering the time- to-event prediction. Thus, we mainly compare crDNN with its variant mcDNN and CRSA. The tremendous dominance of crDNN over mcDNN can be observed when the hierarchical ordinal regularization is taken into account. The evident supe- riority still holds true for the comparison with CRSA.

The difference in the performance of the proposed method and baselines is also evidenced by the statistical significance test, as detailed in Table IV. Altogether, the proposed method is able to discriminate different loan statuses and predict the time-to-event of loans more accurately compared with baselines.

5) Return on Investment Under Naive Selection Strategy: Apart from the aforementioned comparisons purely based on the risk probability, we further perform in-depth exploration in ROI for this section. ROI is of high concern to investors, which is closely correlated with competing risks in funded loans.

Equated monthly installment (EMI) is commonly adopted as a payment scheme for P2P lending.4 Formally, the monthly installment is given by A = P((r (1 + r )T )/((1 + r )T − 1)), where P is the principal (funded amount in LC) and r is the monthly interest rate. T is the total number of monthly installments, which is also the scheduled term of loans. The annual interest rate in this data set should be transformed to monthly interest rate by being divided by 12. Essentially, a larger proportion of each payment is set aside for interest at the beginning of the amortization schedule compared with the end of the schedule. For ease of notation, we omit some subscriptions but retain unambiguity in the following formulas.

The monthly paid interest Ik can be calculated mathemati- cally as

Ik =

⎧ ⎪⎪⎨

⎪⎪⎩

P × r, k = 1 ⎡

⎣ P(1 + r )k−1 − A k−2∑

j =0 (1 + r ) j

⎤

⎦ r, 2 ≤ k ≤ T .

(10)

Now, given a loan x , the probability of risk category v to which it belongs is pv (x ) as derived in 7, where v ∈ {d1, . . . , dT −1, p1, . . . , pT }.

Afterward, the estimated ROI for loan x can be formulated as follows5:

R̂OI = 1 P

{

A T −1∑

k=1 pdk (x ) +

T∑

k=1 p pk (x )[ P + Ik ] − P

}

. (11)

Equation (11) can also be applied to estimate the expected ROI provided by mcDNN and CRSA. Regarding mcDNN, pv (x ) can be replaced by output probability of the corresponding class. For CRSA, the cumulative incidence of hazard can

4http://www.investopedia.com/terms/e/equated_monthly_installment.asp. 5Standard net present value calculation approach is not adopted here as no

records of cash flow are available in the data set.

be converted to monthly hazard rate since it is discrete. To be specific, we have charge-off and prepayment hazard rates rt 1(x ) and rt 2(x ) for loan x on monthly payment t . Since they are conditional probabilities, the corresponding pv (x ) for CRSA can be derived as follows:

pv (x ) =

⎧ ⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩

r11(x ), v = d1 r12(x ), v = p1 rk1(x )

k−1∏

t =1 [1 − rt 1(x ) − rt 2(x )], v = dk

rk2(x ) k−1∏

t =1 [1 − rt 1(x ) − rt 2(x )], v = pk

rT 2(x ) T −1∏

t =1 [1 − rt 1(x ) − rt 2(x )], v = pT

(12)

where k ∈ {1, . . . , T − 1}. With the help of (11), we utilize a naive selection strategy to compare the ground-truth ROI of selected loans based on crDNN against that of mcDNN and CRSA. In regard to the grading system of LC and LR, we make use of the estimated probability of charge-off to select loans, since they are unable to provide the corresponding categorical probability for the estimation of ROI. Basically, the naive strategy is to set up a parameter topRate to control the number of selected loans and then choose loans from all candidates based on given scores (̂ROI or full-payment probability).

We conduct monthly selection on loans and monitor two metrics.

1) The proportion of months with monthly ROI of selected loans being larger than that of all loans on the same months. The monthly ROI on calendar month i is defined by (13). We call months of this kind good months for short.

2) The aggregated ROI across different calendar months is also given by

ROIi = ∑Ri

j =1 �i j − �i j ∑Ri

j =1 �i j

ROI = ∑M

i=1 ∑Ri

j =1 �i j − �i j ∑M

i=1 ∑Ri

j =1 �i j Ri = Ni × topRate (13)

where �i j and �i j are the corresponding total received payments and funded amount of loan j on month i , respectively. Ri and Ni are the number of selected loans and all loans on month i , respectively.

As shown in Fig. 5, mcDNN, CRSA, and crDNN per- form better than both LR and LC over different topRates. Such disparity mainly results from the involvement of more fine-grained category of loans in risks modeling. Multifaceted modeling of competing risks is thus more amenable to esti- mating the real-world ROIs. Particularly, crDNN achieves the highest proportion of good months and the aggregate ROI over different topRates. The superiority of the proposed method against CRSA and mcDNN justifies the necessity of the simul- taneous and explicit modeling of both the competition among

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

1572 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 30, NO. 5, MAY 2019

Fig. 5. Comparisons on the good month and the aggregate ROI for different methods. Each point is averaged over 10-fold cross-validation results.

TABLE V

PERF ORMANCE COMPARI S ON: AUC@PR (C-INDEX) F OR MOVI NG-TI ME WI NDOW

multiple risks and underlying ordinal constraints. It is noted that the CRSA has a better performance than mcDNN in terms of C-index in Table III, whereas this relationship is reversed in Fig. 5. This is because survival analysis focuses more on the estimate of survival time, whereas mcDNN does not impose any grading regularization effects on different fine-grained risk categories from the survival time end. However, mcDNN is able to model the inherent competition among fine-grained risk categories explicitly as mentioned in Section I. To be concrete, the gap between crDNN and CRSA demonstrates that the explicit modeling of the competition within and between multiple risks is beneficial for improving ROI. The disparate performance between crDNN and mcDNN indicates that the internal grading constraint (time-to-event) involved in competing risks modeling coordinates the learning procedure of deep neural networks better. Furthermore, the ROI-topRate curves seem to be relatively steady across different topRates for LC and LR, whereas they exhibit the decreasing trend with the growth of topRate for other three methods. The proportion of good months for LC and LR is around 0.5 over different topRates. This indicates that the selection of loans based on the lowest interest rate or charge-off probability is not a good solution. That is to say, they have the limited guidance toward loan selection in terms of ROI.

6) Moving-Time Window: In addition to the overall 10-fold cross validation, we proceed to perform the evaluation with a moving-time window of test data. Such a preferable test scenario can simulate the real-world situations better. To be specific, loans issued before the cutoff date are used for model development, while those remaining loans are grouped into test data set for empirical evaluation. The distribution of the number of issued loans, however, is highly skewed to recent years. The high concentration of loans toward the end of time period will lead to unevenly distribution of survival time for those closed loans compared with the overall real-world

Fig. 6. C-index comparisons between crDNN and baselines for loans issued on the same months. Each point indicates a monthly comparison.

distribution. In this case, an ROI analysis based on survival time of only closed loans will introduce heavy bias into the model evaluation. Thus, we utilize AUC@PR to evaluate the overall status of closed loans and C-Index to evaluate the survival time of all loans including censored loans. More discussions about evaluation metrics are detailed in Section V.

The cutoff dates and the statistics of associated loans are reported in Table II. The overall empirical eval- uation results for different time windows are provided in Table V. In practice, LC issues new loans regularly four times per day (http://blog.lendingclub.com/investor-updates- and-enhancements), whereas the issued dates of loans in our data set are provided monthly. In addition, the time limit of issued loans for being fully funded is one month. Thus, we conduct more in-depth comparisons on a monthly basis. Such a kind of comparison is designed to simulate the real-world issue of loans even though there is still a slight difference. C-index can incorporate all loans including censored loans, which is an unbiased metric. Thus, we mainly apply C-index to different issued months for fine-grained evaluation, as shown in Fig. 6. Both overall and fine-grained results show that our method is able to outperform baselines.

Therefore, the proposed method is practically appealing based on empirical repeated cross-validation and moving-time window experiments.

7) Saliency Map-Based Feature Analysis: After capturing competing risks, the next step typically is to explore how input features shape the payment dynamics of P2P loans. Modern saliency maps were proposed originally for visual- izing the way how deep convolutional neural networks can be queried regarding the spatial support of a particular class given a specific image [37]. In this paper, we extend saliency maps [37] from convolutional neural networks to the proposed learning architecture for quantifying the contribution of each loan feature in the modeling context of all other features. For a loan with feature vector x0 and a risk category of interest v, the main task is to figure out how elements of x0 shape output

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

TAN et al.: DEEP LEARNING APPROACH TO COMPETING RISKS REPRESENTATION IN P2P LENDING 1573

TABLE VI

MOS T SALI ENT FEATURES CONTRI BUTI NG TO THE FULL PAYMENT OF LOANS

TABLE VII

MOS T SALI ENT FEATURES CONTRI BUTI NG TO THE CHARGE-OFF OF LOANS

probability f vl (x0) of category v. For the proposed learning method, the score f vl (x ) is a highly nonlinear function of input x . f vl (x ), however, can be approximated by a linear function in the closeness of x0 based on the first-order Taylor expansion f vl (x ) ≈ wT (x − x0) + f vl (x0), where w is the first-order derivative of f vl (x ) with respect to the feature vector x at x0 as w = (∂ f vl (x )/x )|x0 . We call w saliency value, which has two points to deliver: 1) the magnitude of the derivative indicates the extent to which the change of the most influential elements of feature vector on the probability of output node v and 2) the direction of each element of the derivative shows whether such a change improves or degrades the probability of output node v.

In this section, we pick output v = p1 for the interpretation study. As per our proposed representation scheme and learning architecture, the output probability is corresponding to whether the given loan is fully paid or charged off. We compute the saliency maps for all loans and then average them. It is observed that different features contribute to the category of p1 in different manners. To be specific, some features of loans are highly correlated with the type of full payment, whereas others are closely related to the type of charge-off, as shown in Tables VI and VII. For instance, a few salient features involve annual income, subgrade, grade, and purpose. It is found that the higher borrowers’ annual incomes are, the more they are prone to pay off their loans. Regarding the grade and subgrades reported by LC, grades A and B and some of their subgrades contribute to the full payment of loans, while grades D–F lead to charged-off loans to some extent. In addition, different borrowing purposes also exhibit a different payment preference. Particularly, loans with the purpose of small business and medical issues might be charged off with higher possibility as opposed to those loans with the wedding, debit consolidation, and the payment of credit cards.

V. DISCUSSION

In this paper, we model the competing risks of time-to- event loans in P2P lending. The proposed framework of their hierarchical ordinal grading takes into account both the loan status and the time of event. Then, deep neural networks are leveraged as the vital classification engine to train the overall

framework. Essentially, multiclass deep neural networks can also be viewed as a simple version of our method without prior ordinal constraints. Such regularizing effects are largely in line with ROI and work well for model learning in practice if fixed payment plans EMI is applied as discussed in Section IV-B5. It is worth noting that the correlation between the survival time-based grading categories and ROI might be degraded when variable payment plans are adopted. Concretely, when a borrower is able to pay higher payment amounts at his/her discretion over the course of payment periods, the investment performance of loans cannot be arranged on their survival time strictly. The variable payment scheme remains the topic of future research.

In addition, the intermediate state transition has been recently incorporated to capture the evolution of mortgage risks [8]. Our framework does not take it into account due to two concerns, i.e., the focus of this paper and data availability. The final goal of this paper is to prioritize the listed loans prior to their issue for investors. The intermediate statuses after the issued date cannot be utilized in this scenario. Lack of trajectory of loan state process in the released historical loans also prevents us from modeling transition probability among different statuses. The concept of status transition probability can be used in the trading of notes in the secondary market of the P2P platform. This is a very promising research topic to explore in the future, which might provide the up-to-date trading suggestions for investors over the course of payment periods [38].

The proposed framework can be thought of as an alterna- tive approach to modeling competing risks and incorporating explicit competition among different categories compared with the classical survival analysis. Hopefully, it aids in unleashing the power of machine learning algorithms for modeling time- to-event loan data beyond online P2P lending. Concretely, the proposed methodology is rather flexible and other clas- sifiers, such as denoising neural networks [39], support vec- tor machine [40], and ensemble learning [41], [42], can be applied naturally.

In Section IV-B5, we adopt a naive strategy to select loans and compare their corresponding ROIs. It demonstrates the appealing investment performance of crDNN compared with other methods. In particular, the superior results for crDNN over mcDNN can be observed. In the scenario of moving-time window experiments, the widely used metric toward accrued return with progressively censored loans being included is the net annualized return (NAR) or its variant,6 which can quantify the performance unbiasedly. As we have no access to actual payment trajectory in the data set, the NAR cannot be calculated here. Fortunately, our internal real-time investment test based on the preliminary version of the proposed method delivers an appealing NAR performance indicated by adjusted NAR of LC so far as it goes. Besides, the portfolio opti- mization or selection is an important research direction, which has also received certain attention in P2P lending study [5]. However, only charge-off is considered in the risk assessment.

6 https://www.lendingclub.com/public/lendersPerformanceHelpPop.action.

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.

1574 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 30, NO. 5, MAY 2019

This paper can be leveraged for the further study of portfolio optimization in online lending [27].

VI. CONCLUSION

In this paper, we try to characterize the competing risks for P2P lending by imposing the hierarchical fine-grained grading constraints on time-to-event loan data. The repre- sentation of competing risks is formulated into a series of binary classification subproblems that are jointly solved with the proposed deep neural network learning architecture. We apply our approach to the large-scale loan data released by LC. The comprehensive empirical experiments demonstrate the outperformance of the proposed approach compared with classical competing risks survival analysis in terms of both risks prediction and associated ROI. The necessity of this paper is also well justified. It can be potentially applied to other related crowdfunding and credit risks problems as well.

ACKNOWLEDGMENT

The authors would like to thank anonymous reviewers for constructive suggestions and comments and S. Patil for proofreading this paper.

REFERENCES

[1] A. Byanjankar, M. Heikkilä, and J. Mezei, “Predicting credit risk in peer-to-peer lending: A neural network approach,” in Proc. IEEE SSCI, Dec. 2015, pp. 719–725.

[2] G. Dong, K. K. Lai, and J. Yen, “Credit scorecard based on logistic regression with random coefficients,” Procedia Comput. Sci., vol. 1, no. 1, pp. 2463–2468, 2010.

[3] D. Liu, D. Brass, Y. Lu, and D. Chen, “Friendships in online peer- to-peer lending: Pipes, prisms, and relational herding,” vol. 39, no. 3, pp. 729–742, 2015.

[4] C. Luo, H. Xiong, W. Zhou, Y. Guo, and G. Deng, “Enhancing invest- ment decisions in P2P lending: An investor composition perspective,” in Proc. ACM SIGKDD, 2011, pp. 292–300.

[5] H. Zhao, Q. Liu, G. Wang, Y. Ge, and E. Chen, “Portfolio selections in P2P lending: A multi-objective perspective,” in Proc. ACM SIGKDD, 2016, pp. 2075–2084.

[6] H. Zhao, L. Wu, Q. Liu, Y. Ge, and E. Chen, “Investment recommen- dation in P2P lending: A portfolio perspective with risk management,” in Proc. ICDM, Dec. 2014, pp. 1109–1114.

[7] Y. Deng, F. Bao, Y. Kong, Z. Ren, and Q. Dai, “Deep direct rein- forcement learning for financial signal representation and trading,” IEEE Trans. Neural Netw. Learn. Syst., vol. 28, no. 3, pp. 653–664, Mar. 2017.

[8] J. Sirignano, A. Sadhwani, and K. Giesecke. (2016). “Deep learning for mortgage risk.” [Online]. Available: https://arxiv.org/abs/1607.02470

[9] E. S. Schwartz and W. N. Torous, “Mortgage prepayment and default decisions: A poisson regression approach,” Real Estate Econ., vol. 21, no. 4, pp. 431–449, 1993.

[10] C. L. Maxam and M. LaCour-Little, “Applied nonparametric regres- sion techniques: Estimating prepayments on fixed-rate mortgage-backed securities,” J. Real Estate Finance Econ., vol. 23, no. 2, pp. 139–160, 2001.

[11] M. Chernov, B. R. Dunn, and F. A. Longstaff, “Macroeconomic-driven prepayment risk and the valuation of mortgage-backed securities,” Rev. Financial Stud., vol. 31, no. 3, pp. 1132–1183, 2017.

[12] M. Marimo, “Survival analysis of bank loans and credit risk prognosis,” Ph.D. dissertation, School Statist. Actuarial Sci., Univ. Witwatersrand, Johannesburg, South Africa, 2015.

[13] J. Cheng, Z. Wang, and G. Pollastri, “A neural network approach to ordinal regression,” in Proc. IEEE IJCNN, Jun. 2008, pp. 1279–1284.

[14] L. Li and H.-T. Lin, “Ordinal regression by extended binary classifica- tion,” in Proc. NIPS, vol. 19, 2006, pp. 865–872.

[15] M. Herzenstein, R. L. Andrews, U. Dholakia, and E. Lyandres, “The democratization of personal consumer loans? Determinants of success in online peer-to-peer lending communities,” School Manage. Res. Paper, Boston Univ., Boston, MA, USA, SSRN Working Paper 1147856, 2008, vol. 14, no. 6.

[16] E. T. Lee and J. Wang, Statistical Methods for Survival Data Analysis, vol. 476. Hoboken, NJ, USA: Wiley, 2003,

[17] J. P. Fine and R. J. Gray, “A proportional hazards model for the subdistribution of a competing risk,” J. Amer. Statist. Assoc., vol. 94, no. 446, pp. 496–509, 1999.

[18] J. D. Kalbfleisch and R. L. Prentice, The Statistical Analysis of Failure Time Data, vol. 360. Hoboken, NJ, USA: Wiley, 2011,

[19] F. Tan, X. Hou, J. Zhang, Z. Wei, Z. Yan, and S.-C. Weng, “A novel risk assessment scheme and practice for peer-to-peer lending,” in Proc. ACM SIGKDD Workshop Data Sci. Fintech, 2018. [Online]. Available: https://fintech.kdd2018.a.intuit.com/papers/DSF2018_paper_tan.pdf

[20] C.-W. Seah, I. W. Tsang, and Y.-S. Ong, “Transductive ordinal regression,” IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 7, pp. 1074–1086, Jul. 2012.

[21] F. Fernández-Navarro, A. Riccardi, and S. Carloni, “Ordinal neural networks without iterative tuning,” IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 11, pp. 2075–2085, Nov. 2014.

[22] F. Fernández-Navarro, P. A. Gutiérrez, C. Hervás-Martínez, and X. Yao, “Negative correlation ensemble learning for ordinal regression,” IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 11, pp. 1836–1849, Nov. 2013.

[23] R. Herbrich, T. Graepel, and K. Obermayer, “Support vector learning for ordinal regression,” in Proc. 9th Int. Conf. Artif. Neural Netw., Sep. 1999, pp. 97–102.

[24] A. Shashua and A. Levin, “Ranking with large margin principle: Two approaches,” in Proc. NIPS, 2003, pp. 961–968.

[25] Z. Niu, M. Zhou, L. Wang, X. Gao, and G. Hua, “Ordinal regression with multiple output CNN for age estimation,” in Proc. CVPR, Jun. 2016, pp. 4920–4928.

[26] F. Tan, Y. Xia, and B. Zhu, “Link prediction in complex networks: A mutual information perspective,” PloS ONE, vol. 9, no. 9, p. e107056, 2014.

[27] F. Tan, C. Cheng, and Z. Wei, “Modeling real estate for school district identification,” in Proc. IEEE ICDM, Dec. 2016, pp. 1227–1232.

[28] F. Tan, K. Du, Z. Wei, H. Liu, C. Qin, and R. Zhu, “Modeling item- specific effects for video click,” in Proc. SIAM, 2018, pp. 639–647.

[29] J. Davis and M. Goadrich, “The relationship between precision-recall and ROC curves,” in Proc. ACM ICML, 2006, pp. 233–240.

[30] F. E. Harrell, K. L. Lee, and D. B. Mark, “Multivariable prognostic models: Issues in developing models, evaluating assumptions and ade- quacy, and measuring and reducing errors,” Statist. Med., vol. 15, no. 4, pp. 361–387, 1996.

[31] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in Proc. COMPSTAT. Physica-Verlag HD, 2010, pp. 177–186.

[32] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958, 2014.

[33] P. C. Austin, D. S. Lee, and J. P. Fine, “Introduction to the analysis of survival data in the presence of competing risks,” Circulation, vol. 133, no. 6, pp. 601–609, 2016.

[34] A. Zhang, “Statistical methods in credit risk modeling,” Ph.D. dissertation, Dept. Statist., Univ. Michigan, Ann Arbor, MI, USA, 2009.

[35] A. Mahani and M. Sharabiani, “Bayesian, and non-Bayesian, cause- specific competing-risk analysis for parametric and non-parametric sur- vival functions: The R package CFC,” 2015.

[36] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in Proc. ICML, vol. 30, no. 1, 2013, p. 3.

[37] K. Simonyan, A. Vedaldi, and A. Zisserman. (2013). “Deep inside convolutional networks: Visualising image classification models and saliency maps.” [Online]. Available: https://arxiv.org/abs/1312.6034

[38] F. Tan, C. Cheng, and Z. Wei, “Time-aware latent hierarchical model for predicting house prices,” in Proc. ICDM, Nov. 2017, pp. 1111–1116.

[39] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” J. Mach. Learn. Res., vol. 11, no. 12, pp. 3371–3408, Dec. 2010.

[40] C. Cortes and V. Vapnik, “Support-vector networks,” Mach. Learn., vol. 20, no. 3, pp. 273–297, 1995.

[41] M.-L. Zhang and Z.-H. Zhou, “A review on multi-label learning algo- rithms,” IEEE Trans. Knowl. Data Eng., vol. 26, no. 8, pp. 1819–1837, Aug. 2014.

[42] Z.-H. Zhou and J. Feng. (2017). “Deep forest: Towards an alter- native to deep neural networks.” [Online]. Available: https://arxiv. org/abs/1702.08835

Authorized licensed use limited to: University of the Cumberlands. Downloaded on October 07,2020 at 18:53:14 UTC from IEEE Xplore. Restrictions apply.