221 Week 3 A /For WIZARD KIM

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 19, NO. 12, DECEMBER 2017 2751

Image-Based Appraisal of Real Estate Properties Quanzeng You , Ran Pang, Liangliang Cao, and Jiebo Luo, Fellow, IEEE

Abstract—Real estate appraisal, which is the process of estimating the price for real estate properties, is crucial for both buyers and sellers as the basis for negotiation and transaction. Traditionally, the repeat sales model has been widely adopted to estimate real estate prices. However, it depends on the design and calculation of a complex economic-related index, which is challenging to estimate accurately. Today, real estate brokers provide easy access to detailed online information on real estate properties to their clients. We are interested in estimating the real estate price from these large amounts of easily accessed data. In particular, we analyze the prediction power of online house pictures, which is one of the key factors for online users to make a potential visiting decision. The development of robust computer vision algorithms makes the analysis of visual content possible. In this paper, we employ a recurrent neural network to predict real estate prices using the state-of-the-art visual features. The experimental results indicate that our model outperforms several other state-of-the-art baseline algorithms in terms of both mean absolute error and mean absolute percentage error.

Index Terms—Deep neural networks, real estate, visual content analysis.

I. INTRODUCTION

R EAL estate appraisal, which is the process of estimatingthe price for real estate properties, is crucial for both buys and sellers as the basis for negotiation and transaction. Real estate plays a vital role in all aspects of our contemporary society. In a report published by the European Public Real Estate Association (EPRA http://alturl.com/7snxx), it was shown that real estate in all its forms accounts for nearly 20% of the economic activity. Therefore, accurate prediction of real estate prices or the trends of real estate prices help governments and companies make informed decisions. On the other hand, for most of the working class, housing has been one of the largest expenses. A right decision on a house, which heavily depends on their judgement on the value of the property, can possibly help them save money or even make profits from their investment in

Manuscript received March 28, 2016; revised February 26, 2017 and April 18, 2017; accepted May 15, 2017. Date of publication June 1, 2017; date of current version November 15, 2017. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Benoit Huet. (Corresponding author: Quanzeng You.)

Q. You and J. Luo are with the Department of Computer Science, Univer- sity of Rochester, Rochester, NY 14623 USA (e-mail: qyou@cs.rochester.edu; jluo@cs.rochester.edu).

R. Pang is with PayPaL, San Jose, CA 95131 USA (e-mail: pangrr89@ gmail.com).

L. Cao is with the Electrical Engineering and Computer Sciences Department, Columbia University, New York, NY 10013 USA, and also with customerser- viceAI, New York, NY 10013 USA (e-mail: liangliang.cao@gmail.com).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TMM.2017.2710804

Fig. 1. Example of homes for sale from Realtor.

their homes. From this perspective, real estate appraisal is also closely related to people’s lives.

Current research from both estate industry and academia has reached the conclusion that real estate value is closely related to property infrastructure [1], traffic [2], online user reviews [3] and so on. Generally speaking, there are several different types of appraisal values. In particular, we are interested in the market value, which refers to the trade price in a competitive Walrasian auction setting [4]. Today, people are likely to trade through real estate brokers, who provide easy access online websites for browsing real estate property in an interactive and convenient way. Fig. 1 shows an example of house listing from Realtor (http://www.realtor.com/), which is the largest real estate broker in North America. From the figure, we see that a typical piece of listing on a real estate property will introduce the infrastructure data in text for the house along with some pictures of the house. Typically, a buyer will look at those pictures to obtain a general idea of the overall property in a selected area before making his next move.

Traditionally, both real estate industry professionals and researchers have relied on a number of factors, such as eco- nomic index, house age, history trade and neighborhood en- vironment [5] and so on to estimate the price. Indeed, these factors have been proved to be related to the house price, which is quite difficult to estimate and sensitive to many different human activities. Therefore, researchers have devoted much effort in

1520-9210 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

2752 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 19, NO. 12, DECEMBER 2017

building a robust house price index [6]–[9]. In addition, quan- titative features including Area, Year, Storeys, Rooms and Cen- tre [10], [11] are also employed to build neural network models for estimating house prices. However, pictures, which is proba- bly the most important factor on a buyer’s initial decision making process [12], have been ignored in this process. This is partially due to the fact that visual content is very difficult to interpret or quantify by computers compared with human beings.

A picture is worth a thousand words. One advantage with im- ages and videos is that they act like universal languages. People with different backgrounds can easily understand the main con- tent of an image or video. In the real estate industry, pictures can easily tell people exactly how the house looks like, which is im- possible to be described in many ways using language. For the given house pictures, people can easily have an overall feeling of the house, e.g. what is the overall construction style, how the neighboring environment looks like. These high-level attributes are difficult to be quantitatively described. On the other hand, today’s computational infrastructure is also much cheaper and more powerful to make the analysis of computationally inten- sive visual content analysis feasible. Indeed, there are existing works on focusing the analysis of visual content for tasks such as prediction [13], [14], and online user profiling [15]. Due to the recently developed deep learning, computers have become smart enough to interpret visual content in a way similar to human beings.

Recently, deep learning has enabled robust and accurate feature learning, which in turn produces the state-of-the-art per- formance on many computer vision related tasks, e.g., digit recognition [16], [17], image classification [18], [19], aesthet- ics estimation [20] and scene recognition [21]. These systems suggest that deep learning is very effective in learning robust features in a supervised or unsupervised fashion. Even though deep neural networks may be trapped in local optima [22], [23], using different optimization techniques, one can achieve the state-of-the-art performance on many challenging tasks men- tioned above.

Inspired by the recent successes of deep learning, in this work we are interested in solving the challenging real estate ap- praisal problem using deep visual features. In particular, for images related tasks, Convolutional Neural Network (CNN) are widely used due to the usage of convolutional layers. It takes into consideration the locations and neighbors of image pixels, which are important to capture useful features for vi- sual tasks. Convolutional Neural Networks [18], [19], [24] have been proved very powerful in solving computer vision related tasks.

We intend to employ the pictures for the task of real es- tate price estimation. We want to know whether visual features, which is a reflection of a real estate property, can help estimate the real estate price. Intuitively, if visual features can charac- terize a property in a way similar to human beings, we should be able to quantify the house features using those visual re- sponses. Meanwhile, real estate properties are closely related to the neighborhood. In this work, we develop algorithms which only rely on: 1) the neighbor information and 2) the attributes from pictures to estimate real estate property price.

To preserve the local relation among properties we employ a novel approach, which employs random walks to generate house sequences. In building the random walk graph, only the locations of houses are utilized. In this way, the problem of real estate appraisal has been transformed into a sequence learn- ing problem. Recurrent Neural Network (RNN) is particularly designed to solve sequence related problems. Recently, RNNs have been successfully applied to challenging tasks including machine translation [25], image captioning [26], and speech recognition [27]. Inspired by the success of RNN, we deploy RNN to learn regression models on the transformed problem.

The main contributions of our work are as follows. 1) To the best of our knowledge, we are the first to quan-

tify the impact of visual content on real estate price es- timation. We attribute the possibility of our work to the newly designed computer vision algorithms, in particular Convolutional Neural Networks (CNNs).

2) We employ random walks to generate house sequences according to the locations of each house. In this way, we are able to transform the problem into a novel sequence prediction problem, which is able to preserve the relation among houses.

3) We employ the novel Recurrent Neural Networks (RNNs) to predict real estate properties and achieve accurate results.

II. RELATED WORK

Real estate appraisal has been studied by both real estate in- dustrial professionals and academia researchers. Earlier work focused on building price indexes for real properties. The semi- nal work in [6] built price index according to the repeat prices of the same property at different times. They employed regression analysis to build the price index, which shows good perfor- mances. Another widely used regression model, Hedonic re- gression, is developed on the assumption that the characteristics of a house can predict its price [7], [8]. However, it is argued that the Hedonic regression model requires more assumptions in terms of explaining its target [28]. They also mentioned that for repeat sales model, the main problem is lack of data, which may lead to failure of the model. Recent work in [9] employed locations and sale price series to build an autoregressive com- ponent. Their model is able to use both single sale homes and repeat sales homes, which can offer a more robust sale price index.

More studies are conducted on employing feed forward neu- ral networks for real estate appraisal [29]–[32]. However, their results suggest that neural network models are unstable even us- ing the same package with different run times [29]. The perfor- mance of neural networks are closely related to the features and data size [32]. Recently, Kontrimas and Verikas [33] empirically studied several different models on selected 12 dimensional fea- tures, e.g., type of the house, size, and construction year. Their results show that linear regression outperforms neural network on their selected 100 houses.

More recent studies in [1] propose a ranking objective, which takes geographical individual, peer and zone dependencies into

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

YOU et al.: IMAGE-BASED APPRAISAL OF REAL ESTATE PROPERTIES 2753

consideration. Their method is able to use various estate related data, which helps improve their ranking results based on prop- erties’ investment values. Furthermore, the work in [3] studied online user’s reviews and mobile users’ moving behaviors on the problem of real estate ranking. Their proposed sparsity regu- larized learning model demonstrated competitive performance.

In contrast, we are trying to solve this problem using the attributes reflected in the visual appearances of houses. In particular, our model does not use the meta data of a house (e.g., size, number of rooms, and construction year). We intend to utilize the location information in a novel way such that our model is able to use the state-of-the-art deep learning for feature extraction (Convolutional Neural Network) and model learning (Recurrent Neural Network).

III. RECURRENT NEURAL NETWORK FOR REAL ESTATE PRICE ESTIMATION

In this section, we present the main components of our frame- work. We describe how to transform the problem into a prob- lem that can be solved by the Recurrent Neural Network. The architecture of our model is also presented.

A. Random Walks

One main feature of real estate properties is its location. In particular, for houses in the same neighborhood, they tend to have similar extrinsic features including traffic, schools and so on. We build an undirected graph G for all the houses collected, where each node vi represent the i-th house in our data set. The similarity sij between house hi and house hj is defined using the Gaussian kernel function, which is a widely used similarity measure1

sij = exp ( dist(hi,hj )

2σ2

) (1)

where dist(hi,hj ) is the geodesic distance between house hi and hj . σ is the hyper-parameter, which controls the similarity decaying velocity with the increase of distance. In all of our experiments, we set σ to 0.5 miles so that houses within the 1.5 (within 3σ) miles will have a relatively larger similarity. The �-neighborhood graph [34] is employed to build G in our implementation. We assign the weight of each edge eij as the similarity sij between house hi and the house hj .

Given this graph G, we can then employ random walks to gen- erate sequences. In particular, every time, we randomly choose one node vi as the root node, then we proportionally jump to its neighboring nodes vj according to the weights between vi and its neighbors. The probability of jumping to node vj is defined as

pj = eji∑

k∈N (i) eki (2)

where N(i) is the set of neighbor nodes of vi. We continue to employ this process until we generate the desired length of se- quence. The employment of random walks is mainly motivated

1[Online]. Available: http://en.wikipedia.org/wiki/Radial_basis_function_ kernel

Algorithm 1: Random Walks Require: H = {h1,h2, . . . ,hn} geo-coordinates of n

houses σ hyper-parameter for Gaussian Kernel t threshold for distance M total number of desired sequences

1: Calculate the Vincenty distance between any pair of houses

2: Calculate the similarity between houses according to the Gaussian kernel function (see (1)).

3: repeat 4: Initialize sc = {} 5: Randomly pick one node hi and add hi to sc 6: set hc = hi 7: while size(sc) < L do 8: Pick hc’s neighbor node hj with probability pj

defined in (2) 9: add hj to sc

10: set hc = hj 11: end whileadd sc to S 12: until size (S) = M 13: return The set of sequence S

by the recent proposed DeepWalk [35] to learn feature represen- tations for graph nodes. It has been shown that random walks can capture the local structure of the graphs. In this way, we can keep the local location structure of houses and build sequences for houses in the graph. Algorithm 1 summarizes the detailed steps for generating sequences from a similarity graph.

We have generated sequences by employing random walks. In each sequence, we have a number of houses, which is related in terms of their locations. Since we build the graph on top of house locations, the houses within the same sequence are highly possible to be close to each other. In other words, the prices of houses in the same sequence are related to each other. We can employ this context for estimating real estate property price, which can be solved by recurrent neural network discussed in following sections.

B. Recurrent Neural Network

With a Recurrent Neural Network (RNN), we are trying to predict the output sequence {y1,y2, . . . ,yT } given the input sequence {x1,x2, . . . ,xT }. Between the input layer and the output layer, there is a hidden layer, which is usually estimated as in

ht = Δ(W i hht−1 + Wxxt + bh). (3)

Δ represents some selected activation function or other com- plex architecture employed to process the input xt and ht. One of the most widely deployed architectures is Long Short-Term Memory (LSTM) cell [36], which can overcome the vanishing and exploding gradient problem [37] when training RNN with gradient descent. Fig. 2 shows the details of a single Long Short- Term Memory (LSTM) block [38]. Each LSTM cell contains an input gate, an output gate and an forget gate, which is also called a memory cell in that it is able to remember the error in

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

2754 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 19, NO. 12, DECEMBER 2017

Fig. 2. Illustration of a single long short-term memory (LSTM) cell.

the error propagation stage [39]. In this way, LSTM is capable of modeling long-range dependencies than conventional RNNs.

For completeness, we give the detailed calculation of ht given input xt and ht−1 in the following equations. Let Wi. , Wf. , W

o . represent the parameters related to input, forget and

output gate respectively. � denotes the element-wise multiplica- tion between two vectors. φ and ψ are some selected activation functions and σ is the fixed logistic sigmoid function. Follow- ing [27], [38], [40], we employ tanh for both φ in (6) and ψ in (8):

it = σ(W i xxt + W

i hht−1 + W

i c ct−1 + bi) (4)

ft = σ(W f x xt + W

f h ht−1 + W

f c ct−1 + bf ) (5)

ct = ft � ct−1 + it � φ(Wcxxt + Wchht−1 + bc) (6) ot = σ(W

o x xt + W

o h ht−1 + W

o c ct + bo) (7)

ht = ot � ψ(ct). (8)

C. Multilayer Bidirectional LSTM

In previous sections, we have discussed the generation of sequences as well as Recurrent Neural Network. Recall that we have built an undirected graph in generating the sequences, which indicates that the price of one house is related to all the houses in the same sequence including those in the later part. Bidirectional Recurrent Neural Network (BRNN) [41] has been proposed to enable the usage of both earlier and future contexts. In bidirectional recurrent neural network, there is an additional backward hidden layer iterating from the last of the sequence to the first. The output layer is calculated by employing both forward and backward hidden layer.

Bidirectional-LSTM (B-LSTM) is a particular type of BRNN, where each hidden node is calculated by the long short-term memory as shown in Fig. 2. Graves et al. [40] have employed Bidirectional-LSTM for speech recognition. Fig. 3 shows the architecture of the bidirectional recurrent neural network. We have two Bidirectional-LSTM layers. During the forward pass of the network, we calculate the response of both the forward and the backward hidden layers in the 1st-LSTM and 2nd-LSTM

Algorithm 2: Training Multi-Layer B-LSTM

Require: H = {h1,h2, . . . ,hn} geo-coordinates of n houses X = {x1,x2, . . . ,xn} features of the n house Y ={y1,y2, . . . ,yn} prices of the n houses

1: S = RandomWalks (see Algorithm 1) 2: Split S into mini-batches 3: repeat 4: Calculate the gradient of L in (9) and update the

parameters using RMSProp. 5: until Convergence 6: return The learned model M

layer respectively. Next, the output (in our problem, the output is the price of each house) of each house is calculated using the output of the 2nd-LSTM layer as input to the output layer.

The objective function for training the Multi-Layer Bidirec- tional LSTM is defined as follows:

L = 1 N

N∑ n= 1

∑ j

‖ ŷij − yij ‖2 (9)

where W is the the set of all the weights between different layers. yij is the actual trade price for the j-th house in the generated i-th sequence and ŷij is the corresponding estimated price for this house.

When training our Multi-Layer B-LSTM model, we employ the RMSProp [42] optimizer, which is an adaptive method for automatically adjust the learning rates. In particular, it normal- izes the gradients by the average of its recent magnitude.

We conduct the back propagation in a mini-batch approach. Algorithm 2 summarizes the main steps for our proposed algorithm.

D. Prediction

In the prediction stage, the first step is also generating se- quence. For each testing house, we add it as a new node into our previously build similarity graph on the training data. Each test- ing house is a new node in the graph. Next, we add edges to the testing nodes and the training nodes. We use the same settings when adding edges to the new �-neighborhood graph. Given the new graph G′, we randomly generate sequences and keep those sequences that contain one and only one testing node. In this way, for each house, we are able to generate many different sequences that contain this house. Fig. 4 shows the idea. Each testing sequence only has one testing house. The remaining nodes in the sequence are the known training houses.

a) Average: The above strategy implies that we are able to build many different sequences for each testing house. To obtain the final prediction price for each testing house, one simple strat- egy is to average the prediction results from different sequences and report the average price as the final prediction price.

IV. EXPERIMENTAL RESULTS

In this section, we discuss how to collect data and evalu- ate the proposed framework as well as several state-of-the-art

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

YOU et al.: IMAGE-BASED APPRAISAL OF REAL ESTATE PROPERTIES 2755

Fig. 3. Multilayer BRNN architecture for real estate price estimation. There are two bidirectional recurrent layers in this architecture. For real estate price estimation, the price of each house is related to all houses in the same sequence, which is the main motivation to employ bidirectional recurrent layers.

Fig. 4. Testing sequence h1 → h2 → · · · → hT . In each testing sequence, there is one and only one testing node in that sequence. The remaining nodes are all come from training data.

approaches. In this work, all the data are collected from Real- tor (http://www.realtor.com/), which is the largest realtor asso- ciation in North America. We collect data from San Jose, CA, one of the most active cities in U.S., and Rochester, NY, one of the least active cities in U.S., over a period of one year. In the next section, we will discuss the details on how to preprocess the data for further experiments.

A. Data Preparation

The data collected from Realtor contains description, school information and possible pictures about each real property as shown in Fig. 1 show. We are particularly interested in employ- ing the pictures of each house to conduct the price estimation. We filter out those houses without image in our data set. Since houses located in the same neighborhood seem to have similar price, the location is another important features in our data set. However, after an inspection of the data, we notice that some of the house price are abnormal. Thus, we preprocess the data by filtering out houses with extremely high or low price compared with their neighborhood.

Table I shows the overall statistics of our dataset after filter- ing. Overall, the city of San Jose has more houses than Rochester on the market (as expected for one of the hottest market in the

TABLE I AVERAGE PRICE PER SQUARE FOOT AND THE STANDARD DEVIATION

(STD) OF THE PRICE OF THE TWO STUDIED CITIES

City # of Houses Avg Price std of Price

San Jose 3064 454.2 132.1 Rochester 1500 76.4 21.2

country). The house prices in the two cities also have significant differences. Fig. 5 shows some of the example house pictures from the two cities, respectively. From these pictures, we ob- serve that houses whose prices are above average typically have larger yards and better curb appeal, and vice versa. The same can be observed among house interior pictures (examples not shown due to space).

Realtor does not provide the exact geo-location for each house. However, geo-location is important for us to build the �-neighborhood graph for random walks. We employ Microsoft Bing Map API (https://msdn.microsoft.com/en-us/library/ ff701715.aspx) to obtain the latitude and longitude for each house given its collected address. Fig. 6 shows some of the houses in our collected data from San Jose and Rochester using the returned geo-locations from Bing Map API.

According to these coordinates, we are able to calcu- late the distance between any pair of houses. In particular, we employ Vincenty distance (https://en.wikipedia.org/wiki/ Vincenty’s_formulae) to calculate the geodesic distances ac- cording to the coordinates. Fig. 7 shows distribution of the dis- tance between any pair of houses in our data set. The distance is less than 4 miles for most randomly picked pair of houses. In building our �-neighborhood graph, we assign an edge between any pair of houses, which has a distance smaller than 5 miles (� = 5 miles).

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

2756 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 19, NO. 12, DECEMBER 2017

Fig. 5. Examples of house pictures of the two cities, respectively. Top row: houses whose prices (per square foot) are above the average of their neighborhood. Bottom row: houses whose prices (per square foot) are below the average of their neighborhood. (a) Rochester. (b) San Jose.

Fig. 6. Distribution of the houses in our collected data for both San Jose and Rochester according to their geo-locations. (a) San Jose, CA, USA (b) Rochester, NY, USA.

Fig. 7. Distribution of distances between different pairs of houses.

B. Feature Extraction and Baseline Algorithms

In our implementation, we experimented with GoogleNet model [43], which is one of the state-of-the-art deep neural architectures. In particular, we use the response from the last avg − pooling layer as the visual features for each image. In this way, we obtain a 1,024 dimensional feature vector for each image. Each house may have several different pictures on dif- ferent angles of the same property. We average features of all the images of the same house (also known as average-pooling)2

to obtain the feature representation of the house. We compare the proposed framework with the following

algorithms. 1) Regression Model (LASSO): Regression model has been

employed to analyze real estate price index [6]. Recently, the results in Fu et al. [3] show that sparse regularization can obtain better performance in real estate ranking. Thus, we choose to use LASSO (http://statweb.stanford.edu/˜tibs/lasso.html), which is a l1-constrained regression model, as one of our baseline algorithms.

2) DeepWalk: Deepwalk [35] is another way of employing random walks for unsupervised feature learning of graphs. The main approach is inspired by distributed word representation learning. In using DeepWalk, we also use �-neighborhood graph with the same settings with the graph we built for generating sequences for B-LSTM. The learned features are also fed into a LASSO model for learning the regression weights. Indeed, deepwalk can be thought as a simpler version of our algorithm, where only the graph structure are employed to learn features. Our framework can employ both the graph structure and other features, i.e. visual attributes, for building regression model.

C. Training a Multilayer B-LSTM Model

With the above mentioned similarity graph, we are able to generate sequences using random walks following the steps described in Algorithm 1. For each city, we randomly split the houses into training (80%) and testing set (20%). Next, we generate sequences using random walks on the training houses only to build our training sequences for Multi-layer B-LSTM.

2We also tried max-pooling. However, the results are not as good as average- pooling. In the following experiments, we report the results using average- pooling.

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

YOU et al.: IMAGE-BASED APPRAISAL OF REAL ESTATE PROPERTIES 2757

TABLE II PREDICTION DEVIATION OF DIFFERENT MODELS FROM THE ACTUAL SALE PRICES

City LASSO DeepWalk RNN-best RNN-avg

MAE MAPE MAE MAPE MAE MAPE MAE MAPE

San Jose 70.79 16.92% 68.05 16.12% 17.98 4.58% 66.3 16.11% Rochester 14.19 24.83% 13.68 23.28% 5.21 9.94% 13.32 22.69%

Note that RNN-best is the upper-bound performance of the RNN based model proposed in this work.

For both cities, we build 200,000 sequences for training, with a length of 10. Similarly, we also generate testing sequences, where each sequence contain one and only one testing house (see Fig. 4). On the average, we randomly generate 100 sequences for each testing house. The B-LSTM model is trained with a batch size of 1024. In our experimental settings, we set the size of the first hidden layer to be 400 and the size of the second hidden layer to be 200.

The evaluation metrics employed are mean absolute error (MAE) and mean absolute percentage error (MAPE). Both of them are popular measures for evaluating the accuracy of pre- diction models. (10) and (11) give the definitions for these two metrics, where pi is the predicted value and ti is the true value for the i-th instance.

MAE = 1 N

N∑ i= 1

|ti − pi| (10)

MAPE = 1 N

N∑ i= 1

|ti − pi ti

| (11)

We use the same training and testing split to evaluate all the approaches. Table II shows the regression results for all the different approaches in the two selected cities. For each testing house, we generate about 100 sequences. In Table II, we report both the best and the average price of the predicted price. For Rochester, the average standard deviation of the predicted prices over all the houses is 5.6, which is 7.33% of the average price in Rochester (see Table I). Comparably, the average standard deviation for San Jose is 34.64, which is 7.63% of the average price in San Jose. The best is the price closest to the true price among all the available sequences for each house.3 Overall, our B-LSTM model outperforms other two baseline algorithms in both cities. All of the evaluation approaches perform better in San Jose than in Rochester in terms of MAPE. This is possible due to the availability of more training data in the city of San Jose. DeepWalk shows slightly better performance than LASSO, which suggests that location is relatively more important than the visual features in the realtor business. This is expected

D. Confidence Level

For each testing house, the proposed model can give a group of predictions. We want to know whether or not the proposed

3This is the upper bound of the prediction results. We choose the closest price using the ground truth price as reference.

Fig. 8. Performance of B-LSTM-avg in different groups. All the testing houses are grouped by the predicted standard deviation. (a) MAE. (b) MAPE.

model can distinguish the confidence level of its prediction. In particular, we group the testing houses evenly into three groups for each city. The first group has the smallest standard deviation of the prediction prices. The second group is the middle one and the last group is the one with the largest standard deviation.

Fig. 8 shows the MAE and MAPE for the different groups. The results show that standard deviation can be viewed as a rough measure of the confidence level of the proposed model on the current testing house. Small standard deviation tends to indicate a high confidence of the model and overall it also suggests a smaller prediction error.

V. CONCLUSION

In this work, we propose a novel framework for real estate appraisal. In particular, the proposed framework is able to take both the location and the visual attributes into consideration. The evaluation of the proposed model on two selected cities suggests the effectiveness and flexibility of the model. Indeed, our work has also offered new approaches of applying deep neural networks on graph structured data. We hope our model can not only give insights on real estate appraisal, but also can inspire others on employing deep neural networks on graph structured data.

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

2758 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 19, NO. 12, DECEMBER 2017

REFERENCES

[1] Y. Fu et al., “Exploiting geographic dependencies for real estate appraisal: A mutual perspective of ranking and clustering,” in Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2014, pp. 1047– 1056.

[2] K. Wardrip, “Public transits impact on housing costs: A review of the literature,” Center for Housing Policy, Washington, DC, USA, Aug. 31, 2011, Center for Housing Policy. [Online]. Avail- able: http://www.reconnectingamerica.org/resource-center/browse- research/2011/public-transit-s-impact-on-housing-costs-a-review-of-the- literature/SearchForm/?Search=Keith+Wardrip

[3] Y. Fu et al., “Sparse real estate ranking with online user reviews and offline moving behaviors,” in Proc. IEEE Int. Conf. Data Mining, 2014, pp. 120–129.

[4] A. Beja and M. B. Goldman, “On the dynamic behavior of prices in disequilibrium,” J. Finance, vol. 35, no. 2, pp. 235–248, 1980.

[5] E. L’Eplattenier, “How to run a comparative market analysis (CMA) the right way.” 2016. [Online]. Available: http://fitsmallbusiness.com/ comparative-market-analysis/

[6] M. J. Bailey, R. F. Muth, and H. O. Nourse, “A regression method for real estate price index construction,” J. Amer. Statist. Assoc., vol. 58, no. 304, pp. 933–942, 1963.

[7] R. Meese and N. Wallace, “Nonparametric estimation of dynamic hedonic price models and the construction of residential housing price indices,” Real Estate Econ., vol. 19, no. 3, pp. 308–332, 1991.

[8] S. T. Anderson and S. E. West, “Open space, residential property values, and spatial context,” Region. Sci. Urban Econ., vol. 36, no. 6, pp. 773–789, 2006.

[9] C. H. Nagaraja et al., “An autoregressive approach to house price modeling,” Annal. Appl. Statist., vol. 5, no. 1, pp. 124–149, 2011.

[10] T. Lasota, Z. Telec, G. Trawiński, and B. Trawiński, “Empirical com- parison of resampling methods using genetic fuzzy systems for a regres- sion problem,” in Intelligent Data Engineering and Automated Learning- IDEAL. New York, NY, USA: Springer, 2011, pp. 17–24.

[11] O. Kempa, T. Lasota, Z. Telec, and B. Trawiński, “Investigation of bagging ensembles of genetic neural networks and fuzzy systems for real estate appraisal,” in Intelligent Information and Database Systems. New York, NY, USA: Springer, 2011, pp. 323–332.

[12] W. Di, N. Sundaresan, R. Piramuthu, and A. Bhardwaj, “Is a picture really worth a thousand words?:-On the role of images in e-commerce,” in Proc. 7th ACM Int. Conf. Web Search Data Mining, 2014, pp. 633–642.

[13] X. Jin, A. Gallagher, L. Cao, J. Luo, and J. Han, “The wisdom of social multimedia: Using flickr for prediction and forecast,” in Proc. Int. Conf. Multimedia, 2010, pp. 1235–1244.

[14] Q. You, L. Cao, Y. Cong, X. Zhang, and J. Luo, “A multifaceted ap- proach to social multimedia-based prediction of elections,” IEEE Trans. Multimedia, vol. 17, no. 12, pp. 2271–2280, Dec. 2015.

[15] Q. You, S. Bhatia, and J. Luo, “A picture tells a thousand words?About you! user interest profiling from user generated visual content,” Signal Process., vol. 124, pp. 45–53, 2016.

[16] Y. LeCun et al., “Backpropagation applied to handwritten zip code recog- nition,” Neural Comput., vol. 1, no. 4, pp. 541–551, 1989.

[17] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, 2006.

[18] D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmid- huber, “Flexible, high performance convolutional neural networks for image classification,” in Proc. 22nd Int. Joint Conf. Artif. Intell., 2011, pp. 1237–1242.

[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proc. 25th Int. Conf. Neural Inf. Process. Syst., 2012, pp. 1097–1105.

[20] X. Lu, Z. Lin, H. Jin, J. Yang, and J. Z. Wang, “Rapid: Rating pictorial aesthetics using deep learning,” in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 457–466.

[21] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning deep features for scene recognition using places database,” in Proc. 27th Int. Conf. Neural Inf. Process. Syst., 2014, pp. 487–495.

[22] G. E. Hinton, “A practical guide to training restricted Boltzmann ma- chines,” Dept. Comput. Sci., Univ. of Toronto, Toronto, ON, Canada, Tech. Rep. UTML TR 2010-003, 2010.

[23] Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade. New York, NY, USA: Springer, 2012, pp. 437–478.

[24] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learn- ing applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.

[25] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine transla- tion by jointly learning to align and translate,” in Proc. Int. Conf. Learn. Represent., 2014. [Online]. Available: https://arxiv.org/abs/ 1409.0473

[26] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neu- ral image caption generator,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3156–3164.

[27] A. Graves, A.-R. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2013, pp. 6645–6649.

[28] F. T. Wang and P. M. Zorn, “Estimating house price growth with repeat sales data: What’s the aim of the game?” J. Housing Econ., vol. 6, no. 2, pp. 93–118, 1997.

[29] E. Worzala, M. Lenk, and A. Silva, “An exploration of neural networks and its application to real estate valuation,” J. Real Estate Res., vol. 10, no. 2, pp. 185–201, 1995.

[30] P. Rossini, “Improving the results of artificial neural network models for residential valuation,” in Proc. 4th Annu. Pacific-Rim Real Estate Soc. Conf., Perth, Western Australia, 1998.

[31] P. Kershaw and P. Rossini, “Using neural networks to estimate constant quality house price indices,” in Proc. Int. Real Estate Soc. Conf., Jan. 26–30, 1999.

[32] N. Nghiep and C. Al, “Predicting housing value: A comparison of multiple regression analysis and artificial neural networks,” J. Real Estate Res., vol. 22, no. 3, pp. 313–336, 2001.

[33] V. Kontrimas and A. Verikas, “The mass appraisal of the real es- tate by computational intelligence,” Appl. Soft Comput., vol. 11, no. 1, pp. 443–448, 2011.

[34] U. Von Luxburg, “A tutorial on spectral clustering,” Statist. Comput., vol. 17, no. 4, pp. 395–416, 2007.

[35] B. Perozzi, R. Al-Rfou, and S. Skiena, “Deepwalk: Online learning of social representations,” in Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2014, pp. 701–710.

[36] F. Gers, “Long short-term memory in recurrent neural networks,” Ph.D. dissertation, Dept. Comput. Sci., École Polytechnique Fédérale de Lau- sanne, Lausanne, Switzerland, 2001.

[37] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training re- current neural networks,” in Proc. 30th Int. Conf. Int. Conf. Mach. Learn., 2013, pp. 1310–1318.

[38] F. A. Gers, N. N. Schraudolph, and J. Schmidhuber, “Learning pre- cise timing with lstm recurrent networks,” J. Mach. Learn. Res., vol. 3, pp. 115–143, 2003.

[39] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.

[40] A. Graves, N. Jaitly, and A.-R. Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in Proc. Workshop Automat. Speech Recog. Understand., 2013, pp. 273–278.

[41] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673–2681, Nov. 1997.

[42] T. Tieleman and G. Hinton, “Lecture 6.5—rmsprop, coursera: Neural networks for machine learning,” University of Toronto, Toronto, ON, Canada, Tech. Rep., 2012.

[43] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun. 2015, pp. 1–9.

Quanzeng You received the B.E. and M.E. degrees from the Dalian University of Technology, Dalian, China, in 2009 and 2012, respectively, and is cur- rently working toward the Ph.D. degree in computer science at the University of Rochester, Rochester, NY, USA. His advisor is Prof. J. Luo.

His research focuses on social multimedia, social networks, and data mining. He is interested in de- veloping effective machine learning algorithms that can help us understand the data. His recent research is high-level visual understanding, including image

captioning and visual sentiment analysis.

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.

YOU et al.: IMAGE-BASED APPRAISAL OF REAL ESTATE PROPERTIES 2759

Ran Pang is currently working toward the M.S. degree in computer science at the University of Rochester, Rochester, NY, USA.

He is interested in artificial intelligence and his re- search focuses on social multimedia and data mining.

Liangliang Cao received the B.E. degree from the University of Science and Technology of China, Hefei, China, in 2003, the M.E. degree from The Chi- nese University of Hong Kong, Hong Kong, China, in 2005, and the Ph.D. degree from the University of Illinois at Urbana-Champaign, Urbana, IL, USA, in 2011.

He is currently a Senior Research Scientist at Ya- hoo! Laboratories, Sunnyvale, CA, USA, and an Ad- junct Faculty at Columbia University, New York, NY, USA. He has authored or coauthored more than 40 pa-

pers in top conferences and journals, including the International Conference on Computer Vision, the Computer Vision and Pattern Recognition Conference, the European Conference on Computer Vision, the Conference on Neural Informa- tion Processing Systems, the ACM Multimedia, the International World Wide Web Conference, the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MA- CHINE INTELLIGENCE, and the PROCEEDINGS OF THE IEEE. His research interests include the intersection of computer vision, multimedia, and big data analytics.

Mr. Cao was the General Chair of the Greater New York Area Multimedia and Vision Meeting in 2012 and 2013. He was an Area Chair of WACV 2014 and ACM Multimedia 2012. He was a Guest Editor of the ACM Transactions on Multimedia Computing, Communications, and Applications, and the Computer Vision and Image Understanding journal.

Jiebo Luo (S’93–M’96–SM’99–F’09) received the B.S. and M.S. degrees in electrical engineering from the University of Science and Technology of China, Hefei, China, in 1989 and 1992, respectively, and the Ph.D. degree in electrical and computer engineer- ing from the University of Rochester, Rochester, NY, USA, in 1995.

He joined the University of Rochester, Rochester, NY, USA, in fall 2011, after more than 15 years at Kodak Research Laboratories, Rochester, NY, USA, where he was a Senior Principal Scientist leading re-

search and advanced development. Prof. Luo is a Fellow of the International Society for Optics and Pho-

tonics, and the International Association for Pattern Recognition. He has been involved in numerous technical conferences, and served as the Program Co-Chair of ACM Multimedia 2010 and the IEEE CVPR 2012. He is the Editor- in-Chief of the Journal of Multimedia, and has served on the Editorial Boards of the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, the IEEE TRANSACTIONS ON MULTIMEDIA, the IEEE TRANSACTIONS ON CIR- CUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, Pattern Recognition, Machine Vision and Applications, and the Journal of Electronic Imaging.

Authorized licensed use limited to: American Public University System. Downloaded on June 09,2020 at 03:46:32 UTC from IEEE Xplore. Restrictions apply.