Designing Ranking Systems for Hotels on Travel Search Engines by Mining User-Generated and Crowdsourced Content

Transcription

1 Unversty of Pennsylvana ScholarlyCommons Operatons, Informaton and Decsons Papers Wharton Faculty Research Desgnng Ranng Systems for Hotels on Travel Search Engnes by Mnng User-Generated and Crowdsourced Content Anndya Ghose Unversty of Pennsylvana Panagots G. Iperots Bebe L Follow ths and addtonal wors at: Part of the Maretng Commons, Other Busness Commons, Recreaton Busness Commons, Sales and Merchandsng Commons, and the Toursm and Travel Commons Recommended Ctaton Ghose, A., Iperots, P. G., & L, B. (2012). Desgnng Ranng Systems for Hotels on Travel Search Engnes by Mnng User-Generated and Crowdsourced Content. Maretng Scence, 31 (3), Ths paper s posted at ScholarlyCommons. For more nformaton, please contact repostory@pobox.upenn.edu.

2 Desgnng Ranng Systems for Hotels on Travel Search Engnes by Mnng User-Generated and Crowdsourced Content Abstract User-generated content on socal meda platforms and product search engnes s changng the way consumers shop for goods onlne. However, current product search engnes fal to effectvely leverage nformaton created across dverse socal meda platforms. Moreover, current ranng algorthms n these product search engnes tend to nduce consumers to focus on one sngle product characterstc dmenson (e.g., prce, star ratng). Ths approach largely gnores consumers' multdmensonal preferences for products. In ths paper, we propose to generate a ranng system that recommends products that provde, on average, the best value for the consumer's money. The ey dea s that products that provde a hgher surplus should be raned hgher on the screen n response to consumer queres. We use a unque data set of U.S. hotel reservatons made over a three-month perod through Travelocty, whch we supplement wth data from varous socal meda sources usng technques from text mnng, mage classfcaton, socal geotaggng, human annotatons, and geomappng. We propose a random coeffcent hybrd structural model, tang nto consderaton the two sources of consumer heterogenety the dfferent travel occasons and dfferent hotel characterstcs ntroduce. Based on the estmates from the model, we nfer the economc mpact of varous locaton and servce characterstcs of hotels. We then propose a new hotel ranng system based on the average utlty gan a consumer receves from stayng n a partcular hotel. By dong so, we can provde customers wth the bestvalue hotels early on. Our user studes, usng ranng comparsons from several thousand users, valdate the superorty of our ranng system relatve to exstng systems on several travel search engnes. On a broader note, ths paper llustrates how socal meda can be mned and ncorporated nto a demand estmaton model n order to generate a new ranng system n product search engnes. We thus hghlght the tght lnages between user behavor on socal meda and search engnes. Our nterdscplnary approach provdes several nsghts for usng machne learnng technques n economcs and maretng research. Keywords user-generated content, socal meda, search engnes, hotels, ranng system, structural models, text mnng, crowdsourcng Dscplnes Maretng Other Busness Recreaton Busness Sales and Merchandsng Toursm and Travel Ths ournal artcle s avalable at ScholarlyCommons:

3 Desgnng Ranng Systems for Hotels on Travel Search Engnes by Mnng User-Generated and Crowd-Sourced Content 1 Anndya Ghose Stern School of Busness, New Yor Unversty, aghose@stern.nyu.edu Panagots G. Iperots Stern School of Busness, New Yor Unversty, panos@stern.nyu.edu Bebe L Stern School of Busness, New Yor Unversty, bl@stern.nyu.edu Abstract User-Generated Content (UGC) s changng the way consumers shop for goods. Based on a unque dataset of hotel reservatons over a 3-month perod from Travelocty.com, we estmate the demand for hotels usng a hybrd random coeffcent structural model that data from a varety of nformaton sources. We obtan user-generated data from three sources: () text of hotel revews from two well-nown travel search engnes, Travelocty.com and Trpadvsor.com, () socal geo-tags from Geonames.org dentfyng the dfferent locaton-based attrbutes of hotels, and () user-contrbuted opnons on the most mportant hotel characterstcs by accessng a wde consumer demographc usng Amazon Mechancal Tur. These data sources are merged wth satellte mages of the dfferent hotel locatons to create one comprehensve dataset summarzng the locaton and servce characterstcs of the hotels n our sample. We use text analyss technques to measure the qualty of avalable revews. Usng these analyses, we quantfy how the extent of subectvty, readablty, complexty and other stylstc features of user-generated revews are assocated wth hotel room sales. Fnally, based on the hybrd model we estmate the weght that consumers place on dfferent locaton and servce-related features of hotels. We extend the basc model to examne nteracton effects between travel purpose, prce, and hotel characterstcs. Busness travelers are the least prce senstve whle toursts are the most prce senstve. Busness travelers have the hghest margnal valuaton for hotels located closer to a hghway and havng easy access to publc transportaton. In contrast, romance travelers have the hghest margnal valuaton for hotels located closer to a beach and those wth a hgh servce ratng. As the ultmate goal of ths research, we use the generated estmates from our model n order to buld a better ranng system for hotel search. Specfcally, we leverage the econometrc analyss and compute the average utlty gan that a consumer gets by stayng n a partcular hotel. We propose to ran the hotels, n response to a search query, usng ths utlty gan, whch s a measure of value that a consumer gets from ths transacton. By dong so, one can provde customers wth the best-value" hotels early on, thereby mprovng the qualty of onlne hotel search compared to exstng systems. Several feld experments n sx maor ctes (New Yor, Los Angeles, San Francsco, Orlando, New Orleans, and Salt Lae Cty), usng 15,600 ranng comparsons from Amazon Mechancal Tur, suggest that our ranng system s superor to exstng systems. 1 We than Susan Athey, Peter Fader, Francos Moreau, Avv Nevo, Mnae Song, Danel Spulber, and Hal Varan for extremely helpful comments that have sgnfcantly mproved the paper. We also than partcpants at NBER IT Economcs & Productvty Worshop, 2010 Worshop on Dgtal Busness Models, 2010 Maretng Scence Conference, Searle Research Symposum on the Economcs and Law of Internet Search at NorthWestern Unversty, Customer Insghts Conference at Yale Unversty, 2010 SCECR conference, 2009 Worshop on Informaton Technology and Systems (WITS), 2009 Worshop on Economcs and Informaton Systems and semnar partcpants at Temple Unversty, Unversty of Mnnesota for helpful comments. Anndya Ghose and Panos Iperots acnowledge the fnancal support from Natonal Scence Foundaton CAREER Awards IIS and IIS , respectvely. Support was also provded through a MSI-Wharton Interactve Meda Grant (WIMI) and a Mcrosoft Vrtual Earth Award. The authors than Travelocty for provdng the data and Uthaman Palanappan for research assstance.

4 2 1. Introducton Consumers today use a varety of nformaton sources n order to learn more about ther potental purchases. It s now wdely acnowledged that local search for hotel accommodatons are a component of general Web searches that are ncreasng n popularty as more and more users search for prces and reserve ther trps onlne. In travel search, a varety of resources are avalable that provde nformaton to potental travelers about the hotels n ther destnaton. Customers try to dentfy hotels that satsfy partcular crtera, such as servce amentes, locaton attrbutes, and so on. Once they dentfy the canddate hotels, customers would typcally loo at the prce and determne f the real value of that hotel matches the correspondng prce. Hence, locatng a hotel wth the specfc desred characterstcs, but wthout compromsng on the value, becomes an mportant queston. Most onlne travel search engnes only provde rudmentary ranng facltes, typcally usng a sngle ranng crteron, such as dstance from the cty center, star ratngs, and prce per nght. Ths approach has qute a few shortcomngs. Frst, t gnores the multdmensonal preferences of consumers, n that a customer s deal choce may consst of several hotel-specfc attrbutes. Second, t does not tae nto account the heterogeneous preferences of consumers towards hotel characterstcs. Gven a user query, the ranng mechansms tend to assume that people's preferences towards the set of hotel characterstcs are homogeneous. Ths leads them to provde an dentcal ranng recommendaton for all customers, regardless of ther age, ncome or purchase context. Thrd, t largely gnores characterstcs related to the locaton of the hotel, for nstance, n terms of proxmty to the beach, or proxmty to a downtown shoppng area. These locaton-based features represent mportant characterstcs that can nfluence the desrablty of a partcular hotel. In ths paper, we propose to buld a system that rans each hotel accordng to the expected utlty gan across the consumer populaton. The advantage of ths system s that t uses consumer utlty theory to desgn a scalar utlty score wth whch to ran hotels whle ncorporatng all of the observed dmensons of hotel qualty. Currently, there are no establshed measures that quantfy the economc mpact of varous nternal (servce) and external (locaton) characterstcs on hotel demand. However, search engnes do have access to a lot of user-generated nformaton not only on ther own ste but across other socal meda stes as well. Such socal meda data can be useful for estmatng the weghts that consumers place on dfferent hotel characterstcs. We use a unque dataset of hotel reservatons from Travelocty.com. The dataset contans complete nformaton on transactons conducted over a 3-month perod from 11/2008 to 1/2009 for 1497 hotels n the Unted States (US). We have data on user-generated content from three sources: () user-generated hotel revews from two well-nown travel search engnes, Travelocty.com and Trpadvsor.com, () socal-geo tags generated by users dentfyng dfferent geographc attrbutes of hotels from Geonames.org, and () user-contrbuted opnons on the most mportant hotel characterstcs usng on-demand surveys and socal

5 3 annotatons from users on Amazon Mechancal Tur (AMT). 2 Moreover, snce some locaton-based characterstcs, such as proxmty to the beach, are not drectly measurable based on UGC, we use mage classfcaton technques to nfer such features from the satellte mages of the area. These dfferent data sources are then merged to create one comprehensve dataset summarzng the locaton and servce characterstcs of all the hotels. Our emprcal modelng and analyses enables us to compute the average utlty gan from a partcular hotel based on the estmaton of prce elastctes and average utltes. Thereafter, we am to generate hotel ranngs that are superor to exstng ranng technques seen n travel search engnes. Our wor nvolves four steps:. Identfy the mportant hotel locaton and servce characterstcs that nfluence hotel demand and collect that data.. Estmate how these hotel characterstcs nfluence demand and quantfy ther margnal effects usng a structural model.. Impute the expected utlty from each hotel based on demand estmaton and generate ranngs based on them v. Valdate our ranng system by conductng feld experments usng AMT. More specfcally, n the frst step, we determne the partcular hotel characterstcs that are most valued by customers, and thus, nfluence the aggregate demand of the hotels. Beyond the drectly observable characterstcs, such as the number of stars, provded by most thrd-party travel webstes, many users also tend to value locaton characterstcs, such as proxmty to the beach, or proxmty to downtown shoppng areas. In our wor, we ncorporate satellte mage classfcaton technques and use both human and computer ntellgence (n the form of socal geo-taggng and text mnng of revews) to nfer these locaton features. In the second step, we use demand estmaton technques (BLP 1995, Berry and Paes 2007, Song 2010) and estmate the economc value assocated wth varous locaton and servce characterstcs. Ths enables us to quanttatvely analyze how each feature nfluences demand and estmate ts mportance relatve to the other features. In the thrd step, after nferrng to the economc sgnfcance of the locaton and servce-based hotel characterstcs, we ncorporate them nto desgnng a hotel ranng system based on the expected utlty gan from a gven hotel. By dong so, we can provde customers wth the best-value" hotels early on, thereby mprovng the qualty of onlne hotel search compared to exstng systems. In the fnal step, we valdate our proposed ranng system by conductng feld experments wth 2 Socal annotaton s an annotaton assocated wth a web resource (e.g., a web page, an onlne mage, etc.). On a socal annotaton system (e.g., the Amazon Mechancal Tur tool n our case), a user can add, modfy or remove nformaton from the web resource wthout modfyng the resource tself. The annotatons can be thought of as a layer on top of the exstng resource, and ths annotaton layer s usually vsble to other users who share the same annotaton system. In such cases, the web annotaton tool s a type of socal software tool.

6 4 users on the popular on-demand socal annotaton ste, AMT across sx dfferent ctes. Our ey results are as follows.. Fve locaton-based characterstcs have a postve mpact on hotel demand: number of external amentes, presence near a beach, presence near publc transportaton, presence near a hghway, and presence near a Downtown. Two locaton-based characterstcs have a negatve mpact on hotel demand: Annual crme rate, and presence near a lae/rver. The textual content and style of revews also demonstrate a statstcally sgnfcant assocaton wth demand. Revews that are less complex, have words wth fewer syllables, and wth fewer spellng errors have a postve nfluence on demand. Revews wth hgher number of characters and wrtten usng smple language are also postvely assocated wth demand. These results suggest that consumers can form an mage about the qualty of a hotel by udgng the qualty of the user-generated revews. Consumers prefer hotels wth revews that contan obectve nformaton (such as factual descrptons of hotels) relatve to subectve nformaton, ndcatng that they do not trust completely hotel-provded descrptons and prefer confrmaton from thrd-partes. Consumers also prefer to stay n hotels wth revews wrtten n a consstent obectve style rather than stayng n a hotel where the user revews dscuss more subectve aspects of the accommodaton.. We extend the basc model to examne nteracton effects between travel purpose, prce, and hotel characterstcs. Our results show that consumer preferences for locaton and servce characterstcs are nfluenced by prce and travel purpose. For nstance, busness travelers are the least prce senstve whle toursts are the most prce senstve. In addton, busness travelers have the hghest margnal valuaton for hotels located closer to a hghway and havng easy access to publc transportaton. In contrast, romance travelers have the hghest margnal valuaton for hotels located closer to a beach and those wth a hgh servce ratng.. A comparson of the model that condtons on the UGC varables wth a model that does not shows that the model wth UGC varables outperformed the latter n both n- and out-of-sample analyses. We conduct addtonal model ft comparsons and fnd that the model s predctve power drops the most when excludng all the locaton varables, followed by the servce varables and then the UGC varables. v. We also conduct several counterfactual experments whch shed lght on how prce cuts affects demand n dfferent locaton envronments and how they affect substtuton patterns across competng hotels. Upon comparng locatons wth Beach and hghway (whch represents the typcal west/south coast settng), and locatons wth Downtown, transportaton and external amentes (whch represents the typcal bg cty settng), we fnd that a prce cut leads to a lower ncrease n demand n a bg cty settng than that n coastlne settng. That s, consumers tend to react much less senstvely to hotel prce changes n a typcal bg cty. In addton, we fnd that the closest substtutes for 4-star hotels are 5-

7 5 star hotels; the closest substtutes for 3-star hotels are 4-star and 2-star hotels; the closest substtutes for 2-star hotels are 1-star hotels. Our ey contrbutons can be summarzed as follows. Frst, we demonstrate the value of usng multple and dverse user-generated data sources towards examnng the economc value of dfferent locaton and servce characterstcs of hotels usng a structural model of demand estmaton. Customers today mae ther decsons n an envronment wth the plethora of avalable data. It s possble that some consumers chec the characterstcs of the hotel usng tourst gudes and mappng applcatons, or consult onlne revew stes to determne the qualty of the hotel and ts amentes. In order to replcate ths decsonmang envronment, we construct an exhaustve dataset, collectng as much nformaton as possble about the hotels n our data, usng a varety of data sources, and a varety of methodologes such as text mnng, on-demand annotatons, and mage classfcaton. We demonstrate the sgnfcance of dfferent sources of data by conduct model ft comparsons between models that condton for one set of varables vs. others. Second, our emprcal estmates enable us to propose a new ranng system for hotel search based on the computaton of expected utlty gan from each hotel. The proposed new ranng system for hotels rans the hotels based on the computaton of expected utlty gan, whch measures the value that a consumer gets from the transacton. The ey noton s that n response to a consumer search query, the system would recommend and ran those hotels hgher that provde a hgher value for money by tang nto account consumers mult-dmensonal preferences. Fnally, to evaluate the qualty of our ranng technque, we conducted a user study toward whch we desgned and executed several feld experments on AMT across sx dfferent marets n the US. Usng more than 15,000 user responses for comparng dfferent ranngs, we show that our proposed ranng performs sgnfcantly better than several baselne-ranng systems that are beng currently used by travel search engnes. A post-expermental survey revealed users strongly preferred the dversty of the retreved results, gven that our lst conssted of a mx of hotels cuttng across several prce and qualty ranges. Ths ndcates that customers prefer a lst of hotels that each specalzes n a varety of characterstcs, rather than a varety of hotels that each specalzes n only one characterstc. Besdes provdng consumers wth drect economc gans, such a ranng system can lead to non-trval reducton n consumer search costs. Furthermore, by drectng the customers to hotels that are better matches for ther nterests, ths can lead to ncreased usage of travel search engnes. The rest of the paper s organzed as follows. Secton 2 dscusses related wor and places our wor n the context of pror lterature. Secton 3 dscusses the wor related to the data preparaton, ncludng the methods used to dentfy mportant hotel characterstcs, the steps undertaen to conduct the surveys on AMT to elct user opnons, and the text mnng technques used to parse user-generated revews. In Sectons 4 and 5, we provde an overvew of our econometrc approach, and dscuss emprcal results,

8 6 respectvely. In Secton 6, we dscuss how one can apply our approach to desgn a real-world applcaton, such as a ranng system for hotel search. In Secton 7, we conclude. 2. Pror Lterature Our paper draws from multple streams of wor. A ey challenge s to brdge the gap between the textual and qualtatve nature of revew content and the quanttatve nature of dscrete choce models. Wth the rapd growth and popularty of the user-generated content on the Web, a new area of research applyng text mnng technque to product revews has emerged. The frst stream of ths research has focused on the sentment analyss of product revews (Hu & Lu 2004, Pang & Lee 2004, Das & Chen 2007). Ths stmulated addtonal research on dentfyng product features n whch consumers expressed ther opnons (Hu & Lu 2004, Scaffd et al. 2007, Snyder & Barzlay 2007). The automated extracton of product attrbutes has also receved attenton n the recent maretng lterature (Lee & Bradlow 2007). The hypothess that product revews affect product sales has receved strong support n pror emprcal studes (for example, Godes and Mayzln 2004, Chevaler and Mayzln 2006, Lu 2006, Dellarocas et al. 2007, Duan et al. 2008, Forman et al. 2008, Moe 2009). However, these studes focus only on numerc revew ratngs (e.g., the valence and volume of revews) n ther emprcal analyss. Only a handful of emprcal studes have formally tested whether the textual nformaton embedded n onlne user-generated content can have an economc mpact (Ghose et al. 2006, Elashberg et al. 2007, Archa et al. 2008, Ghose and Iperots 2010). However, these studes do not focus on estmatng the mpact of user-generated revews n nfluencng sales beyond the effect of numerc revew ratngs. In addton, researchers usng only numerc ratngs have to deal wth ssues le self-selecton bas (L and Htt 2008) and bmodal dstrbuton of revews (Hu et al. 2008). More mportantly, the matchng of consumers to hotels n numercal ratng systems s not random. A consumer only rates the hotel that she frequents (.e. the one that maxmzes her utlty). Consequently, the average star ratng for each hotel need not reflect the populaton average utlty. Due to the above drawbacs, the average numercal star ratng assgned to a product may not convey a lot of nformaton to a prospectve buyer. Therefore, a ey obectve of ths paper s to analyze the extent to whch textual content and lngustc style of user-generated revews can help us understand consumer choce of hotels. Our wor s related to models of demand estmaton. One model that has made a sgnfcant contrbuton to the feld s the random coeffcent logt model, or BLP 1995 (Berry et al. 1995). Due to the lmtatons of the product-level taste shoc n logt models, a new model based on pure product characterstcs has been proposed recently (Berry and Paes 2007). The pure characterstc model (hereafter, PCM) dffers from the BLP model n the sense that t does not contan the product-level taste shoc. It descrbes the consumer heterogenety, purely based on ther dfferent tastes towards ndvdual product characterstcs, wthout consderatons on the tastes of certan products as a whole (.e., brand preference). However n realty, the product-level dosyncratc tastes of dfferent consumers do exst n

9 7 many marets. As ponted out n Song (2010)Song (2010), whether or not one should ntroduce the product-level taste shoc should depend on the context of the maret. Keepng n mnd the two levels of consumer heterogenety ntroduced by (1) dfferent travel categores (.e., famly trp, romance, or busness trp) and (2) dfferent hotel characterstcs, we propose a random coeffcent hybrd structural model to dentfy the latent weght dstrbuton that consumers assgn to each hotel characterstc. The outcome of our analyss enables us to compute the expected utlty gan from each hotel and ran them accordngly on a travel search engne. Fnally, our paper s related to the wor n onlne recommender systems. By generatng a novel ranng approach for hotels, we am to mprove the recommendaton strategy for travel search engnes and provde customers wth the best-value" hotels early on n the search process. In the maretng lterature, several model-based recommendaton systems have been proposed to predct preferences for recommended tems (Ansar et al. 2000, Yng et al. 2005, Bodapat 2008). A more recent trend along ths lne s Adaptve Personalzaton Systems (Ansar and Mela 2003, Rust and Chung 2006, Chung et al. 2009). 3. Data Descrpton Our dataset conssted of observatons from 1479 hotels n the US. We collected data from varous sources to conduct our study. We had 3 months of hotel transacton data from Travelocty.com from November to January , whch contaned the average transacton prce per room per nght and the total number of rooms sold per transacton. Next, we dscuss the data preparaton wor that s requred. Our wor leveraged three types of usergenerated content data: On-demand user-contrbuted opnons through Amazon Mechancal Tur Locaton descrpton based on user-generated geo-taggng and mage classfcaton Servce descrpton based on user-generated product revews We frst dscuss how we leverage Amazon Mechancal Tur to collect nformaton on user preferences for dfferent hotel characterstcs. Ther responses suggest that these characterstcs can be lumped nto two groups: locaton and servce characterstcs. Once we dentfy the set of consumer preferences, we use other nds of user-generated content to nfer the external locaton characterstcs, the nternal servce characterstcs, and the textual characterstcs of hotel revews that can nfluence consumer purchases. For a better understandng of the varables n our settng, we present the data sources, defntons, and summary statstcs of all varables n Tables 1 and Identfcaton of Hotel Characterstcs usng Amazon Mechancal Tur (AMT) Our analyss frst requres nowledge of those aspects of a hotel that are most mportant to consumers. These factors determne the aggregate prces of the hotels. For our research, we wanted to avod mposng

10 8 ourselves the features that we need to consder. Rather, we decded to rely on a survey of potental hotel customers and as them about the hotel aspects that are mportant for ther purchasng decsons. We do ths through an onlne survey of users. In order to reach a wde demographc, we decded to rely on the crowd-sourcng maretplace of Amazon Mechancal Tur (AMT, hereafter). AMT s an onlne maretplace, used to automate the executon of mcro-tass that requre human nterventon (.e., cannot be fully automated usng data mnng tools). Tas requesters post smple mcro-tass, nown as hts (human ntellgence tass), n the maretplace. The maretplace provdes proper control over the tas executon, such as valdaton of the submtted answers, or the ablty to assgn the same tas to several dfferent worers. It also ensures the proper randomzaton of the assgnments of tass to worers wthn a sngle tas type. Each user receves a small monetary compensaton for completng the tas. For our purposes, our man goal was to have a dversty of consumer opnons. Therefore, before usng AMT for our survey, we wanted to ensure that the partcpants are representatve of the overall Internet populaton. Towards ths goal, we constructed a survey, asng AMT worers to gve us nformaton about ther place of orgn and resdence, gender, age, educaton attanment, ncome, martal status, household sze, and number of chldren. We also ased them about the tme that they spend every wee on AMT, the amount of wor that they complete, the payment they receve, and ther reasons for partcpatng on AMT. To ensure that the results were not accdental, we conducted the survey multple tmes, once every month. The results of the surveys were consstent over tme, ndcatng that our fndngs are robust. The results of the survey ndcated that, contrary to popular percepton, most of the worers are based n the Unted States. Typcally, 70%-80% of the worers mar the Unted States as the country of resdence. Overall, the populaton of the worers matched qute ncely the overall populaton of Internet users. More than 60% of the worers had unversty educaton, and more than 15% of them had graduate degrees, ndcatng that the AMT survey partcpants are more educated than the average Internet user n the US. We also notced that the age of the worers vary wdely but wth an overrepresentaton of young ages (21-30). Snce the partcpants are comparatvely younger compared to the overall Internet populaton, ther ncome levels were lower, and they had smaller famles. Overall, despte some dfferences, we see that the AMT populaton s generally representatve of the overall US Internet populaton and more representatve than surveys conducted usng only locally avalable partcpants. 3 We also ased the AMT worers about ther prevous experences wth vsts to and hotel reservatons from Travelocty.com. We found that 92.5% of worers specfed that they have vsted the webste of Travelocty before, and 55% specfed that they have made hotel reservatons through t. 3 In Appendx E, we provde the exact analyss of the survey and a comparson of the demographcs, wth the demographcs of US Internet users, accordng to the data provded by ComScore. To compensate for the dfferences n the populaton, we also stratfed the responses from the sample based on demographcs, and placed approprate weghts on the responses n order for the results to match the composton of the US Internet user populaton.

11 9 Based on these fndngs, we use AMT worers as the populaton to survey to fnd what hotel characterstcs are mportant for consumers when they mae ther purchase decsons. As part of our survey, we ased 100 anonymous AMT users the followng open-ended queston: what are the hotel characterstcs that you consder mportant when choosng a hotel? We grouped and coded the results of the gven answers (Table 1 summarzes the dentfed features) and dentfed two broad categores of hotel characterstcs: 1. Locaton-based hotel characterstcs (such as Near a beach, Near a waterfront (lae/rver), Near publc transportaton, and Near downtown ) 2. Servce-based hotel characterstcs (such as Hotel class, Qualty of servce, Number of nternal amentes ) Next, we descrbe how we use consumer-generated content to collect nformaton about the varables that are ether too dffcult to collect otherwse (e.g., densty of shops around the hotel), or are lely to be very subectve (e.g., qualty of servce ). 3.2 Extracton of Locaton Characterstcs usng Socal Geotaggng and Image Processng For the locaton-based characterstcs, we combne user-generated content wth automatc technques, to be able to scale our data collecton and generate data sets that are comprehensve at the natonal and even nternatonal level (.e., tens or even hundreds of thousands of hotels). A frst, automatc approach s to use a servce le the Mcrosoft Vrtual Earth Interactve SDK, whch enables us to compute locaton characterstcs le Near restaurants and shops for a gven hotel locaton on a map. Usng the automatc API from the Mcrosoft, we can automatcally perform such local search queres. However, the presence of a characterstc le Near a beach, or Near downtown cannot be retreved by exstng mappng servces. To measure such characterstcs, we use a combnaton of usergenerated geo-taggng and automatc classfcaton of satellte mages of areas near each hotel n our dataset. Socal GeoTaggng and AMT-based taggng: The concept of geo-taggng has been popularzed lately by photo sharng webstes, n whch users annotate ther photos wth the exact longtude and lattude of the locaton. The concept has been extended and s now used n w -style webstes, where users annotate maps wth varous types of annotatons such as brdge, lae, par and other smlar tags. In our study, we extracted the locaton characterstcs Near publc transportaton, Near a beach and Near the downtown va the ste Geonames.org. For the characterstcs Near a lae/rver and Near the nterstate hghway, we extracted the features usng on-demand annotatons from a set of worers from AMT. Such geo-taggng and on-demand annotatons enable us to generate a rcher descrpton of the locaton around each hotel, usng features that are not drectly avalable through exstng mappng servces.. Image Classfcaton: However, no matter how comprehensve the taggng s, there can be locatons that are not yet tagged by users. Therefore, we need ways to leverage the tag database, and allow for the

12 10 automatc taggng of areas that lac tags. For ths, we use automatc mage classfcaton technques of satellte mages to tag locaton features that can nfluence hotel demand. Consder, for example, the case where we are tryng to automatcally dentfy whether a hotel s located Near a beach, or Near downtown. Towards ths, we extracted hybrd satellte mages (szed pxels) usng the Vsual Earth Tle System 4, for each of the (thousands) of hotel venues located n the US, wth four dfferent zoom levels for each. These 4 x 1497 mages were used to extract nformaton about the surroundngs of the hotel, through mage classfcaton and human nspecton usng AMT. To automatcally tag satellte mages, we frst needed to tran our classfcaton model. As a tranng set, we used nformaton from two sources: () locatons tagged by users on a socal taggng ste such as Geonames.org or () locatons annotated by users on AMT. We bult the mage classfers as follows: Frst, we randomly selected a set of 121 hotels and requested fve AMT users to label each example accordng to ts correspondng satellte mages from four dfferent zoom levels. The labelers answered whether there s a beach n the mage, or whether the mage s that of a downtown area. We appled a smple maorty votng method to mae the fnal decson from the mult-labels of the example. Second, we traned an SVM classfer on ths dataset and used the traned SVM classfer to classfy the mages that corresponded to the remanng hotels. Pror wor has shown that non-parametrc classfers, such as Neural Networs, Decson Trees, and Support Vector Machnes (SVM) provde better results than parametrc classfers n complex landscapes (Lu and Weng 2007). Therefore, we tested varous non-parametrc classfcaton technques. These nclude () Decson Trees, whch are wdely used for tranng and classfcaton of remotely sensed mage data (due to ther ablty to generate human nterpretable decson rules and ts speed n tranng and classfcaton), and () Support Vector Machnes (SVM), that are hghly accurate and perform well for a wde varety of classfcaton tass (Fuuda and Hrosawa 2001). We conducted a small study to examne the performance of the classfer out-of-sample data. We classfed the out of sample mages usng AMT; our results llustrated that our SVM classfer had an accuracy of 91.2% for the beach mage classfcaton and 80.7% for the downtown mage classfcaton. We also used the C4.5 algorthm for the classfcaton, and notced an accuracy ncrease for Near a beach and a decrease for Near downtown. The man reason for ths s that beach" mages often contan a sand strp," together wth an ocean margn" well dstrbuted n densty. Ths typcally provdes more stable and dstnct textural nformaton for the beach" mages, thus mang them easer to dstngush. 4

13 Extracton of Servce Characterstcs usng Consumer Revews We used two broad characterstcs n the category of servce-based characterstcs: hotel class and number of nternal amentes. Hotel class s an nternatonally accepted standard rangng from 1-5 stars, representng low to hgh hotel grades. Number of nternal amentes s the aggregaton of hotel nternal amentes, such as bedroom qualty (1 bedroom, 2 bedroom etc), hotel staff, food qualty, bathroom amentes and parng faclty. We extracted ths nformaton from the Trpadvsor.com webste usng fully automated parsng. 5 Snce hotel amentes are not lsted explctly on the Trpadvsor.com webste, we retreved them by followng the ln provded on the hotel web page, whch drects the user to one of ts cooperatng partner webstes (.e., Travelocty.com, Orbtz.com, Expeda.com, Prcelne.com, or Hotels.com). 3.5 Extracton of Textual Qualty of Revews We collected customer revews from Travelocty.com. In order to consder the ndrect nfluence of word-of-mouth, we also collected revews from a neutral, thrd party ste - the Trpadvsor.com webste, whch s the world s largest onlne travel communty. We collected all avalable onlne revews and revewers nformaton up to January 31, 2009 (the last date of transactons n our database). Consstent wth pror wor, we use the total number of revews and the numerc revewer ratng to control for word-of-mouth effects. In addton, gven that the actual qualty of revews plays an mportant role n affectng product sales, we looed nto two text style features: subectvty and readablty. Both of them can nfluence consumers purchase decsons (Ghose and Iperots 2010). To capture the revew textual style comprehensvely, we used a multple-tem method for subectvty and readablty. We ncluded two sub-features for subectvty and fve sub-features for readablty, each of whch measures the revew text style. We observed that there are two types of revews, from the stylstc pont of vew. There are revews that lst obectve" nformaton, lstng the characterstcs of the hotel, and gvng an alternate descrpton that confrms (or reects) the descrpton gven by the hotel. The other types of revews are those wth subectve," sentmental nformaton, n whch the revewers gve a very personal descrpton of the hotel, and gve nformaton that, typcally, does not appear n the offcal descrpton of the hotel. We dstngushed the extent of subectve assessments n the revews by dervng a revew-level numercal score for the degree of subectvty. More specfcally, we used the methods from Ghose and 5 Fully automated parsng refers to the approach used to collect nformaton from a webste. Techncally, we bult a crawler that frst saves to the local computer all the nformaton from the web pages on that webste. Then the crawler parses the saved web page fles one at a tme n an automated fashon usng a pre-coded computer program on the local machne.

14 12 Iperots (2010) who buld on the methods n Pang and Lee (2004). In partcular, obectve nformaton s consdered the nformaton that also appears n the hotel-provded descrpton, and subectve s everythng else. To nfer the probablty of revew subectvty, we traned a classfer by usng as obectve documents the hotel-generated descrptons from the webstes of Travelocty and TrpAdvsor. We then randomly retreved 1000 revews to construct the subectve examples of the tranng set. 6 After constructng the classfers, we used the resultng classfcaton models n the remanng, unseen revews. Instead of classfyng each revew as subectve or obectve, we nstead classfed each sentence n each revew as ether obectve" or subectve," eepng the probablty of beng subectve for each sentence. By dong so, we were able to acqure a subectvty confdence score for each sentence n a revew, hence dervng the mean and standard devaton of ths score as the subectvty measurements for that revew. These numercal scores are able to dstngush how lely a revew contans subectve assessments as opposed to obectve descrptons. We also loo nto the mpact of Readablty, whch s a proxy for the dffculty faced by people when readng onlne revews. Past research has shown that easy-readng text mproves comprehenson, retenton, and readng speed, and that the average readng level of the US adult populaton s at the eghth grade level (Whte 2003). Specfcally, for each hotel, we collected all exstng revews to examne the average number of characters per revew, average number of syllables per revew, average number of spellng errors per revew, and the average length of the sentence as a Complexty measurement (total number of characters dvded by the total number of sentences). To avod dosyncratc errors pecular to a specfc readablty metrc, we computed a set of metrcs for each revew. Specfcally, we computed the followng: Automated Readablty Index, Coleman-Lau Index, Flesch Readng Ease, Flesch-Kncad Grade Level, Gunnng and SMOG. For brevty, we only show results wth SMOG Index n the paper although all the other readablty measures yeld smlar results. Furthermore, prevous studes have shown that the socal dentty nformaton of revewers n an onlne communty shapes communty members' udgment of the products. In other words, the prevalence of revewer dsclosure of dentty nformaton s assocated wth changes n product sales (Forman et al. 2008). Therefore, consstent wth pror wor, we nclude the characterstc that captures the level of revewers dsclosure of ther dentty nformaton real name or locaton. More specfcally, ths bnary characterstc descrbes whether or not a revewer had revealed her real name or locaton nformaton on the revewer profle page of Travelocty and Trpadvsor. In sum, there are 5 broad types of characterstcs n ths category: () total number of revews, () overall revew ratng, () revew subectvty (mean and varance), (v) revew readablty (the number of 6 We conducted the tranng process by usng a 4-gram Dynamc Language Model classfer provded by the lngppe toolt. Lngppe s a tool t provded onlne for processng text usng computatonal lngustcs (More nformaton can be found at

15 13 characters, syllables, and spellng errors, complexty and SMOG Index), and (v) the dsclosure dentty nformaton by the revewer. 4 Model In ths secton, we wll dscuss how we develop our random coeffcent structural model and descrbe how we apply t to emprcally estmate the dstrbuton of consumer preferences towards dfferent hotel characterstcs n our settng. 4.1 Random Coeffcent Model Setup Our model s motvated drectly by the model n Song (2010), where the author proposed a hybrd dscrete choce model of dfferentated product demand. Whle Song (2010) had one random coeffcent on prce, we have multple random coeffcents on prces as well as hotel characterstcs. Note that ths hybrd model s a combnaton of the BLP (1995) and the PCM (2007) approaches. It s called a hybrd model because t resembles the random coeffcent logt demand model n descrbng a brand choce (BLP 1995) and the pure characterstcs demand model n descrbng a wthn-brand product choce (PCM 2007). Ths s bascally a dscrete choce model of dfferentated product demand n whch product groups are horzontally dfferentated whle products wthn a gven group are vertcally dfferentated condtonal on product characterstcs. These two types of dfferentaton are dstngushed by a group-level taste shoc, whch s assumed to be dstrbuted..d. wth a Type I extreme-value dstrbuton. Ths taste shoc represents each consumer s specfc preference towards a product group that s not captured by observed or unobserved product characterstcs. Song (2010) refers to a product group that contans vertcally dfferentated products a brand. Ths hybrd model dentfes preference for product characterstcs n a smlar way as the PCM. The man dfference that the hybrd model compares products of each brand on the qualty ladder separately, whle the PCM compares all products on t at the same tme. Hence, the qualty space s much less crowded n the hybrd model. 7 In our context, a hotel travel category represents a brand and the hotels wthn each travel category represent products. In partcular, the maret share functon of hotel wthn travel category can be wrtten as the product of the probablty that travel category s chosen and the probablty that hotel s chosen gven that travel category s chosen. The former probablty s smlar to the choce probablty n BLP, and the latter to that of the PCM.. 7 Ths hybrd model provdes more effcent substtuton patterns accordng to ts basc assumptons and model foundatons. As Song (2009) descrbes, t dstngushes two types of cross substtutons: the wthn-travel category substtuton and the between-travel category substtuton. The former s confned to hotels wthn the same travel category and has the same substtuton pattern as n the PCM. The latter determnes the substtuton pattern for hotels n dfferent travel categores and has a smlar pattern as n BLP but wth a dstnct dfference. That s, mpact of a change (n prce or avalablty) on other travel categores s confned to hotels of smlar qualty. As a result, a hotel wll have fewer substtutes n our model than n the BLP model.

16 14 We defne a consumer s decson-mang behavor as follows. A consumer needs to locate the hotel whose locaton and servce characterstcs best matches her travel purpose. For nstance, f a consumer wants to go on a romantc trp wth a partner, she would be nterested n the set of hotels that are located close to a beach, downtown wth amentes le nghtclubs, restaurants, etc. She s also aware that hotels specalzng n the romance category are more lely to satsfy such locaton and servce needs. Each hotel can belong to one of the followng eght types of ``travel categores: Famly Trp, Busness Trp, Romantc Trp, Tourst Trp, Trp wth Kds, Trp wth Senors, Pet Frendly, and Dsablty Frendly. 8 To capture the heterogenety n consumers travel purpose, we ntroduce an dosyncratc taste shoc at the travel category level. Ths s smlar to the product-level taste shoc n the BLP (1995) model. Each travel category has a hotel that maxmzes a consumer s utlty n that category. We refer to ths as the best hotel n that category. To fnd the best hotel wthn each travel category, we use the pure characterstc model (PCM) proposed by Berry and Paes (2007). The PCM approach s able to capture the vertcal dfferentaton amongst hotels wthn the same travel category. A ratonal consumer chooses a travel category f and only f her utlty from the best hotel n that category exceeds her utlty from the best hotel n any other travel category. Thus, n our model, the utlty for consumer from choosng hotel wth category type n maret t can be represented as llustrated n Equaton (1): u X P, (1) t t t t t Where: represents a consumer, represents hotel wth travel category type ( 1 7), and t represents a hotel maret. In ths model, and are random coeffcents that capture consumers heterogeneous tastes towards dfferent observed hotel characterstcs, X, and towards the average prce per nght, P, respectvely. represents hotel characterstcs unobservable to the econometrcan. wth a superscrpt represents a travel category-level taste shoc. Note that n our model the travel categorylevel shoc s ndependently and dentcally dstrbuted across consumers and travel categores, consstent wth Song (2010). 9 We defne a maret as the combnaton of cty-wee. Correspondngly, the maret share for each hotel s calculated based on the number of rooms sold for that hotel n that cty durng that wee dvded by t 8 Each travel category s defned and chosen accordng to the nformaton gleaned from the webste of TrpAdvsor. TrpAdvsor allows revewers to specfy ther man trp purpose (travel category) whle postng a revew. We have data on all the hotel revews posted by users for a gven hotel rght from the tme the frst revew was posted tll the last date of our transacton dataset (February 2009). A hotel s classfed nto a specfc travel category based on the most frequently mentoned travel purpose by the revewers for that hotel. Hence, each hotel belongs exclusvely to a travel category. 9 Besdes our model whch ncorporates a travel category level taste shoc, there are at least three other plausble modelng approaches n ths context: () a model wth only a hotel-level taste shoc, resemblng the BLP (1995) approach, () a model wth both travel-category and hotel-level taste shocs, wth travel category at the top herarchy, resemblng the nested logt model, and () a model wth no taste shocs ether at the travel category or hotel level, resemblng the PCM (2007) approach. We have estmated all these models and found that our hybrd model provdes the best performance n both precson and devaton. Detals are provded n Secton 5.3.

17 15 the total number of rooms sold from all hotels n that cty durng that wee. For robustness chec, we also tred the combnaton of cty-nght, and tred revenue nstead of room unts as the bass for maret share calculaton. Meanwhle, regardng the sze of maret, we chose two dfferent defntons, whch lead to two dfferent defntons of outsde good. (1) In our man estmaton, we appled the smlar dea as n most demand estmaton lteratures (e.g., Berry, S et.al 1995, Song 2010) by estmatng the potental consumptons n a maret. For example, we defned the potental maret sze as proportonal to the total number of hotels n a certan maret 10. Under such defnton, the outsde good s defned as no purchase from the current choce set. (2) Alternatvely, snce our man dataset comes from two maor sources: Travelocty-generated transacton data and TrpAdvsor revew data. The dataset we used n our analyss s the set of hotels at the ntersecton of the two sources. Ths means that the hotel choce set for each maret ncludes those hotels that not only have a transacton generated va Travelocty, but also have avalable nformaton on user-generated revews on TrpAdvsor. Snce not every hotel that has a Traveloctygenerated transacton s lsted on the TrpAdvsor webste, we defne our outsde good as the set of hotels that are lsted n the orgnal Travelocty transacton data, but not lsted on the TrpAdvsor webste. Furthermore, we also tested other ways of varyng the maret sze. For example, we appled smlar deas as n Song (2007), by ncreasng or decreasng the total sze for each maret by 20%. Based on all the wor above, we found that n our data dfferent defntons for maret sze yeld qualtatvely the same results. Due to the computatonal complexty and data restrcton, estmatng a unque set of weghts for each consumer s ntractable. To mae ths model tractable, we made some further assumptons about One s to assume that these weghts are normally dstrbuted among consumers,.e., ~ (, ) and and ~ (, ). Our goal s then to estmate the means (, ) and the standard devatons (, ) of these two dstrbutons. The means correspond to the set of coeffcents on hotel characterstcs and on hotel prce, whch measures the average weght placed by the consumers. The standard devatons provde a measure of the extent of consumer heterogenety n those weghts. Furthermore, we notce that these heterogenetes result from partcular demographc attrbutes of consumers. For example, the varance n the prce coeffcent s very lely a result of dfferences n ncomes among the consumers. Therefore, we mae addtonal assumptons about the standard devatons: ~ I, where represents the ncome whose dstrbuton can be learned from the consumer I demographcs; ~ v, where v ~ N(0,1) represents some random factor that wll nfluence people s I v preferences towards ndvdual hotel characterstcs. Therefore, we have the followng two forms for the consumer-specfc coeffcents I and v. I v and : We then rewrte our model as follows: u X v I P, t t t v I t t (2.1) 10 We acqured the total number of hotels n each maret va TrpAdvsor.