Using Polarity Scores of Words for Sentence-level Opinion Extraction

Transcription

1 Usg Polarty Scores of Words for Setece-level Opo Extracto Lu-We Ku, Yog-Sheg Lo ad Hs-Hs Che Departmet of Computer Scece ad Iformato Egeerg Natoal Tawa Uversty Tape, Tawa {lwku, Abstract The opo aalyss task s a plot study task NTCIR-6. It cotas the challeges of opo setece extracto, opo polarty judgmet, opo holder extracto ad relevace setece extracto. The three former are ew tasks, ad the latter s prove to be tough TREC. I ths paper, we troduce our system for aalyzg opoated formato. Several formulae are proposed to decde the opo polartes ad stregths of words from composed characters ad the further to process opo seteces. The egato operators are also take to cosderato opo polarty judgmet, ad the opo operators are used as clues to fd the locatos of opo holders. The performace of the opo extracto ad polarty judgmet acheves the f-measure uder the leet metrc ad uder the strct metrc, whch s the secod best of all partcpats. Keywords: Opo Extracto, Setmet Mg 1 Itroducto The processg of opo formato has bee wdely dscussed these days. People are cocered about opos, ad ths makes the techques of opo formato processg practcal. Geerally speakg, opos are dvded to three categores: postve, eutral ad egatve. Opos of dfferet polartes documets are useful refereces or feedbacks for govermets or compaes helpg them mprove ther servces or products [2]. Opos are usually about a theme, ad are vewed after groupg by the target whch opos toward to, the opo holders or the opo polartes. Therefore, for applcatos, spte of the opo setece extracto ad polarty judgmet, the opo holder detfcato ad the relevace judgmet are also mportat. To extract the relevat opo seteces, techques of relevat setece retreval are vtal. Oe of the three major cofereces, TREC, tred to survey these techques by havg the ovelty track. [10] However, ths task s prove to be tough because of the lack of formato oly oe setece. Moreover, extractg opo holders s beyod extractg amed ettes. All amed ettes, proous, ad job ttles are caddates for opo holders. Eve f all these ettes ca be extracted, we stll eed to decde whch of them are holders of opos. I order to group opos of the same holders, techques for aaphor ad coreferece resoluto must be appled. These ssues rase the degree of dffculty of opo formato processg. May researchers have started the study of opo formato processg. Geerally speakg, the ut for opo formato ca be oe documet, oe setece, or a sgle word. Webe, Wlso ad Bell [9] ad Pag, Lee, ad Vathyaatha [6] processed opo documets ad ther setmet or opo polartes. Researches of extractg opos documets of a specfc gere, revews, also use oe documet as ther judgg ut. Dave s ad Hu s researches both focused o extractg opos of 3C product revews [2][3], whle Ba, Padma ad Arold [11] use move revews as expermetal materals. As for seteces, they are the basc ut for a perso to express a complete dea. Rloff ad Webe dstgushed subjectve seteces [7], whle Km ad Hovy proposed a setmet classfer for Eglsh words ad seteces [4]. Of course, the composed opo words must be recogzed frst to process opo documets ad seteces. Rloff, Webe ad Wlso [12] leared opo ous from patters, ad Takamura, Iu ad Okumura [8] adopt a physcal model to decde opo polartes of words. May techques of NLP were also studed for opo formato processg. Mache learg approaches such as Nave Bayes, maxmum etropy classfcato, ad support vector maches have bee vestgated [6]. Both formato retreval [2] ad formato extracto [1] techologes have also bee explored. However, varous metrcs ad testg beds are employed, whch leads to comparable results. Buldg commo testg sets ad evaluato

2 metrcs are always mportat, ad these are what NTCIR provdes. Wth the equvalet testg documets uder the same evaluato metrcs, t s possble to fd the pros ad cos of each techque, ad also the way to ehace the performace. We proposed our method of opo formato processg for NTCIR plot task ths paper. A Chese opo extracto system s troduced, ad the compoets ths system are used to deal wth subtasks of the plot task. Frequecy-based formulae are adopted the system kerel to calculate the opo scores of words ad seteces, whch tell whether seteces are opoated ad f so, ther opo polartes. Evaluato results are show ad compared wth the other partcpats. At last, a dscusso of the performace s also cluded. 2 A Chese Opo Extracto System: CopeOp The Chese opo extracto system for opoated formato (CopeOp) s a web-based system developed from ews documets. Ths system works o a large set of documets. It ca extract setmet words, seteces ad documets. Moreover, opo summarzato s also oe of ts fuctos. Based o opo summares geerated everyday, t tracks opos toward a specfc topc ad geerate a trackg plot for vsualzato. The trackg topc s the format of a query ths system, so the user ca easly fd opos they cocer. The detal framework of ths system s troduced [13], ad a example of the trackg plots t outputs s show Fgure 1. A B C D 20/3/2000: Electo Day Fgure 1. Opos towards four persos presdetal electo Each bar shows the summarzed opo score of oe day, ad the x-axs s the tmele. Black bars show the postve opo scores, ad the gray bars show the egatve opo scores. The frst perso s the presdet elect. From these plots, we ca observe the reputato of four caddates before ad after the electo. Ths trackg system also tracks opos accordg to dfferet formato sources, cludg ews ageces ad the Web. Therefore, the results of opo aalyss ca be appled as a feature to fd the posto of each ews agecy. Ths s a example of queryg wth typhoo. Oe of the advatages of arragg opos wth tme sequece s easly to see the correspodg evet bursts. Obvously there are three typhoos October The frst s No typhoo Aere, the secod s No typhoo Tokage, ad the thrd s No typhoo NOCK-TEN. The secod oe last loger, whle the thrd oe caused greater damage. Ths fgure shows the opos toward typhoos are all egatve, whch s the same as we have expected. Fgure 2. Opos towards typhoos The compoets for extractg opo words ad seteces, ad the decde ther opo polartes, are essetal ths system. Therefore, the documets of NTCIR opo aalyss task are fed to ths system ad processed by these compoets, ad the the extracted opo formato s reported as the expermet result. 3 Opo Extracto Opos are extracted from seteces, whch are the ut defed by NTCIR opo task. Four factors are cosdered whe extractg opo passages ad determg ther tedecy: the setmet words, the opo operators, the opo holders ad the egato operators. We postulate that the opo of the whole s a fucto of the opos of the parts. That s, the opo degree of a setece, whch decdes f ths setece s opoated ad ts polarty, s a fucto of setmet words, egato words, opo operators, ad opo holders. For egato ad opo operators, word lsts are collected. For recogzg setmet words, ther opo scores are calculated. The defto of the opo scores of words s troduced the ext secto. 3.1 Opo Score of Words Setmet words are employed to compute the tedecy of a setece. Itutvely, a Chese setmet dctoary s dspesable. We adopt a Chese opo dctoary NTUSD [14]. NTUSD cossts of 2,812 postve ad 8,276 egatve opo words.

3 However, lookg up dctoares may suffer from the problem of coverage. I our system, a method to lear setmet words ad ther stregths based o ths dctoary s developed. Scores here dcate the stregths. It s postulated that the meag of a Chese setmet word s a fucto of the composte Chese characters. Ths s exactly how people read deogram whe they come to a ew word. A setmet score s the defed for a Chese word by the followg formula. Ths formula, ot oly tells us the possble opo tedecy of a ukow word, but also dcates ther stregth. We start the dscusso from the defto of the formulas of Chese characters. c Pc (2) c f c fc N c (3) f c Where c ad f c deote the frequeces of a character c the postve ad egatve words, respectvely; ad m deote total umber of uque characters postve ad egatve words, respectvely. Formulas (2) ad (3) utlze the probablty of a character postve/egatve words to show ts setmet tedecy. However, there are more egatve words tha postve oes NTUSD. Hece, the frequecy of a character a postve word may ted to be smaller tha that a egatve word. That causes bas for learg, so formulas (2) ad (3) are ormalzed to formulae (4) ad (5). Pc / N c c / c j1 j 1 c f c / j1 c c f / / m j 1 f f / c m j1 m j 1 f f (4) (5) Formula (7) defes that a setmet tedecy of a Chese word w s the average of the setmet scores of the composg characters c 1, c 2,, c p. p 1 S w Sc (7) j p j1 Accordg to these formula, a character wll be gve at most the score 1 ad at least the score 1, ad so s a word. Take the word (good people) ad (bad people) as examples. There are 7,898 characters postve opo words ad 24,212 characters egatve opo words total. The character (people) appears postve opo words 79 tmes ad egatve opo words 265 tmes. Therefore, the opo score of s 0.04, whch s very eutral. The character (good) appears postve opo words 68 tmes ad egatve opo words 52 tmes, ad t s scored Smlarly, the character (bad) appears postve opo words 0 tmes ad egatve opo words 83 tmes, ad t s scored -1. At last, we fd the opo score of (good people) 0.28, whle the opo score of (bad people) I spte of polarty formato, the opo score provdes stregth formato. For example, the Chese word meas wealth. Its setmet score 0.61 s computed from the sum of (rch, 0.75) ad (expesve, 0.48). To determe the cotext polarty, (wealth, 0.61) s stroger tha (have moey, 0.33), whch s aother Chese word descrbg rch a subtler degree. The stregth formato of setmet words help whe fdg the domate setmet words oe setece. The magtude of the opo score of a ukow word s also the dcato of whether t should be couted. I our system, f a word does ot appear the dctoary, that s, t s ukow, oly the word whose opo score s above 0.4 or below 0.4 s take to cosderato,.e. treated as a setmet word. 3.2 Possble Setmet Words Where P c ad N c deote the weghts of c as Opo scores are ot calculated for all words. postve ad egatve characters, respectvely. Sce the seteces are segmeted, the part of speech Formulae (4) ad (5) calculate the possblty of oe formato s used to extract possble setmet character to carry a postve ad egatve meag, words. From observatos, Chese words are respectvely. The dfferece of P c ad N c,.e., P c - composed mostly by more tha oe character, ad N c Formula (6), determes the setmet tedecy of character c oe character tself usually caot express a complete. If t s a postve value, the ths cocept. Here words wth part of speech A character occurs more ofte postve Chese (adjectve), V (verb), Na (proper ou), D (adverb) words tha egatve oes, ad vce versa. A value ad Cbb (cojucto) ad of legth more tha oe close to 0 meas that t s ot a setmet character or are selected for further calculatos of opo scores. t s a eutral setmet character. 3.3 Negato Operator Sc ( P ) c N c (6)

4 Negato operators are words such as (o), (ot), (ever), (ether), (mpossble), etc.. These words reverse the meags of seteces. Moreover, f they modfy setmet words, the opo polartes of these setmet words wll be reversed, too. I CopeOp, 41 egato operators are collected. For each setece, after assurg setmet words by the formula secto 3.1, each egato operator wll egate the opo polarty of the closest setmet word, that s, chage the opo score of that word from S to S. The effect of a egato operator wll ot cross commas, perods, questo marks, semcolos, ad exclamato marks. Ths setece segmets separated by these puctuato marks are referred to as setece fragmets. Negato operators themselves ca also express egatve atttudes. Therefore, f there are o setmet words oe setece fragmet, the scores of the egato operators wth are couted. 3.4 Opo Operator ad Opo Holder Opo operators are hts for extractg opos. Words lke (say), (thk), (beleve) are actos of expressg thoughts. However, ot all seteces cotag opo operators are opoated. For example, The cetral weather bureau says the hghest temperature today s 32 Celsus degree s cosdered a weather report, whle the setece Joh thks today s hot to death s wthout questo a opo. I the experece of developg our system, we foud that usg opo operators as the oly cues for opo extracto acheves the f-measure aroud 0.55 uder the leet metrc ad 0.35 uder the strct metrc. Geerally, opo operators do ot tell the overall opo polartes. The polartes deped o the cotet of opos. For example, seteces Mary told me that her teacher s a good perso ad Mary told me that her teacher s ot good at teachg, the opo polartes have othg to do wth the opo operator told. However, some opo operators do express the atttudes of the holders ad should be cosdered together wth the cotet of opos. For example, the opo operators hope ad support show the postve atttudes towards the followg opos, whle crtcze ad blame show the egatve atttudes. I the curret system, the opo scores of the opo operators are couted whe decdg the opo polartes. Aother mportat fucto of opo operators s to dcate the opo holders. Sce the opo operators are the actos of expressg opos, the subjects pror to opo operators are lkely to be the holders of the correspodg opos. A word pror to a opo operator s cosdered a opo holder of a opo setece by our system f ether oe of the followg two crtera s met: 1. The part of speech s perso ame (Nb_PERSON), orgazato ame (Nb_ORGANIZATION) or persoal (Nh). For example, (Km Dae-Jug) ad (we) could be possble opo holders. 2. The word s class A (huma), type Ae (job) of Cl. (tog2y4c2c2l2, Me et al., 1982). For example, (professor) ad (studet) could be possble opo holders. 3.5 Algorthm Because opo polartes ad opo holders are formato opo seteces, our system extracts opo seteces frst. Oce the opo seteces are foud, ther correspodg polartes ad holders are reported. The algorthm of the opo extracto s show Fgure 3. Algorthm: Opo Setece Extracto 1. For every setece p 2. For every word p, decde whether t s a setmet word. 3. For every egato operator p 4. Fd the earest setmet word, ad reverse ts opo score from S to -S. 5. Extract the caddate of the opo holder f there s ay opo operator. 6. Decde the opo polarty of p by the fucto of setmet words ad the opo holder as follows. S p S opoholder j1 Where S p, S opo-holder, ad S wj are the opo score of setece p, the weght of opo holder, ad the opo score of setmet word w j, respectvely, ad s the total umber of setmet words p. 7. If the absolute value of S p exceeds the specfc threshold, report ths setece as a opo. Report ts polarty accordg to the sg of S p, ad ts opo holder. Fgure 3. Algorthm of Opo Setece Extracto 4 Expermets ad Dscusso S wj

5 The expermet results are show Table 1, 2, 3 ad 4. For opo setece extracto uder the leet metrc, there are two groups of performace. Oe group s of f-measure aroud 0.6, ad the other s aroud f-measure 0.7. Our system (NTU) s the group of f-measure 0.7. The f-measures of all rus ths task are close to each other. I ths group, our system has the best precso If we cosder performace of the opo extracto together wth the polarty judgmet (feld OpAdPolarty), our system acheves the f-measure 0.383, whch s the secod best. For the performace uder the strct metrc, we are stll the secod best. However, we also fd that the system wth hgher precso wll acheve better performace uder the strct metrc. Therefore, the dfferece betwee our system ad the system of CHUK becomes larger uder the strct metrc. We beleve that the most mportat work s to mprove the precso the future. For setece extracto, f there s ay setmet word oe setece, t wll be extracted. Therefore, mssg oe setmet word wll ot fluece the performace much. However, for the polarty judgmet task, every setmet word s mportat. As metoed, to avod ose, sgle character word wll ot be cosdered a setmet word our system. However, there are several opo words whch cossts oly oe character, ad msses occur. I addto, the egato operator wll egate the earest setmet word. Therefore, the fluece of mssg a setmet word wll propagate f the mssg word happes to be the target for egato. The algorthm of dealg wth egato operators s also very mportat the polarty judgmet. Our system egates the closest setmet words. However, we foud that the pror ad the later setmet words are both possble targets for egato, ad the dstace may ot be the most mportat factor choosg the correct oe. Also f the target s wrog, the result s usually wrog. Besdes, sometmes the egato operator fact egates a o-setmet ou, stead of a setmet word. Sce the ou s o-setmet, our algorthm wll gore t ad fd the closest setmet word to egate. To solve ths problem, we may eed a shallow parser to fd the exact targets of egato operators. Smlar to the egato operators, some verbs have the abltes to egate setmet words. However, ths kd of words s ot cosdered as a egato operator by our system ow. For example, the setece fragmet (ed the terror of wars), the verb ed reverses the setmet of terror, therefore ed the terror s actually somethg good. However, our system, (ed), (war) ad (terror) are all egatve. Ths makes ths setece fragmet very egatve ad that s wrog. For the opo holder extracto task, we acheve a relatvely hgh precso wth a low recall. The loss of the precso s mostly due to wrog segmetatos. However, the loss of the recall may cause by the lmtato that the opo holders must appear pror to opo operators. Sce the opo operators are collected maually ad suffer from the coverage problem, may opo holders are ot extracted. Also f the pror word s ot of part of speech Nb_PERSON, Nb_ORGANIZATION, Nh, or a job ame Cl, othg wll be reported, eve though the opo operators are foud. To acheve a better recall, a double-check mechasm may be eeded to re-exam the locato of the opo holders whe ether opo operators or potetal holders are detected. Opoated Relevace OpAdPolarty P R F P R F P R F CHUK ISCAS Gate Gate UMCP UMCP NTU Table 1. Chese opo aalyss leet results Opoated Relevace OpAdPolarty P R F P R F P R F CHUK ISCAS Gate Gate

6 UMCP UMCP NTU Table 2. Chese opo aalyss strct results Leet Strct P R F P R F CHUK ISCAS Gate Gate UMCP UMCP NTU Table 3. Chese opo holders aalyss: setece based Leet Strct P R F P R F CHUK ISCAS Gate Gate UMCP UMCP NTU Table 4. Chese opo holders aalyss: holder based 5 Cocluso ad Future Work Ths paper troduces a Chese opo extracto system. I ths system, opo scores are used to show the opo polartes ad stregths of words. Ths system adopts bottom up formulae whch calculate the opo scores of potetal setmet words seteces from characters. Together wth the egato operators ad opo operators, the polartes ad opo holders ca be decded for all seteces. The expermetal results are satsfactory. The proposed formulae work well geeral cases. However, they are ot good eough some cases. Frst, the polartes of some opo words are cotext depedet. For example, (crease) s postve whe ts object s (salary), whle t s egatve whe ts object s (tax). Moreover, f we cosder the mult-perspectve ssue, (crease tax) may beeft the budget defct, so t s postve for the govermet. Therefore, the opo polartes of these words deped o the roles they play the seteces or documets. Moreover, there are perspectve ssues to be studed the future. Secod, ths method depeds a lot o the part of speech of words. Some words of part of speech ou (Na) are the ou form of adjectves, but some are ot. Aalyzg the compoets those geeral ous, whch are ot the ou form of adjectves or adverbs, s meagless. However, we caot dstgush oe kd from the other our system, ad ths results false alarms. The egato ssue s also mportat the polarty judgmet. To fd the exact target word for egato, a shallow parser s ecessary. I addto, some other words have the same effects as egato operators. For example, words expressg ot to do lke (prevet) or (dscourage). Eve some opo operators ca egate the opos, too. I the future, words of the cocept ot to do should be able to be extracted automatcally. The combatos of the atttudes of the opo operators ad ther correspodg opos should be cosdered together. To fd opo holders are mportat applcatos of extractg opos. Wth opo holders, the publc ot oly ca fd opos of a specfc perso, but also people havg the same atttudes toward a publc ssue ca be grouped. The possble ettes of persos or orgazatos may ot be ecessarly opo holders opo seteces. Sometmes they are the targets crtczed. Therefore,

7 we extract opo holders wth the hts from opo operators. However, t seems too strct. Not oly the coverage of opo holders s lmted, but also the opo holders do ot always appear together wth the opo operators. We foud that sometmes the opo holders appear wth the possessves. For example, setece fragmets lke the opos/atttudes of A are or B s thoughts o are cota opo holders obvously but do ot accompay opo operators. To solve ths problem, we eed to lear more patters the future. The algorthm for relevat setece retreval s ot tegrated yet ths system, because the CopeOp orgally cooperate wth a IR system. Sce all documets are relevat to the selected opo topcs, seteces these testg documets are all treated as relevat to acheve the basele performace ad we focus o the opo related tasks ths year. I the future, for selectg topcal words ad further retrevg relevat seteces, the exstg algorthm, whch works well o TREC materals [5], ca be appled to mprove the performace. We have developed a Chese opo extracto system. Ad the large-scale expermets are doe o the materals from the NTCIR opo plot task. From the evaluato results, we fd the drectos to mprove our techques o opo setece extracto, opo polarty judgmet, opo holder extracto ad relevat setece retreval. The future goal s to ehace our system wth mproved techques ad apply ths system real applcatos. [9] J. Webe, T. Wlso ad M. Bell. Idetfy collocatos for recogzg opos. Proceedgs of ACL/EACL2001 Workshop o Collocato [10] Soboroff, I. ad Harma, D. Overvew of the TREC 2003 ovelty track. Proceedgs of the Twelfth Text REtreval Coferece, Natoal Isttute of Stadards ad Techology, pages [11] Ba, X., Padma, R. ad Arold, E. O learg parsmoous models for extractg cosumer opos. Proceedgs of the 38th Aual Hawa Iteratoal Coferece o System Sceces, Track 3, Volume 03, page [12] Rloff, E., Webe, J. ad Wlso, T. Learg subjectve ous usg extracto patter bootstrappg. Proceedgs of the Seveth Coferece o Natural Laguage Learg, pages [13] Ku, L.-W., Wu, T.-H., Lee, L.-Y. ad Che., H.-H. (2005). Costructo of a evaluato corpus for opo extracto. Proceedgs of the 5th NTCIR Workshop Meetg, pages [14] Ku, L.-W., Lag, Y.-T. ad Che, H.-H. Opo extracto, summarzato ad trackg ews ad blog Corpora. Proceedgs of AAAI-2006 Sprg Symposum o Computatoal Approaches to Aalyzg Weblogs, AAAI Techcal Report. Pages Refereces [1] C. Carde, J. Webe, T. Wlso ad D. Ltma. Combg low-level ad summary represetatos of opos for mult-perspectve questo aswerg. Proceedgs of AAAI Sprg Symposum Workshop, pages [2] K. Dave, S. Lawrece ad D.M. Peock. Mg the peaut gallery: opo extracto ad sematc classfcato of product revews. Proceedgs of 12th Iteratoal Coferece o World Wde Web, pages [3] Mqg Hu ad Bg Lu. Mg ad Summarzg Customer Revews. SIGKDD 2004, pages [4] Soo-M Km ad Eduard Hovy. Determg the Setmet of Opos. Colg, pages [5] L.-W. Ku, L.-Y. L, T.-H. Wu ad H.-H. Che. Major topc detecto ad ts applcato to opo summarzato. SIGIR 2005, pages [6] B. Pag, L. Lee ad S. Vathyaatha. Thumbs up? Setmet classfcato usg mache learg techques. Proceedgs of the 2002 Coferece o EMNLP, pages [7] E. Rloff ad J. Webe. Learg extracto patters for subjectve expressos. Proceedgs of the 2003 Coferece o EMNLP, pages [8] H. Takamura, T. Iu ad M. Okumura. Extractg Sematc Oretatos of Words Usg Sp Model. ACL 2005, pages