Data Classification with Radial Basis Function Networks Based on a Novel Kernel Density Estimation Algorithm

Transcription

1 Paper : TNN CE485 Data Classfcato wt Radal Bass Fucto Networks Based o a Novel Kerel Desty Estato Algort Ye-Je Oyag, Se-Cg Hwag, Yu-Ye Ou, Ce-Yu Ce 3, ad Z-We Ce 4 Departet of Coputer Scece ad Iforato Egeerg Natoal Tawa Uversty, Tape, Tawa, R. O. C. {yoyag *, ye, zwce 4 }@cse.tu.edu.tw {scwag, cyce 3 }@ars.cse.tu.edu.tw Abstract Ts paper presets a ovel learg algort for effcet costructo of te radal bass fucto RBF etworks tat ca delver te sae level of accuracy as te support vector aces SVM data classfcato applcatos. Te proposed learg algort works by costructg oe RBF sub-etwork to approxate te probablty desty fucto of eac class of obects te trag data set. Wt respect to algort desg, te a dstcto of te proposed learg algort s te ovel kerel desty estato algort tat features a average te coplexty of Olog, were s te uber of saples te trag data set. Oe portat advatage of te proposed learg algort, coparso wt te SVM, s tat te proposed learg algort geerally takes far less te to costruct a data classfer wt a optzed paraeter settg. Ts feature s of sgfcace for ay coteporary applcatos, partcular, for tose applcatos wc ew obects are cotuously added to a already large database. Correspodg autor, Tel: ext. 43, Fax:

2 Aoter desrable feature of te proposed learg algort s tat te RBF etwork costructed s capable of carryg out data classfcato wt ore ta two classes of obects oe sgle ru. I oter words, ulke SVM, t does ot eed to voke ecass suc as oe-agast-oe or oe-agast-all for adlg datasets wt ore ta two classes of obects. Te coparso wt SVM s of partcular terest, because t as bee sow a uber of recet studes tat SVM geerally are able to delver ger level of accuracy ta te oter exstg data classfcato algorts. As te proposed learg algort s stace-based, te data reducto ssue s also addressed ts paper. Oe terestg observato ts regard s tat, for all tree data sets used data reducto erets, te uber of trag saples reag after a aïve data reducto ecas s appled s qute close to te uber of support vectors detfed by te SVM software. Ts paper also copares te perforace of te RBF etworks costructed wt te proposed learg algort ad tose costructed wt a covetoal cluster-based learg algort. Te ost terestg observato leared s tat, wt respect to data classfcato, te dstrbutos of trag saples ear te boudares betwee dfferet classes of obects carry ore crucal forato ta te dstrbutos of saples te er parts of te clusters. Key ters: radal bass fucto RBF etwork, kerel desty estato, data classfcato, ace learg, eural etwork.

3 . Itroducto Te radal bass fucto RBF etwork s a specal type of eural etworks wt several dstctve features [, 3, 7]. Sce ts frst proposal, te RBF etwork as attracted a g degree of terest researc coutes. A RBF etwork cossts of tree layers, aely te put layer, te dde layer, ad te output layer. Te put layer broadcasts te coordates of te put vector to eac of te uts te dde layer. Eac ut te dde layer te produces a actvato based o te assocated radal bass fucto. Fally, eac ut te output layer coputes a lear cobato of te actvatos of te dde uts. How a RBF etwork reacts to a gve put stulus s copletely detered by te actvato fuctos assocated wt te dde uts ad te wegts assocated wt te lks betwee te dde layer ad te output layer. RBF etworks ave bee loted ay applcatos ad qute a few learg algorts ave bee proposed [5, 7, 8, 7, 9, 7, 9, 3, 3]. Te probles tat RBF etworks ave bee appled to clude fucto approxato, data classfcato, ad data clusterg. Depedg o te probles tat te learg algorts are desged for, dfferet optzato crtera ay be eployed. Oe of te a applcatos tat RBF etworks ave bee appled to s data classfcato. However, latest developet data classfcato researc as focused ore o support vector aces SVM [3] ta o RBF etworks, because several recet studes ave reported tat SVM geerally are able to delver ger classfcato accuracy ta te oter exstg data classfcato algorts [6, 8, 0]. Neverteless, SVM suffer oe serous drawback. Tat s, te te take to carry out odel selecto could be uacceptably log for soe coteporary applcatos, partcular, for tose applcatos wc ew obects are cotuously added to a already large database. Terefore, ow to edte te odel selecto process as becoe a crtcal ssue for SVM ad as bee addressed by a uber of recet artcles [, 4, 5, ]. However, te approaces tat ave bee proposed so far for edtg te odel selecto process

4 of SVM all lead to lower predcto accuracy. Ayway, ts s a ssue tat deserves cotuous vestgato. Aoter or drawback of SVM s tat ecass suc as oe-agast-oe or oe-agast-all ust be voked to adle datasets wt ore ta two classes of obects. I ts paper, a ovel learg algort s proposed for effcet costructo of te RBF etworks tat ca delver te sae level of accuracy as SVM data classfcato applcatos wtout sufferg te drawbacks of SVM addressed above. I te RBF etworks costructed wt te proposed learg algort, eac actvato fucto assocated wt te dde uts s a spercal or syetrcal Gaussa fucto. I soe artcles, te specfc type of RBF etworks wt spercal Gaussa fuctos s referred to as te spercal Gaussa RBF etwork [33]. For splcty, we wll use spercal Gaussa fucto SGF etworks to refer to te RBF etworks costructed wt te learg algort proposed ts paper. Wt respect to algort desg, te a dstcto of te proposed learg algort s te ovel kerel desty estato algort desged for effcet costructo of te SGF etwork. Te a propertes of te proposed learg algort are suarzed as follow: te SGF etwork costructed wt te proposed learg algort geerally delvers te sae level of classfcato accuracy as te SVM; te average te coplexty for costructg a SGF etwork s bouded by O log, were s total uber of trag saples; te average te coplexty for classfyg ' cog obects s bouded by O' log. v te SGF etwork s capable of carryg out data classfcato wt ore ta two classes of obects oe sgle ru. Tat s, ulke te SVM, te SGF etwork does ot eed to corporate ecass suc as oe-agast-oe or oe-agast-all for adlg data sets wt ore ta two classes of obects. As te SGF etwork costructed wt te proposed learg algort s stace-based, te effcecy ssue sared by alost all stace-based learg algorts ust be addressed. Tat s, a data reducto ecas ust be eployed to reove redudat saples te trag data

5 set order to prove te effcecy of te stace-based classfer. Experetal results reveal tat te aïve data reducto ecas eployed ts paper s able to reduce te sze of te trag data set substatally wt a slgt pact o classfcato accuracy. Oe terestg observato s tat, te tree data sets used erets, te uber of trag saples reag after data reducto s appled ad te uber of support vectors detfed by te SVM software are te sae order. I fact, two out of te tree cases reported ts paper, te dfferece s less ta 5%. Sce data reducto s a crucal ssue for stace-based learg algorts, furter study o ts ssue sould be coducted. Ts paper also copares te perforace of te SGF etworks costructed wt te proposed learg algort ad te RBF etworks costructed wt a covetoal cluster-based learg algort [9]. Te ost terestg observato leared s tat, wt respect to data classfcato, te dstrbutos of saples ear te boudares betwee dfferet classes of obects carry ore crucal forato ta te dstrbutos of saples te er parts of te clusters. Sce te covetoal cluster-based learg algort for RBF etworks places oe radal bass fucto at te ceter of a cluster, te dstrbutos of saples ear te boudares betwee dfferet classes of obects ay ot be accurately odeled. As a result, te RBF etwork costructed wt te covetoal cluster-based learg algort geeral s ot able to delver te sae level of accuracy as tose data classfcato algorts suc as SVM ad te SGF etwork tat lot te dstrbutos of saples ear te boudares betwee dfferet classes of obects. I te followg part of ts paper, secto presets a revew of te related works. Secto 3 presets a overvew of ow data classfcato s coducted wt te proposed learg algort. Secto 4 elaborates te ovel kerel desty estato algort o wc te proposed learg algort s based. Secto 5 dscusses te pleetato ssues ad presets a aalyss of te coplexty. Secto 6 reports te erets coducted to evaluate te perforace of te proposed learg algort. Fally, cocludg rearks are preseted secto 7. 3

6 . Related Works As etoed earler, tere ave bee qute a few learg algorts proposed for RBF etworks. Te learg algort deteres te uber of uts te dde layer, te actvato fuctos assocated wt te dde uts, ad te wegts assocated wt te lks betwee te dde ad output layers. Learg algorts desged for dfferet applcatos ay eploy dfferet optzato crtera. Te geeral ateatcal for of te output uts a RBF etwork s as follows: were f f x w r x, s te fucto correspodg to te -t output ut ad s a lear cobato of, radal bass fuctos r, r,, r. Bascally, tere are two categores of learg algorts proposed for RBF etworks [8, 7, 9]. Te frst category of learg algorts sply places oe radal bass fucto at eac saple [8]. O te oter ad, te secod category of learg algorts attepts to reduce te uber of dde uts te etwork, or equvaletly te uber of radal bass fuctos te lear fucto above [, 9, 4, 6, 5]. Oe prary otvato bed te desg of te secod category of algorts s to prove te effcecy of te learg process, as te covetoal approaces eployed to fgure out te optal paraeter settgs for te RBF etwork volve coputg te verse of a atrx wt deso equal to te uber of dde uts te etwork. As etoed earler, oe of te a applcatos of RBF etworks s data classfcato. Most learg algorts proposed for costructg RBF etwork based classfers coduct a clusterg aalyss o te trag data set ad allocate oe dde ut for eac cluster [8, 7, 9, 5]. Algorts dffer by te clusterg algort eployed ad ow te paraeters of te RBF etwork are set. Te cluster-based approaces effectvely prove te effcecy of te learg algort ad reduce te coplexty of te RBF etwork costructed. However, because te 4

7 cluster-based approaces typcally place oe radal bass fucto at te ceter of eac cluster, te dstrbutos of trag saples ear te boudares betwee dfferet classes of obects ay ot be accurately odeled. As te eretal results preseted secto 6 of ts paper reveals, wt respect to data classfcato, te dstrbutos of saples ear te boudares betwee dfferet classes of obects carry ore crucal forato ta te dstrbutos of saples te er parts of te clusters. As a result, te RBF etwork costructed wt a covetoal cluster-based learg algort geerally s ot able to delver te sae level of accuracy as tose data classfcato algorts suc as SVM ad te SGF etowrk tat lot te dstrbutos of saples ear te boudares betwee dfferet classes of obects. I ts paper, a ovel learg algort for costructg SGF etworks s preseted. Te ateatcal treatet preseted ts paper for te dervato of te learg algort s dfferet fro our prevous work []. Neverteless, bot treatets lot essetally te sae dea ad result te sae equatos. 3. Overvew of Data Classfcato wt te Proposed Learg Algort Ts secto presets a overvew of ow data classfcato s coducted wt te SGF etworks costructed wt te proposed learg algort. Te detals of te learg algorts wll be elaborated te ext secto. Assue tat te obects of cocer are dstrbuted a -desoal vector space ad let f deote te probablty desty fucto tat correspods to te dstrbuto of class- obects te -desoal vector space. Te proposed learg algort costructs oe SGF sub-etwork for approxatg te probablty desty fucto of oe class of obects te trag data set. I te costructo of te SGF etwork, te learg algort places oe spercal Gaussa fucto at eac trag saple. Te geeral for of te SGF etwork based fucto approxators s as follows: 5

8 fˆ v s v w s S, were fˆ s te SGF etwork based fucto approxator for class- trag saples; v s a vector te -desoal vector space; S s te set of class- trag saples; v v s s te dstace betwee vectors v ad s ; v w ad are paraeters to be set by te learg algort. Wt te SGF etwork based fucto approxators, a ew obect located at v wt ukow class s predcted to belog to te class tat gves te axu value of te lkelood fucto defed te followg: S L ˆ v f v, S were S s te set of class- trag saples ad S s te set of trag saples of all classes. Te essetal ssue of te learg algort s to costruct te SGF etwork based fucto approxators. I te ext secto, te ovel kerel desty estato algort desged for effcet costructo of te SGF etwork wll be preseted. For te te beg, let us address ow to estate te value of te probablty desty fucto at a trag saple. Assue tat te saplg desty s suffcetly g. Te, by te law of large ubers statstcs [30], we ca estate te value of te probablty desty fucto f at a class- saple s as follows: f k R s π s S Γ were Rs s te axu dstace betwee s ad ts k earest trag saples of te sae class; 6

9 R π s Γ space; s te volue of a yperspere wt radus Rs a -desoal vector Γ s te Gaa fucto []; v k s a paraeter to be set eter troug cross valdato or by te user. I equato, Rs s detered by oe sgle trag saple ad terefore could be urelable, f te data set s osy. I our pleetato, we use R s defed te followg to replace Rs equato, k R s ˆ s s, k were sˆ, sˆ,..., sˆ k are te k earest trag saples of te sae class as s. Te bass of eployg R s s elaborated Appedx A. 4. Te Proposed Kerel Desty Estato Algort Ts secto elaborates te effcet kerel desty estato algort for costructo of te SGF etwork based fucto approxators. I fact, te proposed kerel desty estato algort s derved wt soe deal assuptos. Terefore, soe sort of adaptato ust be eployed, f te target data set does ot cofor to tese assuptos. I ts secto, we wll frst focus o te dervato of te kerel desty estato algort, provded tat tese deal assuptos are vald. Te adaptato eployed ts paper wll be addressed later. Assue tat we ow wat to derve a approxate probablty desty fucto for te set of class- trag saples. Te proposed kerel desty estato algort places oe spercal Gaussa fucto at eac saple as sow equato. Te callege ow s ow to fgure out te optal w ad values of eac Gaussa fucto. For a trag saple s, te learg algort frst coducts a ateatcal aalyss o a syteszed data set. Te syteszed data 7

10 set s derved fro two deal assuptos ad serves as a aalogy of te dstrbuto of class- trag saples te proxty of s. Te frst assupto s tat te saplg desty te proxty of s s suffcetly g ad, as a result, te varato of te probablty desty fucto at s ad te egborg class- saples approaces 0. Te secod assupto s tat s ad te egborg class- saples are evely spaced by a dstace detered by te value of te probablty desty fucto at s. Fg. sows a exaple of te syteszed data set for a trag saple a -desoal vector space. Te detals of te odel are elaborated te followg. Saple s s located at te org ad te egborg class- saples are located at,,,, were,,, are tegers ad s te average dstace betwee two adacet class- trag saples te proxty of s. How s detered wll be addressed later o. Te values of te probablty desty fucto at all te saples te syteszed data set, cludg s, are all equal to f s. Te value of f s s estated based o equato secto 3. s Fg.. A exaple of te syteszed data set for a trag saple a -desoal vector space. Te proposed kerel desty estato algort begs wt a aalyss o te syteszed data set to fgure out te values of w ad tat ake fucto g defed te followg 8

11 vrtually a costat fucto equal to f s, g x x,,..., L f s. 3 w I oter words, te obectve s to ake g x a good approxator of f x te proxty of s. Let Te, we ave y q y, were y s a w ]} {Mu[ q y]} g x w {Maxu[ q y, y R y R real uber. 4 sce L x,,..., x x x, were x x, x,, x. It s sow Appedx B tat, f, te qy s boud by ± Terefore, wt, g x defed equato 3 s vrtually a costat fucto. I fact, we ca apply bascally te sae procedure preseted Appedx B to fd te upper bouds ad lower bouds of qy wt alteratve te bouds of qy becoes tgter, f ratos. As Table reveals, β s set to a larger value. However, te tgtess of te bouds of qy s ot te oly cocer wt respect to coosg te approprate β value. We wll dscuss aoter effect to cosder later. β Bouds of qy ±

12 ± ±.94 0 Tab. : Te bouds of fucto qy defed equato 4 wt alteratve As t as bee sow tat, wt a approprate ratos. rato, g x defed equato 3 s vrtually a costat fucto, te ext tg to do s to fgure out te approprate value of w tat akes equato 3 satsfed. We ave were g s g 0,..., 0 w L w β L, β. Terefore, we eed to set w as follows, order to ake g x a good approxator of f x te proxty of s : w β f s. If we eploy equato to estate te value of f s, te we ave k Γ w, were λ λ S R s π β So far, we ave fgured out tat f we eploy a approprate rato of. 7 β ad set w accordg to equato 7, we ca ake g x a good approxator of f x te proxty of s. Te oly reag ssue s to derve a closed for of. I ts paper, s set to te average dstace betwee two adacet class- trag saples te proxty of saple s. I a -desoal vector space, te uber of uforly dstrbuted saples, N, a ypercube wt volue V ca be coputed by N V, were α s te spacg betwee two adacet saples. α 0

13 Accordgly, we set k R Γ π s, 8 were ˆ k k R s s s as defed secto 3. Fally, wt equatos 7 ad 8 corporated to equato, we ave te followg approxate probablty desty fucto for class- trag saples: Γ s s s v s s v v ˆ S R S k w f π λ S S s s v λ β, were 9 v s a vector te -desoal vector space, S s te set of class- trag saples, k R Γ π β β s, v β λ. I our study, we ave observed tat, regardless of te value of β, we ave β π β λ. If ts observato ca be proved to be geerally correct, te we ca furter splfy equato 9 ad obta S S f s s v v ˆ π. 0

14 5. Ipleetato Issues ad Aalyss of Te Coplexty Ts secto dscusses te ssues cocerg pleetato of te ovel kerel desty estato algort proposed te prevous secto ad presets a aalyss of te coplexty. Fg. suarzes te dscusso so far by sowg te detaled steps take to create a SGF etwork based data classfer ad ow te SGF etwork works. I procedure ake_classfer preseted Fg., t s assued tat te optal values of te tree paraeters lsted Table ave bee detered troug cross valdato. I te later part of ts secto, we wll exae te cross valdato ssue. k Te paraeter equato. Te uber of earest trag saples cluded evaluatg te values k of te approxate probablty desty fuctos at a put vector accordg to equato 9 or 0. ˆ Te paraeter tat substtutes for equatos ad 7-0. Table. Te paraeters to be set troug cross valdato for te SGF etwork. Wt respect to te pseudo-codes preseted Fg., tere are several practcal ssues to address. Te frst ssue cocers te two deal assuptos o wc te dervato of equatos 9 ad 0 s based,.e. te assuptos fro wc Fg. s derved. If te target data set does ot cofor to tese assuptos, te soe sort of adaptato ust be eployed. Te practce eployed ts paper s to corporate paraeter ˆ Table. I equatos ad 7-0, paraeter s supposed to be set to te uber of attrbutes of te obects te data set. However, because te local dstrbutos of te trag saples ay ot spread all desos ad soe attrbutes ay eve be correlated, we replace tese equatos by ˆ, wc s to be set troug cross valdato. I fact, te process coducted to fgure out te optal value of ˆ also serves to tue w ad, as we also replace equatos 7 ad 8 by ˆ.

15 Procedure ake_classfer Iput: a set of trag saples S {s, s,, s }; paraeter values of k, k, ad ˆ lsted Table ; paraeter value of β. Output: a SGF etwork. Beg for eac class of trag saples { let S be te set of class- trag saples ad costruct a kd-tree for S ; for eac s S { let sˆ, sˆ,..., sˆ k be te k earest trag saples of te sae class as s ; copute k ˆ R s ˆ s s ; ˆ k copute β ˆ k R s π ˆ Γ ; } copute te approxate value of λ ; β costruct a SGF sub-etwork wt te followg output fucto: ˆ β v s v S s S λ fˆ ; ed } Fg.. Te pseudo codes of te proposed learg algort ad te SGF etwork based classfer. to be cotued Procedure predct Iput: a SGF etwork costructed wt te procedure preseted Procedure ake_classfer; a put obect wt coordate v; Output: a predcto of te class of te put obect; Beg let sˆ, sˆ,..., sˆ k be te k earest saples of v te trag data set; 3

16 ax 0; for eac SGF sub-etwork correspodg to oe class of trag saples { let T be te subset of { sˆ, sˆ,..., sˆ k } tat cossts of class- trag saples; S copute te approxate value of L ˆ v f v wt S fˆ ˆ β v s v S s T λ f L v > ax te class ; } retur class; ; ed Fg.. Te pseudo codes of te proposed learg algort ad te SGF etwork based classfer. cotues Aoter paraeter Table ad Fg. tat eeds to address s k. Sce te fluece of a Gaussa fucto decreases oetally as te dstace creases, we coputg te values of te approxate probablty desty fuctos at a gve vector v accordg to equatos 9 or 0, we oly eed to clude a lted uber of earest trag saples of v. Te uber of earest trag saples to be cluded ca be detered troug cross valdato ad s deoted by k. Tere s oe ore practcal ssue to address. I earler dscusso, we etoed tat tere s aoter aspect to cosder selectg te β rato, addto to te tgtess of te bouds of fucto qy defed equato 4. If we exae equatos 9 ad 0, we wll fd tat te value of te approxato fucto at a saple s,.e. f ˆ, s actually a wegted average of te estated saple destes at s ad at ts earby saples of te sae class. Terefore, a sootg effect wll result. A larger s β rato ples tat te sootg effect wll be ore sgfcat. Terefore, t s of terest to vestgate te effect of β. Our erece 4

17 suggests tat, as log as β s set to a value wt [0.6, ], te value of β as o sgfcat effect o classfcato accuracy. Terefore, β s ot cluded Table. As far as te te coplextes of te algorts preseted Fg. are cocered, tere are two separate ssues. Te frst ssue cocers te te take to create a SGF etwork wt trag saples ad te secod ssue cocers te te take to classfy ' obects wt te SGF etwork. I bot ssues, we eed to detfy te earest egbors of a saple. I our pleetato, te kd-tree structure s eployed [6], wc s a data structure wdely used to searc for te earest egbors. Wt ts practce, te average te coplexty for costructg a kd-tree wt trag saples s O log. I procedure ake_classfer preseted Fg., we eed to costruct c kd-trees, f te trag data set cotas c classes of saples. Terefore, te average te coplexty of ts task s bouded by Oc log. Te, we eed to detfy te k earest egbors for eac of te trag saples ad te average te coplexty of ts task s bouded by Ok log. As te two tasks addressed above doate te te coplexty of procedure ake_classfer, te overall te coplexty for te procedure s Oc log k log, or O log, f bot c ad k are regarded as costats. I procedure predct preseted Fg., te te coplexty for classfyg a cog obect s doated by te work to detfy k earest trag saples of te cog obect. Terefore, te average te coplexty for classfyg oe obect s bouded by Ok log ad te overall te coplexty for classfyg ' cog obects s bouded by Ok ' log or O' log, f k s treated as a costat. I te dscusso above, t s assued tat te optal values for te paraeters lsted Table ave bee detered troug cross valdato, before procedure ake_classfer. If a k-fold cross valdato process s coducted [35], te for eac possble cobato of paraeter values we eed to costruct oe SGF etwork based o a subset of trag saples. Te, we eed to voke procedure predct to fgure out ow good ts partcular cobato of paraeter values s. Based o te aalyss of te coplexty preseted above, t s apparet tat te average te 5

18 coplexty of te cross valdato process s bouded by O log, f te uber of possble cobatos of paraeter values s regarded as a costat. 6. Experetal Results ad Dscussos Te erets reported ts secto ave bee coducted to evaluate te perforace of te SGF etworks costructed wt te learg algort proposed ts paper, coparso wt te alteratve data classfcato algorts. Te erets focus o te followg 3 ssues: classfcato accuracy, executo effcecy, ad te effect of data reducto. Te alteratve data classfcato algorts volved te coparso clude SVM, KNN [35], ad te covetoal cluster-based learg algort proposed [9] for RBF etworks. Te learg algort proposed [9] coducts clusterg aalyss o te trag data set ad allocates oe dde ut for eac cluster of trag saples. For splcty, te followg dscusso, we wll use te covetoal RBF etwork to refer to te data classfer costructed wt te learg algort proposed [9] ad te SGF etwork to refer to te data classfer costructed wt te learg algort proposed ts paper. I tese erets, te SVM software used s LIBSVM [0] wt te radal bass kerel ad te oe-agast-oe practce as bee adopted for te SVM, f te data set cotas ore ta classes of obects. Data set # of trag saples # of testg saples satage letter suttle a. Te tree larger data sets. Data set # of saples rs 50 we 78 6

19 vowel 58 seget 30 glass 4 vecle 846 b. Te sx saller data sets. Table 3. Te becark data sets used te erets. Table 3 lsts a caracterstcs of te 9 becark data sets used te erets. All tese data sets are fro te UCI repostory [9]. Te collecto of becark data sets s te sae as tat used [8], except tat DNA s ot cluded. DNA s ot cluded, because t cotas categorcal data ad a exteso of te proposed learg algort s yet to be developed for adlg categorcal data sets. Aog te 9 data sets, tree of te are cosdered as te larger oes, as eac cotas ore ta 5000 saples wt separate trag ad testg subsets. Te reag sx data sets are cosdered as te saller oes ad tere are o separate trag ad testg subsets tese 6 saller data sets. Accordgly, dfferet evaluato practces ave bee eployed for te saller data sets ad for te larger data sets. For te 3 larger data sets, 0-fold cross valdato as bee coducted o te trag set to detere te optal paraeter values to be used te testg pase. O te oter ad, for te 6 saller data sets, te evaluato practce eployed [8] as bee adopted. Wt ts practce, 0-fold cross valdato as bee coducted o te etre data set ad te best result s reported. Terefore, te results reported wt ts practce ust reveal te axu accuracy tat ca be aceved, provded tat a perfect cross valdato ecas s avalable to detfy te optal paraeter values. I tese erets, β equato 9 as bee set to 0.7. Our observato ts regard s tat, as log as β s set to a value wt [0.6, ], te te value of β as o sgfcat effect o classfcato accuracy. O te oter ad, paraeters α ad β te covetoal RBF etwork 7

20 proposed [9] ave bee set to te eurstc values suggested by te autors,.e..05 ad 5, respectvely. Data sets Data classfcato algorts SGF etwork KNN wt KNN wt Covetoal SVM k k 3 RBF etwork. satage 9.30 k 6, k 6, ˆ letter 97. k 8, k 8, ˆ suttle k 8, k, ˆ Avg Table 4. Coparso of classfcato accuracy wt te 3 larger data sets. Table 4 copares te accuracy delvered by alteratve classfcato algorts wt te 3 larger becark data sets. As Table 4 sows, te SGF etwork ad te SVM bascally delver te sae level of accuracy, wc te KNN ad te covetoal RBF etwork are geerally ot able to atc. Table 5 lsts te eretal results wt te 6 saller data sets. Table 5 sows tat te SGF etwork ad te SVM bascally delver te sae level of accuracy for 4 out of tese 6 data sets. Te two exceptos are glass ad vecle. Te results wt tese two data sets suggest tat bot te SGF etwork ad te SVM ave soe bld spots, ad terefore ay ot be able to perfor as well as te oter soe cases. Te eretal results preseted Table 5 also sow tat te SGF etwork ad te SVM geerally delver a ger level of accuracy ta te KNN ad te covetoal RBF etwork. Data sets Data classfcato algorts 8

21 SGF etwork SVM KNN wt k KNN wt k 3 Covetoal RBF etwork. rs k 4, k 4, ˆ we k 3, k 6, ˆ vowel 99.6 k 5, k, ˆ seget 97.7 k 5, k, ˆ Avg glass k 9, k 3, ˆ vecle k 3, k 8, ˆ Avg Table 5. Coparso of classfcato accuracy wt te 6 saller data sets. I te erets tat ave bee reported so far, o data reducto s perfored te costructo of te SGF etwork. As te learg algort proposed ts paper places oe spercal Gaussa fucto at eac trag saple, reoval of redudat trag saples eas tat te SGF etwork costructed wll cota fewer dde uts ad wll operate ore effcetly. Table 6 presets te effect of applyg a aïve data reducto algort to te 3 larger data sets. Te aïve data reducto algort exaes te trag saples oe by oe a arbtrary order. If te trag saple beg exaed ad all of ts 0 earest egbors te reag trag data set belog to te sae class, te te trag saple beg exaed s cosdered as redudat ad wll be deleted. Wt ts practce, trag saples located ear te boudares betwee dfferet classes of obects wll be retaed, wle trag saples located far away fro te boudares wll be deleted. As sow Table 6, te aïve data reducto algort s able to 9

22 reduce te uber of trag saples te suttle data set substatally, wt less ta % of trag saples left. O te oter ad, te reducto rates for satage ad letter are ot as substatal. It s apparet tat te reducto rate s detered by te caracterstcs of te data set. Table 6 also reveals tat applyg te aïve data reducto ecas wll lead to slgtly lower classfcato accuracy. Sce te data reducto ecas eployed ts paper s a aïve oe, tere s roo for proveet wt respect to bot reducto rate ad pact o classfcato accuracy. Ts s a subect uder vestgato. satage letter suttle # of trag saples te orgal data set # of trag saples after data reducto s appled % of trag saples reag 40.9% 5.96%.44% Classfcato accuracy wt te SGF etwork after data reducto s appled 9.5% 96.8% 99.3% Degradato of accuracy due to data reducto 0.5% 0.94% 0.6% Table 6. Effects of applyg a aïve data reducto ecas. Table 7 copares te uber of trag saples reag after data reducto s appled, te ubers of clusters detfed by te covetoal RBF etwork algort, ad te uber of support vectors detfed by te SVM te becark data sets. Tere are several terestg observatos. Frst, for satage ad letter, te uber of trag saples reag after data reducto s appled ad te uber of support vectors detfed by te SVM are alost equal. For suttle, toug te dfferece s larger, te two ubers are stll te sae order. O te oter ad, te uber of clusters detfed by te covetoal RBF etwork algort s cosstetly uc saller ta te uber of trag saples reag after data reducto s appled ad te uber of support vectors detfed by te SVM. Our terpretato of tese observatos s tat bot te SVM ad te aïve date reducto ecas eployed ere attept 0

23 to detfy te trag saples tat are located ear te boudares betwee dfferet classes of obects. Terefore, te ubers wt tese two algorts preseted Table 7 are alost equal or at least te sae order. O te oter ad, sce ultple saples are eeded to precsely descrbe te boudary of a cluster, te uber of trag saples reag after data reducto s appled ad te uber of support vectors detfed by te SVM are geeral uc larger ta te uber of clusters detfed by te covetoal RBF etwork algort. Te results reported Table 7 alog wt te results preseted Table 4 ad Table 5 also suggest tat, wt respect to data classfcato, te dstrbutos of saples ear te boudares betwee dfferet classes of obects carry ore crucal forato ta te dstrbutos of saples te er part of te clusters. Sce te covetoal RBF etwork corporates oe radal bass fucto located at te geoetrc ceter of a cluster to odel te dstrbuto of te trag saples sde te cluster, te accuracy delvered by te covetoal RBF etwork s geerally lower ta tat delvered by te SGF etwork ad te SVM. # of clusters detfed by te # of trag saples after # of support vectors covetoal RBF etwork data reducto s appled detfed by LIBSVM algort satage letter suttle Table 7. Coparso of te uber of trag saples reag after data reducto s appled, te uber of support vectors detfed by te SVM software, ad te uber of clusters detfed by te covetoal RBF etwork algort. Table 8 copares te executo tes of te SGF etwork, te SVM, ad te covetoal RBF etwork wt te 3 larger data sets preseted Table 3. I Table 8, te total tes take to costruct classfers based o te gve trag data sets are lsted te rows arked by

24 Make_classfer. O te oter ad, te tes take by alteratve classfers to predct te classes of te testg saples are lsted te rows arked by Predcto. SGF etwork wtout data SGF etwork wt data Covetoal SVM reducto reducto RBF etwork satage Make letter classfer suttle satage Predcto letter suttle Table 8. Coparso of executo tes secods. As Table 8 reveals, te te take to costruct a SVM classfer wt te odel selecto process eployed [0] s substatally ger ta te te take to costruct a SGF etwork or a covetoal RBF etwork. A detaled aalyss reveals tat t s te odel selecto process tat doates te te take to costruct a SVM classfer. Tese results ply tat te te take to costruct a SVM classfer wt optzed paraeter settg could be uacceptably log for soe coteporary applcatos, partcular, for tose applcatos wc ew obects are cotuously added to a already large database. Te results Table 8 also ply tat, dealg wt tose data sets suc as satage ad letter tat does ot cota a g percetage of redudat trag saples, te SGF etwork s favorable over te SVM. I suc cases, te SGF etwork eoys substatally ger effcecy ta te SVM te ake_classfer pase ad s able to delver te sae level perforace as te SVM ters of bot classfcato accuracy ad te executo te te predcto pase. O te oter ad, f te data set cotas a g percetage of redudat trag saples suc as suttle, te data reducto ust be appled for te SGF etwork or ts effcecy te

25 Predcto pase would suffer. Wt data reducto eployed, te executo te of te SGF etwork te Predcto pase te s coparable wt tat of te SVM. As te corporato of te aïve data reducto ecas ay lead to slgtly lower classfcato accuracy, t s of terest to develop advaced data reducto ecass. Table 8 also sows tat te covetoal RBF etwork geerally eoys ger effcecy coparso wt te SGF etwork wt data reducto ad te SVM te predcto pase. Ts peoeo s due to te fact sow Table 7 tat te uber of clusters detfed by te covetoal RBF etwork algort for a data set s geerally saller ta te uber of trag saples eployed to costruct te SGF etwork after data reducto ad te uber of support vectors detfed by te SVM algort. Neverteless, as etoed earler, because te covetoal cluster-based learg algort for RBF etworks places oe radal bass fucto at te ceter of eac cluster, te dstrbutos of te obects te data set ay ot be accurately odeled. As a result, te covetoal RBF etwork geeral s ot able to delver te sae level of accuracy as te SVM ad te SGF etwork, wc lot te dstrbutos of trag saples ear te boudares betwee dfferet classes of obects. 7. Cocluso I ts paper, a ovel learg algort for costructg SGF etwork based data classfers s proposed. Wt respect to algort desg, te a dstcto of te proposed learg algort s te ovel kerel desty estato algort desged for effcet costructo of te SGF etworks. Te erets preseted ts paper reveal tat te SGF etworks costructed wt te proposed learg algort geerally aceve te sae level of classfcato accuracy as SVM. Oe portat advatage of te proposed learg algort, coparso wt te SVM, s tat te te take to costruct a SGF etwork wt optzed paraeter settg s orally uc less tat te take to costruct a SVM classfer. Aoter desrable feature of te SGF etwork s tat t ca carry out data classfcato wt ore ta two classes of obects oe 3

26 sgle ru. I oter words, t does ot eed to voke ecass suc as oe-agast-oe or oe-agast-all for adlg datasets wt ore ta two classes of obects. Te oter a propertes of te proposed learg algort are as follow: te average te coplexty for costructg a SGF etwork s bouded by O log, were s total uber of trag saples; te average te coplexty for classfyg ' cog obects s bouded by O' log. As te SGF etworks costructed wt te proposed learg algort are stace-based, ts paper also addresses te effcecy ssue sared by alost all stace-based learg algorts. Experetal results reveal tat te aïve data reducto ecas eployed ts paper s able to reduce te sze of te trag data set substatally wt a slgt pact o classfcato accuracy. Oe terestg observato ts regard s tat, for all tree data sets used data reducto erets, te uber of trag saples reag after data reducto s appled s qute close to te uber of support vectors detfed by te SVM software. I suary, te SGF etwork costructed wt te proposed learg algort s favorable over te SVM dealg wt a data set tat does ot cota a g percetage of redudat trag saples. I suc case, te SGF etwork s able to delver te sae level of perforace as te SVM ters of bot accuracy ad te te take te predcto pase, wle requrg substatally less te to costruct a classfer. O te oter ad, f te data set cotas a g percetage of redudat trag saples, te data reducto ust be appled, or te executo te of te SGF etwork would suffer. As te corporato of te aïve data reducto ecas ay lead to slgtly lower classfcato accuracy, t s of terest to develop advaced data reducto ecass. Ts paper also copares te perforace of te SGF etworks costructed wt te proposed learg algort ad te RBF etworks costructed wt a covetoal cluster-based learg algort. Te ost terestg observato leared s tat, wt respect to data classfcato, te dstrbutos of trag saples ear te boudares 4

27 betwee dfferet classes of obects carry ore crucal forato ta te dstrbutos of saples sde te clusters. As a result, te covetoal RBF etwork geerally s ot able to delver te sae level of accuracy as tose learg algorts suc as SVM ad te SGF etwork tat lot te dstrbutos of trag saples ear te boudares betwee dfferet classes of obects. Based o te study preseted ts paper, tere are several ssues tat deserve furter studes, addto to te developet of advaced data reducto ecass etoed above. Oe ssue s te exteso of te proposed learg algort for adlg categorcal data sets. Aoter ssue cocers wy te SGF etwork fals to delver coparable accuracy te vecle test case, wat te bld spot s, ad ow proveets ca be ade. Fally, t s of terest to develop creetal verso of te proposed learg algort to cope wt te ever-growg coteporary databases. Appedx A Assue tat sˆ, sˆ,..., sˆ k are te k earest trag saples of s tat belogs to te sae class as s. If k s suffcetly large ad te dstrbuto of tese k saples te vector space s ufor te we ave k ρr s Γ π, were ρ s te local desty of saples sˆ, sˆ,..., sˆ k te proxty of s. Furterore, we ave k sˆ s R s r π ρ Γ 0 rdr ρr s Γ π, were r Γ π s te surface area of a yperspere wt radus r a -desoal vector space. Terefore, we ave 5

28 6 ˆ k k R s s s. Te rgt-ad sde of te equato above s te eployed ts paper to estate Rs. Appedx B Let y y q, were R ad R are two coeffcets ad y R. We ave y y dy y dq y q '. Sce qy s a syetrc ad perodcal fucto, f we wat to fd te global axu ad u values of qy, we oly eed to aalyze qy wt terval ] 0, [. Let y 0 [0, ad ε y 0, were ad 0 are tegers, ad 0 ε <. We ave ε dt t q q y q ' 0. Let us cosder te specal case wt. Te, we ave dt t t y q ε 0. Let ε dt t t r. Sce t t s a decreasg fucto for t [, ] ad s a creasg fucto for t [, ], we ave 0 r ε ε ; ε r ε ;

29 7 for 0 ad, ε r ε. Terefore, r y q 0 εθ, were θ 0,. If θ 0, te we ave for ay 0 ε <. θ εθ A. O te oter ad, f θ < 0, te we ave for ay 0 ε <. εθ A. Cobg equatos A. ad A., we obta, for all y [0,, θ y q, l Maxu 0. Slarly, we ca sow tat ρ y q, l Mu 0, were ρ. 0,

30 If we set 00,000, te we ave, wt, q y , for y [0,. Refereces [] E. Art, Te Gaa Fucto, New York, Holt, Reart, ad Wsto, 964. [] R. K. Beatso, J. B. Cerre, ad C. T. Mouat, "Fast evaluato of radal bass fuctos: etods for four-desoal polyaoc sples," SIAM Joural o Scetfc Coputg, vol. 3. o. 6, pp. 7-30, 00. [3] R. K. Beatso ad W. A. Lgt, "Fast evaluato of radal bass fuctos: etod for two-desoal polyaroc sples," IMA Joural of Nuercal Aalyss, vol. 7, pp , 997. [4] R. K. Beatso, W. A. Lgt, ad S. Bllgs, "Fast soluto of te radal bass fucto terpolato equatos: doa decoposto etods," SIAM Joural o Scetfc Coputg, vol., o. 5, pp , 000. [5] F. Bellor, A. Face, ad A. Bllat, "A geeral approac to costruct RBF et-based classfer," Proceedgs of te 7 t Europea Syposu o Artfcal Neural Network, pp , 999. [6] J. L. Betley, "Multdesoal bary searc trees used for assocatve searcg," Coucato of te ACM, vol. 8, o. 9, pp , 975. [7] M. Bac, P. Frasco, ad M. Gor, "Learg wtout local a radal bass fucto etworks," IEEE Trasacto o Neural Networks, vol. 6, o. 3, pp , 995. [8] C. M. Bsop, "Iprovg te geeralzato propertes of radal bass fucto eural etworks," Neural Coputato, vol. 3, o. 4, pp l, 99. [9] C. L. Blake ad C. J. Merz, "UCI repostory of ace learg databases," Teccal report, Uversty of Calfora, Departet of Iforato ad Coputer Scece, Irve, CA,

31 [0] C. C. Cag ad C. J. L, "LIBSVM: a lbrary for support vector aces," ttp:// edu.tw/~cl/lbsv, 00. [] O. Capelle, V. Vapk, O. Bousquet, ad S. Mukeree, "Coosg ultple paraeters for support vector aces," Mace Learg, vol. 46, pp. 3-59, 00. [] S. Ce, C. F. N. Cowa, ad P. M. Grat, "Ortogoal least squares learg for radal bass fucto etworks," IEEE Trasactos o Neural Networks, vol., o., pp , 99. [3] N. Crsta ad J. Sawe-Taylor, A Itroducto to Support Vector Maces, Cabrdge Uversty Press, Cabrdge, UK, 000. [4] D. Decoste ad K. Wagstaff, "Alpa seedg for support vector aces," Proceedgs of Iteratoal Coferece o Kowledge Dscovery ad Data Mg, 000. [5] K. Dua, S. S. Keert, ad A. N. Poo, "Evaluato of sple perforace easures for tug SVM yperparaeters," Neurocoputg, vol. 5, pp. 4-59, 003. [6] S. Duas, J. Platt, ad D. Heckera, "Iductve learg algorts ad represetatos for text categorzato," Proceedgs of te Iteratoal Coferece o Iforato ad Kowledge Maageet, pp , 998. [7] G. W. Flake, "Square ut augeted, radally exteded, ultlayer perceptros," Neural Networks: Trcks of te Trade, G. B. Orr ad K. Müller, Eds., pp , 998. [8] C. W. Hsu ad C. J. L, "A coparso of etods for ult-class support vector aces," IEEE Trasactos o Neural Networks, vol. 3, pp , 00. [9] Y. S. Hwag ad S. Y. Bag, "A effcet etod to costruct a radal bass fucto eural etwork classfer," Neural Networks, vol. 0, o. 8, pp , 997. [0] T. Joacs, "Text categorzato wt support vector aces: learg wt ay relevat features," Proceedgs of Europea Coferece o Mace Learg, pp. 37-4, 998. [] V. Keca, Learg ad Soft Coputg: Support Vector Maces, Neural Networks, ad Fuzzy Logc Models, Te MIT Press, Cabrdge, Massacusetts, Lodo, Eglad, 00. 9

32 [] S. S. Keert, "Effcet tug of SVM yperparaeters usg radus/arg boud ad teratve algorts," IEEE Trasactos o Neural Networks, vol. 3, pp. 5-9, 00. [3] T. M. Mtcell, Mace Learg, McGraw-Hll, 997. [4] J. Moody ad C. J. Darke, "Fast learg etworks of locally-tued processg uts," Neural Coputato, vol., pp. 8-94, 989. [5] M. Musav, W. Aed, K. Ca, K. Fars, ad D. Huels, "O te trag of radal bass fucto classfers," Neural Networks, vol. 5, pp , 99. [6] M. J. L. Orr, "Regularsato te selecto of radal bass fucto cetres," Neural Coputato, vol. 7, o. 3, pp , 995. [7] M. J. L. Orr, "Itroducto to radal bass fucto etworks," Teccal report, Ceter for Cogtve Scece, Uversty of Edburg, 996. [8] M. J. Orr, "Optsg te wdts of radal bass fucto," Proceedgs of te Fft Brazla Syposu o Neural Networks, 998, pp [9] M. J. Orr, J. Halla, A. Murray, ad T. Leoard, "Assessg rbf etworks usg delve," Iteratoal Joural of Neural Systes, vol. 0, o. 5, pp , 000. [30] A. Papouls, Probablty, Rado Varables, ad Stocastc Processes, McGraw-Hll, 99. [3] T. Poggo ad F. Gros, "Networks for approxato ad learg," Proceedgs of te IEEE, vol. 78, o. 9, pp , 990. [3] M. J. Powell, "Radal bass fuctos for ultvarable terpolato: a revew," Algort for Approxato, J. C. Maso ad M. G. Cox, Eds, Oxford, U. K.: Oxford Uversty Press, 987, pp [33] B. Scolkopf, K. K. Sug, C. Burges, F. Gros, P. Nyog, T. Poggo, ad V. Vapk, "Coparg support vector aces wt Gaussa kerels to radal bass fucto classfers," IEEE Trasactos o Sgal Processg, vol. 45, o., pp. -8, 997. [34] B. W. Slvera, Desty Estato for Statstcs ad Data Aalyss, Capa ad Hall, Lodo,

33 [35] I. H. Wtte ad E. Frak, Data Mg, Sa Fracsco, CA, Morga Kaufa,