-- CS341 info session is on Thu 3/18 7pm in Gates Final exam logistics

Transcription

1 -- CS341 info session is on Thu 3/18 7pm in Gates Final exam logistics CS246: Mining Massive Datasets Jure Leskovec, Stanford University

2 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 2

3 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 3 Alternate final: Fri 3/14 7:00-10:00pm in Cubberley Auditorium Final: Mon 3/17 12:15-3:15pm NVidia (Lastname starting with A-M) GatesB01 (Lastname starting with N-Z) See Practice finals + Gradiance quizzes are on Piazza Open book, open computer, no internet SCPD students can take the exam at Stanford!

4 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 4 Exam protocol for SCPD students: On Friday 3/14 your exam proctor will receive the PDF of the final exam from SCPD If you take the exam at Stanford: Ask the exam monitor to delete the SCP If you don t take the exam at Stanford: Arrange a 3h slot with your exam monitor You can take the exam anytime but return it in time exam PDF to cs246.mmds@gmail.com by Tuesday 3/15 11:59pm Pacific time

5 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 5

6 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 6 Data mining research project on real data Groups of 3 students We provide interesting data, computing resources (Amazon EC2) and mentoring You provide project ideas Class meets once a week + individual group mentoring Information session: Tuesday 3/18 7:00pm in Gates 104 (there will be pizza!)

7 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 7 Tue 3/18: Info session We will introduce datasets, problems, ideas Students form groups and project proposals Mon 3/24: Project proposals are due We evaluate the proposals Mon 3/31: Admission results 10 to 15 groups/projects will be admitted Mon 5/5, Wed 5/7: Midterm presentations Thu 6/10: Presentations, poster session More info:

8 CS246: Mining Massive Datasets Jure Leskovec, Stanford University

9 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 9 Redundancy leads to a bad user experience Uncertainty around information need => don t put all eggs in one basket How do we optimize for diversity directly?

10 Monday, January 14, 2013 France intervenes Chuck for Defense Argo wins big Hagel expects fight 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 10

11 Monday, January 14, 2013 France intervenes Chuck for Defense Argo wins big New gun proposals 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 11

12 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 12 Idea: Encode diversity as coverage problem Example: Word cloud of news for a single day Want to select articles so that most words are covered

13 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 13

14 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 14 Q: What is being covered? A: Concepts (In our case: Named entities) France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL Hagel expects fight Q: Who is doing the covering? A: Documents

15 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 15 Suppose we are given a set of documents V Each document d covers a set XX dd of words/topics/named entities W For each set of documents A we define FF AA = XX dd dd AA Goal: We want to max AA kk FF(AA) Note: F(A) is a set function: FF AA : SSSSSSSS N

16 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 16 Given universe of elements WW = {ww 11,, ww nn } and sets XX 11,, XX mm WW X 3 X 2 X 4 X 1 W Goal: Find k sets X i that cover the most of W More precisely: Find k sets X i whose size of the union is the largest Bad news: A known NP-complete problem

17 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 17 Simple Heuristic: Greedy Algorithm: Start with AA 00 = { } For ii = 11 kk Take set dd that mmmmmm FF(AA ii 11 {dd}) Let AA ii = AA ii 11 {dd} Example: Eval. FF dd 11,, FF({dd mm }), pick best (say dd 11 ) Eval. FF dd 11 } {dd 22,, FF({dd 11 } {dd mm }), pick best (say dd 11 ) Eval. FF({dd 11, dd 22 } {dd 33 }),, FF({dd 11, dd 22 } {dd mm }), pick best And so on FF AA = XX dd dd AA

18 Goal: Maximize the covered area 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 18

19 Goal: Maximize the covered area 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 19

20 Goal: Maximize the covered area 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 20

21 Goal: Maximize the covered area 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 21

22 Goal: Maximize the covered area 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 22

23 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 23 A B C Goal: Maximize the size of the covered area Greedy first picks A and then C But the optimal way would be to pick B and C

24 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 24 Greedy produces a solution A where: F(A) (1-1/e)*OPT (F(A)>0.63*OPT) [Nemhauser, Fisher, Wolsey 78] Claim holds for functions F( ) with 2 properties: F is monotone: (adding more docs doesn t decrease coverage) if A B then F(A) F(B) and F({})=0 F is submodular: adding an element to a set gives less improvement than adding it to one of its subsets

25 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 25 Definition: Set function F( ) is called submodular if: For all A,B W: F(A) + F(B) F(A B) + F(A B) + B A A B A B +

26 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 26 Diminishing returns characterization Equivalent definition: Set function F( ) is called submodular if: For all A B, s B: F(A d) F(A) F(B d) F(B) Gain of adding d to a small set Gain of adding d to a large set B A + + d d Large improvement Small improvement

27 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 27 F( ) is submodular: A B F(A d) F(A) F(B d) F(B) Gain of adding X d to a small set Natural example: Sets XX 1,, XX mm FF AA = dd AA XX dd (size of the covered area) Claim: FF(AA) is submodular! A Gain of adding X d to a large set B X d X d

28 Submodularity is discrete analogue of concavity F( ) F(B) F(B d) A B F(A d) F(A) Adding d to B helps less than adding it to A! Solution size A F(A d) F(A) F(B d) F(B) Gain of adding X d to a small set Gain of adding X d to a large set 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 28

29 Marginal gain: ΔΔ FF dd AA = FF AA XX dd FF(AA) Submodular: FF AA dd FF AA FF BB dd FF(BB) Concavity: ff aa + dd ff aa ff bb + dd ff(bb) AA BB aa bb F(A) 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 29 A

30 3/11/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 30 Let FF 11 FF mm be submodular and λλ 11 λλ mm > 00 mm then FF AA = ii λλ ii FF ii AA is submodular Submodularity is closed under non-negative linear combinations! This is an extremely useful fact: Average of submodular functions is submodular: FF AA = PP ii FF ii AA ii Multicriterion optimization: FF AA = λλ ii FF ii AA ii

31 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 31 Q: What is being covered? A: Concepts (In our case: Named entities) France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL Hagel expects fight Q: Who is doing the covering? A: Documents

32 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 32 Objective: pick k docs that cover most concepts France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL Enthusiasm for Inauguration wanes Inauguration weekend F(A): the number of concepts covered by A Elements concepts, Sets concepts in docs F(A) is submodular and monotone! We can use greedy to optimize F

33 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 33 Objective: pick k docs that cover most concepts France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL Enthusiasm for Inauguration wanes Inauguration weekend The good: Penalizes redundancy Submodular The bad: Concept importance? All-or-nothing too harsh

34 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 34

35 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 35 Objective: pick k docs that cover most concepts France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL Enthusiasm for Inauguration wanes Inauguration weekend Each concept cc has importance weight ww cc

36 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 36 Document coverage function probability document d covers concept c [e.g., how strongly d covers c] Obama Romney Enthusiasm for Inauguration wanes

37 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 37 Document coverage function: probability document d covers concept c Cover d (c) can model how relevant is concept c for user u Set coverage function: Prob. that at least one document in A covers c Objective: concept weights

38 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 38 The objective function is also submodular Intuitive diminishing returns property Greedy algorithm leads to a (1 1/e) ~ 63% approximation, i.e., a near-optimal solution

39 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 39 Objective: pick k docs that cover most concepts France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL Enthusiasm for Inauguration wanes Inauguration weekend Each concept cc has importance weight ww cc Documents partially cover concepts: ccccccccrr dd (cc)

40 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 40

41 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 41 a b c Greedy Marginal gain: F(A x)-f(a) Greedy algorithm is slow! At each iteration we need to re-evaluate marginal gains of all remaning documents Runtime OO( VV KK) for selecting KK documents d e Add document with highest marginal gain

42 [Leskovec et al., KDD 07] In round ii: So far we have AA ii 11 = {dd 11,, dd ii 11 } Now we pick dd ii = aaaaaa mmmmmm dd VV FF(AA ii 11 {dd}) FF(AA ii 11 ) Greedy algorithm maximizes the marginal benefit ΔΔ ii dd = FF(AA ii 11 {dd}) FF(AA ii 11 ) By submodularity property: FF AA ii dd FF AA ii FF AA jj dd FF AA jj for ii < jj Observation: By submodularity: For every dd VV ΔΔ ii (dd) ΔΔ jj (dd) for ii < jj since AA ii AA jj Marginal benefits ΔΔ ii (dd) only shrink! (as i grows) i (d) j (d) Selecting document d in step i covers more words than selecting d at step j (j>i) 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 42 d

43 [Leskovec et al., KDD 07] 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 43 Idea: Use i as upper-bound on j (j > i) Lazy Greedy: Keep an ordered list of marginal benefits i from previous iteration Re-evaluate i only for top node Re-sort and prune Marginal gain a b c d e A 1 ={a} F(A {d}) F(A) F(B {d}) F(B) A B

44 [Leskovec et al., KDD 07] 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 44 Idea: Use i as upper-bound on j (j > i) Lazy Greedy: Keep an ordered list of marginal benefits i from previous iteration Re-evaluate i only for top node Re-sort and prune Marginal gain a b c d e A 1 ={a} F(A {d}) F(A) F(B {d}) F(B) A B

45 [Leskovec et al., KDD 07] 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 45 Idea: Use i as upper-bound on j (j > i) Lazy Greedy: Keep an ordered list of marginal benefits i from previous iteration Re-evaluate i only for top node Re-sort and prune Marginal gain a d b e c A 1 ={a} A 2 ={a,b} F(A {d}) F(A) F(B {d}) F(B) A B

46 Summary so far: Diversity can be formulated as a set cover Set cover is submodular optimization problem Can be (approximately) solved using greedy algorithm Lazy-greedy gives significant speedup 400 Lower is better running time (seconds) exhaustive search (all subsets) naive greedy Lazy number of blogs selected 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 46

47 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 47 But what about personalization? Election trouble model Songs of Syria Sandy delays Recommendations

48 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 48 We assumed same concept weighting for all users France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL France intervenes Chuck for Defense Argo wins big

49 Each user has different preferences over concepts France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL politico France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL movie buff 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 49

50 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 50 Assume each user has different preference vector over concepts Goal: Learn personal concept weights from user feedback

51 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 51 France Mali Hagel Pentagon Obama Romney Zero Dark Thirty Argo NFL France intervenes Chuck for Defense Argo wins big

52 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 52 Multiplicative Weights algorithm Assume each concept cc has weight ww cc We recommend document dd and receive feedback, say rr = +1 or -1 Update the weights: If cc XX dd then ww cc = ββ rr ww cc If cc XX dd then ww cc = ββ rr ww cc If concept c appears in X d and we received positive feedback r=+1 then we increase the weight w c by multiplying it by ββ (ββ > 11) otherwise we decrease the weight (divide by ββ) Normalize weights so that cc ww cc = 11

53 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 53 Steps of the algorithm: 1. Identify items to recommend from 2. Identify concepts [what makes items redundant?] 3. Weigh concepts by general importance 4. Define item-concept coverage function 5. Select items using probabilistic set cover 6. Obtain feedback, update weights

54 3/10/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, 68