Building Reliable Activity Models Using Hierarchical Shrinkage and Mined Ontology

Transcription

1 Building Reliable Activity Models Using Hierarchical Shrinkage and Mined Ontology Emmanuel Munguia Tapia 1, Tanzeem Choudhury and Matthai Philipose 2 1 Massachusetts Institute of Technology 2 Intel Research Seattle

2 Activity Recognition applications Activity-aware actuation Proactive reminding Ubiquitous healthcare Embedded health assessment

3 Activities of Daily Living (ADLs) Activity Class Personal Appearance Housework Toileting Washing up Appliance Use Taking care or an infant Care of clothes and linen Making a snack Making a drink Oral hygiene 23 classes, 1000s of activities Classes of day-to-day activities that: Indicate cognitive well-being Indicate level of independence e.g. Is elder still able to prepare a pasta?

4 What sensors could we use?

5 Activity inference based on object use Dense sensing: attach sensors directly to objects in the environment. Battery-free wireless stickers (RFIDs) Battery powered sensor nodes

6 Activity inference based on object use Advantages Robust to environmental conditions High level information can be associated with the sensors Who are you? ID # <ID = e3f000e13431, desc = bread basket, manufacturer =, >

7 What inference algorithms could we use?

8 Activity inference using Dynamic Bayesian networks (HMMs) Advantages Easy to incorporate common sense information Efficient inference algorithms To create an activity model we need List of objects used while performing an activity Probability of using the objects e.g p(pot cooking) = 0.7

9 Techniques for constructing activity models: Hand definition prepare pasta list of objects pot stove spoon spaghetti probability of using objects P(pot boiling pasta) 0.2 P(stove boiling pasta) 0.2 P(spoon boiling pasta) 0.1 P(Spaghetti boiling pasta) models

10 Techniques for constructing activity models: Learn from data unlabeled data (objects used) label labeled data prepare pasta 20 times learn custom models

11 Techniques for constructing activity models: Mine activity from the web (Wyatt et. al. AAAI 05) preparing pasta preparing pasta find how to pages WEB extract objects pot stove spoon spaghetti Assemble model model compute usage probability from object occurrences P(pot boiling pasta) 0.2 P(stove boiling pasta) 0.2 P(spoon boiling pasta) 0.1 P(Spaghetti boiling pasta)

12 Judging the level of independence of an elder (1)inappropriate probabilities (2) Missing objects Preparing pasta pot kitchen range spaghetti spoon

13 Judging the level of independence of an elder (1)inappropriate probabilities (2) Missing objects Preparing pasta pot kitchen range spaghetti spoon pan stove macaroni fork

14 Dealing with Incompleteness Exploit common sense information about objects that are functionally similar pot pan spoon fork automatically extract ontology from a hierarchically organized lexical system called WordNet Adapt or improve model probabilities based on object similarity information apply statistical smoothing technique known as shrinkage to update model parameters

15 WordNet: Semantic relationships Microwave#1: electromagnetic wave Synset: microwave Microwave#2: kitchen appliance that cooks food Synset: microwave, microwave oven

16 WordNet: Semantic relationships Microwave#1: electromagnetic wave Synset: microwave Hypernym tree Electromagnetic radiation radiation energy Physical phenomenon Natural phenomenon Phenomenon Microwave#2: kitchen appliance that cooks food Synset: microwave, microwave oven Hypernym tree Kitchen appliance appliance durables Consumer goods object Hypernym: when a word sense is a superset of another

17 WordNet: Semantic relationships How do we select the word sense automatically? Hypernym tree Electromagnetic radiation radiation energy Physical phenomenon Natural phenomenon Phenomenon Hypernym tree Kitchen appliance appliance durables Consumer goods object

18 WordNet: Unique beginners for nouns Thing, entity Non-living thing, object Living thing, organism Natural object Artifact Substance food Plant, flora Animal, fauna Person, human being

19 WordNet: Semantic relationships How do we select the word sense automatically? Hypernym tree Electromagnetic radiation radiation energy Physical phenomenon Natural phenomenon Phenomenon Hypernym tree Kitchen appliance appliance durables Consumer goods object Identify objects that are (1) nouns (2) subset of entity

20 WordNet: Semantic relationships How do we select the word sense automatically? Hypernym tree Electromagnetic radiation radiation energy Physical phenomenon Natural phenomenon Phenomenon Hypernym tree Kitchen appliance appliance durables Consumer goods object Identify objects that are (1) nouns (2) subset of entity

21 Ontology extraction from WordNet From the activity recipe mined for preparing pasta, we have the list of objects used. preparing pasta web Mining process pot stove spoon spaghetti

22 Ontology extraction from WordNet Finding the first hypernym tree that contain the concept entity for all the objects in our list we get pot stove spoon pasta Cooking utensil kitchen appliance cutlery food Utensil home appliance tableware solid Implement appliance ware substance Instrumentality durables article entity Artifact consumer goods artifact commodity object artifact entity entity

23 WordNet ontology generation Object Artifact Substance Cutlery spoon Cooking Utensil Kitchen Appliance solid Food pasta stove pot spaghetti

24 WordNet ontology expansion Object Artifact Substance Cutlery spoon Cooking Utensil Kitchen Appliance solid Food pasta stove pot spaghetti

25 WordNet ontology expansion Object Artifact Substance solid Cutlery Food Cooking Utensil Kitchen Appliance fork spoon knife pasta hotplate stove microwave pan pot poacher linguine spaghetti macaroni

26 WordNet ontology Object Artifact Substance solid Cutlery Food Cooking Utensil Kitchen Appliance fork spoon knife pasta hotplate stove microwave pan pot poacher linguine spaghetti macaroni

27 Ontology extraction from WordNet From the activity recipe mined for preparing pasta, we also have the probabilities of object use preparing pasta pot stove spoon Spaghetti P(pot preparing pasta) 0.2 P(stove preparing pasta) 0.2 P(spoon preparing pasta) 0.1 P(Spaghetti preparing pasta)

28 Ontology: setting probabilities Object Artifact Substance solid Cutlery Food Cooking Utensil Kitchen Appliance fork spoon knife pasta hotplate stove microwave pan pot poacher linguine spaghetti macaroni

29 Ontology: setting probabilities Object Artifact Substance solid Cutlery Food Cooking Utensil Kitchen Appliance fork spoon 0.1 pan knife pot 0.2 poacher hotplate stove 0.2 microwave linguine spaghetti pasta macaroni

30 What if we use macaroni instead? Preparing pasta pot kitchen range spaghetti spoon pan stove macaroni fork

31 How do we propagate probabilities? Object Artifact Substance solid Cutlery Food Cooking Utensil Kitchen Appliance fork spoon pan 0.1 knife pot 0.2 poacher hotplate stove 0.2 microwave linguine spaghetti pasta macaroni?

32 Copy probability from sibling node to sibling node Substance solid Food pasta linguine spaghetti macaroni

33 Copy probability from sibling node to sibling node Substance solid Food pasta linguine spaghetti macaroni

34 Copy probability from sibling node to sibling node Substance solid Food pasta linguine spaghetti macaroni

35 Copy probability from sibling node to sibling node Problem: What happens if the probability for spaghetti is inappropriate or bad? Substance solid Food pasta linguine spaghetti macaroni

36 Count number of links from node to node Problem: What happens if spaghetti is not an immediate sibling of macaroni? Substance solid Food Noodle pasta spaghetti linguine macaroni

37 Count number of links from node to node Problem: What happens if spaghetti is not an immediate sibling of macaroni? Links Substance = 1 solid Food Noodle pasta spaghetti linguine macaroni

38 Count number of links from node to node Problem: What happens if spaghetti is not an immediate sibling of macaroni? Links Substance = 2 solid Food Noodle pasta spaghetti linguine macaroni

39 Count number of links from node to node Problem: What happens if spaghetti is not an immediate sibling of macaroni? Links Substance = 3 solid Food Noodle pasta spaghetti linguine macaroni

40 Taking additional information into account What happens if we also know that somebody preparing pasta can use lasagna? Substance Dish solid Food None of the two previous techniques can take advantage of this extra information lasagna 0.1 linguine spaghetti pasta macaroni?

41 Shrinkage: Key idea Idea: find or improve the parameter estimate of leaf nodes by linearly interpolating the estimates of its ancestor nodes How do we compute estimates for internal nodes?? lasagna 0.1 root Substance solid Food Dish pasta linguine? spaghetti leaf macaroni???

42 Shrinkage: Step two root Answer: Propagate information from leaf nodes to ancestors Substance solid For all internal nodes, compute the sum of all their children leaf nodes lasagna 0.1 Dish Food pasta leaf linguine spaghetti macaroni?

43 Shrinkage: Step two root Propagate information from leaf nodes to ancestors For all internal nodes, compute the sum of all their children leaf nodes lasagna 0.1 Substance Dish 0.1 solid Food pasta 0.6 leaf linguine spaghetti macaroni?

44 Shrinkage: Step two 0.1 Repeating for all internal nodes in ontology, we get lasagna 0.1 root Substance Dish solid Food pasta leaf linguine spaghetti macaroni?

45 Shrinkage: Step two 1 root 0.6 Now, we just focus on the nodes from the leaf to the root ~ p( macaroni ) =? Substance solid Food pasta leaf macaroni?

46 Shrinkage: Step three Reduce dependencies by subtract counts of consecutive nodes 1 root Substance 0.6 solid Food pasta leaf macaroni?

47 Shrinkage: Step three 1 root 0.6 Reduce dependencies by subtract counts of consecutive nodes Substance 0.6 = 0.1 solid Food pasta leaf macaroni?

48 Shrinkage: Step three = 0.4 Reduce dependencies by subtract counts of consecutive nodes 1 root = 0 Substance = 0.6 = = solid Food pasta leaf macaroni?

49 Shrinkage: Information tradeoff 0.4 root 0 Substance How do we combine the information? solid Food ~ p( macaroni ) =? pasta 0 leaf macaroni?

50 Shrinkage: Information tradeoff Generic 0.4 root 0 Substance tradeoff solid Food pasta Specific 0 leaf macaroni?

51 Shrinkage: Information tradeoff Generic 5 λ 4 λ 3 λ λ root 4 0 λ Substance 3 0 λ solid 2 λ λ Food 1 λ 1 λ pasta Specific 0 λ 0 0 λ leaf macaroni?

52 Shrinkage: Information tradeoff Generic 5 λ 4 λ 3 λ λ root 4 0 λ Substance 3 0 λ solid 2 λ λ Food 1 λ 1 λ pasta Specific 0 λ λ level = c level e 0 0 λ leaf macaroni?

53 Shrinkage: Information tradeoff 5 0.4λ root ~ p( macaroni ) = λ λ λ + 0 Substance λ + 1 λ + 0 solid Food pasta leaf macaroni?

54 Advantages of shrinkage 1. Create improved probability estimates for the leaf nodes (objects) of the ontology The effect of this improvement is a reduction in the number of training examples required to achieve a desired accuracy 2. Compute probabilities for objects not present in the models The effect is robustness when objects not present in the activity models are used while performing an activity

55 Performance on data collected from multiple individuals installed 108 RFIDs in real home 9 subjects 126 examples of 26 activities Using RFID glove reader Web mined activity models web Ontology: generated from 108 tagged objects and objects present in web mined activity models. Given trace, infer activities

56 Activities modeled by an HMM A A model#1 Pr(object activity) Single state per activity (26) Uniform prior Self-transition for smoothing Uniform inter-state transition Pr( a Pr( a i i t t+1 1 ) = = a ) = T = 0.8 i ii Pr( ai a j ) = Tij = = n 1 25 P( o a) = mined from web = 0.01 T A1 A2 A26 A A A

57 Activities modeled by an HMM A A model#1 model#2 t t+1 A A Shrinkage over observation matrix using H1 = λ level = e level H 2 = λ =1/ c level c level t t+1

58 Activities modeled by an HMM A A model#1 model#2 t t+1 A A Shrinkage over observation matrix using H1 = λ level = e level H 2 = λ =1/ c level c level t t+1

59 Experiment 1: Improvement of overall accuracy Assemble all the activity examples in a single sequence Infer the most likely state (activity) sequence using Viterbi decoding Compute total accuracy

60 Shrinkage: Learning the weights No shrinkage: 42% With shrinkage: 48.35% Improvement: 15.11% Bootstrapping by learning using 126 sensor traces: 19.2% Accuracy H2 Heuristic constant c model#2 H1 15 model#2 H2 10 model# H1 Heuristic constant c

61 Experiment 2: Robustness to unseen observations Replace m% of the observations in the activity examples by observations of one of their randomly selected sibling nodes in the ontology Brushing teeth Original: light toothpaste Replaced: light tooth_powder floss light floss lamp dentifrice Source of illumination toothpaste tooth_powder light lamp

62 Results: Robustness to unseen observations When replacing 100% No shrinkage: Randomly guessing 3.8% Drops 91.6% Accuracy Accuracy vs percentage of replaced observations Shrinkage: Drops 33% 10 5 model#1 model#2 using shrinkage Percentage of replaced observations

63 Effect of limited training data by simulation Simulations were performed to investigate the impact of shrinkage with limited training data The ontology was generated from a list of 815 objects consists of 4188 nodes 815 leaf nodes has a maximum depth of 14.

64 Experiment: Effect of limited training data log-likelihood model#1 model#2 using shrinkage Number training sequences

65 Conclusion Previous work demonstrated that it is possible to mine useful models of arbitrary day-to-day activities from the web Here we show that it is possible to deal with model incompleteness by incorporating common sense knowledge we compute probabilities for objects not originally present in the models We can improve the probability estimates we can learn higher quality models with less amount of training data Towards a completely unsupervised approach to learning activity models

66 Thank you!