DIR2017. Training Neural Rankers with Weak Supervision. Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Sascha Rothe, Jaap Kamps, and W.

Transcription

1 Training Neural Rankers with Weak Supervision DIR2017 Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Sascha Rothe, Jaap Kamps, and W. Bruce Croft Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

2 Motivation Deep neural nets are data hungry For many tasks, the more data you have, the better your model will be! This amount of data is not always available for many IR tasks Unsupervised neural network based methods. Our idea: Using a well stablished unsupervised methods as training signal. Weak supervision: Connecting symbolic IR with data driven methods

3 General Idea To leverage a large amounts of unsupervised data to infer weak labels and use that signal for learning supervised models as if we had the ground truth labels.

4 Weak supervision for Ranking Pseudo-Labeling BM25 plays the role of pseudo-labeler in our learning scenario. A target collection and a large set of training queries (without relevance judgment), Using the pseudo-labeler to rank/score the documents for each query in the training query set. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

5 Ranking Architectures: Score model The goal in this architecture is to learn a scoring function Linear Regression Loss Point-wise Loss: (linear regression, with MSE) q d

6 Ranking Architectures: Rank model The goal in this architecture is to learn a ranking function Pairwise loss Pair-wise at training/ Point-wise at inference Loss: (Hinge Loss) d $ q d %

7 Ranking Architectures: RankProb model The goal in this architecture is to learn a ranking function Logistic Regression Loss Pair-wise Loss: (logistic regression) d $ q d %

8 Input Representations Dense Vector Representation: Fully Featurized Exactly the BM25 input: Sparse Vector Representation: Bag of words Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

9 Input Representations Embedding Vector Representation Embedding size Joint Embedding Matrix for terms in Query and Document learning representation of terms Compositionally function (From words representation to query/document representation) Vocabulary size w ' w ( w ) Embedding size learning global weight of terms

10 Experimental Setup Target data collections: ClueWeb09 CatB dataset Robust dataset Training Query set: AOL (after some filtering, we got more than 6m queries for each set) Hyper-parameters: Width and depth of the network, learning rate, drop-out, embedding size Optimized using batched GP bandits with an expected improvement acquisition function

11 How do the neural models with different training objectives and input representations compare? Method Robust04 ClueWeb MAP MAP BM Score + Dense ô ô ô ô ô ô Score + Sparse ô ô ô ô ô ô Score + Embed ô Rank + Dense ô ô ô ô ô ô Rank + Sparse ô ô ô ô ô ô Rank + Embed RankProb + Dense ô ô ô ô ô ô RankProb + Sparse ô ô ô ô ô RankProb + Embed

12 How do the neural models with different training objectives and input representations compare? Take Home Message: Method Robust04 ClueWeb MAP MAP 1. Define an objective which enables your model to go BM beyond the imperfection of the weakly annotated data Score + Dense ô ô ô ô ô ô Score (ranking + Sparse instead of ô calibrated ô ô scoring) ô ô ô Score + Embed ô Rank Let + Dense the network ô decide ô about ô the ô representation. ô ô Rank Feeding + Sparse the network ô with ô featurized ô ô input kills ô the ô model Rank + Embed creativity! RankProb + Dense ô ô ô ô ô ô RankProb + Sparse ô ô ô ô ô RankProb + Embed

13 How meaningful are the compositionality weights learned in the embedding vector representation? (a) Robust04 (b) ClueWeb (Pearson Correlation: ) (Pearson Correlation: ) Figure 4: Strong linear correlation between weight learned Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

14 How meaningful are the compositionality weights learned in the embedding vector representation? Take Home Message: By just seeing individual local instances from the data, the network learns such a global statistic. (a) Robust04 (b) ClueWeb (Pearson Correlation: ) (Pearson Correlation: ) Figure 4: Strong linear correlation between weight learned Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

15 How well other alternatives for the embedding and weighting functions in embedding vector representation perform? Table 3: Performance of the rankprob model with variants of the embedding vector representation on di erent datasets. indicates that the improvements over all other models are statistically signi cant, at the 0.05 level using the paired two-tailed t-test, with Bonferroni correction. SIGIR 2017, August 2017, Tokyo, Japan Anon. Table 3: Performance of the rankprob model with variants of the embedding vector representationrobust04 on di erent datasets. Embedding type indicates that the improvements over all other models are statistically signi cant, at the 0.05 level using the paired two-tailed MAP t-test, with Bonferroni correction. Pretrained (external) + Uniform Robust04weighting ClueWeb Pretrained (external) + IDF weighting MAP P@20 ndcg@20 MAP P@20 ndcg@ Pretrained (external) + Weight learning Pretrained (external) + Uniform weighting (target) Uniform weighting Pretrained (external)pretrained + IDF weighting Pretrained (external)pretrained + Weight learning (target) IDF weighting Pretrained (target) + Uniform weighting Pretrained (target) + Weight learning Pretrained (target) + IDF weighting Uniform weighting Pretrained (target) + Learned Weight learning Learned + Uniform weighting Learned + IDF weighting Learned + IDF weighting Learned + Weight learning Learned + Weight learning Embedding type function E: (1) employing pre-trained word embeddings learned from an external corpus (we used Google News), (2) employing pretrained word embeddings learned from the target corpus (using the skip-gram model [27]), and (3) learning embeddings during the network training as it is explained in Section 4.3. Furthermore, for the compositionality function, we tried di erent alternatives: (1) uniform weighting (simple averaging which is a common (a) Robust04approach (b) ClueWeb (a) Robust04 (b) ClueWeb in compositionality function), (2) using IDF as xed weights instead Figure 5: Performance of the rankprob model with learned (Pearson Correlation: ) (Pearson Correlation: ) embedding, pre-trained embedding, and learned embedding of learning the weighting function W, and (3) learning weights Figure 4: Strong linear correlation between weight learned ClueWeb MAP P@20 ndcg@ (a) Robust04 with pre-trained embedding as initialization, with respect to (b) ClueWeb

16 How well other alternatives for the embedding and weighting functions in embedding vector representation perform? Take Home Message: Table 3: Performance of the rankprob model with variants of the embedding vector representation on di erent datasets. indicates that the improvements over all other models are statistically signi cant, at the 0.05 level using the paired two-tailed t-test, with Bonferroni correction. SIGIR 2017, August 2017, Tokyo, Japan Anon. Table 3: Performance of the rankprob model with variants of the embedding vector representationrobust04 on di erent datasets. Embedding type indicates that the improvements over all other models are statistically signi cant, at the 0.05 level using the paired two-tailed MAP t-test, with Bonferroni correction. ClueWeb MAP If you Pretrained get enough data, can learn embedding which is (external) + Uniform weighting you Pretrained (external) + IDF weighting (external) + Weight learning updating them better Pretrained fitted to your task by just based on the Pretrained (target) + Uniform weighting Pretrained (target) + IDF weighting objective of(target) the downstream Pretrained + Weight learning task Embedding type Robust04 MAP P@20 Pretrained (external) + Uniform weighting Pretrained (external) + IDF weighting Pretrained (external) + Weight learning Pretrained (target) + Uniform weighting Pretrained (target) + IDF weighting Uniform weighting Pretrained (target) + Learned Weight learning Learned + Uniform weighting Learned + IDF weighting Learned + IDF weighting Learned + Weight learning Learned + Weight learning ClueWeb ndcg@ MAP P@20 ndcg@ But you need a lot of data: THANKS TO WEAK SUPERVISION! function E: (1) employing pre-trained word embeddings learned from an external corpus (we used Google News), (2) employing pretrained word embeddings learned from the target corpus (using the skip-gram model [27]), and (3) learning embeddings during the network training as it is explained in Section 4.3. Furthermore, for the compositionality function, we tried di erent alternatives: (1) uniform weighting (simple averaging which is a common (a) Robust04approach (b) ClueWeb (a) Robust04 (b) ClueWeb in compositionality function), (2) using IDF as xed weights instead Figure 5: Performance of the rankprob model with learned (Pearson Correlation: ) (Pearson Correlation: ) embedding, pre-trained embedding, and learned embedding of learning the weighting function W, and (3) learning weights Figure 4: Strong linear correlation between weight learned (a) Robust04 with pre-trained embedding as initialization, with respect to (b) ClueWeb

17 How useful is learning with weak supervision as pretraining for supervised ranking? y signicant, at the 0.05 level using the paired two-tailed t-test, with Bonferroni correction. Method Robust04 ClueWeb MAP P@20 ndcg@20 MAP P@20 ndcg@20 Weakly supervised Fully supervised Weakly supervised + Fully supervised

18 How useful is learning with weak supervision as pretraining Take Home for Message: supervised ranking? y signicant, at the 0.05 level using the paired two-tailed t-test, with Bonferroni correction. You want to train a neural network Robust04for your task ClueWeb but you ve Method got just a small amount of supervised data? Weakly supervised You can compensate it by pertaining your network on Fully supervised weakly Weakly supervised annotated + Fully supervised data MAP P@20 ndcg@20 MAP P@20 ndcg@20

19 Avoiding your teacher s mistake! Training a neural ranker with controlled weak supervision MAIN GOAL: Controlling the effect of imperfect weak training instances by down-weighting them. Prediction loss wrt. the weak labels Prediction loss wrt. the weak labels Supervision Layer Supervision Layer Confidence Network Confidence Network Representation Learning Representation Learning Goodness of Representation Learning instances Goodness of instances Weak Annotator Weak Annotator True Labels Weak Annotator True Labels

20 Training Full Supervision mode Weak Supervision mode Prediction loss wrt. the weak labels Prediction loss wrt. the weak labels Supervision Layer Confidence Network Supervision Layer Confidence Network Representation Learning Goodness of instances Representation Learning Goodness of instances Weak Annotator True Labels Weak Annotator True Labels Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

21 Thank you!