DENSELY CONNECTED CONVOLUTIONAL NETWORKS

Transcription

1 Best paper award DENSELY CONNECTED CONVOLUTIONAL NETWORKS Gao Huang*, Zhuang Liu*, Laurens van der Maaten, Kilian Q. Weinberger Cornell University Tsinghua University Facebook AI Research CVPR 2017

2 CONVOLUTIONAL NETWORKS LeNet AlexNet VGG Inception ResNet

3 STANDARD CONNECTIVITY

4 RESNET CONNECTIVITY Identity mappings promote gradient propagation. : Element-wise addition Deep residual learning for image recognition: [He, Zhang, Ren, Sun] (CVPR 2015)

5 DENSE CONNECTIVITY C C C C C : Channel-wise concatenation

6 DENSE AND SLIM C C C C k channels k channels k channels k channels k : Growth Rate

7 FORWARD PROPAGATION x0 h1 x1 h2 x2 x0 x1 h3 x3 h4 x0 x2 x1 x0 Batc ReL Con x4 x3 x2 x1 x0

8 COMPOSITE LAYER IN DENSENET x3 x2 x1 x0 Batch Norm ReLU Convolution (3x3) x4 x3 x2 x1 x0 k channels x5 =h5([x0,, x4])

9 COMPOSITE LAYER IN DENSENET WITH BOTTLENECK LAYER x3 x2 x1 x0 Batch Norm ReLU Convolution (1x1) Batch Norm ReLU Convolution (3x3) x4 lxk channels 4xk channels k channels Higher parameter and computational efficiency

10 DENSENET Dense Block 1 Dense Block 2 Dense Block 3 Convolution Convolution Pooling Convolution Pooling Pooling Linear Output Pooling reduces feature map sizes Feature map sizes match within each block

11 ADVANTAGES OF DENSE CONNECTIVITY

12 ADVANTAGE 1: STRONG GRADIENT FLOW Error Signal Implicit deep supervision Deeply supervised Net: [Lee, Xie, Gallagher, Zhang, Tu] (2015)

13 ADVANTAGE 2: PARAMETER & COMPUTATIONAL EFFICIENCY ResNet connectivity: #parameters: C Input Correlated features hl Output C O(CxC) DenseNet connectivity: k<<c Input Diversified features lxk hl k Output O(lxkxk) k: Growth rate

14 ADVANTAGE 3: MAINTAINS LOW COMPLEXITY FEATURES Standard Connectivity: Classifier uses most complex (high level) features w4 y = w4h4(x) x h1(x) h2(x) h3(x) h4(x) classifier Increasingly complex features

15 ADVANTAGE 3: MAINTAINS LOW COMPLEXITY FEATURES Dense Connectivity: Classifier uses features of all complexity levels C C C C w0 w1 w2 w3 w4 y = w0x + +w1h1(x) +w2h2(x) +w3h3(x) +w4h4(x) x h1(x) h2(x) h3(x) h4(x) classifier Increasingly complex features

16 RESULTS

17 RESULTS ON CIFAR-10 ResNet (110 Layers, 1.7 M) ResNet (1001 Layers, 10.2 M) DenseNet (100 Layers, 0.8 M) DenseNet (250 Layers, 15.3 M) 12.0 With data augmentation 12.0 Without data augmentation Test Error (%) Previous SOTA Previous SOTA

18 RESULTS ON CIFAR-100 ResNet (110 Layers, 1.7 M) ResNet (1001 Layers, 10.2 M) DenseNet (100 Layers, 0.8 M) DenseNet (250 Layers, 15.3 M) With data augmentation Without data augmentation Previous SOTA Test Error (%) Previous SOTA

19 RESULTS ON IMAGENET 28.0 DenseNet ResNet 28.0 DenseNet ResNet ResNet-34 ResNet-34 Top-1 error (%) DenseNet-121 ResNet-50 DenseNet-169 DenseNet-201ResNet-101 ResNet-152 DenseNet-264 DenseNet-264(k=48) Top-1 error (%) DenseNet-121 ResNet-50 DenseNet-169 DenseNet-201 ResNet-101 DenseNet-264 ResNet-152 DenseNet-264(k=48) # Parameters (M) Top-1: 20.27% Top-5: 5.17% GFLOPs

20 MULTI-SCALE DENSENET (Preview) Classifier 1 Classifier 2 Classifier 3 Classifier 4 cat: threshold cat: threshold cat: > threshold Multi-Scale DenseNet: [Huang, Chen, Li, Wu, van der Maaten, Weinberger] (arxiv Preprint: )

21 MULTI-SCALE DENSENET (Preview) Test Input Inference Speed: ~ 2.6x faster than ResNets ~ 1.3x faster than DenseNets Classifier 1 Classifier 2 Classifier 3 Classifier 4 Easy examples Hard examples

22 Memory efficient Torch implementation: Other implementations: Our Caffe Implementation Our memory-efficient Caffe Implementation. Our memory-efficient PyTorch Implementation. PyTorch Implementation by Andreas Veit. PyTorch Implementation by Brandon Amos. MXNet Implementation by Nicatio. MXNet Implementation (supports ImageNet) by Xiong Lin. Tensorflow Implementation by Yixuan Li. Tensorflow Implementation by Laurent Mazare. Tensorflow Implementation (with BC structure) by Illarion Khlestov. Lasagne Implementation by Jan Schlüter. Keras Implementation by tdeboissiere. Keras Implementation by Roberto de Moura Estevão Filho. Keras Implementation (with BC structure) by Somshubra Majumdar. Chainer Implementation by Toshinori Hanya. Chainer Implementation by Yasunori Kudo.

23 REFERENCES Kaiming He, et al. "Deep residual learning for image recognition" CVPR 2016 Chen-Yu Lee, et al. "Deeply-supervised nets" AISTATS 2015 Gao Huang, et al. "Deep networks with stochastic depth" ECCV 2016 Gao Huang, et al. "Multi-Scale Dense Convolutional Networks for Efficient Prediction" arxiv preprint arxiv: (2017) Geoff Pleiss, et al. "Memory-Efficient Implementation of DenseNets, arxiv preprint arxiv: (2017)