Recurrent Knowledge Distillation

Conference paper (2018)

Authors

S. Pintea Pattern Recognition and Bioinformatics
Yue Liu KTH Royal Institute of Technology
Jan van Gemert Pattern Recognition and Bioinformatics
Research Group
Pattern Recognition and Bioinformatics
Copyright: © 2018 S. Pintea, Yue Liu, J.C. van Gemert
DOI: https://doi.org/10.1109/ICIP.2018.8451253
More Info
expand_more

Published Date

2018

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Research Group

Pattern Recognition and Bioinformatics

Abstract

Knowledge distillation compacts deep networks by letting a small student network learn from a large teacher network. The accuracy of knowledge distillation recently benefited from adding residual layers. We propose to reduce the size of the student network even further by recasting multiple residual layers in the teacher network into a single recurrent student layer. We propose three variants of adding recurrent connections into the student network, and show experimentally on CIFAR-10, Scenes and MiniPlaces, that we can reduce the number of parameters at little loss in accuracy.

Files

1805.07170.pdf
(pdf | 0.488 Mb)
Unknown license
08451253.pdf
(pdf | 0.834 Mb)
Unknown license

Download not available