TUTCRIS - Tampereen teknillinen yliopisto

TUTCRIS

Curriculum-based Teacher Ensemble for Robust Neural Network Distillation

Tutkimustuotosvertaisarvioitu

Yksityiskohdat

AlkuperäiskieliEnglanti
Otsikko2019 27th European Signal Processing Conference (EUSIPCO)
KustantajaIEEE
Sivumäärä5
ISBN (elektroninen)978-9-0827-9703-9
ISBN (painettu)978-1-5386-7300-3
DOI - pysyväislinkit
TilaJulkaistu - syyskuuta 2019
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaEUROPEAN SIGNAL PROCESSING CONFERENCE -
Kesto: 1 tammikuuta 1900 → …

Julkaisusarja

NimiEuropean Signal Processing Conference
ISSN (painettu)2219-5491
ISSN (elektroninen)2076-1465

Conference

ConferenceEUROPEAN SIGNAL PROCESSING CONFERENCE
Ajanjakso1/01/00 → …

Tiivistelmä

Neural network distillation is used for transferring the knowledge from a complex teacher network into a lightweight student network, improving in this way the performance of the student network. However, neural distillation does not always lead to consistent results, with several factors affecting the efficiency of the knowledge distillation process. In this paper it is experimentally demonstrated that the selected teacher can indeed have a significant effect on knowledge transfer. To overcome this limitation, we propose a curriculum-based teacher ensemble that allows for performing robust and efficient knowledge distillation. The proposed method is motivated by the way that humans learn through a curriculum, as well as supported by recent findings that hints to the existence of critical learning periods in neural networks. The effectiveness of the proposed approach, compared to various distillation variants, is demonstrated using three image datasets and different network architectures.

Tutkimusalat

Julkaisufoorumi-taso