±à¼ÍƼö: |
±¾ÎÄÀ´Ô´csdn£¬¾í»ýÉñ¾ÍøÂçÈÕÒæÔö³¤µÄÉî¶ÈºÍ³ß´çΪÉî¶ÈѧϰÔÚÒÆ¶¯¶ËµÄ²¿Êð´øÀ´Á˾޴óµÄÌôÕ½£¬CNNÄ£ÐÍѹËõÓë¼ÓËÙ³ÉΪÁËѧÊõ½çºÍ¹¤Òµ½ç¶¼ÖØµã¹Ø×¢µÄÑо¿ÁìÓòÖ®Ò»¡£
|
|
ǰÑÔ
×Ô´ÓAlexNetÒ»¾Ù¶áµÃILSVRC 2012 ImageNetͼÏñ·ÖÀྺÈüµÄ¹Ú¾üºó£¬¾í»ýÉñ¾ÍøÂ磨CNN£©µÄÈȳ±±ãϯ¾íÁËÕû¸ö¼ÆËã»úÊÓ¾õÁìÓò¡£CNNÄ£ÐÍ»ðËÙÌæ´úÁË´«Í³È˹¤Éè¼Æ£¨hand-crafted£©ÌØÕ÷ºÍ·ÖÀàÆ÷£¬²»½öÌṩÁËÒ»Öֶ˵½¶ËµÄ´¦Àí·½·¨£¬»¹´ó·ù¶ÈµØË¢ÐÂÁ˸÷¸öͼÏñ¾ºÈüÈÎÎñµÄ¾«¶È£¬¸üÉõÕß³¬Ô½ÁËÈËÑ۵ľ«¶È£¨LFWÈËÁ³Ê¶±ðÈÎÎñ£©¡£CNNÄ£ÐÍÔÚ²»¶Ï±Æ½ü¼ÆËã»úÊÓ¾õÈÎÎñµÄ¾«¶È¼«ÏÞµÄͬʱ£¬ÆäÉî¶ÈºÍ³ß´çÒ²Ôڳɱ¶Ôö³¤¡£
±í1 ¼¸ÖÖ¾µäÄ£Ð͵ijߴ磬¼ÆËãÁ¿ºÍ²ÎÊýÊýÁ¿¶Ô±È

ËæÖ®¶øÀ´µÄÊÇÒ»¸öºÜÞÏÞεij¡¾°£ºÈç´Ë¾Þ´óµÄÄ£ÐÍÖ»ÄÜÔÚÓÐÏÞµÄÆ½Ì¨ÏÂʹÓ㬸ù±¾ÎÞ·¨ÒÆÖ²µ½Òƶ¯¶ËºÍǶÈëʽоƬµ±ÖС£¾ÍËãÏëͨ¹ýÍøÂç´«Ê䣬µ«½Ï¸ßµÄ´ø¿íÕ¼ÓÃÒ²ÈúܶàÓû§Íû¶øÉúη¡£ÁíÒ»·½Ã棬´ó³ß´çµÄÄ£ÐÍÒ²¶ÔÉ豸¹¦ºÄºÍÔËÐÐËÙ¶È´øÀ´Á˾޴óµÄÌôÕ½¡£Òò´ËÕâÑùµÄÄ£Ð;àÀëʵÓû¹ÓÐÒ»¶Î¾àÀë¡£
ÔÚÕâÑùµÄÇéÐÎÏ£¬Ä£ÐÍСÐÍ»¯Óë¼ÓËÙ³ÉÁËØ½´ý½â¾öµÄÎÊÌâ¡£ÆäʵÔçÆÚ¾ÍÓÐѧÕßÌá³öÁËһϵÁÐCNNÄ£ÐÍѹËõ·½·¨£¬°üÀ¨È¨Öµ¼ôÖµ£¨prunning£©ºÍ¾ØÕóSVD·Ö½âµÈ£¬µ«Ñ¹ËõÂʺÍЧÂÊ»¹Ô¶²»ÄÜÁîÈËÂúÒâ¡£
½üÄêÀ´£¬¹ØÓÚÄ£ÐÍСÐÍ»¯µÄËã·¨´ÓѹËõ½Ç¶ÈÉÏ¿ÉÒÔ´óÖ·ÖΪÁ½Àࣺ´ÓÄ£ÐÍÈ¨ÖØÊýÖµ½Ç¶ÈѹËõºÍ´ÓÍøÂç¼Ü¹¹½Ç¶ÈѹËõ¡£ÁíÒ»·½Ã棬´Ó¼æ¹Ë¼ÆËãËÙ¶È·½Ã棬ÓÖ¿ÉÒÔ»®·ÖΪ£º½öѹËõ³ß´çºÍѹËõ³ß´çµÄͬʱÌáÉýËÙ¶È¡£
±¾ÎÄÖ÷ÒªÌÖÂÛÈçϼ¸Æª´ú±íÐÔµÄÎÄÕºͷ½·¨£¬°üÀ¨SqueezeNet[5]¡¢Deep Compression[6]¡¢XNorNet[7]¡¢Distilling[8]¡¢MobileNet[9]ºÍShuffleNet[10]£¬Ò²¿É°´ÕÕÉÏÊö·½·¨½øÐдóÖ·ÖÀࣺ
±í2 ¼¸ÖÖ¾µäѹËõ·½·¨¼°¶Ô±È

Ò»¡¢SqueezeNet
1.1 Éè¼ÆË¼Ïë
SqueezeNetÊÇF. N. Iandola,S.HanµÈÈËÓÚ2016ÄêµÄÂÛÎÄ¡¶SqueezeNet:
AlexNet-level accuracy with 50x fewer parameters and
< 0.5MB model size¡·ÖÐÌá³öµÄÒ»¸öСÐÍ»¯µÄÍøÂçÄ£Ðͽṹ£¬¸ÃÍøÂçÄÜÔÚ±£Ö¤²»Ëðʧ¾«¶ÈµÄͬʱ£¬½«ÔʼAlexNetѹËõÖÁÔÀ´µÄ510±¶×óÓÒ£¨<
0.5MB£©¡£
SqueezeNetµÄºËÐÄÖ¸µ¼Ë¼ÏëÊÇ¡ª¡ªÔÚ±£Ö¤¾«¶ÈµÄͬʱʹÓÃ×îÉٵIJÎÊý¡£
¶øÕâÒ²ÊÇËùÓÐÄ£ÐÍѹËõ·½·¨µÄÒ»¸öÖÕ¼«Ä¿±ê¡£
»ùÓÚÕâ¸ö˼Ï룬SqueezeNetÌá³öÁË3µãÍøÂç½á¹¹Éè¼Æ²ßÂÔ£º
²ßÂÔ 1.½«3x3¾í»ýºËÌæ»»Îª1x1¾í»ýºË¡£
ÕâÒ»²ßÂԺܺÃÀí½â£¬ÒòΪ1¸ö1x1¾í»ýºËµÄ²ÎÊýÊÇ3x3¾í»ýºË²ÎÊýµÄ1/9£¬ÕâÒ»¸Ä¶¯ÀíÂÛÉÏ¿ÉÒÔ½«Ä£ÐͳߴçѹËõ9±¶¡£
²ßÂÔ 2.¼õСÊäÈëµ½3x3¾í»ýºËµÄÊäÈëͨµÀÊý¡£
ÎÒÃÇÖªµÀ£¬¶ÔÓÚÒ»¸ö²ÉÓÃ3x3¾í»ýºËµÄ¾í»ý²ã£¬¸Ã²ãËùÓоí»ý²ÎÊýµÄÊýÁ¿£¨²»¿¼ÂÇÆ«Öã©Îª£º

ʽÖУ¬NÊǾí»ýºËµÄÊýÁ¿£¬Ò²¼´Êä³öͨµÀÊý£¬CÊÇÊäÈëͨµÀÊý¡£
ʽÖУ¬NÊǾí»ýºËµÄÊýÁ¿£¬Ò²¼´Êä³öͨµÀÊý£¬CÊÇÊäÈëͨµÀÊý¡£
Òò´Ë£¬ÎªÁ˱£Ö¤¼õÐ¡ÍøÂç²ÎÊý£¬²»½ö½öÐèÒª¼õÉÙ3x3¾í»ýºËµÄÊýÁ¿£¬»¹Ðè¼õÉÙÊäÈëµ½3x3¾í»ýºËµÄÊäÈëͨµÀÊýÁ¿£¬¼´Ê½ÖÐCµÄÊýÁ¿¡£
²ßÂÔ 3.¾¡¿ÉÄܵĽ«½µ²ÉÑù·ÅÔÚÍøÂçºóÃæµÄ²ãÖС£
ÔÚ¾í»ýÉñ¾ÍøÂçÖУ¬Ã¿²ãÊä³öµÄÌØÕ÷ͼ£¨feature map£©ÊÇ·ñϲÉÑùÊÇÓɾí»ý²ãµÄ²½³¤»òÕ߳ػ¯²ã¾ö¶¨µÄ¡£¶øÒ»¸öÖØÒªµÄ¹ÛµãÊÇ£º·Ö±æÂÊÔ½´óµÄÌØÕ÷ͼ£¨ÑÓ³Ù½µ²ÉÑù£©¿ÉÒÔ´øÀ´¸ü¸ßµÄ·ÖÀྫ¶È£¬¶øÕâÒ»¹Ûµã´ÓÖ±¾õÉÏÒ²¿ÉÒԺܺÃÀí½â£¬ÒòΪ·Ö±æÂÊÔ½´óµÄÊäÈëÄܹ»ÌṩµÄÐÅÏ¢¾ÍÔ½¶à¡£
ÉÏÊöÈý¸ö²ßÂÔÖУ¬Ç°Á½¸ö²ßÂÔ¶¼ÊÇÕë¶ÔÈçºÎ½µµÍ²ÎÊýÊýÁ¿¶øÉè¼ÆµÄ£¬×îºóÒ»¸öÖ¼ÔÚ×î´ó»¯ÍøÂ羫¶È¡£
1.2 ÍøÂç¼Ü¹¹
»ùÓÚÒÔÉÏÈý¸ö²ßÂÔ£¬×÷ÕßÌá³öÁËÒ»¸öÀàËÆinceptionµÄÍøÂçµ¥Ôª½á¹¹£¬È¡ÃûΪfire
module¡£Ò»¸öfire module °üº¬Ò»¸ösqueeze ¾í»ý²ã£¨Ö»°üº¬1x1¾í»ýºË£©ºÍÒ»¸öexpand¾í»ý²ã£¨°üº¬1x1ºÍ3x3¾í»ýºË£©¡£ÆäÖУ¬squeeze²ã½è¼øÁËinceptionµÄ˼Ï룬ÀûÓÃ1x1¾í»ýºËÀ´½µµÍÊäÈëµ½expand²ãÖÐ3x3¾í»ýºËµÄÊäÈëͨµÀÊý¡£Èçͼ1Ëùʾ¡£

ÆäÖУ¬¶¨Òåsqueeze²ãÖÐ1x1¾í»ýºËµÄÊýÁ¿ÊÇs1x1£¬ÀàËÆµÄ£¬expand²ãÖÐ1x1¾í»ýºËµÄÊýÁ¿ÊÇe1x1£¬
3x3¾í»ýºËµÄÊýÁ¿ÊÇe3x3¡£Áîs1x1 < e1x1+ e3x3´Ó¶ø±£Ö¤ÊäÈëµ½3x3µÄÊäÈëͨµÀÊý¼õС¡£SqueezeNetµÄÍøÂç½á¹¹ÓÉÈô¸É¸ö
fire module ×é³É£¬ÁíÍâÎÄÕ»¹¸ø³öÁËһЩ¼Ü¹¹Éè¼ÆÉϵÄϸ½Ú£º
1.ΪÁ˱£Ö¤1x1¾í»ýºËºÍ3x3¾í»ýºË¾ßÓÐÏàͬ´óСµÄÊä³ö£¬3x3¾í»ýºË²ÉÓÃ1ÏñËØµÄzero-paddingºÍ²½³¤
2.squeeze²ãºÍexpand²ã¾ù²ÉÓÃRELU×÷Ϊ¼¤»îº¯Êý
3.ÔÚfire9ºó²ÉÓÃ50%µÄdropout
4.ÓÉÓÚÈ«Á¬½Ó²ãµÄ²ÎÊýÊýÁ¿¾Þ´ó£¬Òò´Ë½è¼øNIN[11]µÄ˼Ï룬ȥ³ýÁËÈ«Á¬½Ó²ã¶ø¸ÄÓÃglobal
average pooling¡£
1.3 ʵÑé½á¹û
±í3 ²»Í¬Ñ¹Ëõ·½·¨ÔÚImageNetÉϵĶԱÈʵÑé½á¹û[5]

ÉϱíÏÔʾ£¬Ïà±È´«Í³µÄѹËõ·½·¨£¬SqueezeNetÄÜÔÚ±£Ö¤¾«¶È²»Ëð£¨ÉõÖÁÂÔÓÐÌáÉý£©µÄÇé¿öÏ£¬´ïµ½×î´óµÄѹËõÂÊ£¬½«ÔʼAlexNet´Ó240MBѹËõÖÁ4.8MB£¬¶ø½áºÏDeep
Compressionºó¸üÄÜ´ïµ½0.47MB£¬ÍêÈ«Âú×ãÁËÒÆ¶¯¶ËµÄ²¿ÊðºÍµÍ´ø¿íÍøÂçµÄ´«Êä¡£
´ËÍ⣬×÷Õß»¹½è¼øResNet˼Ï룬¶ÔÔÊ¼ÍøÂç½á¹¹×öÁËÐ޸ģ¬Ôö¼ÓÁËÅÔ··ÖÖ§£¬½«·ÖÀྫ¶ÈÌáÉýÁËÔ¼3%¡£
1.4 ËÙ¶È¿¼Á¿
¾¡¹ÜÎÄÕÂÖ÷ÒªÒÔѹËõÄ£ÐͳߴçΪĿ±ê£¬µ«ÎãÓ¹ÖÃÒɵÄÒ»µãÊÇ£¬SqueezeNetÔÚÍøÂç½á¹¹ÖдóÁ¿²ÉÓÃ1x1ºÍ3x3¾í»ýºËÊÇÓÐÀûÓÚËٶȵÄÌáÉýµÄ£¬¶ÔÓÚÀàËÆcaffeÕâÑùµÄÉî¶Èѧϰ¿ò¼Ü£¬ÔÚ¾í»ý²ãµÄǰÏò¼ÆËãÖУ¬²ÉÓÃ1x1¾í»ýºË¿É±ÜÃâ¶îÍâµÄim2col²Ù×÷£¬¶øÖ±½ÓÀûÓÃgemm½øÐоØÕó¼ÓËÙÔËË㣬Òò´Ë¶ÔËٶȵÄÓÅ»¯ÊÇÓÐÒ»¶¨µÄ×÷Óõġ£È»¶ø£¬ÕâÖÖÌáËÙµÄ×÷ÓÃÈÔÈ»ÊÇÓÐÏ޵ģ¬ÁíÍ⣬SqueezeNet²ÉÓÃÁË9¸öfire
moduleºÍÁ½¸ö¾í»ý²ã£¬Òò´ËÈÔÐèÒª½øÐдóÁ¿³£¹æ¾í»ý²Ù×÷£¬ÕâÒ²ÊÇÓ°ÏìËٶȽøÒ»²½ÌáÉýµÄÆ¿¾±¡£
¶þ¡¢Deep Compression
Deep Compression³ö×ÔS.Han 2016 ICLRµÄһƪÂÛÎÄ¡¶Deep Compression:
Compressing Deep Neural Networks with Pruning, Trained
Quantization and Huffman Coding¡·¡£¸ÃÎÄÕ»ñµÃÁËICLR 2016µÄ×î¼ÑÂÛÎĽ±£¬Í¬Ê±Ò²¾ßÓÐÀï³Ì±®Ê½µÄÒâÒ壬ÒýÁìÁËCNNÄ£ÐÍСÐÍ»¯Óë¼ÓËÙÑо¿·½ÏòµÄпñ³±£¬Ê¹µÃÕâÒ»ÁìÓò½üÁ½ÄêÀ´Ó¿ÏÖ³öÁË´óÁ¿µÄÓÅÐ㹤×÷ÓëÎÄÕ¡£
2.1 Ëã·¨Á÷³Ì
ÓëÇ°ÃæµÄ¡°¼Ü¹¹Ñ¹ËõÅÉ¡±µÄSqueezeNet²»Í¬£¬Deep CompressionÊÇÊôÓÚ¡°È¨ÖµÑ¹ËõÅÉ¡±µÄ¡£¶øÁ½ÆªÎÄÕ¾ù³ö×ÔS.HanÍŶӣ¬Òò´ËÁ½ÖÖ·½·¨½áºÏ£¬Ë«½£ºÏèµ£¬¸üÊÇÄÜ´ïµ½µÇ·åÔ켫µÄѹËõЧ¹û¡£ÕâһʵÑé½á¹ûÒ²ÔÚÉϱíÖеõ½ÑéÖ¤¡£
Deep CompressionµÄËã·¨Á÷³Ì°üº¬Èý²½£¬Èçͼ2Ëùʾ£º

1¡¢Pruning(Ȩֵ¼ôÖ¦)
¼ôÖ¦µÄ˼ÏëÆäʵÔçÒÑÔÚÔçÆÚÂÛÎÄÖпÉÒÔ¿ú¼û£¬LeCunµÈÈËÔø¾¾ÍÀûÓüôÖ¦À´Ï¡ÊèÍøÂ磬¼õС¹ýÄâºÏµÄ·çÏÕ£¬ÌáÉýÍøÂç·º»¯ÐÔ¡£
ͼ3ÊÇMNISTÉÏѵÁ·µÃµ½µÄLeNet conv1¾í»ý²ãÖеIJÎÊý·Ö²¼£¬¿ÉÒÔ¿´³ö£¬´ó²¿·ÖȨֵ¼¯ÖÐÔÚ0´¦¸½½ü£¬¶ÔÍøÂçµÄ¹±Ï×½ÏС£¬ÔÚ¼ôÖµÖУ¬½«0Öµ¸½½üµÄ½ÏСµÄȨֵÖÃ0£¬Ê¹ÕâЩȨֵ²»±»¼¤»î£¬´Ó¶ø×ÅÖØÑµÁ·Ê£ÏµķÇÁãȨֵ£¬×îÖÕÔÚ±£Ö¤ÍøÂ羫¶È²»±äµÄÇé¿öÏ´ﵽѹËõ³ß´çµÄÄ¿µÄ¡£
ʵÑé·¢ÏÖÄ£ÐͶԼôÖ¦¸üÃô¸Ð£¬Òò´ËÔÚ¼ôֵʱ½¨ÒéÖð²ãµü´úÐÞ¼ô£¬ÁíÍâÿ²ãµÄ¼ôÖ¦±ÈÀýÈçºÎ×Ô¶¯Ñ¡È¡ÈÔÈ»ÊÇÒ»¸öÖµµÃÉîÈëÑо¿µÄ¿ÎÌâ¡£

2¡¢Quantization (ȨֵÁ¿»¯)
´Ë´¦µÄȨֵÁ¿»¯»ùÓÚȨֵ¾ÛÀ࣬½«Á¬Ðø·Ö²¼µÄȨֵÀëÉ¢»¯£¬´Ó¶ø¼õСÐèÒª´æ´¢µÄȨֵÊýÁ¿¡£
1.³õʼ»¯¾ÛÀàÖÐÐÄ£¬ÊµÑéÖ¤Ã÷ÏßÐÔ³õʼ»¯Ð§¹û×îºÃ£»
2.ÀûÓÃk-meansËã·¨½øÐоÛÀ࣬½«È¨Öµ»®·Öµ½²»Í¬µÄclusterÖУ»
3.ÔÚǰÏò¼ÆËãʱ£¬Ã¿¸öȨֵÓÉÆä¾ÛÀàÖÐÐıíʾ£»
4.ÔÚºóÏò¼ÆËãʱ£¬Í³¼ÆÃ¿¸öclusterÖеÄÌݶȺͽ«Æä·´´«¡£

ͼ4 ȨֵÁ¿»¯Ç°ÏòºÍºóÏò¼ÆËã¹ý³Ì[6]
3¡¢Huffman encoding(»ô·òÂü±àÂë)
»ô·òÂü±àÂë²ÉÓñ䳤±àÂ뽫ƽ¾ù±àÂ볤¶È¼õС£¬½øÒ»²½Ñ¹ËõÄ£Ðͳߴ硣
2.2 Ä£ÐÍ´æ´¢
ǰÊöµÄ¼ôÖ¦ºÍÁ¿»¯¶¼ÊÇΪÁËʵÏÖÄ£Ð͵ĸü½ôÖµÄѹËõ£¬ÒÔʵÏÖ¼õСģÐͳߴçµÄÄ¿µÄ¡£
1.¶ÔÓÚ¼ôÖ¦ºóµÄÄ£ÐÍ£¬ÓÉÓÚÿ²ã´óÁ¿²ÎÊýΪ0£¬ºóÐøÖ»Ð轫·ÇÁãÖµ¼°ÆäÏÂ±ê½øÐд洢£¬ÎÄÕÂÖвÉÓÃCSR£¨Compressed
Sparse Row£©À´½øÐд洢£¬ÕâÒ»²½¿ÉÒÔʵÏÖ9x~13xµÄѹËõÂÊ¡£
2.¶ÔÓÚÁ¿»¯ºóµÄÄ£ÐÍ£¬Ã¿¸öȨֵ¶¼ÓÉÆä¾ÛÀàÖÐÐıíʾ£¨¶ÔÓÚ¾í»ý²ã£¬¾ÛÀàÖÐÐÄÉèΪ256¸ö£¬¶ÔÓÚÈ«Á¬½Ó²ã£¬¾ÛÀàÖÐÐÄÉèΪ32¸ö£©£¬Òò´Ë¿ÉÒÔ¹¹Ôì¶ÔÓ¦µÄÂëÊéºÍϱ꣬´ó´ó¼õÉÙÁËÐèÒª´æ´¢µÄÊý¾ÝÁ¿£¬´Ë²½ÄÜʵÏÖÔ¼3xµÄѹËõÂÊ¡£
3.×îºó¶ÔÉÏÊöѹËõºóµÄÄ£ÐͽøÒ»²½²ÉÓñ䳤»ô·òÂü±àÂ룬ʵÏÖÔ¼1xµÄѹËõÂÊ¡£
2.3 ʵÑé½á¹û
±í4 ²»Í¬ÍøÂç²ÉÓÃDeep CompressionºóµÄѹËõÂÊ[6]

ͨ¹ýSqueezeNet+Deep Compression,¿ÉÒÔ½«Ôʼ240MµÄAlexNetѹËõÖÁ0.47M£¬ÊµÏÖÔ¼510xµÄѹËõÂÊ¡£
2.4 ËÙ¶È¿¼Á¿
¿ÉÒÔ¿´³ö£¬Deep CompressionµÄÖ÷ÒªÉè¼ÆÊÇÕë¶ÔÍøÂç´æ´¢³ß´çµÄѹËõ£¬µ«ÔÚǰÏòʱ£¬Èç¹û½«´æ´¢Ä£ÐͶÁÈëÕ¹¿ªºó£¬²¢Ã»ÓдøÀ´¸ü´óµÄËÙ¶ÈÌáÉý¡£Òò´ËSong
H.µÈÈËרÃÅÕë¶ÔѹËõºóµÄÄ£ÐÍÉè¼ÆÁËÒ»Ì×»ùÓÚFPGAµÄÓ²¼þǰÏò¼ÓËÙ¿ò¼ÜEIE[12]£¬ÓÐÐËȤµÄ¿ÉÒÔÑо¿Ò»Ï¡£
Èý¡¢XNorNet
¶þÖµÍøÂçÒ»Ö±ÊÇÄ£ÐÍѹËõºÍ¼ÓËÙÁìÓò¾¾Ã²»Ë¥µÄÑо¿¿ÎÌâÖ®Ò»¡£½«Ôʼ32λ¸¡µãÐ͵ÄȨֵѹËõµ½1±ÈÌØ£¬ÈçºÎ×î´ó³Ì¶ÈµØ¼õСÐÔÄÜËðʧ¾Í³ÉΪÁËÑо¿µÄ¹Ø¼ü¡£
´ËƪÂÛÎÄÖ÷ÒªÓÐÒÔϼ¸¸ö¹±Ï×£º
1.Ìá³öÁËÒ»¸öBWN£¨Binary-Weight-Network£©ºÍXNOR-Network£¬Ç°ÕßÖ»¶ÔÍøÂç²ÎÊý×ö¶þÖµ»¯£¬´øÀ´Ô¼32xµÄ´æ´¢Ñ¹ËõºÍ2xµÄËÙ¶ÈÌáÉý£¬¶øºóÕß¶ÔÍøÂçÊäÈëºÍ²ÎÊý¶¼×öÁ˶þÖµ»¯£¬ÔÚʵÏÖ32x´æ´¢Ñ¹ËõµÄͬʱ´øÁË58xµÄËÙ¶ÈÌáÉý£»
2.Ìá³öÁËÒ»¸öÐÂÐͶþÖµ»¯È¨ÖµµÄËã·¨£»
3.µÚÒ»¸öÔÚ´ó¹æÄ£Êý¾Ý¼¯ÈçImageNetÉÏÌá½»¶þÖµ»¯ÍøÂç½á¹ûµÄ¹¤×÷£»
4.ÎÞÐèԤѵÁ·£¬¿ÉʵÏÖtraining from scratch¡£
3.1 BWN

¼´×îÓŵĶþÖµ»¯Â˲¨Æ÷ÕÅÁ¿B¼´ÎªÔʼ²ÎÊýµÄ·ûºÅº¯Êý£¬×îÓŵij߶ÈÒò×ÓΪÿ¸öÂ˲¨Æ÷ȨֵµÄ¾ø¶ÔÖµµÄ¾ùÖµ¡£
ѵÁ·Ëã·¨Èçͼ5Ëùʾ£¬ÖµµÃ×¢ÒâµÄÊÇ£¬Ö»ÓÐÔÚǰÏò¼ÆËãºÍºóÏò´«²¥Ê±Ê¹ÓöþÖµ»¯ºóµÄȨֵ£¬ÔÚ¸üвÎÊýʱÒÀȻʹÓÃÔʼ²ÎÊý£¬ÕâÊÇÒòΪÈç¹ûʹÓöþÖµ»¯ºóµÄ²ÎÊý»áµ¼ÖºÜСµÄÌݶÈϽµ£¬´Ó¶øÊ¹µÃѵÁ·ÎÞ·¨ÊÕÁ²¡£
3.2 XNOR-Net
ÔÚXNORÍøÂçÖУ¬ÓÅ»¯µÄÄ¿±êÊǽ«Á½¸öʵÊýÏòÁ¿µÄµã³Ë½üËÆµ½Á½¸ö¶þÖµÏòÁ¿µÄµã³Ë£¬¼´

ÔÚ¾í»ý¼ÆËãÖУ¬ÊäÈëºÍȨֵ¾ùÁ¿»¯³ÉÁ˶þÖµ£¬Òò´Ë´«Í³µÄ³Ë·¨¼ÆËã±ä³ÉÁËÒì»ò²Ù×÷£¬¶ø·Ç¶þÖµ»¯Êý¾ÝµÄ¼ÆËãÖ»Õ¼Á˺ÜСһ²¿·Ö¡£
XNOR-NetÖÐÒ»¸öµäÐ͵ľí»ýµ¥ÔªÈçͼ6Ëùʾ£¬Ó봫ͳµ¥Ôª²»Í¬£¬¸÷Ä£¿éµÄ˳ÐòÓÐÁ˵÷Õû¡£ÎªÁ˼õÉÙ¶þÖµ»¯´øÀ´µÄ¾«¶ÈËðʧ£¬¶ÔÊäÈëÊý¾ÝÊ×ÏȽøÐÐBN¹éÒ»»¯´¦Àí£¬BinActiv²ãÓÃÓÚ¶ÔÊäÈë×ö¶þÖµ»¯£¬½Ó׎øÐжþÖµ»¯µÄ¾í»ý²Ù×÷£¬×îºó½øÐÐpooling¡£

ͼ5 BWNѵÁ·¹ý³Ì[7]
3.3 ʵÑé½á¹û
±í5 ImageNetÉ϶þÖµÍøÂçÓëAlexNet½á¹û¶Ô±È[7]

ÓëALexNetÏà±È£¬BWNÍøÂçÄܹ»´ïµ½¾«¶È»ù±¾²»±äÉõÖÁÂԺã¬XNOR-NetÓÉÓÚ¶ÔÊäÈëÒ²×öÁ˶þÖµ»¯£¬ÐÔÄÜÉÔ½µ¡£
ËÄ¡¢Distilling
DistillingËã·¨ÊÇHintonµÈÈËÔÚÂÛÎÄDistilling the Knowledge
in a Neural NetworkÖÐÌá³öµÄÒ»ÖÖÀàËÆÍøÂçÇ¨ÒÆµÄѧϰËã·¨¡£
4.1 »ù±¾Ë¼Ïë
DistillingÖ±Òë¹ýÀ´¼´ÕôÁ󣬯ä»ù±¾Ë¼ÏëÊÇͨ¹ýÒ»¸öÐÔÄܺõĴóÍøÂçÀ´½ÌÐ¡ÍøÂçѧϰ£¬´Ó¶øÊ¹µÃÐ¡ÍøÂçÄܹ»¾ß±¸¸ú´óÍøÂçÒ»ÑùµÄÐÔÄÜ£¬µ«ÕôÁóºóµÄÐ¡ÍøÂç²ÎÊý¹æÄ£Ô¶Ô¶Ð¡ÓÚÔʼ´óÍøÂ磬´Ó¶ø´ïµ½Ñ¹ËõÍøÂçµÄÄ¿µÄ¡£
ÆäÖУ¬ÑµÁ·Ð¡Ä£ÐÍ(distilled model)µÄÄ¿±êº¯ÊýÓÉÁ½²¿·Ö×é³É
1) Óë´óÄ£ÐÍ(cumbersome model)µÄsoftmaxÊä³öµÄ½»²æìØ(cross entropy)£¬³ÆÎªÈíÄ¿±ê(soft
target)¡£ÆäÖУ¬softmaxµÄ¼ÆËã¼ÓÈëÁ˳¬²ÎÊýζÈT£¬ÓÃÒÔ¿ØÖÆÊä³ö£¬¼ÆË㹫ʽ±äΪ

ζÈTÔ½´ó£¬Êä³öµÄ·Ö²¼Ô½»ººÍ£¬¸ÅÂÊzi/TԽС£¬ìØÔ½´ó£¬µ«ÈôT¹ý´ó£¬»áµ¼Ö½ϴóìØÒýÆðµÄ²»È·¶¨ÐÔÔö¼Ó£¬Ôö¼ÓÁ˲»¿ÉÇø·ÖÐÔ¡£
ÖÁÓÚΪºÎÒªÒÔsoft targetÀ´¼ÆËãËðʧ£¬×÷ÕßÈÏΪ£¬ÔÚ·ÖÀàÎÊÌâÖУ¬ÕæÖµ(groundtruth)ÊÇÒ»¸öÈ·¶¨ÐԵ쬼´one-hot
vector¡£ÒÔÊÖдÊý×Ö·ÖÀàÀ´Ëµ£¬¶ÔÓÚÒ»¸öÊý×Ö3£¬ËüµÄlabelÊÇ3µÄ¸ÅÂÊÊÇ1£¬¶øÊÇÆäËûÊýÖµµÄ¸ÅÂÊÊÇ0£¬¶ø¶ÔÓÚsoft
target£¬ËüÄܱíÕ÷labelÊÇ3µÄ¸ÅÂÊ£¬¼ÙÈçÕâ¸öÊý×ÖдµÄÏñ5£¬»¹¿ÉÒÔ¸ø³ölabelÊÇ5µÄÒ»¶¨¸ÅÂÊ£¬´Ó¶øÌṩ¸ü¶àÐÅÏ¢£¬Èç

2)ÓëÕæÖµ(groundtruth)µÄ½»²æìØ(T=1)
ѵÁ·µÄËðʧΪÉÏÊöÁ½ÏîËðʧµÄ¼ÓȨºÍ£¬Í¨³£µÚ¶þÏîҪСºÜ¶à¡£
4.2 ʵÑé½á¹û
×÷Õ߸ø³öÁËÔÚÓïÒôʶ±ðÉϵÄʵÑé½á¹û¶Ô±È£¬Èçϱí
±í6 ÕôÁóÄ£ÐÍÓëÔʼģÐ;«¶È¶Ô±È[8]

ÉϱíÏÔʾ£¬ÕôÁóºóµÄÄ£Ð͵ľ«È·¶ÈºÍµ¥×Ö´íÎóÂʺÍÓÃÓÚ²úÉúÈíÄ¿±êµÄ10¸öÄ£Ð͵ÄÐÔÄÜÏ൱£¬Ð¡Ä£Ðͳɹ¦µØÑ§µ½ÁË´óÄ£Ð͵Äʶ±ðÄÜÁ¦¡£
4.3 ËÙ¶È¿¼Á¿
DistillingµÄÌá³öÔÏȲ¢·ÇÕë¶ÔÍøÂç¼ÓËÙ£¬¶ø×îÖÕ¼ÆËãµÄЧÂÊÈÔȻȡ¾öÓÚÕôÁóÄ£Ð͵ļÆËã¹æÄ££¬µ«ÀíÂÛÉÏÕôÁóºóµÄСģÐÍÏà¶ÔÔʼ´óÄ£Ð͵ļÆËãËÙ¶ÈÔÚÒ»¶¨³Ì¶ÈÉÏ»áÓÐÌáÉý£¬µ«ËÙ¶ÈÌáÉýµÄ±ÈÀýºÍÐÔÄÜά³ÖµÄȨºâÊÇÒ»¸öÖµµÃÑо¿µÄ·½Ïò¡£
Îå¡¢MobileNet
MobileNetÊÇÓÉGoogleÌá³öµÄÕë¶ÔÒÆ¶¯¶Ë²¿ÊðµÄÇáÁ¿¼¶ÍøÂç¼Ü¹¹¡£¿¼Âǵ½Òƶ¯¶Ë¼ÆËã×ÊÔ´ÊÜÏÞÒÔ¼°ËÙ¶ÈÒªÇóÑÏ¿Á£¬MobileNetÒýÈëÁË´«Í³ÍøÂçÖÐÔÏȲÉÓõÄgroup˼Ï룬¼´ÏÞÖÆÂ˲¨Æ÷µÄ¾í»ý¼ÆËãÖ»Õë¶ÔÌØ¶¨µÄgroupÖеÄÊäÈ룬´Ó¶ø´ó´ó½µµÍÁ˾í»ý¼ÆËãÁ¿£¬ÌáÉýÁËÒÆ¶¯¶ËǰÏò¼ÆËãµÄËÙ¶È¡£
5.1 ¾í»ý·Ö½â
MobileNet½è¼øfactorized convolutionµÄ˼Ï룬½«ÆÕͨ¾í»ý²Ù×÷·Ö³ÉÁ½²¿·Ö£º
1.Depthwise Convolution
ÿ¸ö¾í»ýºËÂ˲¨Æ÷Ö»Õë¶ÔÌØ¶¨µÄÊäÈëͨµÀ½øÐоí»ý²Ù×÷£¬ÈçÏÂͼËùʾ£¬ÆäÖÐMÊÇÊäÈëͨµÀÊý£¬DKÊǾí»ýºË³ß´ç£º

ͼ7 Depthwise Convolution[9]
Depthwise convolutionµÄ¼ÆË㸴ÔÓ¶ÈΪ DKDKMDFDF£¬ÆäÖÐDFÊǾí»ý²ãÊä³öµÄÌØÕ÷ͼµÄ´óС¡£
2.Pointwise Convolution
²ÉÓÃ1x1´óСµÄ¾í»ýºË½«depthwise convolution²ãµÄ¶àͨµÀÊä³ö½øÐнáºÏ£¬ÈçÏÂͼ£¬ÆäÖÐNÊÇÊä³öͨµÀÊý£º

ͼ8 Pointwise Convolution[9]
Pointwise ConvolutionµÄ¼ÆË㸴ÔÓ¶ÈΪ MNDFDF
ÉÏÃæÁ½²½ºÏ³Ædepthwise separable convolution
±ê×¼¾í»ý²Ù×÷µÄ¼ÆË㸴ÔÓ¶ÈΪDKDKMNDFDF
Òò´Ë£¬Í¨¹ý½«±ê×¼¾í»ý·Ö½â³ÉÁ½²ã¾í»ý²Ù×÷£¬¿ÉÒÔ¼ÆËã³öÀíÂÛÉϵļÆËãЧÂÊÌáÉý±ÈÀý£º

¶ÔÓÚ3x3³ß´çµÄ¾í»ýºËÀ´Ëµ£¬depthwise separable convolutionÔÚÀíÂÛÉÏÄÜ´øÀ´Ô¼8¡«9±¶µÄЧÂÊÌáÉý¡£
5.2 Ä£Ðͼܹ¹

MobileNetµÄ¾í»ýµ¥ÔªÈçÉÏͼËùʾ£¬Ã¿¸ö¾í»ý²Ù×÷ºó¶¼½Ó×ÅÒ»¸öBN²Ù×÷ºÍReLU²Ù×÷¡£ÔÚMobileNetÖУ¬ÓÉÓÚ3x3¾í»ýºËÖ»Ó¦ÓÃÔÚdepthwise
convolutionÖУ¬Òò´Ë95%µÄ¼ÆËãÁ¿¶¼¼¯ÖÐÔÚpointwise convolution ÖеÄ1x1¾í»ýÖС£¶ø¶ÔÓÚcaffeµÈ²ÉÓþØÕóÔËËãGEMMʵÏÖ¾í»ýµÄÉî¶Èѧϰ¿ò¼Ü£¬1x1¾í»ýÎÞÐè½øÐÐim2col²Ù×÷£¬Òò´Ë¿ÉÒÔÖ±½ÓÀûÓþØÕóÔËËã¼ÓËÙ¿â½øÐпìËÙ¼ÆË㣬´Ó¶øÌáÉýÁ˼ÆËãЧÂÊ¡£
5.3 ʵÑé½á¹û
±í7 MobileNetÓëÖ÷Á÷´óÄ£ÐÍÔÚImageNetÉϾ«¶È¶Ô±È[9]

ÉϱíÏÔʾ£¬MobileNetÔÚ±£Ö¤¾«¶È²»±äµÄͬʱ£¬Äܹ»ÓÐЧµØ¼õÉÙ¼ÆËã²Ù×÷´ÎÊýºÍ²ÎÊýÁ¿£¬Ê¹µÃÔÚÒÆ¶¯¶ËʵʱǰÏò¼ÆËã³ÉΪ¿ÉÄÜ¡£
Áù¡¢ShuffleNet
ShuffleNetÊÇFace++½ñÄêÌá³öÁËһƪÓÃÓÚÒÆ¶¯¶ËǰÏò²¿ÊðµÄÍøÂç¼Ü¹¹¡£ShuffleNet»ùÓÚMobileNetµÄgroup˼Ï룬½«¾í»ý²Ù×÷ÏÞÖÆµ½Ìض¨µÄÊäÈëͨµÀ¡£¶øÓëÖ®²»Í¬µÄÊÇ£¬ShuffleNet½«ÊäÈëµÄgroup½øÐдòÉ¢£¬´Ó¶ø±£Ö¤Ã¿¸ö¾í»ýºËµÄ¸ÐÊÜÒ°Äܹ»·ÖÉ¢µ½²»Í¬groupµÄÊäÈëÖУ¬Ôö¼ÓÁËÄ£Ð͵ÄѧϰÄÜÁ¦¡£
6.1 Éè¼ÆË¼Ïë
ÎÒÃÇÖªµÀ£¬¾í»ýÖеÄgroup²Ù×÷Äܹ»´ó´ó¼õÉÙ¾í»ý²Ù×÷µÄ¼ÆËã´ÎÊý£¬¶øÕâÒ»¸Ä¶¯´øÀ´ÁËËÙ¶ÈÔöÒæºÍÐÔÄÜά³ÖÔÚMobileNetµÈÎÄÕÂÖÐÒ²µÃµ½ÁËÑéÖ¤¡£È»¶øgroup²Ù×÷Ëù´øÀ´µÄÁíÒ»¸öÎÊÌâÊÇ£ºÌض¨µÄÂ˲¨Æ÷½ö¶ÔÌØ¶¨Í¨µÀµÄÊäÈë½øÐÐ×÷Óã¬Õâ¾Í×è°ÁËͨµÀÖ®¼äµÄÐÅÏ¢Á÷´«µÝ£¬groupÊýÁ¿Ô½¶à£¬¿ÉÒÔ±àÂëµÄÐÅÏ¢¾ÍÔ½·á¸»£¬µ«Ã¿¸ögroupµÄÊäÈëͨµÀÊýÁ¿¼õÉÙ£¬Òò´Ë¿ÉÄÜÔì³Éµ¥¸ö¾í»ýÂ˲¨Æ÷µÄÍË»¯£¬ÔÚÒ»¶¨³Ì¶ÈÉÏÏ÷ÈõÁËÍøÂçÁ˱í´ïÄÜÁ¦¡£
6.2 ÍøÂç¼Ü¹¹
ÔÚ´ËÆª¹¤×÷ÖУ¬ÍøÂç¼Ü¹¹µÄÉè¼ÆÖ÷ÒªÓÐÒÔϼ¸¸ö´´Ðµ㣺
1.Ìá³öÁËÒ»¸öÀàËÆÓÚResNetµÄBottleNeckµ¥Ôª
½è¼øResNetµÄÅÔ··Ö֧˼Ï룬ShuffleNetÒ²ÒýÈëÁËÀàËÆµÄÍøÂçµ¥Ôª¡£²»Í¬µÄÊÇ£¬ÔÚstride=2µÄµ¥ÔªÖУ¬ÓÃconcat²Ù×÷´úÌæÁËadd²Ù×÷£¬ÓÃaverage
pooling´úÌæÁË1x1stride=2µÄ¾í»ý²Ù×÷£¬ÓÐЧµØ¼õÉÙÁ˼ÆËãÁ¿ºÍ²ÎÊý¡£µ¥Ôª½á¹¹Èçͼ10Ëùʾ¡£
2.Ìá³ö½«1x1¾í»ý²ÉÓÃgroup²Ù×÷»áµÃµ½¸üºÃµÄ·ÖÀàÐÔÄÜ
ÔÚMobileNetÖÐÌá¹ý£¬1x1¾í»ýµÄ²Ù×÷Õ¼¾ÝÁËÔ¼95%µÄ¼ÆËãÁ¿£¬ËùÒÔ×÷Õß½«1x1Ò²¸ü¸ÄΪgroup¾í»ý£¬Ê¹µÃÏà±ÈMobileNetµÄ¼ÆËãÁ¿´ó´ó¼õÉÙ¡£
3.Ìá³öÁ˺ËÐĵÄshuffle²Ù×÷½«²»Í¬groupÖеÄͨµÀ½øÐдòÉ¢£¬´Ó¶ø±£Ö¤²»Í¬ÊäÈëͨµÀÖ®¼äµÄÐÅÏ¢´«µÝ¡£
ShuffleNetµÄshuffle²Ù×÷Èçͼ11Ëùʾ¡£

ͼ10 ShuffleNetÍøÂçµ¥Ôª[10]

ͼ11 ²»Í¬group¼äµÄshuffle²Ù×÷[10]
6.3 ʵÑé½á¹û
±í8 ShuffleNetÓëMobileNetÔÚImageNetÉϾ«¶È¶Ô±È [10]

ÉϱíÏÔʾ£¬Ïà¶ÔÓÚMobileNet£¬ShuffleNetµÄǰÏò¼ÆËãÁ¿²»½öÓÐЧµØµÃµ½Á˼õÉÙ£¬¶øÇÒ·ÖÀà´íÎóÂÊÒ²ÓÐÃ÷ÏÔÌáÉý£¬ÑéÖ¤ÁËÍøÂçµÄ¿ÉÐÐÐÔ¡£
6.4 ËÙ¶È¿¼Á¿
×÷ÕßÔÚARMƽ̨É϶ÔÍøÂçЧÂʽøÐÐÁËÑéÖ¤£¬¼øÓÚÄÚ´æ¶ÁÈ¡ºÍÏ̵߳÷¶ÈµÈÒòËØ£¬×÷Õß·¢ÏÖÀíÂÛÉÏ4xµÄËÙ¶ÈÌáÉý¶ÔӦʵ¼Ê²¿ÊðÖÐÔ¼2.6x¡£×÷Õ߸ø³öÁËÓëÔʼAlexNetµÄËٶȶԱȣ¬ÈçÏÂ±í¡£

±í9 ShuffleNetÓëAlexNetÔÚARMƽ̨ÉÏËٶȶԱÈ
[10]
½áÊøÓï
½ü¼¸ÄêÀ´£¬³ýÁËѧÊõ½çÓ¿ÏÖµÄÖî¶àCNNÄ£ÐͼÓËÙ¹¤×÷£¬¹¤Òµ½ç¸÷´ó¹«Ë¾Ò²ÍƳöÁË×Ô¼ºµÄÒÆ¶¯¶ËǰÏò¼ÆËã¿ò¼Ü£¬ÈçGoogleµÄTensorflow¡¢FacebookµÄcaffe2ÒÔ¼°Æ»¹û½ñÄê¸ÕÍÆ³öµÄCoreML¡£ÏàÐŽáºÏ²»¶Ïµü´úÓÅ»¯µÄÍøÂç¼Ü¹¹ºÍ²»¶Ï·¢Õ¹µÄÓ²¼þ¼ÆËã¼ÓËÙ¼¼Êõ£¬Î´À´Éî¶ÈѧϰÔÚÒÆ¶¯¶ËµÄ²¿Ê𽫲»»áÊÇÒ»¸öÄÑÌâ¡£
|