Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
CNN Ä£ÐÍѹËõÓë¼ÓËÙËã·¨×ÛÊö
 
  3114  次浏览      28
 2017-12-07 
 
±à¼­ÍƼö:
±¾ÎÄÀ´Ô´csdn£¬¾í»ýÉñ¾­ÍøÂçÈÕÒæÔö³¤µÄÉî¶ÈºÍ³ß´çΪÉî¶ÈѧϰÔÚÒÆ¶¯¶ËµÄ²¿Êð´øÀ´Á˾޴óµÄÌôÕ½£¬CNNÄ£ÐÍѹËõÓë¼ÓËÙ³ÉΪÁËѧÊõ½çºÍ¹¤Òµ½ç¶¼ÖØµã¹Ø×¢µÄÑо¿ÁìÓòÖ®Ò»¡£

ǰÑÔ

×Ô´ÓAlexNetÒ»¾Ù¶áµÃILSVRC 2012 ImageNetͼÏñ·ÖÀྺÈüµÄ¹Ú¾üºó£¬¾í»ýÉñ¾­ÍøÂ磨CNN£©µÄÈȳ±±ãϯ¾íÁËÕû¸ö¼ÆËã»úÊÓ¾õÁìÓò¡£CNNÄ£ÐÍ»ðËÙÌæ´úÁË´«Í³È˹¤Éè¼Æ£¨hand-crafted£©ÌØÕ÷ºÍ·ÖÀàÆ÷£¬²»½öÌṩÁËÒ»Öֶ˵½¶ËµÄ´¦Àí·½·¨£¬»¹´ó·ù¶ÈµØË¢ÐÂÁ˸÷¸öͼÏñ¾ºÈüÈÎÎñµÄ¾«¶È£¬¸üÉõÕß³¬Ô½ÁËÈËÑ۵ľ«¶È£¨LFWÈËÁ³Ê¶±ðÈÎÎñ£©¡£CNNÄ£ÐÍÔÚ²»¶Ï±Æ½ü¼ÆËã»úÊÓ¾õÈÎÎñµÄ¾«¶È¼«ÏÞµÄͬʱ£¬ÆäÉî¶ÈºÍ³ß´çÒ²Ôڳɱ¶Ôö³¤¡£

±í1 ¼¸ÖÖ¾­µäÄ£Ð͵ijߴ磬¼ÆËãÁ¿ºÍ²ÎÊýÊýÁ¿¶Ô±È

ËæÖ®¶øÀ´µÄÊÇÒ»¸öºÜÞÏÞεij¡¾°£ºÈç´Ë¾Þ´óµÄÄ£ÐÍÖ»ÄÜÔÚÓÐÏÞµÄÆ½Ì¨ÏÂʹÓ㬸ù±¾ÎÞ·¨ÒÆÖ²µ½Òƶ¯¶ËºÍǶÈëʽоƬµ±ÖС£¾ÍËãÏëͨ¹ýÍøÂç´«Ê䣬µ«½Ï¸ßµÄ´ø¿íÕ¼ÓÃÒ²ÈúܶàÓû§Íû¶øÉúη¡£ÁíÒ»·½Ã棬´ó³ß´çµÄÄ£ÐÍÒ²¶ÔÉ豸¹¦ºÄºÍÔËÐÐËÙ¶È´øÀ´Á˾޴óµÄÌôÕ½¡£Òò´ËÕâÑùµÄÄ£Ð;àÀëʵÓû¹ÓÐÒ»¶Î¾àÀë¡£

ÔÚÕâÑùµÄÇéÐÎÏ£¬Ä£ÐÍСÐÍ»¯Óë¼ÓËÙ³ÉÁËØ½´ý½â¾öµÄÎÊÌâ¡£ÆäʵÔçÆÚ¾ÍÓÐѧÕßÌá³öÁËһϵÁÐCNNÄ£ÐÍѹËõ·½·¨£¬°üÀ¨È¨Öµ¼ôÖµ£¨prunning£©ºÍ¾ØÕóSVD·Ö½âµÈ£¬µ«Ñ¹ËõÂʺÍЧÂÊ»¹Ô¶²»ÄÜÁîÈËÂúÒâ¡£

½üÄêÀ´£¬¹ØÓÚÄ£ÐÍСÐÍ»¯µÄËã·¨´ÓѹËõ½Ç¶ÈÉÏ¿ÉÒÔ´óÖ·ÖΪÁ½Àࣺ´ÓÄ£ÐÍÈ¨ÖØÊýÖµ½Ç¶ÈѹËõºÍ´ÓÍøÂç¼Ü¹¹½Ç¶ÈѹËõ¡£ÁíÒ»·½Ã棬´Ó¼æ¹Ë¼ÆËãËÙ¶È·½Ã棬ÓÖ¿ÉÒÔ»®·ÖΪ£º½öѹËõ³ß´çºÍѹËõ³ß´çµÄͬʱÌáÉýËÙ¶È¡£

±¾ÎÄÖ÷ÒªÌÖÂÛÈçϼ¸Æª´ú±íÐÔµÄÎÄÕºͷ½·¨£¬°üÀ¨SqueezeNet[5]¡¢Deep Compression[6]¡¢XNorNet[7]¡¢Distilling[8]¡¢MobileNet[9]ºÍShuffleNet[10]£¬Ò²¿É°´ÕÕÉÏÊö·½·¨½øÐдóÖ·ÖÀࣺ

±í2 ¼¸ÖÖ¾­µäѹËõ·½·¨¼°¶Ô±È

Ò»¡¢SqueezeNet

1.1 Éè¼ÆË¼Ïë

SqueezeNetÊÇF. N. Iandola,S.HanµÈÈËÓÚ2016ÄêµÄÂÛÎÄ¡¶SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5MB model size¡·ÖÐÌá³öµÄÒ»¸öСÐÍ»¯µÄÍøÂçÄ£Ðͽṹ£¬¸ÃÍøÂçÄÜÔÚ±£Ö¤²»Ëðʧ¾«¶ÈµÄͬʱ£¬½«Ô­Ê¼AlexNetѹËõÖÁÔ­À´µÄ510±¶×óÓÒ£¨< 0.5MB£©¡£

SqueezeNetµÄºËÐÄÖ¸µ¼Ë¼ÏëÊÇ¡ª¡ªÔÚ±£Ö¤¾«¶ÈµÄͬʱʹÓÃ×îÉٵIJÎÊý¡£

¶øÕâÒ²ÊÇËùÓÐÄ£ÐÍѹËõ·½·¨µÄÒ»¸öÖÕ¼«Ä¿±ê¡£

»ùÓÚÕâ¸ö˼Ï룬SqueezeNetÌá³öÁË3µãÍøÂç½á¹¹Éè¼Æ²ßÂÔ£º

²ßÂÔ 1.½«3x3¾í»ýºËÌæ»»Îª1x1¾í»ýºË¡£

ÕâÒ»²ßÂԺܺÃÀí½â£¬ÒòΪ1¸ö1x1¾í»ýºËµÄ²ÎÊýÊÇ3x3¾í»ýºË²ÎÊýµÄ1/9£¬ÕâÒ»¸Ä¶¯ÀíÂÛÉÏ¿ÉÒÔ½«Ä£ÐͳߴçѹËõ9±¶¡£

²ßÂÔ 2.¼õСÊäÈëµ½3x3¾í»ýºËµÄÊäÈëͨµÀÊý¡£

ÎÒÃÇÖªµÀ£¬¶ÔÓÚÒ»¸ö²ÉÓÃ3x3¾í»ýºËµÄ¾í»ý²ã£¬¸Ã²ãËùÓоí»ý²ÎÊýµÄÊýÁ¿£¨²»¿¼ÂÇÆ«Öã©Îª£º

ʽÖУ¬NÊǾí»ýºËµÄÊýÁ¿£¬Ò²¼´Êä³öͨµÀÊý£¬CÊÇÊäÈëͨµÀÊý¡£

ʽÖУ¬NÊǾí»ýºËµÄÊýÁ¿£¬Ò²¼´Êä³öͨµÀÊý£¬CÊÇÊäÈëͨµÀÊý¡£

Òò´Ë£¬ÎªÁ˱£Ö¤¼õÐ¡ÍøÂç²ÎÊý£¬²»½ö½öÐèÒª¼õÉÙ3x3¾í»ýºËµÄÊýÁ¿£¬»¹Ðè¼õÉÙÊäÈëµ½3x3¾í»ýºËµÄÊäÈëͨµÀÊýÁ¿£¬¼´Ê½ÖÐCµÄÊýÁ¿¡£

²ßÂÔ 3.¾¡¿ÉÄܵĽ«½µ²ÉÑù·ÅÔÚÍøÂçºóÃæµÄ²ãÖС£

ÔÚ¾í»ýÉñ¾­ÍøÂçÖУ¬Ã¿²ãÊä³öµÄÌØÕ÷ͼ£¨feature map£©ÊÇ·ñϲÉÑùÊÇÓɾí»ý²ãµÄ²½³¤»òÕ߳ػ¯²ã¾ö¶¨µÄ¡£¶øÒ»¸öÖØÒªµÄ¹ÛµãÊÇ£º·Ö±æÂÊÔ½´óµÄÌØÕ÷ͼ£¨ÑÓ³Ù½µ²ÉÑù£©¿ÉÒÔ´øÀ´¸ü¸ßµÄ·ÖÀྫ¶È£¬¶øÕâÒ»¹Ûµã´ÓÖ±¾õÉÏÒ²¿ÉÒԺܺÃÀí½â£¬ÒòΪ·Ö±æÂÊÔ½´óµÄÊäÈëÄܹ»ÌṩµÄÐÅÏ¢¾ÍÔ½¶à¡£

ÉÏÊöÈý¸ö²ßÂÔÖУ¬Ç°Á½¸ö²ßÂÔ¶¼ÊÇÕë¶ÔÈçºÎ½µµÍ²ÎÊýÊýÁ¿¶øÉè¼ÆµÄ£¬×îºóÒ»¸öÖ¼ÔÚ×î´ó»¯ÍøÂ羫¶È¡£

1.2 ÍøÂç¼Ü¹¹

»ùÓÚÒÔÉÏÈý¸ö²ßÂÔ£¬×÷ÕßÌá³öÁËÒ»¸öÀàËÆinceptionµÄÍøÂçµ¥Ôª½á¹¹£¬È¡ÃûΪfire module¡£Ò»¸öfire module °üº¬Ò»¸ösqueeze ¾í»ý²ã£¨Ö»°üº¬1x1¾í»ýºË£©ºÍÒ»¸öexpand¾í»ý²ã£¨°üº¬1x1ºÍ3x3¾í»ýºË£©¡£ÆäÖУ¬squeeze²ã½è¼øÁËinceptionµÄ˼Ï룬ÀûÓÃ1x1¾í»ýºËÀ´½µµÍÊäÈëµ½expand²ãÖÐ3x3¾í»ýºËµÄÊäÈëͨµÀÊý¡£Èçͼ1Ëùʾ¡£

ÆäÖУ¬¶¨Òåsqueeze²ãÖÐ1x1¾í»ýºËµÄÊýÁ¿ÊÇs1x1£¬ÀàËÆµÄ£¬expand²ãÖÐ1x1¾í»ýºËµÄÊýÁ¿ÊÇe1x1£¬ 3x3¾í»ýºËµÄÊýÁ¿ÊÇe3x3¡£Áîs1x1 < e1x1+ e3x3´Ó¶ø±£Ö¤ÊäÈëµ½3x3µÄÊäÈëͨµÀÊý¼õС¡£SqueezeNetµÄÍøÂç½á¹¹ÓÉÈô¸É¸ö fire module ×é³É£¬ÁíÍâÎÄÕ»¹¸ø³öÁËһЩ¼Ü¹¹Éè¼ÆÉϵÄϸ½Ú£º

1.ΪÁ˱£Ö¤1x1¾í»ýºËºÍ3x3¾í»ýºË¾ßÓÐÏàͬ´óСµÄÊä³ö£¬3x3¾í»ýºË²ÉÓÃ1ÏñËØµÄzero-paddingºÍ²½³¤

2.squeeze²ãºÍexpand²ã¾ù²ÉÓÃRELU×÷Ϊ¼¤»îº¯Êý

3.ÔÚfire9ºó²ÉÓÃ50%µÄdropout

4.ÓÉÓÚÈ«Á¬½Ó²ãµÄ²ÎÊýÊýÁ¿¾Þ´ó£¬Òò´Ë½è¼øNIN[11]µÄ˼Ï룬ȥ³ýÁËÈ«Á¬½Ó²ã¶ø¸ÄÓÃglobal average pooling¡£

1.3 ʵÑé½á¹û

±í3 ²»Í¬Ñ¹Ëõ·½·¨ÔÚImageNetÉϵĶԱÈʵÑé½á¹û[5]

ÉϱíÏÔʾ£¬Ïà±È´«Í³µÄѹËõ·½·¨£¬SqueezeNetÄÜÔÚ±£Ö¤¾«¶È²»Ëð£¨ÉõÖÁÂÔÓÐÌáÉý£©µÄÇé¿öÏ£¬´ïµ½×î´óµÄѹËõÂÊ£¬½«Ô­Ê¼AlexNet´Ó240MBѹËõÖÁ4.8MB£¬¶ø½áºÏDeep Compressionºó¸üÄÜ´ïµ½0.47MB£¬ÍêÈ«Âú×ãÁËÒÆ¶¯¶ËµÄ²¿ÊðºÍµÍ´ø¿íÍøÂçµÄ´«Êä¡£

´ËÍ⣬×÷Õß»¹½è¼øResNet˼Ï룬¶ÔÔ­Ê¼ÍøÂç½á¹¹×öÁËÐ޸ģ¬Ôö¼ÓÁËÅÔ··ÖÖ§£¬½«·ÖÀྫ¶ÈÌáÉýÁËÔ¼3%¡£

1.4 ËÙ¶È¿¼Á¿

¾¡¹ÜÎÄÕÂÖ÷ÒªÒÔѹËõÄ£ÐͳߴçΪĿ±ê£¬µ«ÎãÓ¹ÖÃÒɵÄÒ»µãÊÇ£¬SqueezeNetÔÚÍøÂç½á¹¹ÖдóÁ¿²ÉÓÃ1x1ºÍ3x3¾í»ýºËÊÇÓÐÀûÓÚËٶȵÄÌáÉýµÄ£¬¶ÔÓÚÀàËÆcaffeÕâÑùµÄÉî¶Èѧϰ¿ò¼Ü£¬ÔÚ¾í»ý²ãµÄǰÏò¼ÆËãÖУ¬²ÉÓÃ1x1¾í»ýºË¿É±ÜÃâ¶îÍâµÄim2col²Ù×÷£¬¶øÖ±½ÓÀûÓÃgemm½øÐоØÕó¼ÓËÙÔËË㣬Òò´Ë¶ÔËٶȵÄÓÅ»¯ÊÇÓÐÒ»¶¨µÄ×÷Óõġ£È»¶ø£¬ÕâÖÖÌáËÙµÄ×÷ÓÃÈÔÈ»ÊÇÓÐÏ޵ģ¬ÁíÍ⣬SqueezeNet²ÉÓÃÁË9¸öfire moduleºÍÁ½¸ö¾í»ý²ã£¬Òò´ËÈÔÐèÒª½øÐдóÁ¿³£¹æ¾í»ý²Ù×÷£¬ÕâÒ²ÊÇÓ°ÏìËٶȽøÒ»²½ÌáÉýµÄÆ¿¾±¡£

¶þ¡¢Deep Compression

Deep Compression³ö×ÔS.Han 2016 ICLRµÄһƪÂÛÎÄ¡¶Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding¡·¡£¸ÃÎÄÕ»ñµÃÁËICLR 2016µÄ×î¼ÑÂÛÎĽ±£¬Í¬Ê±Ò²¾ßÓÐÀï³Ì±®Ê½µÄÒâÒ壬ÒýÁìÁËCNNÄ£ÐÍСÐÍ»¯Óë¼ÓËÙÑо¿·½ÏòµÄпñ³±£¬Ê¹µÃÕâÒ»ÁìÓò½üÁ½ÄêÀ´Ó¿ÏÖ³öÁË´óÁ¿µÄÓÅÐ㹤×÷ÓëÎÄÕ¡£

2.1 Ëã·¨Á÷³Ì

ÓëÇ°ÃæµÄ¡°¼Ü¹¹Ñ¹ËõÅÉ¡±µÄSqueezeNet²»Í¬£¬Deep CompressionÊÇÊôÓÚ¡°È¨ÖµÑ¹ËõÅÉ¡±µÄ¡£¶øÁ½ÆªÎÄÕ¾ù³ö×ÔS.HanÍŶӣ¬Òò´ËÁ½ÖÖ·½·¨½áºÏ£¬Ë«½£ºÏèµ£¬¸üÊÇÄÜ´ïµ½µÇ·åÔ켫µÄѹËõЧ¹û¡£ÕâһʵÑé½á¹ûÒ²ÔÚÉϱíÖеõ½ÑéÖ¤¡£

Deep CompressionµÄËã·¨Á÷³Ì°üº¬Èý²½£¬Èçͼ2Ëùʾ£º

1¡¢Pruning(Ȩֵ¼ôÖ¦)

¼ôÖ¦µÄ˼ÏëÆäʵÔçÒÑÔÚÔçÆÚÂÛÎÄÖпÉÒÔ¿ú¼û£¬LeCunµÈÈËÔø¾­¾ÍÀûÓüôÖ¦À´Ï¡ÊèÍøÂ磬¼õС¹ýÄâºÏµÄ·çÏÕ£¬ÌáÉýÍøÂç·º»¯ÐÔ¡£

ͼ3ÊÇMNISTÉÏѵÁ·µÃµ½µÄLeNet conv1¾í»ý²ãÖеIJÎÊý·Ö²¼£¬¿ÉÒÔ¿´³ö£¬´ó²¿·ÖȨֵ¼¯ÖÐÔÚ0´¦¸½½ü£¬¶ÔÍøÂçµÄ¹±Ï×½ÏС£¬ÔÚ¼ôÖµÖУ¬½«0Öµ¸½½üµÄ½ÏСµÄȨֵÖÃ0£¬Ê¹ÕâЩȨֵ²»±»¼¤»î£¬´Ó¶ø×ÅÖØÑµÁ·Ê£ÏµķÇÁãȨֵ£¬×îÖÕÔÚ±£Ö¤ÍøÂ羫¶È²»±äµÄÇé¿öÏ´ﵽѹËõ³ß´çµÄÄ¿µÄ¡£

ʵÑé·¢ÏÖÄ£ÐͶԼôÖ¦¸üÃô¸Ð£¬Òò´ËÔÚ¼ôֵʱ½¨ÒéÖð²ãµü´úÐÞ¼ô£¬ÁíÍâÿ²ãµÄ¼ôÖ¦±ÈÀýÈçºÎ×Ô¶¯Ñ¡È¡ÈÔÈ»ÊÇÒ»¸öÖµµÃÉîÈëÑо¿µÄ¿ÎÌâ¡£

2¡¢Quantization (ȨֵÁ¿»¯)

´Ë´¦µÄȨֵÁ¿»¯»ùÓÚȨֵ¾ÛÀ࣬½«Á¬Ðø·Ö²¼µÄȨֵÀëÉ¢»¯£¬´Ó¶ø¼õСÐèÒª´æ´¢µÄȨֵÊýÁ¿¡£

1.³õʼ»¯¾ÛÀàÖÐÐÄ£¬ÊµÑéÖ¤Ã÷ÏßÐÔ³õʼ»¯Ð§¹û×îºÃ£»

2.ÀûÓÃk-meansËã·¨½øÐоÛÀ࣬½«È¨Öµ»®·Öµ½²»Í¬µÄclusterÖУ»

3.ÔÚǰÏò¼ÆËãʱ£¬Ã¿¸öȨֵÓÉÆä¾ÛÀàÖÐÐıíʾ£»

4.ÔÚºóÏò¼ÆËãʱ£¬Í³¼ÆÃ¿¸öclusterÖеÄÌݶȺͽ«Æä·´´«¡£

ͼ4 ȨֵÁ¿»¯Ç°ÏòºÍºóÏò¼ÆËã¹ý³Ì[6]

3¡¢Huffman encoding(»ô·òÂü±àÂë)

»ô·òÂü±àÂë²ÉÓñ䳤±àÂ뽫ƽ¾ù±àÂ볤¶È¼õС£¬½øÒ»²½Ñ¹ËõÄ£Ðͳߴ硣

2.2 Ä£ÐÍ´æ´¢

ǰÊöµÄ¼ôÖ¦ºÍÁ¿»¯¶¼ÊÇΪÁËʵÏÖÄ£Ð͵ĸü½ôÖµÄѹËõ£¬ÒÔʵÏÖ¼õСģÐͳߴçµÄÄ¿µÄ¡£

1.¶ÔÓÚ¼ôÖ¦ºóµÄÄ£ÐÍ£¬ÓÉÓÚÿ²ã´óÁ¿²ÎÊýΪ0£¬ºóÐøÖ»Ð轫·ÇÁãÖµ¼°ÆäÏÂ±ê½øÐд洢£¬ÎÄÕÂÖвÉÓÃCSR£¨Compressed Sparse Row£©À´½øÐд洢£¬ÕâÒ»²½¿ÉÒÔʵÏÖ9x~13xµÄѹËõÂÊ¡£

2.¶ÔÓÚÁ¿»¯ºóµÄÄ£ÐÍ£¬Ã¿¸öȨֵ¶¼ÓÉÆä¾ÛÀàÖÐÐıíʾ£¨¶ÔÓÚ¾í»ý²ã£¬¾ÛÀàÖÐÐÄÉèΪ256¸ö£¬¶ÔÓÚÈ«Á¬½Ó²ã£¬¾ÛÀàÖÐÐÄÉèΪ32¸ö£©£¬Òò´Ë¿ÉÒÔ¹¹Ôì¶ÔÓ¦µÄÂëÊéºÍϱ꣬´ó´ó¼õÉÙÁËÐèÒª´æ´¢µÄÊý¾ÝÁ¿£¬´Ë²½ÄÜʵÏÖÔ¼3xµÄѹËõÂÊ¡£

3.×îºó¶ÔÉÏÊöѹËõºóµÄÄ£ÐͽøÒ»²½²ÉÓñ䳤»ô·òÂü±àÂ룬ʵÏÖÔ¼1xµÄѹËõÂÊ¡£

2.3 ʵÑé½á¹û

±í4 ²»Í¬ÍøÂç²ÉÓÃDeep CompressionºóµÄѹËõÂÊ[6]

ͨ¹ýSqueezeNet+Deep Compression,¿ÉÒÔ½«Ô­Ê¼240MµÄAlexNetѹËõÖÁ0.47M£¬ÊµÏÖÔ¼510xµÄѹËõÂÊ¡£

2.4 ËÙ¶È¿¼Á¿

¿ÉÒÔ¿´³ö£¬Deep CompressionµÄÖ÷ÒªÉè¼ÆÊÇÕë¶ÔÍøÂç´æ´¢³ß´çµÄѹËõ£¬µ«ÔÚǰÏòʱ£¬Èç¹û½«´æ´¢Ä£ÐͶÁÈëÕ¹¿ªºó£¬²¢Ã»ÓдøÀ´¸ü´óµÄËÙ¶ÈÌáÉý¡£Òò´ËSong H.µÈÈËרÃÅÕë¶ÔѹËõºóµÄÄ£ÐÍÉè¼ÆÁËÒ»Ì×»ùÓÚFPGAµÄÓ²¼þǰÏò¼ÓËÙ¿ò¼ÜEIE[12]£¬ÓÐÐËȤµÄ¿ÉÒÔÑо¿Ò»Ï¡£

Èý¡¢XNorNet

¶þÖµÍøÂçÒ»Ö±ÊÇÄ£ÐÍѹËõºÍ¼ÓËÙÁìÓò¾­¾Ã²»Ë¥µÄÑо¿¿ÎÌâÖ®Ò»¡£½«Ô­Ê¼32λ¸¡µãÐ͵ÄȨֵѹËõµ½1±ÈÌØ£¬ÈçºÎ×î´ó³Ì¶ÈµØ¼õСÐÔÄÜËðʧ¾Í³ÉΪÁËÑо¿µÄ¹Ø¼ü¡£

´ËƪÂÛÎÄÖ÷ÒªÓÐÒÔϼ¸¸ö¹±Ï×£º

1.Ìá³öÁËÒ»¸öBWN£¨Binary-Weight-Network£©ºÍXNOR-Network£¬Ç°ÕßÖ»¶ÔÍøÂç²ÎÊý×ö¶þÖµ»¯£¬´øÀ´Ô¼32xµÄ´æ´¢Ñ¹ËõºÍ2xµÄËÙ¶ÈÌáÉý£¬¶øºóÕß¶ÔÍøÂçÊäÈëºÍ²ÎÊý¶¼×öÁ˶þÖµ»¯£¬ÔÚʵÏÖ32x´æ´¢Ñ¹ËõµÄͬʱ´øÁË58xµÄËÙ¶ÈÌáÉý£»

2.Ìá³öÁËÒ»¸öÐÂÐͶþÖµ»¯È¨ÖµµÄËã·¨£»

3.µÚÒ»¸öÔÚ´ó¹æÄ£Êý¾Ý¼¯ÈçImageNetÉÏÌá½»¶þÖµ»¯ÍøÂç½á¹ûµÄ¹¤×÷£»

4.ÎÞÐèԤѵÁ·£¬¿ÉʵÏÖtraining from scratch¡£

3.1 BWN

¼´×îÓŵĶþÖµ»¯Â˲¨Æ÷ÕÅÁ¿B¼´ÎªÔ­Ê¼²ÎÊýµÄ·ûºÅº¯Êý£¬×îÓŵij߶ÈÒò×ÓΪÿ¸öÂ˲¨Æ÷ȨֵµÄ¾ø¶ÔÖµµÄ¾ùÖµ¡£

ѵÁ·Ëã·¨Èçͼ5Ëùʾ£¬ÖµµÃ×¢ÒâµÄÊÇ£¬Ö»ÓÐÔÚǰÏò¼ÆËãºÍºóÏò´«²¥Ê±Ê¹ÓöþÖµ»¯ºóµÄȨֵ£¬ÔÚ¸üвÎÊýʱÒÀȻʹÓÃԭʼ²ÎÊý£¬ÕâÊÇÒòΪÈç¹ûʹÓöþÖµ»¯ºóµÄ²ÎÊý»áµ¼ÖºÜСµÄÌݶÈϽµ£¬´Ó¶øÊ¹µÃѵÁ·ÎÞ·¨ÊÕÁ²¡£

3.2 XNOR-Net

ÔÚXNORÍøÂçÖУ¬ÓÅ»¯µÄÄ¿±êÊǽ«Á½¸öʵÊýÏòÁ¿µÄµã³Ë½üËÆµ½Á½¸ö¶þÖµÏòÁ¿µÄµã³Ë£¬¼´

ÔÚ¾í»ý¼ÆËãÖУ¬ÊäÈëºÍȨֵ¾ùÁ¿»¯³ÉÁ˶þÖµ£¬Òò´Ë´«Í³µÄ³Ë·¨¼ÆËã±ä³ÉÁËÒì»ò²Ù×÷£¬¶ø·Ç¶þÖµ»¯Êý¾ÝµÄ¼ÆËãÖ»Õ¼Á˺ÜСһ²¿·Ö¡£

XNOR-NetÖÐÒ»¸öµäÐ͵ľí»ýµ¥ÔªÈçͼ6Ëùʾ£¬Ó봫ͳµ¥Ôª²»Í¬£¬¸÷Ä£¿éµÄ˳ÐòÓÐÁ˵÷Õû¡£ÎªÁ˼õÉÙ¶þÖµ»¯´øÀ´µÄ¾«¶ÈËðʧ£¬¶ÔÊäÈëÊý¾ÝÊ×ÏȽøÐÐBN¹éÒ»»¯´¦Àí£¬BinActiv²ãÓÃÓÚ¶ÔÊäÈë×ö¶þÖµ»¯£¬½Ó׎øÐжþÖµ»¯µÄ¾í»ý²Ù×÷£¬×îºó½øÐÐpooling¡£

ͼ5 BWNѵÁ·¹ý³Ì[7]

3.3 ʵÑé½á¹û

±í5 ImageNetÉ϶þÖµÍøÂçÓëAlexNet½á¹û¶Ô±È[7]

ÓëALexNetÏà±È£¬BWNÍøÂçÄܹ»´ïµ½¾«¶È»ù±¾²»±äÉõÖÁÂԺã¬XNOR-NetÓÉÓÚ¶ÔÊäÈëÒ²×öÁ˶þÖµ»¯£¬ÐÔÄÜÉÔ½µ¡£

ËÄ¡¢Distilling

DistillingËã·¨ÊÇHintonµÈÈËÔÚÂÛÎÄDistilling the Knowledge in a Neural NetworkÖÐÌá³öµÄÒ»ÖÖÀàËÆÍøÂçÇ¨ÒÆµÄѧϰËã·¨¡£

4.1 »ù±¾Ë¼Ïë

DistillingÖ±Òë¹ýÀ´¼´ÕôÁ󣬯ä»ù±¾Ë¼ÏëÊÇͨ¹ýÒ»¸öÐÔÄܺõĴóÍøÂçÀ´½ÌÐ¡ÍøÂçѧϰ£¬´Ó¶øÊ¹µÃÐ¡ÍøÂçÄܹ»¾ß±¸¸ú´óÍøÂçÒ»ÑùµÄÐÔÄÜ£¬µ«ÕôÁóºóµÄÐ¡ÍøÂç²ÎÊý¹æÄ£Ô¶Ô¶Ð¡ÓÚԭʼ´óÍøÂ磬´Ó¶ø´ïµ½Ñ¹ËõÍøÂçµÄÄ¿µÄ¡£

ÆäÖУ¬ÑµÁ·Ð¡Ä£ÐÍ(distilled model)µÄÄ¿±êº¯ÊýÓÉÁ½²¿·Ö×é³É

1) Óë´óÄ£ÐÍ(cumbersome model)µÄsoftmaxÊä³öµÄ½»²æìØ(cross entropy)£¬³ÆÎªÈíÄ¿±ê(soft target)¡£ÆäÖУ¬softmaxµÄ¼ÆËã¼ÓÈëÁ˳¬²ÎÊýζÈT£¬ÓÃÒÔ¿ØÖÆÊä³ö£¬¼ÆË㹫ʽ±äΪ

ζÈTÔ½´ó£¬Êä³öµÄ·Ö²¼Ô½»ººÍ£¬¸ÅÂÊzi/TԽС£¬ìØÔ½´ó£¬µ«ÈôT¹ý´ó£¬»áµ¼Ö½ϴóìØÒýÆðµÄ²»È·¶¨ÐÔÔö¼Ó£¬Ôö¼ÓÁ˲»¿ÉÇø·ÖÐÔ¡£

ÖÁÓÚΪºÎÒªÒÔsoft targetÀ´¼ÆËãËðʧ£¬×÷ÕßÈÏΪ£¬ÔÚ·ÖÀàÎÊÌâÖУ¬ÕæÖµ(groundtruth)ÊÇÒ»¸öÈ·¶¨ÐԵ쬼´one-hot vector¡£ÒÔÊÖдÊý×Ö·ÖÀàÀ´Ëµ£¬¶ÔÓÚÒ»¸öÊý×Ö3£¬ËüµÄlabelÊÇ3µÄ¸ÅÂÊÊÇ1£¬¶øÊÇÆäËûÊýÖµµÄ¸ÅÂÊÊÇ0£¬¶ø¶ÔÓÚsoft target£¬ËüÄܱíÕ÷labelÊÇ3µÄ¸ÅÂÊ£¬¼ÙÈçÕâ¸öÊý×ÖдµÄÏñ5£¬»¹¿ÉÒÔ¸ø³ölabelÊÇ5µÄÒ»¶¨¸ÅÂÊ£¬´Ó¶øÌṩ¸ü¶àÐÅÏ¢£¬Èç

2)ÓëÕæÖµ(groundtruth)µÄ½»²æìØ(T=1)

ѵÁ·µÄËðʧΪÉÏÊöÁ½ÏîËðʧµÄ¼ÓȨºÍ£¬Í¨³£µÚ¶þÏîҪСºÜ¶à¡£

4.2 ʵÑé½á¹û

×÷Õ߸ø³öÁËÔÚÓïÒôʶ±ðÉϵÄʵÑé½á¹û¶Ô±È£¬Èçϱí

±í6 ÕôÁóÄ£ÐÍÓëԭʼģÐ;«¶È¶Ô±È[8]

ÉϱíÏÔʾ£¬ÕôÁóºóµÄÄ£Ð͵ľ«È·¶ÈºÍµ¥×Ö´íÎóÂʺÍÓÃÓÚ²úÉúÈíÄ¿±êµÄ10¸öÄ£Ð͵ÄÐÔÄÜÏ൱£¬Ð¡Ä£Ðͳɹ¦µØÑ§µ½ÁË´óÄ£Ð͵Äʶ±ðÄÜÁ¦¡£

4.3 ËÙ¶È¿¼Á¿

DistillingµÄÌá³öÔ­ÏȲ¢·ÇÕë¶ÔÍøÂç¼ÓËÙ£¬¶ø×îÖÕ¼ÆËãµÄЧÂÊÈÔȻȡ¾öÓÚÕôÁóÄ£Ð͵ļÆËã¹æÄ££¬µ«ÀíÂÛÉÏÕôÁóºóµÄСģÐÍÏà¶Ôԭʼ´óÄ£Ð͵ļÆËãËÙ¶ÈÔÚÒ»¶¨³Ì¶ÈÉÏ»áÓÐÌáÉý£¬µ«ËÙ¶ÈÌáÉýµÄ±ÈÀýºÍÐÔÄÜά³ÖµÄȨºâÊÇÒ»¸öÖµµÃÑо¿µÄ·½Ïò¡£

Îå¡¢MobileNet

MobileNetÊÇÓÉGoogleÌá³öµÄÕë¶ÔÒÆ¶¯¶Ë²¿ÊðµÄÇáÁ¿¼¶ÍøÂç¼Ü¹¹¡£¿¼Âǵ½Òƶ¯¶Ë¼ÆËã×ÊÔ´ÊÜÏÞÒÔ¼°ËÙ¶ÈÒªÇóÑÏ¿Á£¬MobileNetÒýÈëÁË´«Í³ÍøÂçÖÐÔ­ÏȲÉÓõÄgroup˼Ï룬¼´ÏÞÖÆÂ˲¨Æ÷µÄ¾í»ý¼ÆËãÖ»Õë¶ÔÌØ¶¨µÄgroupÖеÄÊäÈ룬´Ó¶ø´ó´ó½µµÍÁ˾í»ý¼ÆËãÁ¿£¬ÌáÉýÁËÒÆ¶¯¶ËǰÏò¼ÆËãµÄËÙ¶È¡£

5.1 ¾í»ý·Ö½â

MobileNet½è¼øfactorized convolutionµÄ˼Ï룬½«ÆÕͨ¾í»ý²Ù×÷·Ö³ÉÁ½²¿·Ö£º

1.Depthwise Convolution

ÿ¸ö¾í»ýºËÂ˲¨Æ÷Ö»Õë¶ÔÌØ¶¨µÄÊäÈëͨµÀ½øÐоí»ý²Ù×÷£¬ÈçÏÂͼËùʾ£¬ÆäÖÐMÊÇÊäÈëͨµÀÊý£¬DKÊǾí»ýºË³ß´ç£º

ͼ7 Depthwise Convolution[9]

Depthwise convolutionµÄ¼ÆË㸴ÔÓ¶ÈΪ DKDKMDFDF£¬ÆäÖÐDFÊǾí»ý²ãÊä³öµÄÌØÕ÷ͼµÄ´óС¡£

2.Pointwise Convolution

²ÉÓÃ1x1´óСµÄ¾í»ýºË½«depthwise convolution²ãµÄ¶àͨµÀÊä³ö½øÐнáºÏ£¬ÈçÏÂͼ£¬ÆäÖÐNÊÇÊä³öͨµÀÊý£º

ͼ8 Pointwise Convolution[9]

Pointwise ConvolutionµÄ¼ÆË㸴ÔÓ¶ÈΪ MNDFDF

ÉÏÃæÁ½²½ºÏ³Ædepthwise separable convolution

±ê×¼¾í»ý²Ù×÷µÄ¼ÆË㸴ÔÓ¶ÈΪDKDKMNDFDF

Òò´Ë£¬Í¨¹ý½«±ê×¼¾í»ý·Ö½â³ÉÁ½²ã¾í»ý²Ù×÷£¬¿ÉÒÔ¼ÆËã³öÀíÂÛÉϵļÆËãЧÂÊÌáÉý±ÈÀý£º

¶ÔÓÚ3x3³ß´çµÄ¾í»ýºËÀ´Ëµ£¬depthwise separable convolutionÔÚÀíÂÛÉÏÄÜ´øÀ´Ô¼8¡«9±¶µÄЧÂÊÌáÉý¡£

5.2 Ä£Ðͼܹ¹

MobileNetµÄ¾í»ýµ¥ÔªÈçÉÏͼËùʾ£¬Ã¿¸ö¾í»ý²Ù×÷ºó¶¼½Ó×ÅÒ»¸öBN²Ù×÷ºÍReLU²Ù×÷¡£ÔÚMobileNetÖУ¬ÓÉÓÚ3x3¾í»ýºËÖ»Ó¦ÓÃÔÚdepthwise convolutionÖУ¬Òò´Ë95%µÄ¼ÆËãÁ¿¶¼¼¯ÖÐÔÚpointwise convolution ÖеÄ1x1¾í»ýÖС£¶ø¶ÔÓÚcaffeµÈ²ÉÓþØÕóÔËËãGEMMʵÏÖ¾í»ýµÄÉî¶Èѧϰ¿ò¼Ü£¬1x1¾í»ýÎÞÐè½øÐÐim2col²Ù×÷£¬Òò´Ë¿ÉÒÔÖ±½ÓÀûÓþØÕóÔËËã¼ÓËÙ¿â½øÐпìËÙ¼ÆË㣬´Ó¶øÌáÉýÁ˼ÆËãЧÂÊ¡£

5.3 ʵÑé½á¹û

±í7 MobileNetÓëÖ÷Á÷´óÄ£ÐÍÔÚImageNetÉϾ«¶È¶Ô±È[9]

ÉϱíÏÔʾ£¬MobileNetÔÚ±£Ö¤¾«¶È²»±äµÄͬʱ£¬Äܹ»ÓÐЧµØ¼õÉÙ¼ÆËã²Ù×÷´ÎÊýºÍ²ÎÊýÁ¿£¬Ê¹µÃÔÚÒÆ¶¯¶ËʵʱǰÏò¼ÆËã³ÉΪ¿ÉÄÜ¡£

Áù¡¢ShuffleNet

ShuffleNetÊÇFace++½ñÄêÌá³öÁËһƪÓÃÓÚÒÆ¶¯¶ËǰÏò²¿ÊðµÄÍøÂç¼Ü¹¹¡£ShuffleNet»ùÓÚMobileNetµÄgroup˼Ï룬½«¾í»ý²Ù×÷ÏÞÖÆµ½Ìض¨µÄÊäÈëͨµÀ¡£¶øÓëÖ®²»Í¬µÄÊÇ£¬ShuffleNet½«ÊäÈëµÄgroup½øÐдòÉ¢£¬´Ó¶ø±£Ö¤Ã¿¸ö¾í»ýºËµÄ¸ÐÊÜÒ°Äܹ»·ÖÉ¢µ½²»Í¬groupµÄÊäÈëÖУ¬Ôö¼ÓÁËÄ£Ð͵ÄѧϰÄÜÁ¦¡£

6.1 Éè¼ÆË¼Ïë

ÎÒÃÇÖªµÀ£¬¾í»ýÖеÄgroup²Ù×÷Äܹ»´ó´ó¼õÉÙ¾í»ý²Ù×÷µÄ¼ÆËã´ÎÊý£¬¶øÕâÒ»¸Ä¶¯´øÀ´ÁËËÙ¶ÈÔöÒæºÍÐÔÄÜά³ÖÔÚMobileNetµÈÎÄÕÂÖÐÒ²µÃµ½ÁËÑéÖ¤¡£È»¶øgroup²Ù×÷Ëù´øÀ´µÄÁíÒ»¸öÎÊÌâÊÇ£ºÌض¨µÄÂ˲¨Æ÷½ö¶ÔÌØ¶¨Í¨µÀµÄÊäÈë½øÐÐ×÷Óã¬Õâ¾Í×è°­ÁËͨµÀÖ®¼äµÄÐÅÏ¢Á÷´«µÝ£¬groupÊýÁ¿Ô½¶à£¬¿ÉÒÔ±àÂëµÄÐÅÏ¢¾ÍÔ½·á¸»£¬µ«Ã¿¸ögroupµÄÊäÈëͨµÀÊýÁ¿¼õÉÙ£¬Òò´Ë¿ÉÄÜÔì³Éµ¥¸ö¾í»ýÂ˲¨Æ÷µÄÍË»¯£¬ÔÚÒ»¶¨³Ì¶ÈÉÏÏ÷ÈõÁËÍøÂçÁ˱í´ïÄÜÁ¦¡£

6.2 ÍøÂç¼Ü¹¹

ÔÚ´ËÆª¹¤×÷ÖУ¬ÍøÂç¼Ü¹¹µÄÉè¼ÆÖ÷ÒªÓÐÒÔϼ¸¸ö´´Ðµ㣺

1.Ìá³öÁËÒ»¸öÀàËÆÓÚResNetµÄBottleNeckµ¥Ôª

½è¼øResNetµÄÅÔ··Ö֧˼Ï룬ShuffleNetÒ²ÒýÈëÁËÀàËÆµÄÍøÂçµ¥Ôª¡£²»Í¬µÄÊÇ£¬ÔÚstride=2µÄµ¥ÔªÖУ¬ÓÃconcat²Ù×÷´úÌæÁËadd²Ù×÷£¬ÓÃaverage pooling´úÌæÁË1x1stride=2µÄ¾í»ý²Ù×÷£¬ÓÐЧµØ¼õÉÙÁ˼ÆËãÁ¿ºÍ²ÎÊý¡£µ¥Ôª½á¹¹Èçͼ10Ëùʾ¡£

2.Ìá³ö½«1x1¾í»ý²ÉÓÃgroup²Ù×÷»áµÃµ½¸üºÃµÄ·ÖÀàÐÔÄÜ

ÔÚMobileNetÖÐÌá¹ý£¬1x1¾í»ýµÄ²Ù×÷Õ¼¾ÝÁËÔ¼95%µÄ¼ÆËãÁ¿£¬ËùÒÔ×÷Õß½«1x1Ò²¸ü¸ÄΪgroup¾í»ý£¬Ê¹µÃÏà±ÈMobileNetµÄ¼ÆËãÁ¿´ó´ó¼õÉÙ¡£

3.Ìá³öÁ˺ËÐĵÄshuffle²Ù×÷½«²»Í¬groupÖеÄͨµÀ½øÐдòÉ¢£¬´Ó¶ø±£Ö¤²»Í¬ÊäÈëͨµÀÖ®¼äµÄÐÅÏ¢´«µÝ¡£

ShuffleNetµÄshuffle²Ù×÷Èçͼ11Ëùʾ¡£

ͼ10 ShuffleNetÍøÂçµ¥Ôª[10]

ͼ11 ²»Í¬group¼äµÄshuffle²Ù×÷[10]

6.3 ʵÑé½á¹û

±í8 ShuffleNetÓëMobileNetÔÚImageNetÉϾ«¶È¶Ô±È [10]

ÉϱíÏÔʾ£¬Ïà¶ÔÓÚMobileNet£¬ShuffleNetµÄǰÏò¼ÆËãÁ¿²»½öÓÐЧµØµÃµ½Á˼õÉÙ£¬¶øÇÒ·ÖÀà´íÎóÂÊÒ²ÓÐÃ÷ÏÔÌáÉý£¬ÑéÖ¤ÁËÍøÂçµÄ¿ÉÐÐÐÔ¡£

6.4 ËÙ¶È¿¼Á¿

×÷ÕßÔÚARMƽ̨É϶ÔÍøÂçЧÂʽøÐÐÁËÑéÖ¤£¬¼øÓÚÄÚ´æ¶ÁÈ¡ºÍÏ̵߳÷¶ÈµÈÒòËØ£¬×÷Õß·¢ÏÖÀíÂÛÉÏ4xµÄËÙ¶ÈÌáÉý¶ÔӦʵ¼Ê²¿ÊðÖÐÔ¼2.6x¡£×÷Õ߸ø³öÁËÓëԭʼAlexNetµÄËٶȶԱȣ¬ÈçÏÂ±í¡£

±í9 ShuffleNetÓëAlexNetÔÚARMƽ̨ÉÏËÙ¶È¶Ô±È [10]

½áÊøÓï

½ü¼¸ÄêÀ´£¬³ýÁËѧÊõ½çÓ¿ÏÖµÄÖî¶àCNNÄ£ÐͼÓËÙ¹¤×÷£¬¹¤Òµ½ç¸÷´ó¹«Ë¾Ò²ÍƳöÁË×Ô¼ºµÄÒÆ¶¯¶ËǰÏò¼ÆËã¿ò¼Ü£¬ÈçGoogleµÄTensorflow¡¢FacebookµÄcaffe2ÒÔ¼°Æ»¹û½ñÄê¸ÕÍÆ³öµÄCoreML¡£ÏàÐŽáºÏ²»¶Ïµü´úÓÅ»¯µÄÍøÂç¼Ü¹¹ºÍ²»¶Ï·¢Õ¹µÄÓ²¼þ¼ÆËã¼ÓËÙ¼¼Êõ£¬Î´À´Éî¶ÈѧϰÔÚÒÆ¶¯¶ËµÄ²¿Ê𽫲»»áÊÇÒ»¸öÄÑÌâ¡£

   
3114 ´Îä¯ÀÀ       28
Ïà¹ØÎÄÕÂ

»ùÓÚͼ¾í»ýÍøÂçµÄͼÉî¶Èѧϰ
×Ô¶¯¼ÝÊ»ÖеÄ3DÄ¿±ê¼ì²â
¹¤Òµ»úÆ÷ÈË¿ØÖÆÏµÍ³¼Ü¹¹½éÉÜ
ÏîĿʵս£ºÈçºÎ¹¹½¨ÖªÊ¶Í¼Æ×
 
Ïà¹ØÎĵµ

5GÈ˹¤ÖÇÄÜÎïÁªÍøµÄµäÐÍÓ¦ÓÃ
Éî¶ÈѧϰÔÚ×Ô¶¯¼ÝÊ»ÖеÄÓ¦ÓÃ
ͼÉñ¾­ÍøÂçÔÚ½»²æÑ§¿ÆÁìÓòµÄÓ¦ÓÃÑо¿
ÎÞÈË»úϵͳԭÀí
Ïà¹Ø¿Î³Ì

È˹¤ÖÇÄÜ¡¢»úÆ÷ѧϰ&TensorFlow
»úÆ÷ÈËÈí¼þ¿ª·¢¼¼Êõ
È˹¤ÖÇÄÜ£¬»úÆ÷ѧϰºÍÉî¶Èѧϰ
ͼÏñ´¦ÀíËã·¨·½·¨Óëʵ¼ù