±à¼ÍƼö: |
±¾ÎÄÀ´×Ôinfoq.com£¬±¾ÎÄÃèÊöµÄÊǶÔGitHubÏÈǰʵÏֵĸ߿ÉÓÃÐÔ£¨HA£©½â¾ö·½°¸µÄµü´úºÍ¸Ä½ø¡£ |
|
GitHubʹÓÃMySQL×÷ΪËùÓзÇgitÏîÄ¿µÄÖ÷ÒªÊý¾Ý´æ´¢£¬Òò´ËMySQLµÄ¿ÉÓÃÐÔ¶ÔÓÚGitHubµÄÔËάÀ´ËµÖÁ¹ØÖØÒª¡£Õ¾µã±¾Éí¡¢GitHubµÄAPI¡¢Éí·ÝÑéÖ¤µÈ¶¼ÐèÒªÊý¾Ý¿â·ÃÎÊ¡£ÎÒÃÇÔËÐжà¸öMySQL¼¯ÈºÀ´·þÎñÎÒÃǵIJ»Í¬·þÎñºÍÈÎÎñ¡£ÎÒÃǵļ¯ÈºÊ¹ÓþµäµÄÖ÷-¸±ÉèÖã¬ÆäÖм¯ÈºµÄµ¥¸ö½Úµã£¨Ö÷½Úµã£©Äܹ»½ÓÊÜд²Ù×÷¡£ÆäËü¼¯Èº½Úµã£¨¸±½Úµã£©Òì²½¸üÐÂÖ÷½ÚµãµÄ±ä¸ü²¢·þÎñÎÒÃǵĶÁÁ÷Á¿¡£
Ö÷½ÚµãµÄ¿ÉÓÃÐÔÌØ±ðµØÖØÒª¡£Ö÷½Úµã²»¿ÉÓÃʱ£¬¼¯Èº¾Í²»ÄܽÓÊÜд²Ù×÷£ºÈκÎÐèÒª³Ö¾Ã»¯µÄд²Ù×÷¶¼²»Äܱ»³Ö¾Ã»¯¡£Èκδ«ÈëµÄ±ä¸ü£¬ÀýÈçÌá½»´úÂë¡¢ÌáÎÊÌâ¡¢Óû§´´½¨¡¢´úÂëÉó²é¡¢Ð½¨´úÂë¿âµÈµÈ£¬¶¼»áʧ°Ü¡£
ΪÁËÖ§³Öд²Ù×÷£¬ÎÒÃÇÏÔÈ»ÐèÒªÓÐÒ»¸ö¿ÉÓõÄд½Úµã£¬¼´¼¯ÈºµÄÖ÷½Úµã¡£µ«Í¬ÑùÖØÒªµÄÊÇ£¬ÎÒÃÇÐèÒªÄܹ»Ê¶±ð£¬»òÕß·¢ÏÖ£¬ÄǸö½Úµã¡£
Óöµ½Ò»¸ö¹ÊÕÏʱ£¬±ÈÈçÖ÷½Úµã±ÀÀ£µÄ³¡¾°£¬ÎÒÃDZØÐëÈ·±£´æÔÚÒ»¸öеÄÖ÷½Úµã£¬²¢ÇÒÄܹ»¿ìËÙͨ¸æÆäÉí·Ý¡£¼ì²â¹ÊÕÏ¡¢ÔËÐйÊÕϻָ´ÒÔ¼°Í¨¸æÐÂÖ÷½ÚµãÉí·ÝËù»¨·ÑµÄʱ¼ä×é³ÉÁË×Üå´»úʱ¼ä¡£
±¾ÎIJûÊöÁËGitHubµÄMySQL¸ß¿ÉÓÃÐÔºÍÖ÷·þÎñ·¢ÏÖ½â¾ö·½°¸£¬Õâ¸ö·½°¸Ê¹µÃÎÒÃÇÄܹ»¿É¿¿µØ½øÐпçÊý¾ÝÖÐÐÄÔËά¡¢¿Ë·þÊý¾ÝÖÐÐĸôÀëµÄÓ°Ï첢ʵÏÖ¹ÊÕÏʱµÄ¶Ìå´»úʱ¼ä¡£
¸ß¿ÉÓÃÐÔÄ¿±ê
±¾ÎÄÃèÊöµÄ½â¾ö·½°¸ÊǶÔGitHubÏÈǰʵÏֵĸ߿ÉÓÃÐÔ£¨HA£©½â¾ö·½°¸µÄµü´úºÍ¸Ä½ø¡£Ëæ×ÅÎÒÃǹæÄ£µÄÀ©´ó£¬ÎÒÃǵÄMySQL
HA²ßÂÔ±ØÐëÊÊÓ¦±ä»¯¡£ÎÒÃÇÏ£Íû¶ÔÎÒÃǵÄMySQLºÍGitHubµÄÆäËü·þÎñÔËÓÃÏàËÆµÄHA²ßÂÔ¡£
µ±¿¼ÂǸ߿ÉÓÃÐԺͷþÎñ·¢ÏÖʱ£¬Ò»Ð©ÎÊÌâ¿ÉÒÔÖ¸µ¼ÄãÕÒµ½Ò»¸öÇ¡µ±µÄ½â¾ö·½°¸¡£ÕâЩÎÊÌâ°üÀ¨µ«²»ÏÞÓÚ£º
ÄãÄÜÈÝÈ̵Äå´»úʱ¼äÊǶà¾Ã£¿
±ÀÀ£¼ì²âµÄ¿É¿¿ÐÔÈçºÎ£¿ÄãÄÜÈÝÈ̼ÙÑôÐÔ£¨¹ýÔç½øÐйÊÕϻָ´£©Âð£¿
¹ÊÕϻָ´µÄ¿É¿¿ÐÔÈçºÎ£¿ËüÔÚÄÄЩÇé¿öÏ»áʧ°Ü£¿
½â¾ö·½°¸¿çÊý¾ÝÖÐÐÄÄÜÁ¦ÈçºÎ£¿ÔÚµÍÑӳٺ͸ßÑÓ³ÙÍøÂçµÄÄÜÁ¦ÈçºÎ£¿
½â¾ö·½°¸Äܿ˷þÍêÕûµÄÊý¾ÝÖÐÐĹÊÕÏ»òÍøÂç¸ôÀëµÄÓ°ÏìÂð£¿
Èç¹ûÓеϰ£¬Ê²Ã´»úÖÆÄܹ»·ÀÖ¹»ò¼õÇáÄÔÁÑÏÖÏó£¨Á½¸ö·þÎñÆ÷¶¼Ðû³ÆÊÇÖ¸¶¨¼¯ÈºµÄÖ÷½Úµã£¬¶¼¶ÀÁ¢µØ±Ë´ËÎÞÒâʶµØ½ÓÊÜд²Ù×÷£©£¿
ÄãÄܹ»³ÐÊÜÊý¾Ý¶ªÊ§Â𣿵½Ê²Ã´³Ì¶È£¿
ΪÁË˵Ã÷ÉÏÊöһЩÎÊÌ⣬ÈÃÎÒÃÇÏÈ¿´Ò»ÏÂÎÒÃÇ֮ǰµÄHAµü´úÒÔ¼°ÎªÊ²Ã´ÎÒÃÇÒª¸Ä±äËü¡£
Ô¶Àë»ùÓÚVIPºÍDNSµÄ·þÎñ·¢ÏÖ
ÔÚÎÒÃÇ֮ǰµÄµü´úÖУ¬ÎÒÃÇʹÓãº
orchestratorÓÃÓÚ¹ÊÕϼàÌýºÍ¹ÊÕϻָ´
VIPºÍDNSÓÃÓÚ·¢ÏÖÖ÷½Úµã
ÔÚÄǸöµü´úÖУ¬¿Í»§¶Ëͨ¹ýʹÓÃÒ»¸öÃû³Æ£¬ÀýÈçmysql-writer-1.github.netÀ´·¢ÏÖд½Úµã¡£Õâ¸öÃû³Æ½âÎöΪÖ÷½Úµã»ñÈ¡µÄÐéÄâIPµØÖ·£¨Virtual
IP address£¬VIP£©¡£
Òò´Ë£¬Æ½³£µÄʱºò£¬¿Í»§¶Ë»áÖ»½âÎöÕâ¸öÃû³Æ£¬Á¬½Ó½âÎöµ½µÄIPµØÖ·£¬È»ºóÕÒµ½ÕýÔÚÁíÒ»¶Ë¼àÌýµÄÖ÷½Úµã¡£
Õâ¸ö¸±±¾ÍØÆË£¬¿çÔ½3¸ö²»Í¬µÄÊý¾ÝÖÐÐÄ£º

µ±·¢ÉúÒ»¸öÖ÷½Úµã¹ÊÕÏʼþʱ£¬Ò»¸öеķþÎñÆ÷£¨¸±±¾Ö®Ò»£©£¬±ØÐë±»ÌáÉýΪÖ÷½Úµã¡£
orchestrator½«¼à²âµ½Ò»¸ö¹ÊÕÏ£¬ÌáÉýÒ»¸öеÄÖ÷½Úµã£¬È»ºó²ÉÈ¡Ðж¯ÖØÐ·ÖÅäÃû³Æ/VIP¡£¿Í»§¶Ë²¢²»×¼È·µØÖªµÀÖ÷½ÚµãµÄÉí·Ý£ºËüÃÇËùÖªµÀµÄÖ»ÊÇÒ»¸öÃû³Æ£¬¶øÄǸöÃû³ÆÏÖÔÚÒ»¶¨½âÎöµ½ÁËеÄÖ÷½Úµã¡£È»¶ø£¬×¢Ò⣺
VIPÊÇÐ×÷µÄ£ºËüÃDZ»Êý¾Ý¿â·þÎñÆ÷±¾ÉíÉùÃ÷ºÍÓµÓС£ÎªÁË»ñÈ¡»òÊÍ·ÅÒ»¸öVIP£¬Ò»¸ö·þÎñÆ÷±ØÐë·¢ËÍÒ»¸öARPÇëÇó¡£ÔÚÐÂÌáÉýµÄÖ÷½Úµã»ñÈ¡Õâ¸öVIP֮ǰ£¬ÓµÓÐÕâ¸öVIPµÄ·þÎñÆ÷±ØÐëÏÈÊÍ·ÅÕâ¸öVIP¡£ÕâÓÐһЩ²»ÈçÈËÒâµÄЧ¹û£º
Ò»¸ö¹ÊÕϻָ´²Ù×÷°´Ë³ÐòÊ×ÏÈ»áÇëÇó¹ÒµôµÄÖ÷½ÚµãÊÍ·ÅVIP£¬È»ºóÇëÇóÐÂÌáÉýµÄÖ÷½Úµã»ñÈ¡Õâ¸öVIP¡£µ«Èç¹ûÀϵÄÖ÷½ÚµãÎÞ·¨·ÃÎÊ»òÕ߾ܾøÊÍ·ÅVIPÄØ£¿¼ÙÉèÔÚÄÇ̨·þÎñÆ÷ÉÏÒ»¿ªÊ¼·¢ÉúÁËÒ»¸ö¹ÊÕÏ£¬ÄÇôËüÒ²ºÜ¿ÉÄܲ»»á¼°Ê±ÏìÓ¦»òÕ߸ù±¾²»ÏìÓ¦¡£
ÎÒÃÇ»áÃæÁÙÁÑÄÔ´¦¾³£ºÁ½¸öÖ÷»úÉù³ÆÓµÓÐÏàͬµÄVIP¡£²»Í¬µÄ¿Í»§¶Ë¸ù¾Ý×î¶ÌÍøÂç·¾¶£¬¿ÉÄÜ»áÁ¬½Óµ½ÆäÖÐÈκÎÒ»¸ö·þÎñÆ÷¡£
Õâ¸öÎÊÌâµÄ¸ùÔ´ÊÇÒÀÀµÓÚÁ½¸ö¶ÀÁ¢·þÎñÆ÷µÄºÏ×÷£¬¶øÕâÖÖÉèÖÃÊDz»¿É¿¿µÄ¡£
¼´Ê¹¾ÉµÄÖ÷½ÚµãºÏ×÷£¬Õâ¸ö¹¤×÷Á÷Ò²ÀË·ÑÁ˱¦¹óµÄʱ¼ä£ºÇл»µ½ÐµÄÖ÷½Úµãʱ£¬ÐèÒªµÈ´ýÓë¾ÉµÄÖ÷½ÚµãµÄÁªÏµ¡£
¶øÇÒµ±VIP¸Ä±äʱ£¬²»Äܱ£Ö¤ÏÖÓеĿͻ§¶ËÓë¾ÉµÄ·þÎñÆ÷µÄÁ¬½Ó¶Ï¿ª£¬´Ó¶øµ¼ÖÂÎÒÃÇÈÔÃæÁÙÁÑÄÔ´¦¾³¡£
ÔÚÎÒÃǵÄÉèÖÃÖУ¬VIPÓëÎïÀíµØÖ·°ó¶¨¡£ËüÃÇÊôÓÚÒ»¸ö½»»»»ú»ò·ÓÉÆ÷¡£Òò´Ë£¬ÎÒÃÇÖ»Äܽ«VIPÖØÐ·ÖÅ䏸Ï໥¶¨Î»µÄ·þÎñÆ÷¡£ÌرðÊÇ£¬ÔÚijЩÇé¿öÏ£¬ÎÒÃDz»Äܽ«VIP·ÖÅ䏸ÔÚ²»Í¬Êý¾ÝÖÐÐÄÌáÉýµÄ·þÎñÆ÷£¬²¢ÇÒ±ØÐë¸ü¸ÄDNS¡£
DNSµÄ±ä»¯ÐèÒª¸ü³¤µÄʱ¼äÀ´´«²¥¡£¿Í»§¶Ë»áΪÁËÔ¤ÅäÖÃʱ¼ä¶ø»º´æDNSÃû³Æ¡£Ò»¸ö¿çÊý¾ÝÖÐÐĵĹÊÕÏÒâζןü³¤µÄå´»úʱ¼ä£ºÈÃËùÓпͻ§¶ËÒâʶµ½ÐÂÖ÷½ÚµãµÄÉí·ÝÐèÒª»¨·Ñ¸ü³¤Ê±¼ä¡£
½ö½öÕâЩÏÞÖÆ¾Í×ãÒÔ´ÙʹÎÒÃÇȥѰÕÒÒ»ÖÖеĽâ¾ö·½°¸£¬µ«»¹Óиü¶àµÄ¹ËÂÇ£º
Ö÷½Úµãͨ¹ýpt-heartbeat·þÎñÀ´×ÔÎÒ×¢ÈëÐÄÌø£¬´Ó¶ø´ïµ½ÑÓ³Ù²âÁ¿ºÍ½ÚÁ÷¿ØÖƵÄÄ¿µÄ¡£Õâ¸ö·þÎñ±ØÐëÔÚÐÂÌáÉýµÄÖ÷½ÚµãÉÏ¿ªÆô¡£Èç¹û¿ÉÄܵϰ£¬Õâ¸ö·þÎñ»áÔھɵÄÖ÷½ÚµãÉϻᱻ¹ØÍ£¡£
ͬÑùµØ£¬Pseudo-GTID×¢ÈëÊÇÓÉÖ÷½Úµã×ÔÎÒ¹ÜÀíµÄ¡£ËüÐèÒªÔÚеÄÖ÷½ÚµãÉÏ¿ªÆô£¬¶øÇÒ×îºÃÔھɵÄÖ÷½ÚµãÉϹØÍ£¡£
еÄÖ÷½Úµã±»ÉèÖÃΪ¿ÉдµÄ¡£Èç¹û¿ÉÄܵϰ£¬¾ÉµÄÖ÷½Úµã±»ÉèÖÃΪread_only¡£
ÕâЩ¶îÍâµÄ²½ÖèµÄÖ´ÐÐʱ¼ä¹¹³ÉÁË×Üå´»úʱ¼äµÄÒ»²¿·Ö£¬²¢ÇÒÒýÈëÁËËüÃÇ×Ô¼ºµÄ¹ÊÕϺͳåÍ»¡£
Õâ¸ö½â¾ö·½°¸ÊÇÓÐЧµÄ£¬¶øÇÒGitHubÒѾÓÐÔËÐÐÁ¼ºÃµÄ·Ç³£³É¹¦µÄMySQL¹ÊÕϻָ´´ëÊ©£¬µ«ÊÇÎÒÃÇÏëÒªÔÚÒÔÏ·½ÃæÌáÉýÎÒÃǵĸ߿ÉÓÃÐÔ£º
²»ÒÀÀµÊý¾ÝÖÐÐÄ¡£
¿Ë·þÊý¾ÝÖÐÐĹÊÕϵÄÓ°Ïì¡£
ÒÆ³ý²»¿É¿¿µÄÐ×÷¹¤×÷Á÷¡£
¼õÉÙ×Üå´»úʱ¼ä¡£
¾¡¿ÉÄÜʵÏÖÎÞËð¹ÊÕϻָ´¡£
GitHubµÄ¸ß¿ÉÓÃÐÔ·½°¸£ºorchestrator¡¢ConsulºÍGLB
ÎÒÃǵÄвßÂÔÒÔ¼°¸½´øµÄ¸Ä½ø£¬½â¾ö»òÕß¼õÇáÁËÉÏÊöµÄÐí¶àµ£ÓÇ¡£ÔÚĿǰµÄ¸ß¿ÉÓÃÐÔÉèÖÃÖУº
orchestratorÓÃÀ´ÔËÐйÊÕϼàÌýºÍ¹ÊÕϻָ´¡£ÎÒÃÇʹÓÃÁËÈçÏÂͼËùʾµÄÒ»¸ö¿çÊý¾ÝÖÐÐĵÄorchestrator/raft¡£
Hashicorp¹«Ë¾µÄÓÃÓÚ·þÎñ·¢ÏÖµÄConsul¡£
×÷Ϊ¿Í»§¶ËºÍд²Ù×÷½ÚµãÖ®¼äµÄ´úÀí²ãµÄGLB/HAProxy¡£
ÓÃÓÚÍøÂç·ÓɵÄanycast¡£

ÐÂÉèÖÃÍêȫɾ³ýÁËVIPºÍDNS¸ü¸Ä¡£²¢ÇÒÔÚÒýÈë¸ü¶à×é¼þµÄͬʱ£¬ÎÒÃÇʹµÃ×é¼þ½âñî²¢¼ò»¯ÁËÈÎÎñ£¬»¹Ê¹ÓÃÁËÎȽ¡µÄ½â¾ö·½°¸¡£Ïê½âÈçÏ£º
Ò»¸öÆÕͨµÄ¹¤×÷Á÷
ƽ³££¬Appͨ¹ýGLB/HAProxyÁ¬½Óµ½Ð´²Ù×÷½Úµã¡£
App²»»áÒâʶµ½Ö÷½ÚµãµÄÉí·Ý¡£ºÍÒÔǰһÑù£¬ËüÃÇʹÓÃÒ»¸öÃû³Æ¡£ÀýÈ磬cluster1µÄÖ÷½Úµã»áÊÇmysql-writer-1.github.net¡£È»¶ø£¬ÔÚÎÒÃÇĿǰµÄÉèÖÃÖУ¬Õâ¸öÃû³Æ»á±»½âÎöµ½Ò»¸öÈβ¥£¨anycast£©IP¡£
ͨ¹ýanycast·½·¨£¬Õâ¸öÃû³ÆÔÚÈκεط½¶¼±»½âÎöΪÏàͬµÄIP£¬µ«ÊÇÁ÷Á¿»á¸ù¾Ý¿Í»§¶ËλÖ÷ֱð½øÐзÓÉ¡£ÌØ±ðµØ£¬ÎÒÃǵÄÿ¸öÊý¾ÝÖÐÐͼÔÚ¶à¸öÇøÓò²¿ÊðÁËGLB£¨ÎÒÃǵĸ߿ÉÓøºÔؾùºâ£©¡£µ½mysql-writer-1.github.netµÄÁ÷Á¿Í¨³£Â·Óɵ½±¾µØÊý¾ÝÖÐÐĵÄGLB¼¯Èº¡£Òò´Ë£¬ËùÓеĿͻ§¶Ë¶¼ÊÇÓɱ¾µØ´úÀí·þÎñµÄ¡£
ÎÒÃÇÔÚHAProxyÉÏÔËÐÐGLB¡£ÎÒÃǵÄHAProxyÓÐд²Ù×÷³Ø£ºÃ¿¸öMySQL¼¯ÈºÒ»¸ö³Ø£¬¶øÃ¿¸ö³ØÓÐÒ»¸öºó¶Ë·þÎñÆ÷×÷ΪÕâ¸ö¼¯ÈºµÄÖ÷½Úµã¡£ËùÓеÄGLB/HAProxyÇøÓòÔÚËùÓеÄÊý¾ÝÖÐÐͼӵÓÐÏàͬµÄд²Ù×÷³Ø£¬¶øËüÃǶ¼Ö¸ÏòÕâЩ³ØÖÐÍêÈ«ÏàͬµÄºó¶Ë·þÎñÆ÷¡£Òò´Ë£¬Èç¹ûÒ»¸öAppÏëÒªÏòmysql-writer-1.github.netдÈ룬Õâ¸úËüÓëÄĸöGLB·þÎñÆ÷Á¬½ÓÎ޹ء£Ëü½«×ÜÊDZ»Â·Óɵ½cluster1Ö÷½Úµã¡£
¾ÍApp¶øÑÔ£¬·þÎñ·¢ÏÖÔÚGLBÖÕÖ¹£¬²¢ÇÒÓÀÔ¶²»ÐèÒªÖØÐ·¢ÏÖ¡£Á÷Á¿¶¼ÊÇÔÚGLBÉÏ·Óɵ½ÕýÈ·µÄÄ¿µÄµØ¡£
ÄÇô£¬GLBÈçºÎÖªµÀ½«ÄÄЩ·þÎñÆ÷×÷Ϊºó¶ËÁÐ±í£¬ÒÔ¼°ÎÒÃÇÈçºÎ½«¸ü¸Ä´«²¥µ½GLB£¿
ConsulµÄ·þÎñ·¢ÏÖ
Consul×÷ΪһÖÖ·þÎñ·¢ÏÖ½â¾ö·½°¸¶øÎÅÃû£¬²¢ÇÒ»¹ÌṩDNS·þÎñ¡£È»¶øÔÚÎÒÃǵĽâ¾ö·½°¸ÖУ¬ÎÒÃÇÓÃËü×÷Ϊһ¸ö¸ß¿ÉÓõļüÖµ¶Ô£¨KV£©´æ´¢Æ÷¡£
ÎÒÃÇʹÓÃConsulµÄKV´æ´¢Æ÷дÈ뼯ȺÖ÷½ÚµãµÄÉí·Ý¡£¶ÔÓÚÿ¸ö¼¯Èº£¬¶¼ÓÐÒ»Ì×KV¼Ç¼±íÃ÷¼¯ÈºµÄÖ÷½ÚµãµÄfqdn¡¢port¡¢ipv4ºÍipv6¡£
ÿ¸öGLB/HAProxy½Úµã¶¼ÔËÐÐconsul-template£ºÒ»¸ö¼àÌýConsulÊý¾Ý±ä»¯µÄ·þÎñ£¨ÔÚÎÒÃǵݸÀýÖУºÊÇÖ¸¼¯ÈºÖ÷½ÚµãÊý¾ÝµÄ±ä»¯£©¡£console-template»áÉú³ÉÒ»¸öÓÐЧµÄÅäÖÃÎļþ£¬²¢ÇÒÄܹ»»ùÓÚÅäÖÃµÄ±ä»¯ÖØÐ¼ÓÔØHAProxy¡£
Òò´Ë£¬ConsulÖÐÿ¸öÖ÷½ÚµãÉí·ÝµÄ¸Ä±ä¶¼±»Ã¿¸öGLB/HAProxy¹Û²â£¬È»ºóÖØÐÂÅäÖÃ×ÔÉí£¬½«ÐµÄÖ÷½ÚµãÉèÖÃΪһ¸ö¼¯ÈºµÄºó¶Ë³ØµÄµ¥¸öʵÌ壬ȻºóÖØÐ¼ÓÔØÒÔ·´Ó³ÄÇЩ±ä»¯¡£
ÔÚGitHub£¬ÎÒÃÇÔÚÿ¸öÊý¾ÝÖÐÐͼÓÐÒ»¸öConsulÉèÖ㬶øÇÒÿ¸öÉèÖö¼ÊǸ߿ÉÓõġ£È»¶ø£¬ÕâЩÉèÖÃÊDZ˴˶ÀÁ¢µÄ¡£ËüÃDz»»á±Ë´Ë¸´ÖÆ£¬Ò²²»¹²ÏíÈκÎÊý¾Ý¡£
ÄÇô£¬ConsulÊÇÈçºÎµÃÖª±ä»¯µÄÄØ£¿ÕâЩÐÅÏ¢ÓÖÊÇÈçºÎ¿çƽ̨·Ö²¼µÄÄØ£¿
orchestrator/raft
ÎÒÃÇÔËÐÐÒ»¸öorchestrator/raftÉèÖãºorchestrator½Úµãͨ¹ýraft¹²Ê¶Ï໥ͨÐÅ¡£ÎÒÃÇÿ¸öÊý¾ÝÖÐÐÄÓÐ1µ½2¸öorchestrator½Úµã¡£
orchestrator¸ºÔð¹ÊÕϼì²â¡¢MySQL¹ÊÕϻָ´²¢½«Ö÷½ÚµãµÄ±ä¸ü֪ͨConsul¡£¹ÊÕϻָ´Óɵ¥¸öorchestrator/raftÁìµ¼½Úµãά»¤£¬µ«ÊǼ¯ÈºÏÖÔÚÓÐÒ»¸öеÄÖ÷½ÚµãÕâ¸ö±ä¸üÏûÏ¢ÊÇͨ¹ýraft»úÖÆ´«²¥¸øËùÓÐorchestrator½ÚµãµÄ¡£
µ±orchestrator½Úµã½ÓÊÕµ½Ö÷½Úµã±ä¸üÏûϢʱ£¬ËüÃǶ¼»á֪ͨËûÃǵı¾µØConsulÉèÖãºËüÃǸ÷×Ôµ÷ÓÃÒ»´ÎKVд²Ù×÷¡£ÓµÓÐ1¸öÒÔÉÏorchestratorµÄÊý¾ÝÖÐÐĽ«»áÏòConsulÓжà´Î£¨µÈͬµÄ£©Ð´²Ù×÷¡£
ÕûºÏ¹¤×÷Á÷
ÔÚÒ»¸öÖ÷½Úµãå´»ú³¡¾°£º
orchestrator½Úµã¼à²âµ½¹ÊÕÏ¡£
orchestrator/raftÁìµ¼¿ªÊ¼Ò»´Î»Ö¸´´ëÊ©£¬ÌáÉýÒ»¸öеÄÖ÷½Úµã¡£
orchestrator/raft½«Ö÷½Úµã±ä¸üͨ¸æ¸øËùÓÐraft¼¯Èº½Úµã
ÿ¸öorchestrator/raft³ÉÔ±½ÓÊÕµ½Ò»¸öÁìµ¼±ä¸ü֪ͨ¡£ËüÃǸ÷×ÔÔÚ±¾µØConsulµÄKV´æ´¢Æ÷ÖиüÐÂеÄÖ÷½ÚµãµÄÉí·Ý¡£
ÿ¸öGLB/HAProxy¶¼ÔËÐÐÁËconsul-template£¬¼à¿ØConsulµÄKV´æ´¢Öеıä¸ü£¬È»ºóÖØÐÂÅäÖúͼÓÔØHAProxy¡£
¿Í»§¶ËÁ÷Á¿±»Öض¨Ïòµ½ÐµÄÖ÷½Úµã¡£
ÿ¸ö×é¼þ¶¼Ö°ÔðÇåÎú£¬¶øÇÒÕû¸öÉè¼Æ¼È½âñîÓÖ¼òµ¥¡£orchestrator²»ÐèÒªÖªµÀ¸ºÔؾùºâÆ÷¡£Consul²»ÐèÒªÖªµÀÐÅÏ¢À´×ÔÄÄÀï¡£´úÀíÖ»¹ØÐÄConsul¡£¿Í»§¶ËÖ»¹ØÐÄ´úÀí¡£
´ËÍ⣺
ÎÞÐè´«²¥DNS±ä¸ü¡£
ûÓÐTTL¡£
Õâ¸öÁ÷³Ì²»ÐèÒª¹ÒµôµÄÖ÷½ÚµãµÄºÏ×÷¡£ËüºÜ´ó³Ì¶ÈÉϱ»ºöÂÔÁË¡£
¸ü¶àϸ½Ú
ΪÁ˽øÒ»²½±£ÕÏÕâ¸öÁ÷³Ì£¬ÎÒÃÇ»¹×öÁËÈçϹ¤×÷£º
HAProxyÅäÖÃÁËÒ»¸ö·Ç³£¶ÌµÄhard-stop-after¡£µ±ËüÓÃд²Ù×÷³ØÖеÄÒ»¸öеĺó¶Ë·þÎñÆ÷ÖØÐ¼ÓÔØÊ±£¬Ëü»á×Ô¶¯ÖÕÖ¹ÈκÎÏÖ´æµÄÓë¾ÉµÄÖ÷½ÚµãµÄÁ¬½Ó¡£
ͨ¹ýhard-stop-after£¬ÎÒÃÇÉõÖÁ²»ÐèÒªÀ´×Ô¿Í»§¶ËµÄÅäºÏ£¬¶øÇÒÕâÑù¼õÇáÁËÁÑÄÔ³¡¾°¡£ÖµµÄ×¢ÒâµÄÊÇ£¬Õâ²¢²»ÊÇÑÏÃܵģ¬ÔÚÎÒÃÇɱËÀ¾ÉÁ¬½Ó֮ǰ»á¹ýÈ¥Ò»¶Îʱ¼ä¡£µ«ÊÇÔÚÄÇÖ®ºó£¬ÎÒÃǾͿÉÒÔ·ÅÐIJ»»á³öÏÖÁîÈËÌÖÑáµÄÒâÍâ¡£
ÎÒÃDz¢Ã»ÓÐÑϸñÒªÇóConsulÔÚËùÓÐʱ¼ä¶¼ÊÇ¿ÉÓõġ£ÊÂʵÉÏ£¬ÎÒÃÇÖ»ÐèÒªËüÔÚ¹ÊÕϻָ´Ê±¿ÉÓá£Èç¹ûConsulÅöÇɹҵôÁË£¬GLB»á¼ÌÐøÊ¹ÓÃÉÏ´ÎÒÑÖªµÄÖµ²Ù×÷£¬²»»á²ÉÈ¡¾çÁÒµÄÐж¯¡£
GLB±»ÉèÖÃÀ´ÑéÖ¤ÐÂÌáÉýµÄÖ÷½ÚµãµÄÉí·Ý¡£ÀàËÆÓÚÎÒÃǵÄÉÏÏÂÎĸÐÖªMySQL³Ø£¬ÔÚºó¶Ë·þÎñÆ÷½øÐмì²é£¬À´È·ÈÏËüȷʵÊÇÒ»¸öд²Ù×÷½Úµã¡£Èç¹ûÅöÇÉɾ³ýÁËConsulÖеÄÖ÷½ÚµãÐÅÏ¢£¬Ã»ÓÐÎÊÌ⣻¿Õ°×µÄÌõÄ¿»á±»ºöÂÔ¡£Èç¹ûÎÒÃÇÔÚConsulÖÐÎóдÈëÁËÒ»¸ö·ÇÖ÷½Úµã·þÎñÆ÷µÄÃû³Æ£¬Ã»ÓÐÎÊÌ⣻GLB»á¾Ü¾ø¸üÐÂËü²¢Ê¹ÓÃÉÏ´ÎÒÑÖªµÄ״̬ÔËÐС£
ÎÒÃÇ»áÔÚÏÂÃæÕ½ÚÖнøÒ»²½½â¾öµ£ÓDz¢×·Çó¸ß¿ÉÓÃÐÔÄ¿±ê¡£
orchestrator/raft¹ÊÕϼì²â
orchestratorʹÓÃÒ»ÖÖÕûÌå·½°¸À´¼ì²â¹ÊÕÏ£¬Òò´ËÊǷdz£¿É¿¿µÄ¡£ÎÒÃDz»¹Û²â¼ÙÑôÐÔ£ºÎÒÃDz»»á¹ýÔçÆô¶¯¹ÊÕϻָ´£¬Òò´Ë²»»áÔâÊܲ»±ØÒªµÄå´»úʱ¼ä¡£
orchestrator/raft½øÒ»²½½â¾öÁËÒ»¸öÍêÕûµÄÊý¾ÝÖÐÐÄÍøÂç¸ôÀëµÄÇé¿ö£¨¼´Êý¾ÝÖÐÐÄΧÀ¸£©¡£Êý¾ÝÖÐÐÄÍøÂç¸ôÀë»áÒýÆð»ìÏý£ºÄǸöÊý¾ÝÖÐÐÄÖеķþÎñÆ÷Äܹ»±Ë´ËͨÐÅ¡£ÊÇËüÃÇÓëÆäËüÊý¾ÝÖÐÐÄÍøÂç¸ôÀëÁË£¿»¹ÊÇÆäËüÊý¾ÝÖÐÐı»ÍøÂç¸ôÀëÁË£¿
ÔÚÒ»¸öorchestrator/raftÉèÖÃÖУ¬raftÁìµ¼½ÚµãÊÇÔËÐйÊÕϻָ´µÄ½Úµã¡£Áìµ¼½ÚµãÊÇÖ¸»ñµÃ´ó¶àÊýȺÌåÖ§³ÖµÄ½Úµã¡£ÎÒÃǵÄorchestrator½Úµã²¿Êð¾ÍÊÇÕâÑù£¬Ã»Óе¥¸öÊý¾ÝÖÐÐÄÕ¼´ó¶àÊýÖ§³Ö£¬ÈκÎn-1¸öÊý¾ÝÖÐÐÄÕ¼´ó¶àÊýÖ§³Ö¡£
ÔÚÒ»¸öÍêÕûµÄÊý¾ÝÖÐÐÄÍøÂç¸ôÀëʼþÖУ¬ÄǸöÊý¾ÝÖÐÐÄÖеÄorchestrator½ÚµãÓëÆäËûÊý¾ÝÖÐÐÄÖеĶԵȽڵã¶Ï¿ªÁ¬½Ó¡£Òò´Ë£¬ÔÚ¸ôÀëµÄÊý¾ÝÖÐÐÄÖеÄorchestrator½Úµã²»ÄܳÉΪraft¼¯ÈºµÄÁìµ¼½Úµã¡£Èç¹ûÈκÎÕâÖÖ½ÚµãÅöÇɳÉΪÁìµ¼½Úµã£¬ËüÒ²»áÏĄ̂¡£Ò»¸öеÄÁìµ¼½Úµã»á´ÓÆäËüÊý¾ÝÖÐÐÄ·ÖÅä¡£Õâ¸öÁìµ¼½Úµã½«»ñµÃËùÓÐÆäËüÊý¾ÝÖÐÐĵÄÖ§³Ö£¬¶øÕâЩÊý¾ÝÖÐÐÄÄܹ»±Ë´ËͨÐÅ¡£
Òò´Ë£¬orchestrator½Úµã¾ÍÊÇÍøÂç¸ôÀëµÄÊý¾ÝÖÐÐÄÖ®ÍâµÄÒ»¸ö½Úµã¡£ÔÚÒ»¸ö¸ôÀëµÄÊý¾ÝÖÐÐÄÓ¦¸ÃÓÐÒ»¸öÖ÷½Úµã£¬orchestrator½«Æô¶¯¹ÊÕϻָ´£¬ÓÿÉÓÃÊý¾ÝÖÐÐÄÖ®Ò»ÀïµÄÒ»¸ö·þÎñÆ÷È¡´úËü¡£ÎÒÃÇͨ¹ý½«¾ö²ßίÍиø·Ç¸ôÀëÊý¾ÝÖÐÐÄÖеÄȺÌåÀ´¼õÇáÊý¾ÝÖÐÐĸôÀë¡£
¸ü¿ìµÄͨ¸æ
¿ÉÒÔͨ¹ý¸ü¿ìËÙµØÍ¨¸æÖ÷½Úµã±ä¸üÀ´½øÒ»²½¼õÉÙ×Üå´»úʱ¼ä¡£ÕâÈçºÎʵÏÖÄØ£¿
µ±orchestrator¿ªÊ¼¹ÊÕϻָ´Ê±£¬Ëü¹Û²â¿É±»ÌáÉýµÄ·þÎñÆ÷Ⱥ¡£Àí½â¸´ÖÆÔÔò²¢×ñ´Ó°µÊ¾ºÍÏÞÖÆ£¬Äܹ»»ùÓÚ×î¼Ñ×ö·¨×÷³öÓÅ»¯µÄ¾ö²ß¡£
ÐèÒªÒâʶµ½£¬¿ÉÓÃÓÚÌáÉýµÄ·þÎñÆ÷Ò²ÊÇÒ»¸öÀíÏëµÄºòÑ¡Õߣ¬ÀýÈ磺
ûÓÐʲô¿ÉÒÔ×èÖ¹·þÎñÆ÷µÄÌáÉý£¨¶øÇÒÓû§ÒѾDZÔÚ°µÊ¾ÕâЩ·þÎñÆ÷ÊÇÊ×Ñ¡ÌáÉý¶ÔÏó£©
ÕâЩ·þÎñÆ÷Äܹ»½«ÆäËùÓеÄÐֵܽڵã×÷Ϊ¸´ÖÆÆ·
ÔÚÕâÖÖÇé¿ö£¬orchestratorÊ×ÏȽ«·þÎñÆ÷ÉèÖÃΪ¿ÉдµÄ£¬È»ºóѸËÙͨ¸æ·þÎñÆ÷µÄÌáÉý£¨Ð´ÈëConsul
KV£©£¬Í¬Ê±Òì²½¿ªÊ¼ÐÞ¸´¸´ÖÆÊ÷£¨Õâ¸ö²Ù×÷ͨ³£»á»¨·Ñ¸ü¶àʱ¼ä£©¡£
ºÜ¿ÉÄܵ±ÎÒÃǵÄGLB·þÎñÆ÷ÍêÈ«ÖØÐ¼ÓÔØÊ±£¬¸´ÖÆÊ÷ÒѾÍêºÃÎÞËðÁË£¬µ«Õâ²»ÊÇÑϸñ±ØÐèµÄ¡£·þÎñÆ÷¿ÉÒÔ½ÓÊÕд²Ù×÷£¡
°ëͬ²½¸´ÖÆ
ÔÚMySQLµÄ°ëͬ²½¸´ÖÆÖУ¬ÔÚ±ä¸üÒѾÌá½»µ½Ò»¸ö»ò¶à¸ö¸±±¾Ö®Ç°£¬Ö÷·þÎñÆ÷²»»á³ÐÈÏÕâ¸öÊÂÎñÌá½»¡£ÕâÌṩÁËÒ»ÖÖʵÏÖÎÞËð¹ÊÕϻָ´µÄ·½·¨£ºÈκÎÌá½»µ½Ö÷½ÚµãµÄ±ä¸ü¶¼ÒѾӦÓûòÕߵȴý±»Ó¦Óõ½Ä³¸ö¸±±¾¡£
Ò»ÖÂÐÔ°éËæ×ųɱ¾£º¿ÉÓÃÐÔ·çÏÕ¡£Èç¹ûûÓи±±¾È·ÈÏÊÕµ½±ä¸ü£¬Ö÷½Úµã»á×èÈû²¢ÇÒд²Ù×÷»áÍ£¶Ù¡£ÐÒÔ˵ÄÊÇ£¬ÓÐÒ»¸ö³¬Ê±ÅäÖ㬳¬¹ý³¬Ê±Ê±¼ä£¬Ö÷½ÚµãÄܹ»»Ö¸´µ½Òì²½¸´ÖÆÄ£Ê½£¬Ê¹µÃд²Ù×÷ÔٴοÉÓá£
ÎÒÃǽ«ÎÒÃǵij¬Ê±ÅäÖÃÉèÖÃΪһ¸öºÏÀíµÄµÍÖµ£º500ms¡£Õâ×ã¹»½«Ö÷½ÚµãµÄ±ä¸ü´«µÝ¸ø±¾µØÊý¾ÝÖÐÐĸ±±¾ÒÔ¼°Ô¶³ÌµÄÊý¾ÝÖÐÐÄ¡£ÓÐÁËÕâ¸ö³¬Ê±£¬ÎÒÃǾͿÉÒÔ¹Û²âÍêÃÀµÄ°ëͬ²½ÐÐΪ£¨²»»Ø¹öµ½Òì²½¸´ÖÆ£©£¬Í¬Ê±ÔÚÈ·ÈÏʧ°ÜµÄÇé¿öÏ»á¸ÐÊܵ½Ò»¸ö¿É½ÓÊܵķdz£¶ÌµÄ×èÈûʱ¼ä¡£
ÎÒÃÇÔÚ±¾µØÊý¾ÝÖÐÐĸ±±¾ÉÏÆôÓðëͬ²½£¬¶øÇÒÔÚÖ÷½Úµã¹ÒµôʼþÖУ¬ÎÒÃÇÆÚÍû£¨¾¡¹Ü²¢²»ÑϸñÇ¿ÖÆ£©ÎÞËð¹ÊÕϻָ´¡£µ«ÊÇ£¬ÎÒÃDz»»áÆÚÍûÒ»¸öÍêÕûµÄÊý¾ÝÖÐÐĹÊÕϵÄÎÞËð¹ÊÕϻָ´£¬ÒòΪËüµÄ´ú¼Û·Ç³£´ó¡£
ÔÚ½øÐаëͬ²½³¬Ê±ÊµÑéʱ£¬ÎÒÃÇ»¹¹Û²ìµ½Ò»ÖÖ¶ÔÎÒÃÇÓÐÀûµÄÏÖÏó£ºÎÒÃÇÄܹ»ÔÚÖ÷½Úµã¹ÊÕÏÖÐÓ°ÏìÀíÏëµÄºòÑ¡ÕßµÄÉí·Ý¡£Í¨¹ýÔÚÖ¸¶¨·þÎñÆ÷ÉÏÆôÓðëͬ²½²¢½«ËüÃDZê¼ÇΪºòÑ¡Õߣ¬ÎÒÃÇÄܹ»Í¨¹ýÓ°Ïì¹ÊÕϽá¹ûÀ´¼õÉÙ×Üå´»úʱ¼ä¡£ÎÒÃÇÔÚʵÑéÖй۲쵽£¬ÎÒÃÇͨ³£Äܹ»ÌáÉýÀíÏëµÄºòÑ¡Õß²¢Òò´Ë¿ìËÙ½øÐÐͨ¸æ¡£
ÐÄÌø×¢Èë
ÎÒÃÇÑ¡ÔñÔÚÈκεط½ÈκÎʱ¼ä¹ÜÀípt-heartbeat·þÎñµÄ¿ªÆô/¹Ø±Õ£¬¶ø²»ÊÇÖ»ÔÚÌáÉý/½µ¼¶µÄÖ÷½ÚµãÉϹÜÀípt-heartbeat·þÎñµÄ¿ªÆô/¹Ø±Õ¡£ÕâÐèҪһЩ²¹¶¡£¬¸Ä±äËüÃǵÄread_only״̬»òÕßÍêÈ«±¼À££¬ÒÔ±ãʹpt-heartbeatÓë·þÎñÆ÷Ò»Ö¡£
ÔÚÎÒÃǵ±Ç°ÉèÖÃÖУ¬pt-heartbeat·þÎñÔËÐÐÔÚÖ÷½ÚµãºÍ¸±±¾ÉÏ¡£ÔÚÖ÷½ÚµãÉÏ£¬ËüÃÇÉú³ÉÐÄÌøÊ¼þ¡£ÔÚ¸±±¾ÉÏ£¬ËüÃDZêʶ·þÎñÆ÷ÊÇread_only²¢ÖÜÆÚÐÔ¼ì²éËüÃǵÄ״̬¡£Ò»µ©Ò»¸ö·þÎñÆ÷±»ÌáÉýΪÖ÷½Úµã£¬ÄǸö·þÎñÆ÷ÉϵÄpt-heartbeat½«Æä±êʶΪ¿ÉдµÄ£¬²¢¿ªÊ¼×¢ÈëÐÄÌøÊ¼þ¡£
orchestratorËùÓÐȨίÍÐ
ÎÒÃǽøÒ»²½Î¯Íиøorchestrator£º
Preudo-GTID×¢Èë
½«ÌáÉýµÄÖ÷½ÚµãÉèÖÃΪ¿ÉдµÄ²¢Çå³ýËüµÄ¸´ÖÆ×´Ì¬
Èç¹û¿ÉÄܵϰ£¬½«¾ÉµÄÖ÷½ÚµãÉèÖÃΪread_only
ÔÚеÄÖ÷½ÚµãÉÏ£¬Õâ¼õÉÙÁËĦ²Á¡£±»ÌáÉýµÄÖ÷½ÚµãÃ÷ÏÔÐèÒªÊÇ»îÔ¾µÄºÍ¿É·ÃÎʵ쬷ñÔòÎÒÃDz»»áÌáÉýËü¡£ÄÇô£¬¿ÉÒÔÈÃorchestratorÖ±½Ó½«±ä¸üÓ¦Óõ½ÌáÉýµÄÖ÷½ÚµãÉÏ¡£
ÏÞÖÆºÍȱÏÝ
´úÀí²ãʹµÃAppÒâʶ²»µ½Ö÷½ÚµãµÄÉí·Ý£¬Í¬Ê±Ëü»¹¶ÔÖ÷½ÚµãÆÁ±ÎÁËAppµÄÉí·Ý¡£Ö÷½Úµã¿´µ½µÄ¶¼ÊÇÀ´×Ô´úÀí²ãµÄÁ¬½Ó£¬¶øÎÒÃǶªÊ§ÁËÕæÕýÁ¬½ÓÀ´Ô´µÄÐÅÏ¢¡£
Ëæ×Å·Ö²¼Ê½ÏµÍ³µÄ·¢Õ¹£¬ÎÒÃÇÈÔÈ»ÃæÁÙδ´¦Àí¹ýµÄ³¡¾°¡£
ÓÈÆäÊÇ£¬ÔÚÒ»¸öÊý¾ÝÖÐÐĸôÀ볡¾°ÖУ¬¼ÙÉèÖ÷½ÚµãÊÇÔÚ¸ôÀëµÄÊý¾ÝÖÐÐÄ£¬ÄǸöÊý¾ÝÖÐÐĵÄAppÈÔÈ»Äܹ»ÏòÖ÷½ÚµãдÈë¡£Ò»µ©ÍøÂç»Ö¸´£¬Õâ¿ÉÄܵ¼ÖÂ״̬²»Ò»Ö¡£ÎÒÃÇͨ¹ý´Ó·Ç³£¹ÂÁ¢µÄÊý¾ÝÖÐÐÄʵÏÖÒ»¸ö¿É¿¿µÄSTONITHÀ´¼õÇáÕâÖÖÁÑÄÔÏÖÏó¡£Ïñ֮ǰһÑù£¬Ö÷½Úµã½µ¼¶Ö®Ç°»á¾¹ýһЩʱ¼ä£¬²¢ÇÒ»á´æÔÚÒ»¶Îʱ¼äµÄÁÑÄÔ¡£±ÜÃâÁÑÄÔÏÖÏóµÄÔËά³É±¾·Ç³£¸ß¡£
´æÔÚ¸ü¶à³¡¾°£º¹ÊÕϻָ´Ê±Consulå´»ú£»²¿·ÖÊý¾ÝÖÐÐĸôÀ룻ÆäËü³¡¾°µÈ¡£ÎÒÃÇÃ÷°×£¬ÔÚÕâÖÖÐÔÖʵķֲ¼Ê½ÏµÍ³ÖУ¬²»¿ÉÄܹرÕËùÓеÄ©¶´£¬Òò´ËÎÒÃǹØ×¢×îÖØÒªµÄ³¡¾°¡£
½á¹û
ÎÒÃǵÄorchestrator/GLB/ConsulÉèÖÃÌṩÁË£º
¿É¿¿µÄ¹ÊÕϼì²â
²»ÒÀÀµÊý¾ÝÖÐÐĵĹÊÕϻָ´
µäÐ͵ÄÎÞËð¹ÊÕϻָ´
Êý¾ÝÖÐÐÄÍøÂç¸ôÀëÖ§³Ö
¼õÇáÁÑÄÔÏÖÏ󣨸ü¶à¹¤×÷ÈÔÔÚ½øÐÐÖУ©
ûÓкÏ×÷ÒÀÀµ
´ó²¿·Ö³¡¾°ÏµÄ10-13ÃëµÄ×Üå´»úʱ¼ä
×Üå´»úʱ¼äÔڷdz£Éٵij¡¾°Ï»᳤´ï20Ã룬ÔÚ¼«¶ËÇé¿öÏ»᳤´ï25Ãë¡£
½áÂÛ
orchestratoion/proxy/service-discovery·¶Ê½ÔÚ½âñî¼Ü¹¹ÖÐʹÓÃÁËÖÚËùÖÜÖªÇÒÁîÈËÐÅÀµµÄ×é¼þ£¬Ê¹µÃËü¸üÈÝÒײ¿Êð¡¢ÔËάºÍ¹Û²â£¬²¢ÇÒÿ¸ö×é¼þ¶¼¿ÉÒÔ¶ÀÁ¢µØÀ©´ó»òËõС¹æÄ£¡£ÎÒÃǽ«²»¶Ï²âÊÔÎÒÃǵÄÉèÖ㬴Ӷø²»¶ÏѰÇó¸Ä½ø¡£ |