дblogºÍд´úÂëÒ»Ñù£¬¸Õ¿ªÊ¼¶¼ÊDz»ÍêÃÀµÄ£¬ÐèÒª²»¶ÏµÄÐÞÕýºÍÖØ¹¹£¬Èç¹û´ó¼ÒÔÚÔĶÁ±¾blogÖз¢ÏÖÈκÎÎÊÌâºÍÒÉÎÊ£¬¶¼»¶ÓÌÖÂÛ»òÅÄש¡£
1 ÐÔÄܵ÷Óżò½é
1.1ΪʲôҪ½øÐÐÐÔÄܵ÷ÓÅ£¿
1.1.1 ±àдµÄÐÂÓ¦ÓÃÉÏÏßǰÔÚÐÔÄÜÉÏÎÞ·¨Âú×ãÐèÇó£¬Õâ¸öʱºòÐèÒª¶Ôϵͳ½øÐÐÐÔÄܵ÷ÓÅ
1.1.2 Ó¦ÓÃϵͳÔÚÏßÉÏÔËÐкóËæ×ÅϵͳÊý¾ÝÁ¿µÄ²»¶ÏÔö³¤¡¢·ÃÎÊÁ¿µÄ²»¶ÏÉÏÉý£¬ÏµÍ³µÄÏìÓ¦ËÙ¶Èͨ³£Ô½À´Ô½Âý£¬²»Âú×ãÒµÎñÐèÒª£¬Õâ¸öʱºòÒ²ÐèÒª¶Ôϵͳ½øÐÐÐÔÄܵ÷ÓÅ
1.2 ÐÔÄܵ÷ÓŰüÀ¨ÄÇЩ·½Ã棿
1.2.1 ̸µ½ÏµÍ³»ò²úÆ·µÄÐÔÄܵ÷ÓÅ£¬¿ÉÒÔ´Ó¹ãÒåºÍÏÁÒåÁ½¸ö·¶Î§À´Àí½â¡£
´Ó¹ãÒåµÄ²ãÃæÀ´¿´£¬¾Í²»½öÏÞÓÚ³ÌÐòÄÚ²¿ÁË£¬ÒòΪÔì³ÉϵͳÐÔÄÜÎÊÌâµÄÆ¿¾±ºÜ¿ÉÄÜÀ´Ô´ÓÚ·½·½ÃæÃ棬¶øÕâÖÖÇé¿öÍùÍùÊÇÐÔÄܵ÷ÓÅºÜÆÕ±éµÄÇé¿ö£¬
ÏÂÃæ¾Í´Ó¹ãÒåµÄ·¶Î§Ï¸·Ö³É¼¸¸ö½Ç¶ÈÀ´½øÐвûÊö£¬¼ûͼ-1 ¹ãÒåÐÔÄܵ÷ÓÅÓã¹Çͼ

ͼ-1 ¹ãÒåÐÔÄܵ÷ÓÅÓã¹Çͼ
1.2.2´ÓÏÁÒåµÄ·¶³ëÀ´¿´£¬ÐÔÄܵ÷ÓÅÖ÷ÒªÊÇָͨ¹ýÐÞ¸ÄÈí¼þ³ÌÐòÂß¼¡¢½á¹¹µÈ¼¼ÊõÊÖ¶ÎÌáÉýÈí¼þ²úÆ·µÄ¸÷ÏîÐÔÄÜÖ¸±ê£¬ÈçÏìӦʱ¼äµÈ¡£±¾ÎÄÖØÒªÊÇ´ÓÏÁÒåµÄ·¶³ëÀ´¿´¡£
´ÓÏÁÒåµÄ·¶³ë¿´£¬ÐÔÄܵ÷ÓÅ¿ÉÒÔ´Ó Ó²¼þ(¼ÆËã»úÌåϵ»ú¹¹)¡¢²Ù×÷ϵͳ(OS\JVM)¡¢Îļþϵͳ¡¢ÍøÂçͨÐÅ¡¢Êý¾Ý¿âϵͳ¡¢Öмä¼þ¡¢Ó¦ÓóÌÐò±¾ÉíµÈ·½ÃæÈëÊÖ¡£
ÕâÀïÖ÷Òª¹Ø×¢JVM¡¢Öмä¼þ¡¢Ó¦ÓóÌÐòµÄÐÔÄܵ÷ÓÅ¡£
1.3 ÐÔÄܵIJο¼Ö¸±ê
Ö´ÐÐʱ¼ä£ºÒ»¶Î´úÂë´Ó¿ªÊ¼ÔËÐе½ÔËÐнáÊøËùʹÓõÄʱ¼ä¡£
CPUʱ¼ä£º(Ëã·¨)º¯Êý»òÕßÏß³ÌÕ¼ÓÃCPUµÄʱ¼ä¡£
ÄÚ´æ·ÖÅ䣺³ÌÐòÔÚÔËÐÐʱռÓõÄÄÚ´æ¿Õ¼ä¡£
´ÅÅÌÍÌÍÂÁ¿£ºÃèÊöI/OµÄʹÓÃÇé¿ö¡£
ÍøÂçÍÌÍÂÁ¿£ºÃèÊöÍøÂçµÄʹÓÃÇé¿ö¡£
ÏìӦʱ¼ä£ºÏµÍ³¶ÔijÓû§ÐÐΪ»òÕß¶¯×÷×ö³öÏìÓ¦µÄʱ¼ä¡£ÏìӦʱ¼äÔ½¶Ì£¬ÐÔÄܺá£
2 ÐÔÄܵ÷ÓŲ½Öè
2.1ÐÔÄܵ÷ÓŲ½Öèͼ

ͼ-2ÐÔÄܵ÷ÓŲ½Öèͼ
µ÷ÓÅǰÊ×ÏÈÒª×öµÄÊǺâÁ¿ÏµÍ³ÏÖ×´£¬Õâ°üÀ¨Ä¿Ç°ÏµÍ³µÄÇëÇó´ÎÊý¡¢ÏìӦʱ¼ä¡¢×ÊÔ´ÏûºÄµÈÐÅÏ¢£¬ÀýÈçBϵͳĿǰ95%µÄÇëÇóÏìӦʱ¼äΪ1Ãë¡£
ÔÚÓÐÁËϵͳÏÖ×´ºó¿ÉÉ趨µ÷ÓÅÄ¿±ê£¬Í¨³£µ÷ÓÅÄ¿±êÊǸù¾ÝÓû§ËùÄܽÓÊܵÄÏìÓ¦ËÙ¶È»òϵͳËùÓµÓеĻúÆ÷ÒÔ¼°ËùÖ§³ÅµÄÓû§Á¿¹ÀËã³öÀ´µÄ£¬ÀýÈç
É趨µ÷ÓÅÄ¿±ê£º95%µÄÇëÇóÒªÔÚ500msÄÚ·µ»Ø¡£
ÔÚÉ趨Á˵÷ÓÅÄ¿±êºó£¬ÐèÒª×öµÄÊÇѰÕÒÐÔÄÜÆ¿¾±£¬ÕâÒ»²½×îÖØÒªµÄÊÇÕÒ³öÔì³ÉĿǰϵͳÐÔÄܲ»×ãµÄ×î´óÆ¿¾±µã¡£ÕÒ³öºó£¬¿É½áºÏһЩ¹¤¾ßÀ´ÕÒ³öÔì³ÉÆ¿¾±µãµÄ´úÂ룬µ½´Ë²ÅÍê³ÉÁËÕâ¸ö²½Öè¡£
ÔÚÕÒµ½ÁËÔì³ÉÆ¿¾±µãµÄ´úÂëºó£¬¿ªÊ¼½øÐÐÐÔÄܵ÷ÓÅ¡£Í¨³£ÐèÒª·ÖÎöÆäÐèÇó»òÒµÎñ³¡¾°£¬È»ºó½áºÏһЩÓÅ»¯µÄ¼¼ÇÉÈ·¶¨ÓÅ»¯µÄ²ßÂÔ£¬ÓÅ»¯²ßÂÔ»ò¼ò»ò·±£¬Ñ¡ÔñÆäÖÐÊÕÒæ±È(ÓÅ»¯ºóµÄÔ¤ÆÚЧ¹û/ÓÅ»¯ÐèÒª¸¶³öµÄ´ú¼Û)×î¸ßµÄÓÅ»¯·½°¸£¬½øÐÐÓÅ»¯¡£
ÓÅ»¯²¿Êðºó£¬¼ÌÐøºâÁ¿ÏµÍ³µÄ×´¿ö£¬ÈçÒѴﵽĿ±ê£¬Ôò¿É½áÊø´Ë´Îµ÷ÓÅ£¬ÈçÈÔδ´ïµ½Ä¿±ê£¬ÔòÒª¿´ÊÇ·ñ²úÉúÁËеÄÐÔÄÜÆ¿¾±¡£»ò¿ÉÒÔ¿¼ÂǼÌÐø³¢ÊÔÉÏÒ»²½ÖÐÖÆ¶¨µÄÆäËûÓÅ»¯·½°¸£¬Ö±µ½´ï³Éµ÷ÓÅÄ¿±ê»òÂÛÖ¤ÔÚĿǰµÄÌåϵ½á¹¹ÉÏÎÞ·¨´ïµ½µ÷ÓÅÄ¿±êΪֹ¡£
3 ÐÔÄܵ÷ÓÅ˼·
ÐÔÄܵ÷ÓŵIJ½ÖèÖ÷ÒªÓУººâÁ¿ÏµÍ³ÏÖ×´¡¢É趨µ÷ÓÅÄ¿±ê¡¢Ñ°ÕÒÐÔÄÜÆ¿¾±¡¢ÐÔÄܵ÷ÓÅ£¬ÑéÖ¤ÊÇ·ñ´ïµ½µ÷ÓÅÄ¿±ê¡£
±¾ÎĵµÖ÷Òª¹Ø×¢µ÷ÓŲ½ÖèÖÐµÄ Ñ°ÕÒÐÔÄÜÆ¿¾± ºÍ ÐÔÄܵ÷Õû(ÓÅ) Á½¸ö¹Ø¼üµÄ½×¶Î½×¶Î£¬Ö÷Ҫ˼·¼ûÏÂͼ£ºÍ¼-3
ѰÕÒÐÔÄÜÆ¿¾±ºÍÐÔÄܵ÷ÓÅÏî½á¹¹·Ö½âͼ

ͼ-3 ѰÕÒÐÔÄÜÆ¿¾±ºÍÐÔÄܵ÷ÓÅÏî½á¹¹·Ö½âͼ
4 ѰÕÒÐÔÄÜÆ¿¾±
ѰÕÒÐÔÄÜÆ¿¾±·Ö¸öÁ½²¿·Ö£ºÑ°ÕÒ¹ý¶ÈÏûºÄ×ÊÔ´µÄ´úÂë ºÍ Ñ°ÕÒδ³ä·ÖʹÓÃ×ÊÔ´µ«³ÌÐòÖ´ÐÐÂýµÄÔÒòºÍ´úÂë¡£
ͨ³£ÐÔÄÜÆ¿¾±µÄ±íÏóÊÇ×ÊÔ´ÏûºÄ¹ý¶àÍⲿ´¦ÀíϵͳµÄÐÔÄܲ»×㣻»òÕß×ÊÔ´ÏûºÄ²»¶àµ«³ÌÐòµÄÏìÓ¦ËÙ¶ÈÈ´ÈÔ´ï²»µ½ÒªÇó¡£
ÕâÀï×ÊÔ´Ö÷ÒªÊÇÖ¸£ºÏûºÄÔÚCPU¡¢ÄÚ´æ¡¢ÎļþIO¡¢ÍøÂçIOµÈ·½Ãæ×ÊÔ´¡£»úÆ÷µÄ×ÊÔ´ÊÇÓÐÏ޵쬵±Ä³×ÊÔ´ÏûºÄ¹ý¶àʱ£¬Í¨³£»áÔì³ÉϵͳµÄÏìÓ¦ËÙ¶ÈÂý¡£
Íⲿ´¦ÀíϵͳµÄÐÔÄܲ»×ãÖ÷ÒªÊÇÖ¸Ëùµ÷ÓÃµÄÆäËûϵͳÌṩµÄ¹¦ÄÜ(ÈçÊý¾Ý¿â²Ù×÷µÄÏìÓ¦ËٶȲ»¹»)£»Ëùµ÷ÓÃµÄÆäËûϵͳÐÔÄܲ»×ã¶àÊýÇé¿öÏÂÒ²ÊÇ×ÊÔ´ÏûºÄ¹ý¶à£¬µ«³ÌÐòµÄÐÔÄܲ»×ãÔì³ÉµÄ£»
Êý¾Ý¿â²Ù×÷ÐÔÄܲ»×ãͨ³£¿ÉÒÔ¸ù¾ÝÊý¾Ý¿âµÄsqlÖ´ÐÐËÙ¶È¡¢Êý¾Ý¿â»úÆ÷µÄIOPS¡¢Êý¾Ý¿âµÄActive SessionsµÈ·ÖÎö³öÀ´¡£
ѰÕÒδ³ä·ÖʹÓÃ×ÊÔ´µ«³ÌÐòÖ´ÐÐÂýµÄÔÒòºÍ´úÂ룺×ÊÔ´ÏûºÄ²»¶à¡¢µ«³ÌÐòµÄÏìÓ¦ËÙ¶ÈÈÔÈ»´ï²»µ½ÒªÇóµÄÖ÷ÒªÔÒòÊdzÌÐò´úÂëÔËÐÐЧÂʲ»¹»¸ß¡¢Î´³ä·ÖʹÓÃ×ÊÔ´»ò³ÌÐò½á¹¹²»ºÏÀí¡£
¶ÔÓÚJavaÓ¦ÓöøÑÔ£¬Ñ°ÕÒÐÔÄÜÆ¿¾±µÄ·½·¨Í¨³£ÎªÊ×ÏÈ·ÖÎö×ÊÔ´µÄÏûºÄ£¬È»ºó½áºÏOSºÍJavaµÈһЩ·ÖÎö¹¤¾ßÀ´²éÕÒ³ÌÐòÖÐÔì³É×ÊÔ´ÏûºÄ¹ý¶àµÄ´úÂë¡£
4.1CPUÏûºÄ·ÖÎö
ÔÚLinuxÖУ¬CPUÖ÷ÒªÓÃÓÚ Öжϡ¢Äں˽ø³ÌÒÔ¼°Óû§½ø³ÌµÄÈÎÎñ´¦Àí£¬ÓÅÏȼ¶Îª£ºÖжÏ
> Äں˽ø³Ì > Óû§½ø³Ì£¬ÔÚѧϰÈçºÎ·ÖÎöCPUÏûºÄ×´¿öǰ£¬ÓÐÈý¸öÖØÒª¸ÅÄî½»´úһϡ£
4.1.1 ÉÏÏÂÎÄÇл»¡¢ÔËÐжÓÁС¢ÀûÓÃÂÊ
4.1.1.1ÉÏÏÂÎÄÇл»
ÿ¸öCPU(»ò¶àºËCPUÖеÄÿºËCPU)ÔÚͬһʱ¼äÖ»ÄÜÖ´ÐÐÒ»¸öÏ̡߳£ Linux²ÉÓõÄÊÇÇÀռʽµ÷¶È£º¼´ÎªÃ¿¸öÏ̷߳ÖÅäÒ»¶¨µÄÖ´ÐÐʱ¼ä£¬µ±µ½´ïÖ´ÐÐʱ¼ä¡¢Ïß³ÌÖÐÓÐIO×èÈû»ò¸ßÓÅÏȼ¶Ïß³ÌÒªÖ´ÐÐʱ£¬Linux½«Çл»Ö´ÐеÄỊ̈߳¬ÔÚÇл»Ê±Òª´æ´¢Ä¿Ç°³ÌÐòµÄÖ´ÐÐ״̬PCB(Program
Control Block)£¬²¢»Ö¸´ÒªÖ´ÐеÄÏ̵߳Ä״̬£¬Õâ¸ö¹ý³Ì¾Í³ÆÎªÉÏÏÂÎÄÇл»¡£
¶ÔÓÚJavaÓ¦Ó㬵äÐ͵ÄÊÇÔÚ½øÐÐÎļþIO²Ù×÷¡¢ÍøÂçIO²Ù×÷¡¢ËøµÈ´ý»òÏß³ÌSleepʱ£¬µ±Ç°Ïß³Ì»á½øÈë×èÈû»òÐÝÃß״̬£¬´Ó¶ø´¥·¢ÉÏÏÂÎÄÇл»£¬ÉÏÏÂÎÄÇл»¹ý¶à»áÔì³ÉÄÚºËÕ¼¾Ý½Ï¶àµÄCPUʹÓã¬Ê¹µÃÓ¦ÓõÄÏìÓ¦ËÙ¶ÈϽµ¡£
4.1.1.2ÔËÐжÓÁÐ
ÿ¸öCPUºË¶¼Î¬»¤ÁËÒ»¸ö¿ÉÔËÐеÄÏ̶߳ÓÁУ¬ÀýÈçÒ»¸ö4ºËµÄCPU, JavaÓ¦ÓÃÖÐÆô¶¯ÁË8¸öỊ̈߳¬ÇÒÕâ8¸öÏ̶߳¼´¦ÓÚ¿ÉÔËÐÐ״̬£¬ÄÇôÔÚ·ÖÅ䯽¾ùµÄÇé¿öÏÂÿ¸öCPUÖеÄÔËÐжÓÁÐÀï¾Í»áÓÐÁ½¸öÏ̡߳£Í¨³£¶øÑÔ£¬ÏµÍ³µÄloadÖ÷ÒªÓÉCPUµÄÔËÐжÓÁÐÀ´¾ö¶¨£¬¼ÙÉèÒÔÉÏ×´¿öά³ÖÁË1·ÖÖÓ£¬ÄÇôÕâ1·ÖÖÓÄÚϵͳµÄload¾Í»áÊÇ2£¬µ«ÓÉÓÚloadÊǸö¸´ÔÓµÄÖµ£¬Òò´ËÒ²²»ÊǾø¶ÔµÄ£¬ÔËÐжÓÁÐÖµÔ½´ó£¬¾ÍÒâζ×ÅÏß³ÌÒªÏûºÄÔ½³¤µÄʱ¼ä²ÅÄÜÖ´ÐÐÍê¡£
Linux System and NewWork Performance MonitoringÖн¨Òé¿ØÖÆÔÚÿ¸öCPUºËÉϵÄÔËÐжÓÁÐΪ1-3¸ö¡£
4.1.1.3ÀûÓÃÂÊ
CPUÀûÓÃÂÊΪCPUÔÚÓû§½ø³Ì¡¢Äں˽ø³Ì¡¢Öжϴ¦Àí¡¢IOµÈ´ýÒÔ¼°¿ÕÏÐÎå¸ö²¿·ÖʹÓõİٷֱȣ¬ÕâÎå¸öÖµÊÇÓÃÀ´·ÖÎöCPUÏûºÄÇé¿öµÄ¹Ø¼üÖ¸±ê¡£
Linux System and NewWork Performance Monitoring Öн¨ÒéÓû§½ø³ÌµÄCPUÏûºÄ/Äں˵ÄCPUÏûºÄµÄ±ÈÀýÔÚ
65%-70% / 30%-35%
4.1.2Linux¹Û²âCPUÏûºÄ״̬µÄ¹¤¾ß£ºperf¡¢top¡¢vmstat¡¢pidstat¡¢sar¡¢pcpu¡¢ps
Hh -eo tid
4.1.2.0 ¹¤¾ß perf ÐÔÄܲâÊÔ¹¤¾ß
sudo apt-get install linux-tools-common
sudo apt-get install linux-tools-3.13.0-27-generic
4.1.2.1¹¤¾ßtop
Óù¤¾ßSSHµÇ½µ½Linux ÉϺó£¬ÔÚ×Ö·û½çÃæÏÂÊäÈëtopÃüÁîºó¼´¿É²é¿´CPUµÄÏûºÄÇé¿ö£¬CPUµÄÐÅÏ¢ÔÚTOPÊÓͼµÄÉÏÃæ¼¸ÐÐÖÐ
ͼ-4 Top²é¿´CPUʹÓÃÇé¿öͼ£ºµÚËÄÐб»¾ØÐÎΧÆðÀ´µÄ²¿·Ö£ºÆäÖÐ 0.2% us ±íʾÓû§½ø³Ì´¦ÀíËùÕ¼µÄ°Ù·Ö±È£»0.7%
sy ±íʾΪÄÚºËÏ̴߳¦ÀíËùÕ¼µÄ°Ù·Ö±È£» 0.0% ni ±íʾ±»niceÃüÁî¸Ä±äÓÅÏȼ¶µÄÈÎÎñËùÕ¼µÄ°Ù·Ö±È£»99.0%
id ±íʾCPUµÄidle¿ÕÏÐʱ¼äËùÕ¼µÄ°Ù·Ö±È£» 0.0% wa ±íʾÔÚÖ´ÐеĹý³ÌÖеȴýIOËùÕ¼µÄ°Ù·Ö±È£»
0.2% hi ±íʾӲ¼þÖжÏËùÕ¼µÄ°Ù·Ö±È£» 0.0% si ±íʾÈí¼þÖжÏËùÕ¼µÄ°Ù·Ö±È¡£

ͼ-4 Top²é¿´CPUʹÓÃÇé¿öͼ
linuxϲ鿴¸ù¾Ý½ø³Ì²é¿´Ï̵߳ķ½·¨
1¡¢cat /proc/${pid}/status
2¡¢pstree -p ${pid}
3¡¢3.1 top -p ${pid} ÔÙtop ÖмüÈë[Shift]-[H]×éºÏ¼ü¡£
3.2 »òÕßÖ±½ÓÊäÈë top -bH -d 3 -p ${pid}
top -H ÊÖ²áÖÐ˵£º-H : Threads toggle
¼ÓÉÏÕâ¸öÑ¡ÏîÆô¶¯top£¬topÒ»ÐÐÏÔʾһ¸öÏ̡߳£·ñÔò£¬ËüÒ»ÐÐÏÔʾһ¸ö½ø³Ì¡£
1 ÏÈÓÃps + grepÕÒ³ö¸ÃËÀµÄ½ø³Ìpid£¬±ÈÈç 30420
2 top -H -p 30420£¬(topÈ»ºóshift+H¿ÉÒÔ¿´³öij¸öÏß³Ì)ËùÓиýø³ÌµÄÏ̶߳¼ÁгöÀ´ÁË¡£¿´¿´ÄĸöÏß³ÌpidÕ¼ÓÃ×î¶à£¬È»ºó½«Õâ¸öpidת»»Îª16½øÖÆ£¬Èç
44bf£¬×¢ÒâҪСд
3 jstack 30420 | less£¬È»ºó²éÕÒ nid=0x44bf£¬£¬ÕÒµ½ÁË
4¡¢ps xH
ÊÖ²áÖÐ˵£ºH Show threads as if they were
processes
ÕâÑù¿ÉÒԲ鿴ËùÓдæÔÚµÄÏ̡߳£
5¡¢ps -mp <PID>
ÊÖ²áÖÐ˵£ºm Show threads after processes
ÕâÑù¿ÉÒԲ鿴һ¸ö½ø³ÌÆðµÄÏß³ÌÊý¡£
4.1.2.2¹¤¾ßpidstat
pidstat ÊÇSYSSTATÖеŤ¾ß.£¬ÈçÐèʹÓÃpidstat,ÇëÏȰ²×°SYSSTAT:http://www.icewalkers.com/Linux/Software/59040/sysstat.html
°²×°·½·¨: root$ sudo apt-get install sysstat
SYSSTAT°üº¬¹¤¾ßÓУºsysstat, sar, sadf, iostat,
pidstat,mpstat, nfsiostat and cifsiostat commands for
Linux.
ʹÓÃÀý×Ó£º
sar -n DEV 1 100
sar -n DEV 1 100 | grep lo
ÊäÈëpidstat 1 2,ÔÚconsoleÉϽ«»áÿ¸ô1ÃëÊä³öĿǰ»î¶¯½ø³ÌµÄCPUÏûºÄ×´¿ö£¬¹²Êä³ö2´Î£¬½á¹ûͼ-5
pidstatʹÓÃʾÀýͼ Ëùʾ£º

ͼ-5 pidstatʹÓÃʾÀýͼ
ÆäÖÐCPU±íʾµÄΪµ±Ç°½ø³ÌËùʹÓõ½µÄCPU¸öÊý£¬
ÈçÐë²é¿´½ø³ÌÖÐÏ̵߳ÄCPUÏûºÄ×´¿ö£¬¿ÉÒÔÊäÈë pidstat -p [PID]
-t 1 5 ÕâÑùµÄ·½Ê½À´²é¿´£¬Ö´ÐкóµÄÊä³öÈçÏ£º

ͼ-6
ͼÖеÄTID¼´ÎªÏß³ÌID,½ÏÖ®topÃüÁʽ¶øÑÔ£¬ pidstatµÄºÃ´¦Îª¿É²é¿´Ã¿¸öÏ̵߳ľßÌåCPUÀûÓÃÂʵÄ×´¿ö(
ÀýÈç%usr ¡¢%system )
4.1.2.3CPUÏûºÄ·½ÃæÏà¹Ø¹¤¾ß½éÉÜ
³ýÁËtop¡¢pidstatÍâ¡¢linuxÖл¹¿ÉÒÔʹÓÃvmstatÀ´²ÉÑù(ÀýÈçÿÃë vmstat 1)²é¿´CPUµÄÉÏÏÂÎÄÇл»¡¢ÔËÐжÓÁС¢ÀûÓÃÂʵľßÌåÐÅÏ¢¡£
ps Hh -eo tid,pcpu ·½Ê½Ò²¿ÉÓÃÀ´²é¿´¾ßÌåÏ̵߳ÄCPUÏûºÄ×´¿ö£»sarÀ´²é¿´Ò»¶¨Ê±¼ä·¶Î§ÄÚÒÔ¼°ÀúÊ·µÄcpuÏûºÄ×´¿öÐÅÏ¢¡£
SYSSTAT°üº¬¹¤¾ßÓУºSysstat, sar, sadf, iostat,
pidstat,mpstat, nfsiostat and cifsiostat commands for
Linux.
The sysstat package contains the sar,
sadf, iostat, nfsiostat, cifsiostat, pidstat and mpstat
commands for Linux.
The sar command collects and reports
system activity information. The information collected
by sar can be saved in a file in a binary format for
future inspection. The statistics reported by sar concern
I/O transfer rates, paging activity, process-related
activities, interrupts, network activity, memory and
swap space utilization, CPU utilization, kernel activities
and TTY statistics, among others.
The sadf command may be used to display
data collected by sar in various formats (CSV, XML,
database-friendly, etc.).
The iostat command reports CPU utilization
and I/O statistics for disks and network filesystems.
The pidstat command reports statistics
for Linux tasks (processes).
The mpstat command reports global and
per-processor statistics. Both UP and SMP machines are
fully supported.
Sysstat has also support for hotplug
CPU's, and for National Language (NLS).
The nfsiostat command reports I/O statistics
for network filesystems.
The cifsiostat command reports I/O statistics
for CIFS filesystems.
µ±CPUÏûºÄÑÏÖØÊ±£¬Ö÷ÒªÌåÏÖÔÚus¡¢sy¡¢wa »ò hiµÄÖµ±ä¸ß£¬ waµÄÖµÊÇIOµÈ´ýÔì³ÉµÄ£¬hiµÄÖµ±ä¸ßÖ÷ҪΪӲ¼þÖÕ¶ËÔì³ÉµÄ£¬ÀýÈçÍø¿¨½ÓÊÜÊý¾ÝƵ·±µÄ×´¿ö¡£
¶ÔÓÚJavaÓ¦ÓöøÑÔ£¬CPUÏûºÄÑÏÖØÖ÷ÒªÌåÏÖÔÚus¡¢syÁ½¸öÖµÉÏ£¬·Ö±ð¿´¿´JavaÓ¦ÓÃÔÚÕâÁ½¸öÖµ¸ßµÄÇé¿öÏÂÓ¦ÈçºÎÕÒµ½¶ÔÓ¦Ôì³ÉÆ¿¾±µÄ´úÂë¡£
4.1.3 us (CPUµÄÓû§½ø³Ì´¦ÀíËùÕ¼µÄ°Ù·Ö±È)
µ±usÖµ¹ý¸ßʱ£¬±íʾÔËÐеÄÓ¦ÓÃÏûºÄÁ˴󲿷ֵÄCPU. ÔÚÕâÖÖÇé¿öÏ£¬¶ÔÓÚJavaÓ¦ÓöøÑÔ£¬×îÖØÒªµÄÊÇÕÒµ½¾ßÌåÏûºÄCPUµÄÏ̼߳°ËøÖ´ÐеĴúÂë¡£
pstackÏÔʾÿ¸ö½ø³ÌµÄÕ»¸ú×Ù
pstreeÒÔÊ÷½á¹¹ÏÔʾ½ø³Ì
·½·¨ÈçÏ£º
Ê×ÏÈͨ¹ýLinuxÌṩµÄÃüÁî(È磺top/jpsÕÒµ½java½ø³Ìpid£¬È»ºópidstatͨ¹ý½ø³ÌpidÕÒµ½¶ÔÓ¦µÄÏß³Ìtid)ÕÒµ½ÏûºÄCPUÑÏÖØµÄÏ̼߳°ÆäID,½«´ËÏß³ÌIDת»¯ÎªÊ®Áù½øÖƵÄÖµ¡£
Æä´Îͨ¹ýkill -3 [javapid]»òjstack µÄ·½Ê½dump³öÓ¦ÓõÄjavaÏß³ÌÐÅÏ¢£¬Í¨¹ý֮ǰת»¯³öµÄÊ®Áù½øÖƵÄÖµÕÒµ½¶ÔÓ¦µÄnidÖµµÄÏ̡߳£´ËÏ̼߳´ÎªÏûºÄCPUµÄỊ̈߳¬ÔÚ²ÉÑùʱÐë¶àÖ´Ðм¸´ÎÉÏÊöµÄ¹ý³Ì£¬ÒÔÈ·±£ÕÒµ½ÕæÊµµÄÏûºÄCPUµÄÏ̡߳£
ѰÕÒ¹ý³Ì£ºµÚÒ»²½£ºjpsÕÒµ½java½ø³ÌºÅPID=2418, È»ºópidstat
-p 2418 -t 1 5 | less ÕÒµ½ÏûºÄCPU¶àµÄÏ̺߳Åtid=2428,È»ºó½«tid=2428(Ê®½øÖÆ)
ת»»ÎªÊ®Áù½øÖÆÊý£º97C.

ͼ-7 jpsÃüÁîͼ

ͼ-8 pidstat ÕÒµ½ÏûºÄCPU¸ßµÄtidÏß³ÌID
µÚ¶þ²½£ºjstack 2418 > jstack03.txt ½«Ïß³ÌdumpÐÅϢдÈëÎļþjstack03.txt,
È»ºó½«Ê®Áù½øÖƵÄÏß³ÌÊý£º97C Ϊ79CΪ²éѯÌõ¼þ£¬µ½jstack03.txtÎļþÖвéÕÒµ½nid= 97CµÄỊ̈߳¬¸ÃÏ̼߳´ÎªÏûºÄCPU¶àµÄÏ̡߳£

ͼ-9 Ïß³ÌÕ»dumpÐÅÏ¢
´ÓÉÏ¿ÉÒÔ¿´³ö£¬Ö÷ÒªÊÇConsumeCPUTaskµÄÖ´ÐÐÏûºÄÁËCPU,µ«ÓÉÓÚjstackÐèҪʱ¼ä£¬Òò´Ë»ùÓÚjstack²¢²»Ò»¶¨ÄÜ·ÖÎö³öÕæÕýµÄÏûºÄCPUµÄ´úÂëÊÇÄÄÐС£
ÀýÈçÔÚÒ»¸ö²Ù×÷ÖÐÑ»·µ÷ÓÃÁ˺ܶàÆäËûµÄ²Ù×÷£¬ÈçÆäËûµÄ²Ù×÷ÿ´Î¶¼±È½Ï¿ì£¬µ«ÓÉÓÚÑ»·Ì«¶à´Î£¬Ôì³ÉÁËCPUÏûºÄ£¬ÔÚÕâÖÖÇé¿öÏÂjstackÊÇÎÞ·¨²¶×½³öÀ´µÄ¡£×î¼Ñ·½Ê½ÊÇͨ¹ý
intel vtuneÀ´½øÐзÖÎö£¬vtuneÊÇÉÌÒµÈí¼þ(IntelоƬµÄÒ»¿îÐÔÄÜ·ÖÎöÈí¼þ,ÓÃAMDоƬµÄͬѧ¾Í²»ÄÜÓÃÁË)¡£
ÔÚ²»Ê¹ÓÃvtuneµÄÇé¿öÏ£¬ÔòÖ»ÄÜͨ¹ýÈÏÕæ²é¿´Õû¸öÏß³ÌÖÐÖ´Ðе͝×÷À´·ÖÎöÔÒò£¬ÀýÈçÔÚ´ÓÉÏͼÖУ¬¿ÉÒÔ¿´³öConsumeCPUTaskÒ»Ö±´¦ÓÚÔËÐÐ״̬£¬¿ÉÒÔ·ÖÎöConsumeCPUTaskÕâ¸öÏ߳̾ßÌåÔÚ×öµÄ¶¯×÷£¬´ÓÆä´úÂë¿É¿´³öÕæ¸öÏß³ÌÒ»Ö±´¦ÓÚÔËÐйý³ÌÖУ¬ÖÐ;ûÓÐIOÖжϡ¢ËøµÈ´ýÏÖÏó£¬Òò´ËÔì³ÉÁËCPUÏûºÄÑÏÖØ¡£
¸ÃʾÀýÔ´´úÂëUsHighOfCpuDemo.javaÏÂÔØµØÖ·Îª£º¼û²Î¿¼²¿·Ö
4.1.4 sy (ÄÚºËÏ̴߳¦ÀíËùÕ¼µÄ°Ù·Ö±È)
µ±syÖµ¸ßʱ±íʾLinux»¨·ÑÁ˸ü¶àµÄʱ¼äÔÚ½øÐÐÏß³ÌÇл»£¬JavaÓ¦ÓÃÔì³ÉÕâÖÖÏÖÏóµÄÖ÷ÒªÔÒòÊÇÆô¶¯µÄÏ̱߳Ƚ϶࣬ÇÒÕâЩÏ̶߳àÊý¶¼´¦ÓÚ²»¶ÏµÄ×èÈû(ÀýÈçËøµÈ´ý¡¢IOµÈ´ý״̬)ºÍÖ´ÐÐ״̬µÄ±ä»¯¹ý³ÌÖУ¬Õâ¾Íµ¼ÖÂÁ˲Ù×÷ϵͳҪ²»¶ÏµØÇл»Ö´ÐеÄỊ̈߳¬²úÉú´óÁ¿µÄÉÏÏÂÎÄÇл»¡£ÔÚÕâÖÖ×´¿öÏ£¬¶ÔJavaÓ¦ÓöøÑÔ£¬×îÖØÒªµÄÊÇÕÒ³öÏ̲߳»¶ÏÇл»×´Ì¬µÄÔÒò£¬¿É²ÉÓõķ½·¨ÎªÍ¨¹ý
kill -3 [javapid] »òjstack -l [javapid] µÄ·½Ê½dump³öjavaÓ¦ÓóÌÐòµÄÏß³ÌÐÅÏ¢£¬²é¿´Ï̵߳Ä״̬ÐÅÏ¢ÒÔ¼°ËøÐÅÏ¢£¬ÕÒ³öµÈ´ý״̬»òËø¾ºÕù¹ý¶àµÄÏ̡߳£
µÚÒ»²½ÔËÐÐ [root@localhost example]# java
-jar SyHighOfCpuDemo.jar £¬ topͼÈçÏ£º CPUµÄ syʹÓñȽϸß

ͼ-10
ÔÚÓÃvmstat -n 3 Ö´ÐÐһϿ´Ò»ÏÂÉÏÏÂÎÄÇл»Çé¿ö

ͼ-11
ÓÉÉÏ¿ÉÖª£¬CPUÔÚcs(ÄÚºËÏß³ÌÉÏÏÂÎÄÇл») ÒÔ¼°syÉÏÏûºÄºÜ´ó¡£
µÚ¶þ²½ ÔËÐÐʱ²ÉÓÃjstack -l ²é¿´³ÌÐòµÄÏß³Ì×´¿ö£¬¿ÉÒÔ¿´µ½Æô¶¯Á˺ܶàỊ̈߳¬²¢ÇҺܶàµÄÏ̶߳¼¾³£´¦ÓÚ
TIMED_WAITING(on object monitor)״̬¡¢WAITING (on object
monitor)ºÍRunnable״̬µÄÇл»ÖС£
ͨ¹ýon object monitor¶ÔÓ¦µÄ¶ÑÕ»ÐÅÏ¢£¬¿É²éÕÒµ½ÏµÍ³ÖÐËø¾ºÕù¼¤ÁҵĴúÂ룬ÕâÊÇÔì³Éϵͳ¸ü¶àʱ¼äÏûºÄÔÚÏß³ÌÉÏÏÂÎÄÇл»µÄÔÒò¡£
¾ßÌåTIMED_WAITING(on object monitor)״̬¡¢WAITING
(on object monitor)ºÍRunnable״̬ÈçÏÂͼ

ͼ-12

ͼ-13

ͼ-14
¸ÃʾÀýÔ´´úÂëSyHighOfCpuDemo.javaÏÂÔØµØÖ·Îª£º¼û²Î¿¼
4.2ÄÚ´æÏûºÄ·ÖÎö
JavaÓ¦ÓöÔÓÚÄÚ´æµÄʹÓðüÀ¨Á½·½Ãæ JVM¶ÑÄÚ´æ ºÍ JVM¶ÑÍâÄÚ´æ¡£
JavaÓ¦ÓöÔÄÚ´æµÄÏûºÄÉÏÖ÷ÒªÊÇÔÚJVM¶ÑÄÚ´æÉÏ¡£ÔÚÕýʽ»·¾³ÖУ¬¶àÊýJavaÓ¦Óö¼»á½« -Xms ºÍ -XmxÉèΪÏàͬµÄÖµ£¬±ÜÃâÔËÐÐÆÚÒª²»¶ÏÉêÇëÄÚ´æ¡£
ĿǰJavaÓ¦ÓÃÖ»ÓÐÔÚ´´½¨Ï̺߳ÍʹÓÃDirect ByteBuffer
ʱ²Å»á²Ù×÷JVM¶ÑÍâµÄÄÚ´æJVM,Òò´ËÔÚÄÚ´æÏûºÄ·½Ãæ×îÖµµÃ¹Ø×¢µÄÊÇJVMÄÚ´æµÄÏûºÄ×´¿ö¡£¶ÔÓÚJVMÄÚ´æÏûºÄ×´¿ö·ÖÎöµÄ·½·¨¹¤¾ßÓУºJVM£¨jmap¡¢jstat¡¢mat¡¢visualvmµÈ·½·¨£©¡£JVMÄÚ´æÏûºÄ¹ý¶à»áµ¼ÖÂGCÖ´ÐÐÆµ·±£¬CPUÏûºÄÔö¼Ó£¬Ó¦ÓÃÏ̵߳ÄÖ´ÐÐËÙ¶ÈÑÏÖØÏ½µ£¬ÉõÖÁÔì³ÉOutOfMemoryError,×îÖÕµ¼ÖÂJava½ø³ÌÍ˳ö¡£
sudo -u admin /opt/taobao/java/bin/jmap -F -dump:live,format=b,file=/home/admin/dumpcrash_20150909_2.log 6606 sudo -u admin /opt/company/java/bin/jmap -F -dump:file=/home/user/dumpcrash_xxx.log 1234 /opt/company/java/bin/jmap -F -dump:file=/home/user/dumpcrash_xxx.log 1234 jmap -histo 13321> dump_platform.core jmap -dump:live,format=b,file=heap_platform.core 13321 jmap ¨Cdump:file=<ÎļþÃû>,format=b [pid] |
·ÖÎöjmapµÄdumpÎļþ: Eclipse Memory Analyzer
http://www.eclipse.org/mat/downloads.php
¶ÔÓÚJVM¶ÑÒÔÍâµÄÄÚ´æ·½ÃæµÄÏûºÄ£¬×îΪֵµÃ¹Ø×¢µÄÊÇswapµÄÏûºÄ ÒÔ¼°
ÎïÀíÄÚ´æµÄÏûºÄ£¬ÕâÁ½·½ÃæµÄÏûºÄ¶¼¿É»ùÓÚ²Ù×÷ϵͳµÄÃüÁîÀ´²é¿´¡£

·½·¨ÇøMethod AreaÄÚ´æ´óС¶ÔÓ¦µÄÅäÖòÎÊýÊÇ:-XX:PermSize
-XX:MaxPermSize Õâ¸öÅ׳öOutOfMemoryError: PermGen space
(³£Á¿³Ø»òÀàÌ«¶à£¬Èç·´Éä,CGLibµÈ)
Java¶ÑHeapÄÚ´æ´óС¶ÔÓ¦µÄÅäÖòÎÊýÊÇ:-Xms -Xmx Õâ¸öÅ׳öOutOfMemoryError:
Java heap space
ÐéÄâ»úÕ»VM stackÄÚ´æ´óС¶ÔÓ¦µÄÅäÖòÎÊýÊÇ:-Xss Õâ¸öÅ׳öStackOverflowError;
OutOfMemoryError:unable to create new native thread
±¾µØ·½·¨Õ»Native Method stackÄÚ´æ´óС¶ÔÓ¦µÄÅäÖòÎÊýÊÇ:-XX:DirectoryByteBuffer
-XX:MaxDirectoryByteBuff Õâ¸öÅ׳öOutOfMemoryError: sun.misc.Unsafe.allocateMemory(Native
Method)
ÔÚJavaÓïÑÔÖУ¬¶ÔÏó·ÃÎÊÊÇÈçºÎ½øÐеÄÄØ£¿
¶ÔÏó·ÃÎÊÔÚJavaÓïÑÔÖÐÎÞ´¦²»Ôڵ쬼´Ê¹×îÆÕͨµÄ³ÌÐòÐÐΪ£¬Ò²»áÉæ¼°JavaÕ»¡¢Java¶Ñ¡¢·½·¨ÇøÕâÈý¸ö×îÖØÒªÄÚ´æÇøÓòÖ®¼äµÄ¹ØÁª¹ØÏµ£¬ÈçÏ´úÂ룺
Object obj = new Object();
¼ÙÉèÕâ¾ä´úÂë³öÏÖÔÚ·½·¨ÌåÖУ¬ÄÇ¡°Object obj¡±Õⲿ·ÖµÄÓïÒ彫»á·´Ó³µ½¡¾JavaÕ»¡¿µÄ±¾µØ±äÁ¿±íÖУ¬×÷Ϊһ¸öreferenceÀàÐÍÊý¾Ý³öÏÖ¡£
¶ø¡°new Object()¡±Õⲿ·ÖµÄÓïÒ彫»á·´Ó³µ½¡¾Java¶Ñ¡¿ÖУ¬ÐγÉÒ»¿é´æ´¢ÁËObjectÀàÐÍËùÓÐʵÀýÊý¾ÝÖµ(Instance
Data,¶ÔÏóÖи÷¸öʵÀý×ֶεÄÊý¾Ý)µÄ½á¹¹»¯Äڴ棬¸ù¾Ý¾ßÌåÀàÐÍÒÔ¼°ÐéÄâ»úʵÏֵĶÔÏóÄÚ´æ²¼¾Ö(Object
Memory Layout)µÄ²»Í¬£¬Õâ¿éÄÚ´æµÄ³¤¶ÈÊDz»¹Ì¶¨µÄ¡£
ÁíÍâÔÚJava¶ÑÖл¹±ØÐë°üº¬ÄܲéÕÒµ½´Ë¶ÔÏóÀàÐÍÊý¾Ý(Èç¶ÔÏóÀàÐÍ¡¢¸¸ÀࡢʵÏֵĽӿڡ¢·½·¨µÈ)µÄµØÖ·ÐÅÏ¢£¬ÕâЩÀàÐÍÊý¾ÝÔò´æ´¢ÔÚ¡¾·½·¨ÇøMethod
Area¡¿ÖС£
Àà¼ÓÔØµÄ²½Ö裺

4.2.1ÄÚ´æÏûºÄ·ÖÎö¹¤¾ß½éÉÜ
ÔÚLinuxÖпÉͨ¹ývmstat¡¢sar¡¢top¡¢pidstat, pmapµÈ·½Ê½À´²é¿´
swapºÍÎïÀíÄÚ´æµÄÏûºÄ×´¿ö¡£
4.2.1.1 vmstat
ÔÚÃüÁîÐÐÖÐÊäÈëvmstat,ÆäÖеÄÐÅÏ¢ºÍÄÚ´æÏà¹ØµÄÖ÷ÒªÊÇmemoryϵÄswpd¡¢free¡¢buff¡¢cacheÒÔ¼°swapϵÄsi
ºÍ so.

ͼ-15
ÆäÖÐswpdÊÇÖ¸ÐéÄâÄÚ´æÒÑʹÓõIJ¿·Ö£¬µ¥Î»Îªkb£»free±íʾ¿ÕÏеÄÎïÀíÄڴ棻buffer±íʾÓÃÓÚ»º³åµÄÄڴ棻cache±íʾÓÃÓÚ×÷Ϊ»º´æµÄÄÚ´æ
swapϵÄsiÊÇָÿÃë´Ódisk¶ÁÖÁÄÚ´æµÄÊý¾ÝÁ¿£»soÊÇָÿÃë´ÓÄÚ´æÖÐдÈëdiskµÄÊý¾ÝÁ¿¡£
swpdÖµ¹ý¸ßͨ³£ÊÇÓÉÓÚÎïÀíÄÚ´æ²»¹»Ó㬲Ù×÷ϵͳ½«ÎïÀíÄÚ´æÖеÄÒ»²¿·ÖÊý¾ÝתΪ·ÅÈëÓ²ÅÌÉϽøÐд洢£¬ÒÔÌÚ³ö×ã¹»µÄ¿Õ¼ä¸øµ±Ç°ÔËÐеijÌÐòʹÓá£
ÔÚĿǰÔËÐеijÌÐò±ä»¯ºó£¬¼´´ÓÓ²ÅÌÉÏÖØÐ¶ÁÈ¡Êý¾Ýµ½ÄÚ´æÖУ¬ÒÔ±ã»Ö¸´³ÌÐòµÄÔËÐУ¬Õâ¸ö¹ý³Ì»á²úÉúswap
IO, Òò´Ë¿´swapµÄÏûºÄÇé¿öÖ÷Òª¹Ø×¢µÄÊÇswapIOµÄ×´¿ö£¬ÈçswapIO·¢ÉúµÃ½ÏƵ·±£¬ÄÇôÑÏÖØÓ°ÏìϵͳµÄÐÔÄÜ¡£
ÓÉÓÚJavaÓ¦ÓÃÊǵ¥½ø³ÌÓ¦Óã¬Òò´ËÖ»ÒªJVMµÄÄÚ´æÉèÖò»Êǹý´ó£¬ÊDz»»á²Ù×÷µ½swapÇøÓòµÄ¡£ÎïÀíÄÚ´æÏûºÄ¹ý¸ß¿ÉÄÜÊÇÓÉÓÚJVMÄÚ´æÉèÖùý´ó¡¢´´½¨µÄJavaÏ̹߳ý¶à»òͨ¹ýDirect
ByteBufferÍùÎïÀíÄÚ´æÖзÅÖÃÁ˹ý¶àµÄ¶ÔÏóÔì³ÉµÄ¡£
4.2.2.2 sar
ͨ¹ýsarµÄ-r²ÎÊý¿É²é¿´ÄÚ´æµÄÏûºÄ×´¿ö£¬ÀýÈçsar -r 2 5

ͼ-16
ÎïÀíÄÚ´æÏà¹ØµÄÐÅÏ¢Ö÷ÒªÊÇkbmemfree¡¢kbmemused¡¢%memused¡¢kbbuffers¡¢kbcached,
µ±ÎïÀíÄÚ´æÓпÕÏÐʱ£¬Linux»áʹÓÃһЩÎïÀíÄÚ´æÓÃÓÚbufferºÍcache, ÒÔÌáÉýϵͳµÄÔËÐÐЧÂÊ£¬Òò´Ë¿ÉÒÔÈÏΪϵͳÖпÉÓõÄÎïÀíÄÚ´æÎª£º
kbmemfree + kbmemused + kbcached.
sarÏà±ÈvmstatµÄºÃ´¦ÊÇ¿ÉÒÔ²éѯÀúÊ·×´¿ö£¬ÒÔ¸ü¼Ó׼ȷµØ·ÖÎöÇ÷ÊÆ×´¿ö£¬ÀýÈç
sar -r -f /temp/log/sa/sa12
vmstat ºÍ sarµÄ¹²Í¬ÈõµãÊDz»ÄÜ·ÖÎö½ø³ÌËùÕ¼ÓõÄÄÚ´æÁ¿¡£
4.2.2.3 top
ͨ¹ýtop¿É²é¿´½ø³ÌËùÏûºÄµÄÄÚ´æÁ¿£¬²»¹ýtopÖп´µ½µÄJava½ø³ÌÏûºÄµÄÄÚ´æ¡£ÒòΪJava½ø³ÌÊǰüÀ¨ÁËJVMÒÑ·ÖÅäµÄÄÚ´æ¼ÓÉÏJavaÓ¦ÓÃËùºÄ·ÑµÄJVMÒÔÍâµÄÎïÀíÄڴ棬Õâ»áµ¼ÖÂtopÖп´µ½Java½ø³ÌËùÏûºÄµÄÄÚ´æ´óСÓпÉÄܳ¬¹ý
-Xmx ¼ÓÉÏ -XX:MaxPermSizeÉèÖõÄÄÚ´æ´óС£¬²¢ÇÒJava³ÌÐòÔÚÆô¶¯ºóÒ²Ö»ÊÇÕ¼¾ÝÁË-XmsµÄµØÖ·¿Õ¼ä£¬µ«²¢Ã»ÓÐÕ¼¾Ýʵ¼ÊµÄÄڴ棬ֻÓÐÔÚÏàÓ¦µÄµØÖ·¿Õ¼ä±»Ê¹Óùýºó²Å±»¼ÆÈëÏûºÄµÄÄÚ´æÖС£Òò´Ë´¿´âµÄ¸ù¾ÝtopºÜÄÑÅжϳöJava½ø³ÌÏûºÄµÄÄÚ´æÖÐÓжàÉÙÊÇÊôÓÚJVMµÄ£¬ÓжàÉÙÊÇÊôÓÚJVMÍâµÄÄÚ´æ¡£
Ò»¸öС¼¼ÇÉÊǶÔÓÉÓÚÄÚ´æÂú¶ø·¢Éú¹ýFull GCµÄÓ¦ÓöøÑÔ(²»ÊÇÖ÷¶¯µ÷ÓÃSystem.gcµÄÓ¦ÓÃ)£¬¶àÊýÇé¿öÏÂ(ÀýÈçÓÉÓÚ²úÉúµÄ¶ÔÏó¹ý´óµ¼ÖÂÖ´ÐÐFull
GC²¢Å׳öOutOfMemoryErrorµÄÏÖÏó¾ÍÒª³öÍâ)¿ÉÒÔÈÏΪÆäJava½ø³ÌÖÐÏÔʾ³öÀ´µÄÄÚ´æÏûºÄÖµ¼´Îª
JVM -XmxµÄÖµ + ÏûºÄµÄJVMÍâµÄÄÚ´æÖµ¡£

ͼ-17
4.2.2.4pidstat
ͨ¹ý pidÒ²¿ÉÒԲ鿴½ø³ÌËùÏûºÄµÄÄÚ´æÁ¿£¬ÃüÁî¸ñʽΪ: pidstat
-r -p [pid] [interval] [times], ÀýÈçÈçÏ£º

ͼ-18
²é¿´¸Ã½ø³ÌËùÕ¼ÓõÄÎïÀíÄÚ´æRSS(Resident Set Size)ºÍÐéÄâÄÚ´æµÄ´óСVSZ(virtual
memory size)¡£
´ÓÒÔÉϵö¹¤¾ßÀ´¿´£¬×î¼ÑµÄÄÚ´æÏûºÄ·ÖÎö·½·¨ÊǽáºÏtop»òpidstat,ÒÔ¼°JVMµÄÄÚ´æ·ÖÎö¹¤¾ßÀ´¹²Í¬·ÖÎöÄÚ´æÏûºÄ×´¿ö¡£
ÏÂÃæÍ¨¹ýÀý×Ó·Ö±ðչʾJavaÓ¦ÓöÔÎïÀíÄÚ´æµÄÏûºÄºÍ¶ÔJVM¶ÑÄÚµÄÏûºÄ¡£
4.2.2 JVM¶ÑÍâÄÚ´æÏûºÄ·ÖÎö
»ùÓÚDirect ByteBuffer¿ÉÒÔºÜÈÝÒ×µØÊµÏÖ¶ÔÎïÀíÄÚ´æµÄÖ±½Ó²Ù×÷£¬¶øÎÞÐëºÄ·ÑJVM
heapÇø¡£
Àý×ÓÖУ¬ÎªÁ˸üÇåÎúµØ¹Û²ìÄÚ´æµÄ±ä»¯Çé¿ö£¬·ÅÈëÁ˶à¸öThread.sleep,¼ÓÉÏ
-Xms140 -Xmx140²ÎÊýÖ´ÐÐÉÏÃæµÄ´úÂ룬ÔÚÖ´Ðйý³ÌÖнáºÏtopÃüÁîºÍjstatÃüÁî²é¿´java½ø³ÌÕ¼ÓÃÄÚ´æµÄ´óСÒÔ¼°JVM
heapµÄ±ä»¯Çé¿ö¡£
[root@localhost ~]# jps 4661 jar 4670 Jps [root@localhost ~]# pidstat -r -p 4661 1 100 |
jpsÕÒµ½java½ø³Ì£¬jstat -gcutil [pid] 1000
10
´úÂëʾÀý£ºHighNonHeapOfMemoryDemo.java
½áºÏÉÏÃætopºÍjstat¹Û²ìµ½µÄ״̬£¬¿ÉÒÔ²é³ö direct bytebufferÏûºÄµÄÊÇJVM
heap ÍâµÄÎïÀíÄÚ´æ¡£µ«ËüͬÑùÊÇ»ùÓÚGC·½Ê½À´Êͷŵģ¬Í¬Ê±Ò²¿ÉÒÔ¿´³öJVM heapÒ»µ©Ê¹Óú󣬼´Ê¹½øÐÐÁËGC,½øÐÐÖÐÈÔÈ»»áÏÔʾ֮ǰÆäËùÏûºÄµÄÄÚ´æ´óС£¬Òò´ËJVMÄÚ´æÖоßÌåµÄÏûºÄ×´¿ö±ØÐëͨ¹ýJDKÌṩµÄÃüÁî²Å¿ÉÒÔ׼ȷ·ÖÎö¡£
³ýÁËDirect Bytebuffer·½Ê½¶ÔJVMÍâÎïÀíÄÚ´æµÄÏûºÄÍ⣬´´½¨Ïß³ÌÒ²»áÏûºÄÒ»¶¨´óСµÄÄÚ´æ¡£ÕâÒ»·½ÃæÈ¡¾öÓÚ-Xss¶ÔÓ¦ÖµµÄ´óС£¬ÁíÒ»·½ÃæÒ²È¡¾öÓÚÏß³ÌstackµÄÉî¶È£¬µ±Ïß³ÌÍ˳öʱ£¬ÆäËùÕ¼ÓõÄÄڴ潫×Ô¶¯ÊÍ·Å¡£
4.2.3JVM¶ÑÄÚ´æÏûºÄ·ÖÎö£¨ÐèÒªÖØµã²¹³ä£©
JavaÓ¦ÓÃÖгýÁËDirect Bytebuffer¡¢´´½¨Ï̵߳ȲÙ×÷JVMÍâÎïÀíÄÚ´æµÄ·½·¨Í⣬´ó¶àÊý¶¼ÊǶÔÓÚJVM
heapÇøµÄÏûºÄ¡£
ÔÚJava³ÌÐò³öÏÖÄÚ´æÏûºÄ¹ý¶à¡¢GCƵ·±»òOutOfMemoryErrorµÄÇé¿öºó£¬ÒªÊ×ÏÈ·ÖÎöÆäËùºÄ·ÑµÄÊÇJVMÍâµÄÎïÀíÄڴ滹ÊÇJVM
heapÇø¡£ ÈçΪJVMÍâµÄÎïÀíÄڴ棬ÔòÒª·ÖÎö³ÌÐòÖÐÏ̵߳ÄÊýÁ¿ÒÔ¼°Direct BytebufferµÄʹÓÃÇé¿ö£»Èç¹ûΪJVM
heapÇø£¬ÔòÒª½áºÏJDKÌṩµÄ¹¤¾ß»òÍⲿµÄ¹¤¾ß·ÖÎö³ÌÐòÖеľßÌå¶ÔÏóµÄÄÚ´æÕ¼ÓÃÇé¿ö¡£
4.3ÎļþIOÏûºÄ·ÖÎö
4.3.1 ÎļþIO·ÖÎöµÄ³£ÓÃÃüÁî
LinuxÔÚ²Ù×÷Îļþʱ£¬½«Êý¾Ý·ÅÈëÎļþ»º´æÇø£¬Ö±µ½ÄÚ´æ²»¹»»òϵͳҪÊÍ·ÅÄÚ´æ¸øÓû§½ø³ÌʹÓÃʱ£¬²ÅдÈëÎļþÖС£Òò´ËÔڲ鿴LinuxÄÚ´æ×´¿öʱ¾³£»á·¢ÏÖ¿ÉÓÃ(free)µÄÎïÀíÄÚ´æ²»¶à£¬µ«cachedÓÃÁ˺ܶ࣬ÕâÊÇLinuxÌáÉýÎļþIOËٶȵÄÒ»ÖÖ×ö·¨£¬ÔÚÕâÖÖ×ö·¨Ï£¬ÈçÎïÀí¿ÕÏÐÄÚ´æ¹»Óã¬Í¨³£ÔÚLinuxÉÏÖ»ÓÐдÎļþºÍµÚÒ»´Î¶ÁÈ¡Îļþʱ»á²úÉúÕæÕýµÄÎļþIO.
ÔÚLinuxÖÐÒª¸ú×ÙÏ̵߳ÄÎļþIOµÄÏûºÄ£¬Ö÷Òª·½·¨ÊÇͨ¹ýpidstatÀ´²éÕÒ¡£
4.3.1.1 pidstat
ÊäÈëÈ磺pidstat -d -t -p [pid] 1 100 ÀàËÆµÄÃüÁî¼´¿É²é¿´Ï̵߳ÄIOÏûºÄ×´¿ö£¬±ØÐëÔÚ2.6.20ÒÔÉϰ汾µÄÄں˻ò°²×°SYSSTAT¹¤¾ßºó²Å»áÓÐЧ£¬Ö´ÐкóµÄЧ¹ûÈçÏÂͼ

ͼ-18
ÆäÖÐkB_rd/s±íʾÿÃë¶ÁÈ¡µÄkBÊý£¬kB_wr/s±íʾÿÃëдÈëµÄkBÊý¡£
ûÓа²×°pidstat»òÄں˰汾²»ÔÚ2.6.20ÒÔºóµÄ°æ±¾µÄÇé¿ö£¬¿Éͨ¹ýiostatÀ´²é¿´£¬µ«iostatÖ»Äܲ鿴Õû¸öϵͳµÄÎļþIOÏûºÄÇé¿ö£¬ÎÞ·¨¸ú×Ùµ½½ø³ÌµÄÎļþIOÏûºÄÇé¿ö¡£
4.3.1.2 iostat
Ö±½ÓÊäÈëiostatÃüÁ¿É²é¿´¸÷¸öÉ豸µÄIOÀúÊ·×´¿ö£¬ÈçÏÂͼ£º

ͼ-19
ÔÚÉÏÃæµÄ¼¸ÏîÖ¸±êÖУ¬ÆäÖÐDevice±íʾÉ豸¾íÃû³Æ»ò·ÖÇøÃû£»tpsÊÇÿÃëµÄIOÇëÇóÊý£¬ÕâÒ²ÊÇIOÏûºÄÇé¿öÖÐÖµµÃ¹Ø×¢µÄÊý×Ö£»kB_read/s±íʾÿÃëÖÓ¶ÁµÄkBÊý£»
kB_wrtn/s ±íʾÿÃëÖÓдµÄkBÊý£» kB_read×ܹ²¶ÁÈ¡µÄÊýÁ¿£» kB_wrtn×ܹ²Ð´ÈëµÄÊýÁ¿£»
³ýÁËÉÏÃæµÄ·½Ê½Í⣬»¹¿Éͨ¹ýÊäÈëiostat -x xvda 3 5 ÕâÑùµÄ·½Ê½À´¶¨Ê±²ÉÑù²é¿´IOµÄÏûºÄ×´¿ö£¬Ö´ÐÐÈçÏ£º

ͼ-20
ÆäÖÐÖµµÃ¹Ø×¢µÄÖ÷ÒªÓÐ: r/s±íʾÿÃëÖÓ¶ÁµÄÇëÇóÊý£»w/s±íʾÿÃëÖÓдµÄÇëÇóÊý£»await±íʾƽ¾ùÿ´ÎIO²Ù×÷µÄµÈ´ýʱ¼ä£¬µ¥Î»ÎªºÁÃ룻avgqu-sz
±íʾµÈ´ýÇëÇóµÄ¶ÓÁÐµÄÆ½¾ù³¤¶È£»svctm±íʾƽ¾ùÿ´ÎÉ豸ִÐÐIO²Ù×÷µÄʱ¼ä£»%util±íʾÓаٷÖÖ®¶àÉÙÓÃÓÚIO²Ù×÷¡£
ÔÚʹÓÃiostat²é¿´IOµÄÏûºÄʱ£¬Ê×ÏÈÒª¹Ø×¢µÄÊÇCPUÖеÄiowait%ËùÕ¼µÄ°Ù·Ö±È£¬µ±iowaitÕ¼¾ÝÁËÖ÷ÒªµÄ°Ù·Ö±Èʱ£¬¾Í±íʾҪ¹Ø×¢IO·½ÃæµÄÏûºÄ×´¿öÁË£¬
Õâʱ¿ÉÒÔͨ¹ýiostat -x ·½Ê½À´ÏêϸµØ²é¿´¾ßÌåÇé¿ö£¬²Î¼ûÏÂͼ£º

ͼ-21
4.3.2 ÎļþIOÏûºÄ·ÖÎö°¸Àý
µ±ÎļþIOÏûºÄ¹ý¸ßʱ£¬¶ÔÓÚJavaÓ¦ÓÃ×îÖØÒªµÄÊÇÕÒµ½Ôì³ÉÎļþIOÏûºÄ¸ßµÄ´úÂ룬ѰÕÒµÄ×î¼Ñ·½°¸ÎªÍ¨¹ýpidstatÖ±½ÓÕÒµ½ÎļþIO²Ù×÷¶àµÄÏ̡߳£Ö®ºó½áºÏjstackÕÒµ½¶ÔÓ¦µÄJava´úÂ룬ÈçûÓÐpidstat,Ò²¿ÉÖ±½Ó¸ù¾ÝjstackµÃµ½µÄÏß³ÌÐÅÏ¢À´·ÖÎöÆäÖÐÎļþIO²Ù×÷½Ï¶àµÄÏ̡߳£
JavaÓ¦ÓÃÔì³ÉÎļþIOÏûºÄÑÏÖØÖ÷ÒªÊǶà¸öÏß³ÌÐèÒª½øÐдóÁ¿ÄÚÈÝдÈë(ÀýÈçÆµ·±µÄÈÕ־дÈë)µÄ¶¯×÷£»»ò´ÅÅÌÉ豸±¾ÉíµÄ´¦ÀíËÙ¶ÈÂý£»»òÎļþϵͳÂý£»»ò²Ù×÷µÄÎļþ±¾ÉíÒѾºÜ´óÔì³ÉµÄ¡£
Àý×Ó£¬Í¨¹ýÍùÒ»¸öÎļþÖв»¶ÏµØÔö¼ÓÄÚÈÝ£¬ÎļþÔ½À´Ô½´ó£¬Ôì³ÉдËÙ¶ÈÂý£¬×îÖÕIOWaitÖµ¸ß¡£´úÂëʵÀý¼û²Î¿¼Öи½¼þ£ºIOWaitHighDemo.jar

ͼ-22
´Óͼ-22¹Ø×¢´ÓtopÖеÄ%iowaitֵƫ¸ß¿ÉÒÔÅжÏÕ¼¾ÝÁ˺ܶàCPU,
½áºÏiostatµÄÐÅÏ¢À´¿´£¬Ö÷ÒªÊÇwkB/s±È½Ï¸ßÖ÷ÒªÊÇдµÄÏûºÄ£¬²¢ÇÒ»¨·ÑÔÚawaitÉϵÄʱ¼äÒªÔ¶´óÓÚsvctmµÄʱ¼ä¡£ÖÁÓÚÊÇʲô¶¯×÷µ¼ÖÂÁËiowait,ÈÔÈ»ÐèÒª¶ÔÓ¦ÓõÄÏß³ÌdumpÀ´·ÖÎö,ÕÒ³öÆäÖеÄIO²Ù×÷Ïà¹ØµÄ¶¯×÷¡£Ê¹ÓÃpidstatÕÒµ½IO¶ÁдÁ¿´óµÄÏß³ÌID,È»ºó½áºÏjstackÉú³ÉµÄÏß³ÌdumpÎļþ£¬¼´¿ÉÕÒµ½ÏàÓ¦µÄÏûºÄÎļþIO¶àµÄ¶¯×÷¡£
jstack²Ù×÷½ø³ÌÏ̵߳ÄdumpÎļþÈçÏ£º

ͼ-23
´ÓÉÏÃæµÄÏ̶߳ÑÕ»ÖУ¬¿É¿´µ½Ïß³ÌÍ£ÁôÔÚÁËFileOutputStream.writeBytes(Native
Method)Õâ¸öNative·½Ê½ÉÏ£¬Õâ¸ö·½·¨Ëù×öµÄ¶¯×÷Ϊ½«Êý¾ÝдÈëÎļþÖУ¬Ò²¾ÍÊÇËùҪѰÕÒµÄIO²Ù×÷Ïà¹ØµÄ¶¯×÷¡£¼ÌÐø¸ú×Ù¶ÑÕ»ÍùÉϲéÕÒ£¬ÖªµÀ²éÕÒµ½ÏµÍ³ÖеĴúÂ룬ÀýÈçÀý×ÓÖеÄIOWaitHighDemo.javaµÚ57ÐУ¬ÏÂͼ(ͼ-24)ÊÇIOWaitHighDemo.javaµÚ57ÐÐÏà¹Ø´úÂë

ͼ-24
|