Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
ʹÓÃDocker´î½¨ELKÈÕ־ϵͳ
 
  2040  次浏览      31
 2018-1-12  
 
±à¼­ÍƼö:
±¾ÎÄÀ´×ÔÓÚ½È×Ó,ÎÄÕÂÏêϸ½éÉÜÁËʹÓÃDocker´î½¨ELKÈÕ־ϵͳµÄÈ«¹ý³Ì£º×ÜÌå¼Ü¹¹Í¼¡¢¾µÏñ×¼±¸¡¢°²×°²½Öè¡£

֮ǰÓñ¾µØ°æ±¾°²×°ÁËELKÖ®ºó£¬¾ÍûÓÐÔÙȥŪËüÁË¡£Äêµ×ûÄÇô棬ÐÄÀïÒ»Ö±µë¼Ç£¬ËùÒÔ×î½üÓÖ¿ªÊ¼ÕÛÌÚÁË¡£È¥elastic¹ÙÍø¿´Ò»Ï£¬¹ûÈ»°æ±¾µÛ¾ÍÊǰ汾µÛ£¬Ò»ÖÜÒ»°æ±¾¡£ÏÖÔÚÎÒÓõİ汾ÊÇ»ùÓÚ6.1.1°æ±¾µÄ¡£

Ä¿±ê

  1. ÊÕ¼¯JavaÈÕÖ¾Îļþ£¬²¢ÇÒ¸ù¾ÝÎļþµÄ²»Í¬½«ÈÕÖ¾·ÖÀ࣬±ÈÈ磺¶©µ¥ÈÕÖ¾£¬¿Í»§ÈÕÖ¾µÈ¡£
  2. ÈÕÖ¾Îļþ¶àÐд¦Àí

×ÜÌå¼Ü¹¹Í¼

×¼±¸¾µÏñ

6.0Ö®ºó¹Ù·½¿ªÊ¼×Ô¼ºÎ¬»¤¾µÏñ°æ±¾:https://www.docker.elastic.co/¡£ÕÒµ½ÐèÒªµÄELK¾µÏñµØÖ·£¬pullÏÂÀ´¾ÍºÃÁË¡£¹Ù·½pullÏÂÀ´Ö®ºó¾µÏñÃûÌ«³¤ÁË£¬ËùÒÔÎÒ½«¾µÏñÈ«²¿ÖØÐ´òÁËtag£¬ÃüÁdocker tag docker.elastic.co/elasticsearch/elasticsearch:6.1.1 elasticsearch:latest¡£

ʹÓÃdocker images²é¿´£º

°²×°docker°æ±¾ElasticSearch

ÔÚelasticsearchµÄdocker°æ±¾ÎĵµÖУ¬¹Ù·½Ìáµ½ÁËvm.max_map_countµÄÖµÔÚÉú²ú»·¾³×îÉÙÒªÉèÖóÉ262144¡£ÉèÖõķ½Ê½ÓÐÁ½ÖÖ

  1. ÓÀ¾ÃÐÔµÄÐÞ¸Ä,ÔÚ/etc/sysctl.confÎļþÖÐÌí¼ÓÒ»ÐУº
grep vm.max_map_count /etc/sysctl.conf # ²éÕÒµ±Ç°µÄÖµ¡£
vm.max_map_count=262144 # Ð޸ĻòÕßÐÂÔö
  1. ÕýÔÚÔËÐеĻúÆ÷£º
sysctl -w vm.max_map_count=262144

Ö®ºóÎÒÃÇÖ´ÐÐÃüÁ±©Â¶ÈÝÆ÷µÄ9200£¬9300¶Ë¿Ú£¬·½±ãÎÒÃÇÔÚÆäËü¼¯Æ÷ÉÏ¿ÉÒÔͨ¹ýÀàËÆhead²å¼þÈ¥×öesË÷ÒýµÄ²Ù×÷µÈ¡£Ö´ÐÐÃüÁîΪ£º

docker run -p 9200:9200 -p 9300:9300 --name elasticsearch -e "discovery.type=single-node" elasticsearch

Èç¹ûʵ¼ÊʹÓÃÖУ¬¿ÉÄÜÐèÒªÉèÖü¯ÈºµÈ²Ù×÷¡£Òòʵ¼ÊÇé¿ö¶ø¶¨¡£Èç¹ûÄãÐèÒª´æ´¢ÀúÊ·Êý¾Ý£¬ÄÇô¾Í¿ÉÄÜÐèÒª½«dataĿ¼±£´æµ½±¾µØ£¬Ê¹ÓÃ-v£¬»òÕßmount²ÎÊý¹ÒÔØ±¾µØÒ»¸öĿ¼¡£

°²×°docker°æ±¾kibana

kibanaµÄ×÷ÓÃÖ÷ÒªÊǰïÖúÎÒÃǽ«ÈÕÖ¾Îļþ¿ÉÊÓ»¯¡£±ãÓÚÎÒÃDzÙ×÷£¬Í³¼ÆµÈ¡£ËüÐèÒªES·þÎñ£¬ËùÒÔÎÒÃǽ«²¿ÊðºÃµÄesºÍkibana¹ØÁªÆðÀ´£¬Ö÷ÒªÓõ½µÄ²ÎÊýÊÇ--link:

docker run -d -p 5601:5601 --link elasticsearch -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana

ʹÓÃlink²ÎÊý£¬»áÔÚkibanaÈÝÆ÷hostsÎļþÖмÓÈëelasticsearch ipµØÖ·£¬ÕâÑùÎÒÃǾÍÖ±½Óͨ¹ý¶¨ÒåµÄnameÀ´·ÃÎÊes·þÎñÁË¡£

°²×°logstashºÍfilebeat

Ç°ÃæµÄkibanaºÍESµÄ°²×°£¬Èç¹ûÎÒÃÇÔÚ¿ª·¢»·¾³Öв¢²»ÐèҪ̫¶àµÄ¹Ø×¢ËûÃǵÄÏêϸÅäÖᣵ«ÊÇlogstashºÍfilebeatÎÒÃÇÐèҪעÒâÏÂËüµÄÅäÖã¬ÒòΪÕâÁ½ÕßÊÇÎÒÃÇÍê³ÉÐèÇóµÄÖØÒªµã¡£

logstashÎÒÃÇÖ»ÈÃËü½øÐÐÈÕÖ¾´¦Àí£¬´¦ÀíÍêÖ®ºó½«ÆäÊä³öµ½elasticsearch¡£

filebeatÊÇÒ»¸öÇáÁ¿¼¶ÊÕ¼¯Æ÷£¬ÎÒÃÇʹÓÃËüÀ´ÊÕ¼¯JavaÈÕÖ¾£¬½«²»Í¬Îļþ¼ÐϵÄÈÕÖ¾½øÐÐtag£¬´¦Àí¶àÐÐÈÕÖ¾ÐÐΪ(Ö÷ÒªÕë¶ÔJavaÒì³£ÐÅÏ¢)£¬Ö®ºó·¢Ë͸ølogstash¡£

ÈÕÖ¾µÄÎļþ¸ñʽ´ó¸Å¾ÍÊÇ£ºDATE LOG-LEVEL LOG-MESSAGE£¬¸ñʽÊÇÔÚlog4j.propertiesÖж¨ÒåµÄ¡£ÄãÒ²¿ÉÒÔ×Ô¼º¶¨ÒåÊä³ö¸ñʽ¡£

ÏÖÔÚÎÒÃǶ¨Òålogstash.conf,Ö÷ÒªÔÚlogstashÖÐʹÓÃgrok filter²å¼þ¡£

logstash.conf:

input {
beats {
#host => "localhost"
port => "5043"
}
}
filter {
if [fields][doc_type] == 'order' {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{JAVALOGMESSAGE:msg}" }
}
}

if [fields][doc_type] == 'customer' { # ÕâÀïдÁ½¸öÒ»ÑùµÄgrok£¬Êµ¼ÊÉÏ¿ÉÄܳöÏÖ¶àÖÖ²»Í¬µÄÈÕÖ¾¸ñʽ£¬ÕâÀï×ö¸öÌáʾ¶øÒÑ,µ±È»Èç¹ûÊÇÏàͬµÄ¸ñʽ£¬ÕâÀï¿ÉÒÔ²»Ð´µÄ
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{JAVALOGMESSAGE:msg}" }
}
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{[fields][doc_type]}-%{+YYYY.MM.dd}"
}
}

ÔÚlogstash.confÖУ¬ÎÒÃÇÖ÷ҪʹÓÃ[fields][doc_type]À´±êÃ÷ÈÕÖ¾µÄÀàÐÍ£¬Õâ¸öֵʵÔÚfilebeatÖж¨ÒåµÄ¡£

ÏÖÔÚÎÒÃǼٶ¨ÐèÒªÊÕ¼¯Á½¸öĿ¼ÏµÄÈÕÖ¾Îļþ£º/home/user/elk/customer/*.log£¬/home/user/elk/order/*.log£º

customer.log:

2017-12-26 10:05:56,476 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:07:23,529 INFO WarehouseController: 271 - findWarehouseList,json{"formJSON": {"userId":"885769620971720708"},"requestParameterMap":{},"requestAttrMap" :{"name":"asdf","user":"8857696","ip":"183.63.112.1","source":"asdfa", "customerId":"885768861337128965","IMEI":"863267033748196", "sessionId":"xm1cile2bcmb15wtqmjno7tgz", "sfUSCSsadDDD":"asdf/10069&ADR&1080&1920&OPPO R9s Plus&Android6.0.1", "URI":"/warehouse-service/appWarehouse/findByCustomerId.apec", "encryptType":"2", "requestStartTime":3450671468321405}}
2017-12-26 10:07:23,650 INFO WarehouseServiceImpl:325 - warehouse list:8,warehouse str: [{"addressDetail":"nnnnnnnn","areaId":"210624","areaNa":""}]
2017-12-26 10:10:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:15:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:20:56,478 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:05:56,476 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:07:23,529 INFO WarehouseController:271 - findWarehouseList,json{"formJSON":{"userId":"885769620971720708"}}]
2017-12-26 10:10:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:15:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:20:56,478 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration

order.log:

2017-12-26 11:29:19,374 INFO WebLogAspect:53 -- ÇëÇó:18,SPEND TIME:0
2017-12-26 11:38:20,404 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:41:07,754 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 12:38:58,683 INFO RedisClusterConfig:107 -- //// --- Æô¶¯µ¥µãRedis ---
2017-12-26 12:39:00,325 DEBUG ApplicationContextRegister:26 --
2017-12-26 12:39:06,961 INFO NoticeServiceApplication:57 -- Started NoticeServiceApplication in 17.667 seconds (JVM running for 18.377)
2017-12-26 11:27:56,577 INFO WebLogAspect:51 -- ÇëÇó:19,RESPONSE:"{\"data\":null,\"errorCode\":\"\",\"errorMsg\":\"\",\"repeatAct\":\"\", \"succeed\":true}"
2017-12-26 11:27:56,577 INFO WebLogAspect:53 -- ÇëÇó:19,SPEND TIME:1
2017-12-26 11:28:09,829 INFO WebLogAspect:42 -- ÇëÇó:20, URL:http://192.168.7.203:30004/sr/flushCache
2017-12-26 11:28:09,830 INFO WebLogAspect:43 -- ÇëÇó:20,HTTP_METHOD:POST
2017-12-26 11:28:09,830 INFO WebLogAspect:44 -- ÇëÇó:20,IP:192.168.7.98
2017-12-26 11:28:09,830 INFO WebLogAspect:45 -- ÇëÇó:20, CLASS_METHOD:com.notice.web.estrictController
2017-12-26 11:28:09,830 INFO WebLogAspect:46 -- ÇëÇó:20,METHOD:flushRestrict
2017-12-26 11:28:09,830 INFO WebLogAspect:47 -- ÇëÇó:20,ARGS:["{\n}"]
2017-12-26 11:28:09,830 DEBUG SystemRestrictController:231 -- Ë¢ÐÂȨÏÞÏÞÖÆÁ´
2017-12-26 11:38:20,404 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:41:07,754 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:41:40,664 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:43:38,224 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:47:49,141 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:51:02,525 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:52:28,726 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:53:55,301 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:54:26,717 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 11:58:48,834 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 12:38:51,126 INFO NoticeServiceApplication:664 -- The following profiles are active: test
2017-12-26 12:38:58,683 INFO RedisClusterConfig:107 -- //// --- Æô¶¯µ¥µãRedis ---
2017-12-26 12:39:00,325 DEBUG ApplicationContextRegister:26 -- ApplicationContextRegister.setApplicationContext: applicationContextorg.springframework.boot.context. embedded.AnnotationConfigEmbeddedWebApplicationContext@5f150435: startup date [Tue Dec 26 12:38:51 CST 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@63c12fb0
2017-12-26 12:39:06,961 INFO NoticeServiceApplication:57 -- Started NoticeServiceApplication in 17.667 seconds (JVM running for 18.377)

ÈÕÖ¾µÄÎļþ¸ñʽ´ó¸Å¾ÍÊÇ£ºDATE LOG-LEVEL LOG-MESSAGE£¬¸ñʽÎÒÃÇÊÇÔÚlog4j.propertiesÖж¨ÒåµÄ¡£Äã¿ÉÒÔ×Ô¼º¶¨Ò壬×Ô¶¨Òå×¢ÒâÐÞ¸Älogstash.confÖеÄgrok¾ÍºÃ¡£

Ö®ºó½â¾öÎÒÃǵÄfilebeatÒª½â¾öµÄÎÊÌ⣺ÊÕ¼¯ÈÕÖ¾£¬´¦Àí¶àÐÐÈÕÖ¾£¬¸øÈÕÖ¾´ò±êÇ©¡£ÔÚfilebeat.ymlÖÐ,Èç϶¨Ò壺

filebeat.yml

filebeat.prospectors:
- paths:
- /home/user/elk/logs/order/*.log
multiline:
pattern: ^\d{4}
negate: true
match: after
fields:
doc_type: order
- paths:
- /home/user/elk/logs/customer/*.log
multiline:
pattern: ^\d{4}
negate: true
match: after
fields:
doc_type: customer
output.logstash: # Êä³öµØÖ·
hosts: ["logstash:5043"]
  1. ÊÕ¼¯ÈÕÖ¾£ºÖ±½ÓʹÓÃprospector¶¨Î»²¢ÇÒ´¦ÀíÈÕÖ¾Îļþ¡£
  2. ¶àÐÐÈÕÖ¾: ¸ù¾ÝÈÕÖ¾¸ñʽ£¬ÎÒÃÇ¿ªÍ·¶¼ÊÇyyyy£¬ÀàËÆÓë´¿4¸öÊý×Ö£¬ËùÒÔÎÒÃÇʹÓÃmultile²å¼þ£¬×öÅäÖþͺ᣹ٷ½µÄÎĵµÍ¦ÏêϸµÄ£¬Ö÷Òª¾ÍÊÇʵ¼ù£ºfilebeat multiline https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
  3. ´ò±êÇ©£ºÕâ¸öÊÇ×îÖØÒªµÄ£¬Ö÷ÒªµÄÄ¿µÄÊÇÈÃlogstashÖªµÀfilebeat·¢Ë͸øËüµÄÏûÏ¢ÊÇÄǸöÀàÐÍ£¬È»ºólogstash·¢Ë͵½esµÄʱºò£¬ÎÒÃÇ¿ÉÒÔ½¨Á¢Ïà¹ØË÷Òý¡£ÕâÀïµÄfieldsÊÇÄÚÖõģ¬doc_typeÊÇ×Ô¶¨ÒåµÄ¡£

֮ǰµÄdocument_type ÔÚ5.5.0ÖоÍÒѾ­·ÏÆúÁË¡£https://www.elastic.co/guide/en/beats/libbeat/6.1/release-notes-5.5.0.html#_deprecated_6

Á˽âÕâЩ֮ºó£¬ÎÒÃÇÆô¶¯ÎÒÃǵÄlogstashºÍfilebeat¡£

Æô¶¯docker°æ±¾µÄlogstash£º

docker run -it --name logstash --link elasticsearch -d -v ~/elk/yaml/logstash.conf:/usr/share/logstash/pipeline/logstash.conf logstash

Æô¶¯filebeat£¬½«Îļþ¹ÒÔØµ½ÈÝÆ÷ÖУ¬ÕâÀïÒ²¿ÉÒÔÓÐÆäËüµÄ´¦Àí·½·¨£¬Äã¿ÉÒÔ¸ù¾Ý×Ô¼ºµÄÐèÇóÀ´¡£

docker run --name filebeat -d --link logstash -v ~/elk/yaml/filebeat.yml:/usr/share/filebeat/filebeat.yml -v ~/elk/logs/:/home/logs/ filebeat

×îºó¼ÇµÃÔÚkibanaÀïÃæ½¨Á¢Ë÷Òý(create index)µÄʱºò£¬Ä¬ÈÏʹÓõÄÊÇlogstash£¬¶øÎÒÃÇÊÇ×Ô¶¨ÒåµÄdoc_type,ËùÒÔÄãÐèÒªÊäÈëorder*,customer*ÕâÑù¾Í¿ÉÒÔ½¨Á¢Á½¸öË÷ÒýÁË¡£

Ö®ºó¾Í¿ÉÒÔÔÚkibanaµÄDiscoveryÀïÃæ¿´µ½ÄãÅäÖõÄÁË

Èç¹ûÄãÖ±½ÓÓÃÎÒµÄlog£¬Ç뽫ʱ¼äÉÔ΢¸Äһϣ¬2017-12-26¸ÄΪµ±ÌìʵÑéÄêÔ¡£

ÉÏÃæµÄÃüÁîÎÒ¶¼×Ô¼ºÊµ¼ù¹ý£¬ÊÇ¿ÉÒÔÓõģ¬×¢ÒâÏÂ-v²ÎÊý¹ÒÔØµÄ¼¸¸ö±¾µØÅ̵ĵØÖ·¡£»¹ÓÐfilebeatÊÕ¼¯µÄµØÖ·¡£

ÅäÖÃÎļþµØÖ·²Ö¿â£ºÊ¹ÓÃDocker´î½¨ELKÈÕ־ϵͳ£¨https://github.com/chenzhijun/elk£©£¬²Ö¿âÅäÓÐdocker-compose.ymlÎļþÔÚ¸ùĿ¼ÏÂÖ±½ÓÔËÐÐdocker-compose up ¾Í¿ÉÒÔ¿´µ½Êµ¼ÊЧ¹ûÁË(¼ÇµÃ¸ÄÏÂÈÕ־ʱ¼ä)¡£

   
2040 ´Îä¯ÀÀ       31
Ïà¹ØÎÄÕÂ

ÔÆ¼ÆËãµÄ¼Ü¹¹
¶ÔÔÆ¼ÆËã·þÎñÄ£ÐÍ
ÔÆ¼ÆËãºËÐļ¼ÊõÆÊÎö
Á˽âÔÆ¼ÆËãµÄ©¶´
Ïà¹ØÎĵµ

ÔÆ¼ÆËã¼ò½é
ÔÆ¼ÆËã¼ò½éÓëÔÆ°²È«
ÏÂÒ»´úÍøÂç¼ÆËã--ÔÆ¼ÆËã
ÈídzÎöÔÆ¼ÆËã
Ïà¹Ø¿Î³Ì

ÔÆ¼ÆËãÔ­ÀíÓëÓ¦ÓÃ
ÔÆ¼ÆËãÓ¦ÓÃÓ뿪·¢
CMMIÌåϵÓëʵ¼ù
»ùÓÚCMMI±ê×¼µÄÈí¼þÖÊÁ¿±£Ö¤