±à¼ÍƼö: |
±¾ÎÄÀ´×ÔÓÚ½È×Ó,ÎÄÕÂÏêϸ½éÉÜÁËʹÓÃDocker´î½¨ELKÈÕ־ϵͳµÄÈ«¹ý³Ì£º×ÜÌå¼Ü¹¹Í¼¡¢¾µÏñ×¼±¸¡¢°²×°²½Öè¡£ |
|
֮ǰÓñ¾µØ°æ±¾°²×°ÁËELKÖ®ºó£¬¾ÍûÓÐÔÙȥŪËüÁË¡£Äêµ×ûÄÇô棬ÐÄÀïÒ»Ö±µë¼Ç£¬ËùÒÔ×î½üÓÖ¿ªÊ¼ÕÛÌÚÁË¡£È¥elastic¹ÙÍø¿´Ò»Ï£¬¹ûÈ»°æ±¾µÛ¾ÍÊǰ汾µÛ£¬Ò»ÖÜÒ»°æ±¾¡£ÏÖÔÚÎÒÓõİ汾ÊÇ»ùÓÚ6.1.1°æ±¾µÄ¡£
Ä¿±ê
- ÊÕ¼¯JavaÈÕÖ¾Îļþ£¬²¢ÇÒ¸ù¾ÝÎļþµÄ²»Í¬½«ÈÕÖ¾·ÖÀ࣬±ÈÈ磺¶©µ¥ÈÕÖ¾£¬¿Í»§ÈÕÖ¾µÈ¡£
- ÈÕÖ¾Îļþ¶àÐд¦Àí
×ÜÌå¼Ü¹¹Í¼

×¼±¸¾µÏñ
6.0Ö®ºó¹Ù·½¿ªÊ¼×Ô¼ºÎ¬»¤¾µÏñ°æ±¾:https://www.docker.elastic.co/¡£ÕÒµ½ÐèÒªµÄELK¾µÏñµØÖ·£¬pullÏÂÀ´¾ÍºÃÁË¡£¹Ù·½pullÏÂÀ´Ö®ºó¾µÏñÃûÌ«³¤ÁË£¬ËùÒÔÎÒ½«¾µÏñÈ«²¿ÖØÐ´òÁËtag£¬ÃüÁdocker tag docker.elastic.co/elasticsearch/elasticsearch:6.1.1 elasticsearch:latest¡£
ʹÓÃdocker images²é¿´£º

°²×°docker°æ±¾ElasticSearch
ÔÚelasticsearchµÄdocker°æ±¾ÎĵµÖУ¬¹Ù·½Ìáµ½ÁËvm.max_map_countµÄÖµÔÚÉú²ú»·¾³×îÉÙÒªÉèÖóÉ262144¡£ÉèÖõķ½Ê½ÓÐÁ½ÖÖ
- ÓÀ¾ÃÐÔµÄÐÞ¸Ä,ÔÚ/etc/sysctl.confÎļþÖÐÌí¼ÓÒ»ÐУº
grep vm.max_map_count /etc/sysctl.conf # ²éÕÒµ±Ç°µÄÖµ¡£
vm.max_map_count=262144 # Ð޸ĻòÕßÐÂÔö |
- ÕýÔÚÔËÐеĻúÆ÷£º
sysctl -w vm.max_map_count=262144 |
Ö®ºóÎÒÃÇÖ´ÐÐÃüÁ±©Â¶ÈÝÆ÷µÄ9200£¬9300¶Ë¿Ú£¬·½±ãÎÒÃÇÔÚÆäËü¼¯Æ÷ÉÏ¿ÉÒÔͨ¹ýÀàËÆhead²å¼þÈ¥×öesË÷ÒýµÄ²Ù×÷µÈ¡£Ö´ÐÐÃüÁîΪ£º
docker run -p 9200:9200 -p 9300:9300 --name elasticsearch -e "discovery.type=single-node" elasticsearch |
Èç¹ûʵ¼ÊʹÓÃÖУ¬¿ÉÄÜÐèÒªÉèÖü¯ÈºµÈ²Ù×÷¡£Òòʵ¼ÊÇé¿ö¶ø¶¨¡£Èç¹ûÄãÐèÒª´æ´¢ÀúÊ·Êý¾Ý£¬ÄÇô¾Í¿ÉÄÜÐèÒª½«dataĿ¼±£´æµ½±¾µØ£¬Ê¹ÓÃ-v£¬»òÕßmount²ÎÊý¹ÒÔØ±¾µØÒ»¸öĿ¼¡£
°²×°docker°æ±¾kibana
kibanaµÄ×÷ÓÃÖ÷ÒªÊǰïÖúÎÒÃǽ«ÈÕÖ¾Îļþ¿ÉÊÓ»¯¡£±ãÓÚÎÒÃDzÙ×÷£¬Í³¼ÆµÈ¡£ËüÐèÒªES·þÎñ£¬ËùÒÔÎÒÃǽ«²¿ÊðºÃµÄesºÍkibana¹ØÁªÆðÀ´£¬Ö÷ÒªÓõ½µÄ²ÎÊýÊÇ--link:
docker run -d -p 5601:5601 --link elasticsearch -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana |
ʹÓÃlink²ÎÊý£¬»áÔÚkibanaÈÝÆ÷hostsÎļþÖмÓÈëelasticsearch ipµØÖ·£¬ÕâÑùÎÒÃǾÍÖ±½Óͨ¹ý¶¨ÒåµÄnameÀ´·ÃÎÊes·þÎñÁË¡£
°²×°logstashºÍfilebeat
Ç°ÃæµÄkibanaºÍESµÄ°²×°£¬Èç¹ûÎÒÃÇÔÚ¿ª·¢»·¾³Öв¢²»ÐèҪ̫¶àµÄ¹Ø×¢ËûÃǵÄÏêϸÅäÖᣵ«ÊÇlogstashºÍfilebeatÎÒÃÇÐèҪעÒâÏÂËüµÄÅäÖã¬ÒòΪÕâÁ½ÕßÊÇÎÒÃÇÍê³ÉÐèÇóµÄÖØÒªµã¡£
logstashÎÒÃÇÖ»ÈÃËü½øÐÐÈÕÖ¾´¦Àí£¬´¦ÀíÍêÖ®ºó½«ÆäÊä³öµ½elasticsearch¡£
filebeatÊÇÒ»¸öÇáÁ¿¼¶ÊÕ¼¯Æ÷£¬ÎÒÃÇʹÓÃËüÀ´ÊÕ¼¯JavaÈÕÖ¾£¬½«²»Í¬Îļþ¼ÐϵÄÈÕÖ¾½øÐÐtag£¬´¦Àí¶àÐÐÈÕÖ¾ÐÐΪ(Ö÷ÒªÕë¶ÔJavaÒì³£ÐÅÏ¢)£¬Ö®ºó·¢Ë͸ølogstash¡£
ÈÕÖ¾µÄÎļþ¸ñʽ´ó¸Å¾ÍÊÇ£ºDATE LOG-LEVEL LOG-MESSAGE£¬¸ñʽÊÇÔÚlog4j.propertiesÖж¨ÒåµÄ¡£ÄãÒ²¿ÉÒÔ×Ô¼º¶¨ÒåÊä³ö¸ñʽ¡£
ÏÖÔÚÎÒÃǶ¨Òålogstash.conf,Ö÷ÒªÔÚlogstashÖÐʹÓÃgrok filter²å¼þ¡£
logstash.conf:
input { beats { #host => "localhost" port => "5043" } } filter { if [fields][doc_type] == 'order' { grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{JAVALOGMESSAGE:msg}" } } } if [fields][doc_type] == 'customer' { # ÕâÀïдÁ½¸öÒ»ÑùµÄgrok£¬Êµ¼ÊÉÏ¿ÉÄܳöÏÖ¶àÖÖ²»Í¬µÄÈÕÖ¾¸ñʽ£¬ÕâÀï×ö¸öÌáʾ¶øÒÑ,µ±È»Èç¹ûÊÇÏàͬµÄ¸ñʽ£¬ÕâÀï¿ÉÒÔ²»Ð´µÄ grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{JAVALOGMESSAGE:msg}" } } } } output { stdout { codec => rubydebug } elasticsearch { hosts => [ "localhost:9200" ] index => "%{[fields][doc_type]}-%{+YYYY.MM.dd}" }
} |
ÔÚlogstash.confÖУ¬ÎÒÃÇÖ÷ҪʹÓÃ[fields][doc_type]À´±êÃ÷ÈÕÖ¾µÄÀàÐÍ£¬Õâ¸öֵʵÔÚfilebeatÖж¨ÒåµÄ¡£
ÏÖÔÚÎÒÃǼٶ¨ÐèÒªÊÕ¼¯Á½¸öĿ¼ÏµÄÈÕÖ¾Îļþ£º/home/user/elk/customer/*.log£¬/home/user/elk/order/*.log£º
customer.log:
2017-12-26 10:05:56,476 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration
2017-12-26 10:07:23,529 INFO WarehouseController: 271 - findWarehouseList,json{"formJSON": {"userId":"885769620971720708"},"requestParameterMap":{},"requestAttrMap" :{"name":"asdf","user":"8857696","ip":"183.63.112.1","source":"asdfa", "customerId":"885768861337128965","IMEI":"863267033748196", "sessionId":"xm1cile2bcmb15wtqmjno7tgz", "sfUSCSsadDDD":"asdf/10069&ADR&1080&1920&OPPO R9s Plus&Android6.0.1", "URI":"/warehouse-service/appWarehouse/findByCustomerId.apec", "encryptType":"2", "requestStartTime":3450671468321405}}
2017-12-26 10:07:23,650 INFO WarehouseServiceImpl:325 - warehouse list:8,warehouse str: [{"addressDetail":"nnnnnnnn","areaId":"210624","areaNa":""}]
2017-12-26 10:10:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration 2017-12-26 10:15:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration 2017-12-26 10:20:56,478 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration 2017-12-26 10:05:56,476 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration 2017-12-26 10:07:23,529 INFO WarehouseController:271 - findWarehouseList,json{"formJSON":{"userId":"885769620971720708"}}] 2017-12-26 10:10:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration 2017-12-26 10:15:56,477 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration 2017-12-26 10:20:56,478 INFO ConfigClusterResolver:43 - Resolving eureka endpoints via configuration |
order.log:
2017-12-26 11:29:19,374 INFO WebLogAspect:53 -- ÇëÇó:18,SPEND TIME:0 2017-12-26 11:38:20,404 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:41:07,754 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 12:38:58,683 INFO RedisClusterConfig:107 -- //// --- Æô¶¯µ¥µãRedis --- 2017-12-26 12:39:00,325 DEBUG ApplicationContextRegister:26 -- 2017-12-26 12:39:06,961 INFO NoticeServiceApplication:57 -- Started NoticeServiceApplication in 17.667 seconds (JVM running for 18.377)
2017-12-26 11:27:56,577 INFO WebLogAspect:51 -- ÇëÇó:19,RESPONSE:"{\"data\":null,\"errorCode\":\"\",\"errorMsg\":\"\",\"repeatAct\":\"\", \"succeed\":true}"
2017-12-26 11:27:56,577 INFO WebLogAspect:53 -- ÇëÇó:19,SPEND TIME:1
2017-12-26 11:28:09,829 INFO WebLogAspect:42 -- ÇëÇó:20, URL:http://192.168.7.203:30004/sr/flushCache
2017-12-26 11:28:09,830 INFO WebLogAspect:43 -- ÇëÇó:20,HTTP_METHOD:POST 2017-12-26 11:28:09,830 INFO WebLogAspect:44 -- ÇëÇó:20,IP:192.168.7.98
2017-12-26 11:28:09,830 INFO WebLogAspect:45 -- ÇëÇó:20, CLASS_METHOD:com.notice.web.estrictController
2017-12-26 11:28:09,830 INFO WebLogAspect:46 -- ÇëÇó:20,METHOD:flushRestrict 2017-12-26 11:28:09,830 INFO WebLogAspect:47 -- ÇëÇó:20,ARGS:["{\n}"] 2017-12-26 11:28:09,830 DEBUG SystemRestrictController:231 -- Ë¢ÐÂȨÏÞÏÞÖÆÁ´ 2017-12-26 11:38:20,404 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:41:07,754 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:41:40,664 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:43:38,224 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:47:49,141 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:51:02,525 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:52:28,726 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:53:55,301 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:54:26,717 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 11:58:48,834 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 12:38:51,126 INFO NoticeServiceApplication:664 -- The following profiles are active: test 2017-12-26 12:38:58,683 INFO RedisClusterConfig:107 -- //// --- Æô¶¯µ¥µãRedis ---
2017-12-26 12:39:00,325 DEBUG ApplicationContextRegister:26 -- ApplicationContextRegister.setApplicationContext: applicationContextorg.springframework.boot.context. embedded.AnnotationConfigEmbeddedWebApplicationContext@5f150435: startup date [Tue Dec 26 12:38:51 CST 2017]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@63c12fb0
2017-12-26 12:39:06,961 INFO NoticeServiceApplication:57 -- Started NoticeServiceApplication in 17.667 seconds (JVM running for 18.377) |
ÈÕÖ¾µÄÎļþ¸ñʽ´ó¸Å¾ÍÊÇ£ºDATE LOG-LEVEL LOG-MESSAGE£¬¸ñʽÎÒÃÇÊÇÔÚlog4j.propertiesÖж¨ÒåµÄ¡£Äã¿ÉÒÔ×Ô¼º¶¨Ò壬×Ô¶¨Òå×¢ÒâÐÞ¸Älogstash.confÖеÄgrok¾ÍºÃ¡£
Ö®ºó½â¾öÎÒÃǵÄfilebeatÒª½â¾öµÄÎÊÌ⣺ÊÕ¼¯ÈÕÖ¾£¬´¦Àí¶àÐÐÈÕÖ¾£¬¸øÈÕÖ¾´ò±êÇ©¡£ÔÚfilebeat.ymlÖÐ,Èç϶¨Ò壺
filebeat.yml
filebeat.prospectors: - paths: - /home/user/elk/logs/order/*.log multiline: pattern: ^\d{4} negate: true match: after fields: doc_type: order - paths: - /home/user/elk/logs/customer/*.log multiline: pattern: ^\d{4} negate: true match: after fields: doc_type: customer output.logstash: # Êä³öµØÖ·
hosts: ["logstash:5043"]
|
- ÊÕ¼¯ÈÕÖ¾£ºÖ±½ÓʹÓÃprospector¶¨Î»²¢ÇÒ´¦ÀíÈÕÖ¾Îļþ¡£
- ¶àÐÐÈÕÖ¾: ¸ù¾ÝÈÕÖ¾¸ñʽ£¬ÎÒÃÇ¿ªÍ·¶¼ÊÇyyyy£¬ÀàËÆÓë´¿4¸öÊý×Ö£¬ËùÒÔÎÒÃÇʹÓÃmultile²å¼þ£¬×öÅäÖþͺ᣹ٷ½µÄÎĵµÍ¦ÏêϸµÄ£¬Ö÷Òª¾ÍÊÇʵ¼ù£ºfilebeat multiline https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
- ´ò±êÇ©£ºÕâ¸öÊÇ×îÖØÒªµÄ£¬Ö÷ÒªµÄÄ¿µÄÊÇÈÃlogstashÖªµÀfilebeat·¢Ë͸øËüµÄÏûÏ¢ÊÇÄǸöÀàÐÍ£¬È»ºólogstash·¢Ë͵½esµÄʱºò£¬ÎÒÃÇ¿ÉÒÔ½¨Á¢Ïà¹ØË÷Òý¡£ÕâÀïµÄfieldsÊÇÄÚÖõģ¬doc_typeÊÇ×Ô¶¨ÒåµÄ¡£
֮ǰµÄdocument_type ÔÚ5.5.0ÖоÍÒѾ·ÏÆúÁË¡£https://www.elastic.co/guide/en/beats/libbeat/6.1/release-notes-5.5.0.html#_deprecated_6
Á˽âÕâЩ֮ºó£¬ÎÒÃÇÆô¶¯ÎÒÃǵÄlogstashºÍfilebeat¡£
Æô¶¯docker°æ±¾µÄlogstash£º
docker run -it --name logstash --link elasticsearch -d -v ~/elk/yaml/logstash.conf:/usr/share/logstash/pipeline/logstash.conf logstash |
Æô¶¯filebeat£¬½«Îļþ¹ÒÔØµ½ÈÝÆ÷ÖУ¬ÕâÀïÒ²¿ÉÒÔÓÐÆäËüµÄ´¦Àí·½·¨£¬Äã¿ÉÒÔ¸ù¾Ý×Ô¼ºµÄÐèÇóÀ´¡£
docker run --name filebeat -d --link logstash -v ~/elk/yaml/filebeat.yml:/usr/share/filebeat/filebeat.yml -v ~/elk/logs/:/home/logs/ filebeat |
×îºó¼ÇµÃÔÚkibanaÀïÃæ½¨Á¢Ë÷Òý(create index)µÄʱºò£¬Ä¬ÈÏʹÓõÄÊÇlogstash£¬¶øÎÒÃÇÊÇ×Ô¶¨ÒåµÄdoc_type,ËùÒÔÄãÐèÒªÊäÈëorder*,customer*ÕâÑù¾Í¿ÉÒÔ½¨Á¢Á½¸öË÷ÒýÁË¡£
Ö®ºó¾Í¿ÉÒÔÔÚkibanaµÄDiscoveryÀïÃæ¿´µ½ÄãÅäÖõÄÁË
Èç¹ûÄãÖ±½ÓÓÃÎÒµÄlog£¬Ç뽫ʱ¼äÉÔ΢¸Äһϣ¬2017-12-26¸ÄΪµ±ÌìʵÑéÄêÔ¡£
ÉÏÃæµÄÃüÁîÎÒ¶¼×Ô¼ºÊµ¼ù¹ý£¬ÊÇ¿ÉÒÔÓõģ¬×¢ÒâÏÂ-v²ÎÊý¹ÒÔØµÄ¼¸¸ö±¾µØÅ̵ĵØÖ·¡£»¹ÓÐfilebeatÊÕ¼¯µÄµØÖ·¡£
ÅäÖÃÎļþµØÖ·²Ö¿â£ºÊ¹ÓÃDocker´î½¨ELKÈÕ־ϵͳ£¨https://github.com/chenzhijun/elk£©£¬²Ö¿âÅäÓÐdocker-compose.ymlÎļþÔÚ¸ùĿ¼ÏÂÖ±½ÓÔËÐÐdocker-compose up ¾Í¿ÉÒÔ¿´µ½Êµ¼ÊЧ¹ûÁË(¼ÇµÃ¸ÄÏÂÈÕ־ʱ¼ä)¡£
|