Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
ÈçºÎʹÓÃPrometheusÇáËÉʵÏÖ¼¯Èº¼à¿Ø£¿
 
×÷ÕߣºCloudox_
  1590  次浏览      27
 2020-3-13
 
±à¼­ÍƼö:
±¾ÎÄÖ÷Òª½éÉÜÁË¿ªÔ´¼à¿Ø¿ò¼ÜPrometheusµÄ°²×°¡¢ÅäÖá¢Ìí¼Ó¼à¿ØµÈÏêϸµÄʵÏÖ¼¯Èº¼à¿Ø¹ý³Ì£¬Ï£Íû¶ÔÄúµÄѧϰÓÐËù°ïÖú¡£
±¾ÎÄÀ´×ÔÓÚ51cto£¬ÓÉ»ðÁú¹ûÈí¼þLuca±à¼­¡¢ÍƼö¡£

1¡¢Prometheus ¶Ô±È Zabbix

ºÍ Zabbix ÀàËÆ£¬Prometheus Ò²ÊÇÒ»¸ö½üÄê±È½Ï»ðµÄ¿ªÔ´¼à¿Ø¿ò¼Ü£¬ºÍ Zabbix²»Í¬Ö®´¦ÔÚÓÚ Prometheus Ïà¶Ô¸üÁé»îµã£¬Ä£¿é¼ä±È½Ï½âñ±ÈÈç¸æ¾¯Ä£¿é¡¢´úÀíÄ£¿éµÈµÈ¶¼¿ÉÒÔÑ¡ÔñÐÔÅäÖ᣷þÎñ¶ËºÍ¿Í»§¶Ë¶¼ÊÇ¿ªÏä¼´Ó㬲»ÐèÒª½øÐа²×°¡£ZabbixÔòÊÇÒ»Ì×°²×°°ÑËùÓж«Î÷¶¼ÅªºÃ£¬ºÜÅÓ´óÒ²ºÜ·±ÔÓ¡£

Zabbix µÄ¿Í»§¶Ë agent ¿ÉÒԱȽϷ½±ãµÄͨ¹ý½Å±¾À´¶ÁÈ¡»úÆ÷ÄÚÊý¾Ý¿â¡¢ÈÕÖ¾µÈÎļþÀ´×öÉϱ¨¡£¶ø Prometheus µÄÉϱ¨¿Í»§¶ËÔò·ÖΪ²»Í¬ÓïÑÔµÄSDKºÍ²»Í¬ÓÃ;µÄ exporter Á½ÖÖ£¬±ÈÈçÈç¹ûÄãÒª¼à¿Ø»úÆ÷״̬¡¢mysqlÐÔÄܵȣ¬ÓдóÁ¿ÒѾ­³ÉÊìµÄ exporter À´Ö±½Ó¿ªÏäʹÓã¬Í¨¹ýhttpͨÐÅÀ´¶Ô·þÎñ¶ËÌṩÐÅÏ¢Éϱ¨(serverÈ¥pullÐÅÏ¢);

¶øÈç¹ûÄãÏëÒª¼à¿Ø×Ô¼ºµÄÒµÎñ״̬£¬ÄÇôÕë¶Ô¸÷ÖÖÓïÑÔ¶¼Óйٷ½»òÆäËûÈËдºÃµÄ sdk ¹©ÄãʹÓ㬶¼±È½Ï·½±ã£¬²»ÐèÒªÏȰÑÊý¾Ý´æÈëÊý¾Ý¿â»òÈÕÖ¾ÔÙ¹© zabbix-agent ²É¼¯¡£

Zabbix µÄ¿Í»§¶Ë¸ü¶àÊÇÖ»×öÉϱ¨µÄÊÂÇ飬push ģʽ¡£¶ø Prometheus ÔòÊǿͻ§¶Ë±¾µØÒ²»á´æ´¢¼à¿ØÊý¾Ý£¬·þÎñ¶Ë¶¨Ê±À´À­È¡ÏëÒªµÄÊý¾Ý¡£½çÃæÀ´Ëµ zabbix ±È½Ï³Â¾É£¬¶ø prometheus ±È½ÏÐÂÇҷdz£¼ò½à£¬¼ò½àµ½Ö»ÄÜËãÒ»¸ö²âÊÔºÍÅäÖÃÆ½Ì¨¡£ÒªÏë»ñµÃÁ¼ºÃµÄ¼à¿ØÌåÑ飬´îÅä Grafana »¹ÊǶþÕߵıØ×ß֮·¡£

2¡¢°²×° Prometheus

Prometheus ÓкܶàÖÖ°²×°·½Ê½£¬¿ÉÒÔÔÚ¹ÙÍø¿´µ½£¬ÕâÀïÖ»½éÉÜÏÂÔØ°²×°°ü½âѹµÄ·½Ê½£¬ÒòΪ Prometheus ÊÇ¡°¿ªÏä¼´Óᱵģ¬Ò²¾ÍÊÇ˵½âѹ°²×°°üºó¾Í¿ÉÒÔÖ±½ÓʹÓÃÁË£¬²»ÐèÒªÔÙÖ´Ðа²×°³ÌÐò£¬ºÜ·½±ã¡£

¿ÉÒÔÈ¥ Prometheus µÄ¹ÙÍøÏÂÔØÒ³Ãæ»ñÈ¡×îа汾µÄÐÅÏ¢£¬±ÈÈçÏÖÔÚµÄ×îа汾ÊÇ2.7.2£¬ÄǾÍÏÂÔØÏàӦϵͳµÄ°²×°°ü£¬È»ºó½âѹ

$ wget https://github.com/prometheus/prometheus /releases/download/v2.7.2/prometheus-2.7.2.linux-amd64.tar.gz
$ tar xvfz prometheus-2.7.2.linux-amd64.tar.gz

½âѹºóµ±Ç°Ä¿Â¼¾Í»á³öÏÖÒ»¸öÏàÓ¦µÄÎļþ¼Ð£¬½øÈë¸ÃÎļþ¼Ð£¬È»ºó¾Í¿ÉÒÔÖ±½ÓÔËÐÐPrometheus serverÁË!

$ cd prometheus-2.7.2.linux-amd64
// ²é¿´°æ±¾
$ ./prometheus --version
// ÔËÐÐserver
$ ./prometheus --config.file=prometheus.yml

ÃüÁîÖÐµÄ prometheus.yml ÎļþÆäʵ¾ÍÊÇÅäÖÃÎļþ£¬Ò²ÔÚµ±Ç°Ä¿Â¼Ï£¬ÔÚÆäÖпÉÒÔÅäÖÃһЩ¶«Î÷¡£

3¡¢ÅäÖà Prometheus

ÉÏÎÄ˵ÁË£¬prometheus.yml ÊÇÅäÖÃÎļþ£¬´ò¿ª¿ÉÒÔ¿´µ½²»¶àµÄ¼¸Ê®ÐÐÎÄ×Ö£¬ÀàËÆÏÂÃæ£º

$ cat prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ['localhost:9090']

Æä´óÖ·ÖΪËIJ¿·Ö£º

global£ºÈ«¾ÖÅäÖã¬ÆäÖÐ scrape_interval ±íʾץȡһ´ÎÊý¾ÝµÄ¼ä¸ôʱ¼ä£¬evaluation_interval ±íʾ½øÐи澯¹æÔò¼ì²âµÄ¼ä¸ôʱ¼ä;

alerting£º¸æ¾¯¹ÜÀíÆ÷(Alertmanager)µÄÅäÖã¬Ä¿Ç°»¹Ã»Óа²×°Alertmanager;

rule_files£º¸æ¾¯¹æÔòÓÐÄÄЩ;

scrape_configs£º×¥È¡¼à¿ØÐÅÏ¢µÄÄ¿±ê¡£Ò»¸ö job_name ¾ÍÊÇÒ»¸öÄ¿±ê£¬Æä targets ¾ÍÊDzɼ¯ÐÅÏ¢µÄ IP ºÍ¶Ë¿Ú¡£ÕâÀïĬÈÏ¼à¿ØÁË Prometheus ×Ô¼º£¬¿ÉÒÔͨ¹ýÐÞ¸ÄÕâÀïÀ´ÐÞ¸Ä Prometheus µÄ¼à¿Ø¶Ë¿Ú¡£

Prometheus µÄÿ¸ö exporter ¶¼»áÊÇÒ»¸öÄ¿±ê£¬ËüÃÇ¿ÉÒÔÉϱ¨²»Í¬µÄ¼à¿ØÐÅÏ¢£¬±ÈÈç»úÆ÷״̬£¬»òÕß mysql ÐÔÄܵȵȣ¬²»Í¬ÓïÑÔ sdk Ò²»áÊÇÒ»¸öÄ¿±ê£¬ËüÃÇ»áÉϱ¨Äã×Ô¶¨ÒåµÄÒµÎñ¼à¿ØÐÅÏ¢¡£

4¡¢Prometheus ½çÃæ

ÔËÐкó£¬ÔÚä¯ÀÀÆ÷·ÃÎÊ[»úÆ÷IP:¶Ë¿Ú]¾Í¿ÉÒԲ鿴 Prometheus µÄ½çÃæÁË£¬ÕâÀïµÄ»úÆ÷IPÊÇÄãÔËÐÐ Prometheus µÄ»úÆ÷£¬¶Ë¿ÚÊÇÉÏÃæÅäÖÃÎļþÖÐÅäÖÃµÄ¼à¿Ø×Ô¼ºµÄ¶Ë¿Ú¡£´ò¿ªºó½çÃæÈçÏ£º

Èç¹û·ÃÎʲ»ÁË£¬¿´¿´ÊDz»ÊǶ˿ÚûÓдò¿ª»òÕßÔÊÐíÍâÍø·ÃÎÊ¡£

½çÃæ·Ç³£¼òµ¥(ËùÒÔÎÒÃÇ»¹ÐèÒªGrafana)£¬ÉÏÃæ±êÇ©À¸ÖУ¬AlertsÊǸ澯¹ÜÀíÆ÷£¬ÔÝʱ»¹Ã»°²×°¡£GraphÊDz鿴¼à¿ØÏîµÄͼ±í£¬Ò²ÊÇ·ÃÎʺóµÄĬÈÏÒ³Ãæ£¬StatusÖпÉÒԲ鿴һЩÅäÖá¢¼à¿ØÄ¿±ê¡¢¸æ¾¯¹æÔòµÈ¡£

ÔÚ Graph Ò³Ãæ£¬ÓÉÓÚÎÒÃÇĬÈÏÒѾ­¼à¿ØÁË Prometheus ×Ô¼º£¬ËùÒÔÒѾ­¿ÉÒԲ鿴һЩ¼à¿ØÍ¼±í£¬±ÈÈçÔÚÊäÈë¿òÊäÈë¡°promhttp_metric_handler_requests_total¡±£¬Ö´ÐÐExecute£¬ÏÂÃæµÄС±êÇ©ÖÐÇл»µ½Graph¾ÍÄÜ¿´µ½¡°/metrics¡±·ÃÎÊ´ÎÊýµÄÕÛÏßͼ¡£

5¡¢Ìí¼Ó»úÆ÷״̬¼à¿Ø

ÎÒÃdz¢ÊÔÌí¼ÓµÚÒ»¸ö¼à¿Øexporter¡ª¡ª¼à¿Øµ±Ç°»úÆ÷×ÔÉíµÄ״̬£¬°üÀ¨Ó²ÅÌ¡¢CPU¡¢Á÷Á¿µÈ¡£ÒòΪPrometheusÒѾ­ÓÐÁ˺ܶàÏֳɵij£ÓÃexporter£¬ËùÒÔÎÒÃÇÖ±½ÓÓÃÆäÖеÄnode_exporter¡£×¢ÒâÁË£¬ÕâÀïÃû×ÖËäÈ»½Ðnode_exporter£¬µ«¸únodejsûÓÐÈκιØÏµ£¬ÔÚPrometheus¿´À´£¬Ò»Ì¨»úÆ÷»òÕß˵һ¸ö½Úµã¾ÍÊÇÒ»¸önode£¬ËùÒÔ¸ÃexporterÊÇÔÚÉϱ¨µ±Ç°½ÚµãµÄ״̬¡£node_exporter±¾ÉíÒ²ÊÇÒ»¸öhttp·þÎñ£¬¿ÉÒÔ¹© prometheus server µ÷ÓÃ(pull)À´»ñÈ¡¼à¿ØµÄÐÅÏ¢£¬°²×°·½·¨Í¬ÑùÊÇÏÂÔØ°²×°°üºó½âѹֱ½ÓÔËÐУº

// ÏÂÔØ×îа汾£¬¿ÉÒÔÔÚgithubµÄreleaseÖжÔ×îа汾ÓÒ¼ü»ñÈ¡ÏÂÔØÁ´½Ó
$ wget https://github.com/prometheus/node_exporter/ releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz
// ½âѹ
$ tar xvfz node_exporter-0.17.0.linux-amd64.tar.gz
// ½øÈë½âѹ³öµÄĿ¼
$ cd node_exporter-0.17.0.linux-amd64
// ÔËÐÐ¼à¿Ø²É¼¯·þÎñ
$ ./node_exporter

ÔËÐкó¿ÉÒÔ¿´µ½ÔÚ¼àÌý9100¶Ë¿Ú¡£ÕâÑù¾Í¿ÉÒԲɼ¯ÁË£¬ÏÖÔÚÏÈ·ÃÎÊÊÔÊÔÄܲ»ÄÜÓÐûÓгɹ¦ÔËÐУºÕâÀïÒ²¿ÉÒÔ¿´³öÆäʵÿ¸öexporter±¾Éí¶¼ÊÇÒ»¸öhttp·þÎñ£¬server¶Ë»á¶¨Ê±À´·ÃÎÊ»ñÈ¡¼à¿ØÐÅÏ¢¡£·ÃÎʳɹ¦µÄ»°£¬ÎÒÃÇÈ¥prometheusµÄÅäÖÃÎļþ(prometheus.yml)ÖУ¬¼ÓÉÏÕâ¸ötarget£º

scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'server'
static_configs:
- targets: ['localhost:9100']

¿ÉÒÔ¿´µ½£¬¾ÍÊÇÔÚ scrape_configs Ä£¿éÖмÓÒ»¸öjob£¬ÃüºÃÃû£¬ÅäÖúüàÌýµÄIPºÍ¶Ë¿Ú¼´¿É£¬È»ºóÖØÐÂÔËÐÐ prometheus£¬ÔÚ±êÇ©À¸µÄ Status ¡ú Targets ÖпÉÒÔ¿´µ½¶àÁËÒ»¸ö£º

Èç¹ûÐÂ¼ÓµÄ target µÄ status ÊÇ¡°UP¡±µÄ»°£¬¾Í˵Ã÷¼àÌý³É¹¦ÁË¡£

´ËʱȥGraphÖУ¬ÊäÈë¿òÊäÈënode¿ÉÒÔ·¢ÏÖÓкܶà node ¿ªÍ·µÄ¼à¿ØÏîÁË£¬¶¼ÊǺͻúÆ÷״̬Óйصģ¬¿ÉÒÔ×Ô¼ºÖ´Ðп´Ò»¿´¡£

°²×° Grafana

ÒòΪ Prometheus µÄ½çÃæ¿´ÆðÀ´·Ç³£¼òµ¥£¬ËùÒÔÎÒÃÇ»¹ÐèÒª Grafana Õâ¸ö·Ç³£Ç¿´óÒ²ÊÇ×î³£ÓÃµÄ¼à¿ØÕ¹Ê¾¿ò¼Ü¡£

ÎÒÃÇ»¹ÊÇÓÃÏÂÔØ¶þ½øÖưüµÄ·½Ê½À´½øÐа²×°£¬ÕâÖÖ·½Ê½²»ÐèÒªÄ㵱ǰµÄlinuxÓû§ÓµÓÐsudoȨÏÞ£¬Ò²²»ÐèÒªÄãÖªµÀ root ÃÜÂë¡£Èç¹ûÄãÓÐÕâЩȨÏÞ£¬ÄǾÍʹÓà yum µÈÆäËûÖ±½ÓµÄ°²×°·½Ê½°É£¬°²×°ËµÃ÷¼û Grafana µÄ¹Ù·½°²×°Ò³Ãæ

ÎÒÃÇÖ±½ÓÏÂÔØ²¢½âѹ£º

$wget [https://dl.grafana.com/oss/release/grafana-6.0.0.linux-amd64.tar.gz](https://dl.grafana.com/oss/release/grafana-6.0.0.linux-amd64.tar.gz)
$ tar -zxvf grafana-6.0.0.linux-amd64.tar.gz
Õâ¸öÒ³Ãæ¸ø³öµÄÊÇ×îа汾µÄ°²×°ÃüÁÓÒÉϽǿÉÒÔÑ¡ÔñÇл»ÆäËû°æ±¾µÄ°²×°ÃüÁî¡£

½âѹºó»á³öÏÖ grafana-6.0.0 Ŀ¼£¬½øÈë¸ÃĿ¼£¬È»ºó¾Í¿ÉÒÔÔËÐÐ Grafana ÁË£º

$ cd grafana-6.0.0
// Æô¶¯Grafana¡£$ ./bin/grafana-server web

ͨ¹ýlogÐÅÏ¢¿ÉÒÔ¿´µ½GrafanaĬÈÏÔËÐÐÔÚ3000¶Ë¿Ú£¬Õâ¸öÒ²¿ÉÒÔͨ¹ýÅäÖÃÎļþ½øÐÐÐ޸ģº´´½¨ÃûΪ custom.ini µÄÅäÖÃÎļþ£¬Ìí¼Óµ½ conf Îļþ¼Ð£¬¸´ÖÆ conf/defaults.ini Öж¨ÒåµÄËùÓÐÉèÖã¬È»ºóÐÞ¸Ä×Ô¼ºÏëÒªÐ޸ĵġ£

ÔÚ Grafana չʾ¼à¿ØÐÅÏ¢

°²×°²¢Æô¶¯Grafanaºó£¬ä¯ÀÀÆ÷ÊäÈë IP:3000 À´·ÃÎÊGrafana£¬¹ÜÀíÔ±Õ˺ÅÃÜÂëĬÈÏÊÇadmin/admin¡£Ê״εǽ»áÈÃÄãÐ޸ĹÜÀíÔ±ÃÜÂ룬Ȼºó¾Í¿ÉÒԵǼ²é¿´ÁË¡£

ÔÚ½çÃæ×ó±ßÊÇÒ»ÊúÅÅÑ¡ÏѡÔñÉèÖÃͼ±êÖеÄData Source£¬Ìí¼ÓPrometheusµÄÊý¾ÝÔ´£¬URL ¾ÍÌîÉÏÃæÄã¸ø Prometheus Server ÉèÖõÄip+¶Ë¿ÚºÅ¾ÍÐÐÁË£¬Èç¹ûû¸Ä¹ýÇÒÔÚ±¾»úÔËÐеϰ£¬ÄǾÍÊÇ localhost:9090¡£

´Ëʱ¿ÉÒÔÌí¼Ó dashboard£¬Ò²¾ÍÊÇ¼à¿ØÃæ°åÁË£¬ÔÚ¸ÕÅäºÃµÄ Prometheus Data SourceµÄÉèÖÃÖÐÓÐÒ»¸ö±êÇ©¾ÍÊÇ dashboard£¬ÎÒÃǵ¼Èë Prometheus 2.0 Stats Õâ¸öÃæ°å£¬¾ÍÄÜ¿´µ½ÎÒÃÇPrometheusµÄһЩ»ù±¾¼à¿ØÇé¿öÁË£¬ÕâÆäʵ¾ÍÊǵ¼ÈëÁËÒ»¸ö±ðÈËдºÃµÄÃæ°åÅäÖ㬲¢ÇÒÁ¬½ÓÎÒÃÇ×Ô¼º Prometheus µÄ¼à¿ØÊý¾Ý×öչʾ¡£»¹¼ÇµÃÎÒÃÇÉÏÃæ»¹ÔËÐÐÁËÒ»¸ö node exporter °É£¬ÏÖÔÚÎÒÃÇÕ¹ÏÖÒ»ÏÂÕâ¸ö¼à¿ØÐÅÏ¢£¬×ó±ßÊúÅŵã»÷¼ÓºÅͼ±êÖÐµÄ Import£¬À´µ¼ÈëÆäËû±ðÈËдºÃµÄÃæ°å¡£ÔÚGrafanaµÄ¹Ù·½Ãæ°åÒ³ÃæÆäʵ¿ÉÒÔ¿´µ½ºÜ¶à±ðÈËÅäÖúõÄÃæ°å£¬ÎÒÃÇÕÒµ½×Ô¼ºÏëÒªµÄÃæ°å£¬±ÈÈçÕâ¸önode exporter µÄ£º

¸´ÖÆÓÒ±ßÄǸöÃæ°åID£¬È»ºóÔÚ Import ½çÃæÊäÈëID£¬Load ºóÅäÖúÃÊý¾ÝԴΪÎÒÃǵÄPrometheus£¬¾Í¿ÉÒÔ³öÏÖÎÒÃÇ×Ô¼º»úÆ÷µÄ״̬¼à¿ØÃæ°åÁË£¬ºÜìÅ¿á°É¡£

Õâ¸öÃæ°åÐèÒª°²×°Ò»¸ö±ýͼµÄ²å¼þ(Ò³ÃæÉÏÓÐ˵Ã÷)£¬°²×° Grafana ²å¼þµÄ·½·¨Îª£º

// ½øÈëGrafana/binĿ¼
./grafana-cli plugins install [²å¼þÃû]
// °²×°³É¹¦ºóÖØÆôGrafana

Ãæ°åÖеÄÿ¸öͼ¶¼ÊÇ¿ÉÒԱ༭µÄ£¬Ò²¿ÉÒÔÉèÖø澯£¬Grafana ¸æ¾¯Ö§³Ö¶àÖÖ·½Ê½£¬ÎÒÃÇ×î³£ÓõľÍÊÇÓʼþºÍ webhook ÁË£¬ËùÒÔÆäʵ²»Ì«ÐèÒªÓà Prometheus µÄ¸æ¾¯¡£

   
1590 ´Îä¯ÀÀ       27
Ïà¹ØÎÄÕÂ

DevOpsתÐÍÈÚÈëµ½ÆóÒµÎÄ»¯
DevOps ÄÜÁ¦Ä£ÐÍ¡¢Ñݽø¼°°¸ÀýÆÊÎö
»ùÓÚ DevOps ÀíÄîµÄ˽ÓÐ PaaS ƽ̨ʵ¼ù
΢Èí¿ª·¢ÍŶӵÄDevOpsʵ¼ùÆôʾ
Ïà¹ØÎĵµ

DevOpsÇý¶¯Ó¦ÓÃÔËά±ä¸ïÓë´´ÐÂ
ÔËά¹ÜÀí¹æ»®
ÈçºÎʵÏÖÆóÒµÓ¦Óò¿Êð×Ô¶¯»¯
ÔËά×Ô¶¯»¯Êµ¼ù֮·
Ïà¹Ø¿Î³Ì

×Ô¶¯»¯ÔËά¹¤¾ß£¨»ùÓÚDevOps£©
»¥ÁªÍøÔËάÓëDevOps
MySQLÐÔÄÜÓÅ»¯¼°ÔËάÅàѵ
ITϵͳÔËά¹ÜÀí