HAproxy日志的转换(与nignix,或者标准的HTTP日志)

2014 年 2 月 10 日1,2710

本站主要内容均为原创,转帖需注明出处www.alexclouds.net

       通常HAproxy的日志要处理才便于统计和分析,因为常用的AWSTAT或者其他WEB日志分析工具并不能很好的支持HAPROXY日志。那么要分析日志数据,就有两条路可走

1、HAproxy配套的haproxy_log_analysis

2、Custom HAproxy Log format

       第一种就参见 Google搜索haproxy_log_analysis。 第二种要在配置文件里写一些格式

例如这里我把HAproxy日志改为Nginx日志。

listen pg_for_production 0.0.0.0:80
        mode http
        option  httplog
        option httpclose
        option forwardfor
        capture request header Host len 32
        log-format %ci\ -\ -\ [%t]\ "%r"\ %ST\ %B\ "_"\ "_"

        server production 118.x.x.x:80 weight 256

出来的日志是这样的,基本上是标准的Ngnix日志!:

Feb 10 16:41:29 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.777] "GET /wp-content/uploads/2013/05/hadoop_thumb.jpg HTTP/1.1" 200 18635 "_" "_"
Feb 10 16:41:29 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.785] "GET /wp-content/uploads/2013/03/mysql-nosql1_thumb.jpg HTTP/1.1" 200 31930 "_" "_"
Feb 10 16:41:29 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.803] "GET /wp-content/themes/lianyue_theme/image/image_post/23.jpg HTTP/1.1" 200 30056 "_" "_"
Feb 10 16:41:29 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.821] "GET /wp-content/uploads/2012/12/yyffgltx.jpg HTTP/1.1" 200 52615 "_" "_"
Feb 10 16:41:29 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.940] "GET /wp-content/uploads/2012/12/IDC-ISP-1024x699.jpg HTTP/1.1" 200 109824 "_" "_"
Feb 10 16:41:29 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.959] "GET /wp-content/themes/lianyue_theme/image/image_post/21.JPG HTTP/1.1" 200 57043 "_" "_"
Feb 10 16:41:30 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:30.027] "GET /wp-content/themes/lianyue_theme/image/image_post/20.JPG HTTP/1.1" 200 77705 "_" "_"
Feb 10 16:41:30 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:30.098] "GET /wp-content/themes/lianyue_theme/image/bg.jpg HTTP/1.1" 200 18347 "_" "_"
Feb 10 16:41:31 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:29.607] "GET /wp-content/themes/lianyue_theme/image/image_post/43.jpg HTTP/1.1" 200 781947 "_" "_"
Feb 10 16:41:34 localhost haproxy[21683]: 118.242.238.18 - - [10/Feb/2014:16:41:34.109] "GET /?p=1574 HTTP/1.1" 200 22079 "_" "_"

 

       这里面有些说法,看下面的来自HAproxy网站的详细说明,照着下面的说明自己写就行。

 

Custom log format

The directive log-format allows you to customize the logs in http mode and tcp
mode. It takes a string as argument.

HAproxy understands some log format variables. % precedes log format variables.
Variables can take arguments using braces ('{}'), and multiple arguments are
separated by commas within the braces. Flags may be added or removed by
prefixing them with a '+' or '-' sign.

Special variable "%o" may be used to propagate its flags to all other
variables on the same format string. This is particularly handy with quoted
string formats ("Q").

If a variable is named between square brackets ('[' .. ']') then it is used
as a sample expression rule (see section 7.3). This it useful to add some
less common information such as the client's SSL certificate's DN, or to log
the key that would be used to store an entry into a stick table.

Note: spaces must be escaped. A space character is considered as a separator.
In order to emit a verbatim '%', it must be preceded by another '%' resulting
in '%%'. HAProxy will automatically merge consecutive separators.

Flags are :
  * Q: quote a string
  * X: hexadecimal representation (IPs, Ports, %Ts, %rt, %pid)

Example:

log-format %T\ %t\ Some\ Text
log-format %{+Q}o\ %t\ %s\ %{-Q}r

At the moment, the default HTTP format is defined this way :

    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ \
               %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r

the default CLF format is defined this way :

    log-format %{+Q}o\ %{-Q}ci\ -\ -\ [%T]\ %r\ %ST\ %B\ \"\"\ \"\"\ %cp\ \
               %ms\ %ft\ %b\ %s\ \%Tq\ %Tw\ %Tc\ %Tr\ %Tt\ %tsc\ %ac\ %fc\ \
               %bc\ %sc\ %rc\ %sq\ %bq\ %CC\ %CS\ \%hrl\ %hsl

and the default TCP format is defined this way :

    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ \
               %ac/%fc/%bc/%sc/%rc\ %sq/%bq

Please refer to the table below for currently defined variables :

  +---+------+-----------------------------------------------+-------------+
  | R | var  | field name (.2.2 and .2.3 for description)  | type        |
  +---+------+-----------------------------------------------+-------------+
  |   | %o   | special variable, apply flags on all next var |             |
  +---+------+-----------------------------------------------+-------------+
  |   | %B   | bytes_read           (from server to client)  | numeric     |
  | H | %CC  | captured_request_cookie                       | string      |
  | H | %CS  | captured_response_cookie                      | string      |
  |   | %H   | hostname                                      | string      |
  |   | %ID  | unique-id                                     | string      |
  |   | %ST  | status_code                                   | numeric     |
  |   | %T   | gmt_date_time                                 | date        |
  |   | %Tc  | Tc                                            | numeric     |
  |   | %Tl  | local_date_time                               | date        |
  | H | %Tq  | Tq                                            | numeric     |
  | H | %Tr  | Tr                                            | numeric     |
  |   | %Ts  | timestamp                                     | numeric     |
  |   | %Tt  | Tt                                            | numeric     |
  |   | %Tw  | Tw                                            | numeric     |
  |   | %U   | bytes_uploaded       (from client to server)  | numeric     |
  |   | %ac  | actconn                                       | numeric     |
  |   | %b   | backend_name                                  | string      |
  |   | %bc  | beconn      (backend concurrent connections)  | numeric     |
  |   | %bi  | backend_source_ip       (connecting address)  | IP          |
  |   | %bp  | backend_source_port     (connecting address)  | numeric     |
  |   | %bq  | backend_queue                                 | numeric     |
  |   | %ci  | client_ip                 (accepted address)  | IP          |
  |   | %cp  | client_port               (accepted address)  | numeric     |
  |   | %f   | frontend_name                                 | string      |
  |   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
  |   | %fi  | frontend_ip              (accepting address)  | IP          |
  |   | %fp  | frontend_port            (accepting address)  | numeric     |
  |   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
  |   | %hr  | captured_request_headers default style        | string      |
  |   | %hrl | captured_request_headers CLF style            | string list |
  |   | %hs  | captured_response_headers default style       | string      |
  |   | %hsl | captured_response_headers CLF style           | string list |
  |   | %ms  | accept date milliseconds                      | numeric     |
  |   | %pid | PID                                           | numeric     |
  | H | %r   | http_request                                  | string      |
  |   | %rc  | retries                                       | numeric     |
  |   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
  |   | %s   | server_name                                   | string      |
  |   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
  |   | %si  | server_IP                   (target address)  | IP          |
  |   | %sp  | server_port                 (target address)  | numeric     |
  |   | %sq  | srv_queue                                     | numeric     |
  | S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
  | S | %sslv| ssl_version (ex: TLSv1)                       | string      |
  |   | %t   | date_time      (with millisecond resolution)  | date        |
  |   | %ts  | termination_state                             | string      |
  | H | %tsc | termination_state with cookie status          | string      |
  +---+------+-----------------------------------------------+-------------+

    R = Restrictions : H = mode http only ; S = SSL only

0 0