实时日志分析之二:Openresty

Openresty是一个基于 Nginx 与 Lua 的高性能 Web 平台,其内部集成了大量精良的 Lua 库、第三方模块以及大多数的依赖项。用于方便地搭建能够处理超高并发、扩展性极高的动态 Web 应用、Web 服务和动态网关。
文档:https://moonbingbing.gitbooks.io/openresty-best-practices/content/index.html

环境

  • 系统:centos7
  • ip:192.168.2.236

准备

  • 优化linux内核参数
    sudo vim /etc/sysctl.conf 加入如下内容
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    net.ipv4.ip_forward = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.accept_source_route = 0
    kernel.sysrq = 0
    kernel.core_uses_pid = 1
    net.ipv4.tcp_syncookies = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    net.ipv4.tcp_max_tw_buckets = 6000
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_rmem = 4096 87380 4194304
    net.ipv4.tcp_wmem = 4096 16384 4194304
    net.core.wmem_default = 8388608
    net.core.rmem_default = 8388608
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.core.netdev_max_backlog = 262144
    net.core.somaxconn = 262144
    net.ipv4.tcp_max_orphans = 3276800
    net.ipv4.tcp_max_syn_backlog = 262144
    net.ipv4.tcp_timestamps = 0
    net.ipv4.tcp_synack_retries = 1
    net.ipv4.tcp_syn_retries = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_mem = 94500000 915000000 927000000
    net.ipv4.tcp_fin_timeout = 1
    net.ipv4.tcp_keepalive_time = 30
    net.ipv4.ip_local_port_range = 1024 65000
    vm.max_map_count=655360
    vm.swappiness = 1

sudo vim /etc/security/limits.conf 加入如下内容

1
2
3
4
5
6
7
8
# max number of open file descriptors
* soft nofile 32768
# max number of open file descriptors
* hard nofile 65536
# max number of processes
* soft nproc 32768
# max number of processes
* hard nproc 65536

sudo vim /etc/security/limits.d/90-nproc.conf 加入如下内容

1
2
3
4
# max number of processes
* soft nproc 32768
# max number of processes
* hard nproc 65536

重启服务器

1
sudo reboot

安装

1
2
3
4
5
6
7
8
sudo yum -y update
sudo useradd nginx -s /sbin/nologin -M # 新增nginx用户(禁止用户用于ssh登录,且不创建home目录)
wget https://openresty.org/download/openresty-1.11.2.1.tar.gz
tar -zxvf openresty-1.11.2.1.tar.gz
cd openresty-1.11.2.1
./configure --user=nginx --group=nginx --with-http_stub_status_module
gmake
sudo gmake install

默认openresty安装目录:/usr/local/openresty
默认nginx安装目录:/usr/local/openresty/nginx

配置

创建nginx log目录

1
2
sudo mkdir -p /var/log/nginx
sudo chown -R nginx:nginx /var/log/nginx

增加权限,使nginx其可绑定80端口

1
sudo setcap 'cap_net_bind_service=+ep' /usr/local/openresty/nginx/sbin/nginx

demo

下面做一个”hello world”项目openrestydemo
开发代码目录结构:
mahua

把代码放在服务器/opt/目录下,且保证nginx用户对/opt/openrestydemo目录下的lua脚本有访问权限(很重要)

  • openrestydemo/conf/nginx.conf代码:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    user nginx;
    worker_rlimit_nofile 65536;
    events {
    use epoll;
    worker_connections 40000;
    multi_accept on;
    }
    http {
    sendfile on;
    tcp_nopush on;
    open_file_cache max=2048 inactive=20s;
    open_file_cache_valid 60s;
    tcp_nodelay on;
    client_body_timeout 12s;
    client_header_timeout 12s;
    keepalive_timeout 15s;
    send_timeout 10s;
    types_hash_max_size 2048;
    include /usr/local/openresty/nginx/conf/mime.types;
    default_type application/octet-stream;
    underscores_in_headers on;
    log_not_found off;
    open_log_file_cache max=1000 inactive=20s valid=1m min_uses=2;
    log_format click '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" "$wesub"';
    access_log off;
    error_log /var/log/nginx/error.log error;
    gzip on;
    gzip_disable "msie6";
    gzip_min_length 1k;
    gzip_vary on;
    gzip_comp_level 2;
    gzip_buffers 16 8k;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;
    # lua代码路径
    lua_package_path 'lua/?.lua;/opt/openrestydemo/lua/?.lua;;';
    server {
    listen 80;
    server_name 192.168.2.236;
    set_by_lua_block $wesub {
    return '-'
    }
    location ^~ /v1/ad/click/ {
    default_type text/html;
    content_by_lua_file lua/click.lua;
    # access_log /var/log/nginx/we_click_access.log click buffer=16k; # 如果访问量较大,这里加buffer配置
    access_log /var/log/nginx/we_click_access.log click;
    }
    location /favicon.ico {
    return 204;
    access_log off;
    }
    }
    }
  • openrestydemo/lua/click.lua代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    -- 引用示例
    local json = require('utils.json')
    -- 模拟数据
    ngx.var.wesub = '3fe63613-83b1-474b-bd29-2d21d8709980:,12345:,45678:,XY:,0.252:,0.252:,2:,sub1::sub2,::,sub3'
    ngx.status=200
    ngx.say('hello world')
    return
  • openrestydemo/lua/utils/json.lua代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    -----------------------
    -- json相关操作 --
    -----------------------
    local _M = {}
    -- json decode
    function _M.decode(str)
    local json = require("cjson")
    local data = nil
    ok, data = pcall(function(str) return json.decode(str) end, str)
    return ok, data
    end
    -- json encode
    function _M.encode(tabl)
    local json = require("cjson")
    local data = nil
    ok, data = pcall(function(tabl) return json.encode(tabl) end, tabl)
    return ok, data
    end
    return _M
  • 启动openresty

    1
    sudo /usr/local/openresty/nginx/sbin/nginx -p /opt/openrestydemo

访问“http://192.168.2.236/v1/ad/click/?pubid=123&campid=456&gaid=abcdefg”
在/var/log/nginx/we_click_access.log中即可看到类似如下的日志:

1
192.168.2.182 - - [01/Sep/2017:07:53:30 +0000] "GET /v1/ad/click/?pubid=123&campid=456&gaid=abcdefg HTTP/1.1" 200 43 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36" "-" "3fe63613-83b1-474b-bd29-2d21d8709980:,12345:,45678:,XY:,0.252:,0.252:,2:,sub1::sub2,::,sub3"

这便是我们后面要实时分析的一条原始日志示例

备注:
reload openresty命令: sudo /usr/local/openresty/nginx/sbin/nginx -p /opt/openrestydemo -s reload
stop openresty命令: sudo /usr/local/openresty/nginx/sbin/nginx -p /opt/openrestydemo -s stop

关于openresty开发的具体使用,大家自行参考
https://moonbingbing.gitbooks.io/openresty-best-practices/content/index.html
下一步,我们将建立kafka集群,并介绍其基础使用

坚持原创技术分享,您的支持将鼓励我继续创作!

热评文章