검색결과 리스트
글
lightsail wordpress [pool wordpress] server reached pm.max_children setting (25) warninng
트래픽은 큰 차이가 안나는데 갑자기 cpu 사용량이 치솟으면서 사이트가 먹통이 되는 현상이 발생하였다.
그 당시는 외부에서 발표중이여서 서버 재부팅으로 그 순간을 넘겼는데 그 후 같은 현상이 반복되서 원인 파악에 들어갔다.
php-fpm 에러로그 파일 위치
/opt/bitnami/php/var/log/php-fpm.log
php-fpm 세팅 파일 위치
/opt/bitnami/php/etc/php-fpm.conf
bitnami@ip:/opt/bitnami/php/etc$ sudo cat /opt/bitnami/php/var/log/php-fpm.log
[15-May-2020 12:52:48] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 13 total children
[15-May-2020 12:52:56] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 14 total children
[15-May-2020 12:53:00] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 18 total children
[15-May-2020 12:53:03] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 22 total children
[15-May-2020 12:53:07] WARNING: [pool wordpress] server reached pm.max_children setting (25), consider raising it
[15-May-2020 12:56:03] WARNING: [pool wordpress] server reached pm.max_children setting (25), consider raising it
[16-May-2020 06:16:58] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 13 total children
[16-May-2020 06:17:07] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 3 idle, and 14 total children
[16-May-2020 06:17:08] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 3 idle, and 15 total children
[16-May-2020 07:52:26] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 2 idle, and 21 total children
[16-May-2020 07:52:29] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 2 idle, and 23 total children
[16-May-2020 07:52:34] WARNING: [pool wordpress] server reached pm.max_children setting (25), consider raising it
워드프레스 pool쪽에서 문제가 생긴듯하다.
참고로 php-fpm에서 사용하는 설정 파일(conf) 찾는법
bitnam$ ps aux | grep php-fpm
root 3476 0.0 0.2 146396 9340 ? Ss 08:20 0:01 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
daemon 3477 0.6 0.6 153868 25196 ? S 08:20 4:15 php-fpm: pool wordpress
daemon 3478 0.6 0.6 154064 25380 ? S 08:20 4:09 php-fpm: pool wordpress
해당 설정파일을 가면 마지막에 두 개의 pool 설정 파일을 include 하고 있다.
bitnam$ tail /opt/bitnami/php/etc/php-fpm.conf
include=/opt/bitnami/apps/wordpress/conf/php-fpm/pool.conf
include=/opt/bitnami/apps/phpmyadmin/conf/php-fpm/pool.conf
워드프레스 쪽의 설정 파일을 확인해본다.
bitnami@ip$ cat /opt/bitnami/apps/wordpress/conf/php-fpm/pool.conf
[wordpress]
listen=/opt/bitnami/php/var/run/wordpress.sock
include=/opt/bitnami/php/etc/common-dynamic.conf
include=/opt/bitnami/apps/wordpress/conf/php-fpm/php-settings.conf
include=/opt/bitnami/php/etc/environment.conf
pm=dynamic
여기서 다시 common-dynamic.conf를 include하고 있다.
다시 그 파일 설정으로 들어가면 common.conf 파일을 include하고 있다.
common.conf 파일까지 가야지 드디어
bitnami$ cat /opt/bitnami/php/etc/common-dynamic.conf
include=/opt/bitnami/php/etc/common.conf
pm=dynamic
pm.max_children=5
pm.start_servers=2
pm.min_spare_servers=1
pm.max_spare_servers=3
include=/opt/bitnami/php/etc/bitnami/common.conf
bitnami$ cat /opt/bitnami/php/etc/bitnami/common.conf
;
; Bitnami PHP-FPM Configuration
; Copyright 2019 Bitnami.com All Rights Reserved
;
; Note: This file will be modified on server size changes
;
pm.max_children=25
pm.start_servers=4
pm.min_spare_servers=4
pm.max_spare_servers=10
pm.max_requests=5000
각각의 옵션 설명은 아래 설정 파일의 설명을 참고하면 된다.
; Choose how the process manager will control the number of child processes.
; Possible Values:
; static - a fixed number (pm.max_children) of child processes;
; dynamic - the number of child processes are set dynamically based on the
; following directives. With this process management, there will be
; always at least 1 children.
; pm.max_children - the maximum number of children that can
; be alive at the same time.
; pm.start_servers - the number of children created on startup.
; pm.min_spare_servers - the minimum number of children in 'idle'
; state (waiting to process). If the number
; of 'idle' processes is less than this
; number then some children will be created.
; pm.max_spare_servers - the maximum number of children in 'idle'
; state (waiting to process). If the number
; of 'idle' processes is greater than this
; number then some children will be killed.
; ondemand - no children are created at startup. Children will be forked when
; new requests will connect. The following parameter are used:
; pm.max_children - the maximum number of children that
; can be alive at the same time.
; pm.process_idle_timeout - The number of seconds after which
; an idle process will be killed.
; Note: This value is mandatory.
pm.max_children 수를 50으로 증가시키고 php-fpm을 재시작해서 변경 사항을 반영한다.
sudo /opt/bitnami/ctlscript.sh restart php-fpm
그런데 이 포스팅을 작성하면서 cpu 사용량이 조금씩 상승하고 있어서 아파치 접속 로그를 확인해봤더니
아래와 같이 존재하지 않는 경로를 요청하는 수상한 ip들이 보여서 검색해보니 최근에 스팸 등으로 블랙리스트에 신고된 아이피였다.
https://cleantalk.org/blacklists/173.212.194.14
bitnami$ tail -f /opt/bitnami/apache2/logs/access_log
173.212.194.14 - - [17/May/2020:00:22:28 +0900] "GET /product-tag/med/?product_orderby=popularity&product_count=24&product_view=grid&product_order=asc HTTP/1.1" 301 -
173.212.194.14 - - [17/May/2020:00:22:47 +0900] "GET /?product_orderby=popularity&product_count=24&product_view=grid&product_order=asc HTTP/1.1" 200 22179
이것만은 원인은 아니겠지만 그래도 원인 중 하나일 것 같아서 워드프레스 보안 플러그인 설정 변경도 해야겠다.
php-fpm status 쉽게 확인하는 법
; By default the status page output is formatted as text/plain. Passing either
; 'html', 'xml' or 'json' in the query string will return the corresponding
; output syntax. Example:
; http://www.foo.bar/status
; http://www.foo.bar/status?json
; http://www.foo.bar/status?html
; http://www.foo.bar/status?xml
;
; By default the status page only outputs short status. Passing 'full' in the
; query string will also return status for each pool process.
; Example:
; http://www.foo.bar/status?full
; http://www.foo.bar/status?json&full
; http://www.foo.bar/status?html&full
; http://www.foo.bar/status?xml&full
; The Full status returns for each process:
; pid - the PID of the process;
; state - the state of the process (Idle, Running, ...);
; start time - the date and time the process has started;
; start since - the number of seconds since the process has started;
; requests - the number of requests the process has served;
; request duration - the duration in µs of the requests;
; request method - the request method (GET, POST, ...);
; request URI - the request URI with the query string;
; content length - the content length of the request (only with POST);
; user - the user (PHP_AUTH_USER) (or '-' if not set);
; script - the main script called (or '-' if not set);
; last request cpu - the %cpu the last request consumed
; it's always 0 if the process is not in Idle state
; because CPU calculation is done when the request
; processing has terminated;
; last request memory - the max amount of memory the last request consumed
; it's always 0 if the process is not in Idle state
; because memory calculation is done when the request
; processing has terminated;
; If the process is in Idle state, then informations are related to the
; last request the process has served. Otherwise informations are related to
; the current request being served.
; Example output:
; ************************
; pid: 31330
; state: Running
; start time: 01/Jul/2011:17:53:49 +0200
; start since: 63087
; requests: 12808
; request duration: 1250261
; request method: GET
; request URI: /test_mem.php?N=10000
; content length: 0
; user: -
; script: /home/fat/web/docs/php/test_mem.php
; last request cpu: 0.00
; last request memory: 0
;
; Note: There is a real-time FPM status monitoring sample web page available
; It's available in: /bitnami/lamp73stack-linux-x64/output/php/share/php/fpm/status.html
;
; Note: The value must start with a leading slash (/). The value can be
; anything, but it may not be a good idea to use the .php extension or it
; may conflict with a real PHP file.
; Default Value: not set
;pm.status_path = /status
그리고 응답이 늦어지는 상황을 좀 더 모니터링 하기 위해 slowlog를 설정한다
; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
;slowlog = log/$pool.log.slow
; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_slowlog_timeout = 0
최종 변경사항들은 아래와 같다.
pm.status_path = /status
slowlog = log/$pool.log.slow
request_slowlog_timeout = 0
참고: https://www.tecmint.com/enable-monitor-php-fpm-status-in-nginx/
https://stackoverflow.com/questions/15465333/php-fpm-processes-monitoring-profiling
https://docs.bitnami.com/aws/apps/wordpress-pro/configuration/configure-phpfpm-processes/
