lightsail wordpress [pool wordpress] server reached pm.max_children setting (25) warninng




트래픽은 큰 차이가 안나는데 갑자기 cpu 사용량이 치솟으면서 사이트가 먹통이 되는 현상이 발생하였다. 

그 당시는 외부에서 발표중이여서 서버 재부팅으로 그 순간을 넘겼는데 그 후 같은 현상이 반복되서 원인 파악에 들어갔다. 


php-fpm 에러로그 파일 위치

/opt/bitnami/php/var/log/php-fpm.log


php-fpm 세팅 파일 위치

/opt/bitnami/php/etc/php-fpm.conf




bitnami@ip:/opt/bitnami/php/etc$ sudo cat /opt/bitnami/php/var/log/php-fpm.log

[15-May-2020 12:52:48] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 13 total children

[15-May-2020 12:52:56] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 14 total children

[15-May-2020 12:53:00] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 18 total children

[15-May-2020 12:53:03] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 22 total children

[15-May-2020 12:53:07] WARNING: [pool wordpress] server reached pm.max_children setting (25), consider raising it

[15-May-2020 12:56:03] WARNING: [pool wordpress] server reached pm.max_children setting (25), consider raising it

[16-May-2020 06:16:58] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 3 idle, and 13 total children

[16-May-2020 06:17:07] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 3 idle, and 14 total children

[16-May-2020 06:17:08] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 3 idle, and 15 total children

[16-May-2020 07:52:26] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 2 idle, and 21 total children

[16-May-2020 07:52:29] WARNING: [pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 2 idle, and 23 total children

[16-May-2020 07:52:34] WARNING: [pool wordpress] server reached pm.max_children setting (25), consider raising it



워드프레스 pool쪽에서 문제가 생긴듯하다. 



참고로 php-fpm에서 사용하는 설정 파일(conf) 찾는법 


bitnam$ ps aux | grep php-fpm

root      3476  0.0  0.2 146396  9340 ?        Ss   08:20   0:01 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)

daemon    3477  0.6  0.6 153868 25196 ?        S    08:20   4:15 php-fpm: pool wordpress

daemon    3478  0.6  0.6 154064 25380 ?        S    08:20   4:09 php-fpm: pool wordpress



해당 설정파일을 가면 마지막에 두 개의 pool 설정 파일을 include 하고 있다. 




bitnam$ tail /opt/bitnami/php/etc/php-fpm.conf


include=/opt/bitnami/apps/wordpress/conf/php-fpm/pool.conf


include=/opt/bitnami/apps/phpmyadmin/conf/php-fpm/pool.conf


워드프레스 쪽의 설정 파일을 확인해본다. 


bitnami@ip$ cat /opt/bitnami/apps/wordpress/conf/php-fpm/pool.conf

[wordpress]

listen=/opt/bitnami/php/var/run/wordpress.sock

include=/opt/bitnami/php/etc/common-dynamic.conf

include=/opt/bitnami/apps/wordpress/conf/php-fpm/php-settings.conf

include=/opt/bitnami/php/etc/environment.conf

pm=dynamic




여기서 다시 common-dynamic.conf를 include하고 있다. 

다시 그 파일 설정으로 들어가면 common.conf 파일을 include하고 있다. 

common.conf 파일까지 가야지 드디어 


bitnami$ cat /opt/bitnami/php/etc/common-dynamic.conf

include=/opt/bitnami/php/etc/common.conf

pm=dynamic

pm.max_children=5

pm.start_servers=2

pm.min_spare_servers=1

pm.max_spare_servers=3


include=/opt/bitnami/php/etc/bitnami/common.conf




bitnami$ cat /opt/bitnami/php/etc/bitnami/common.conf

;

; Bitnami PHP-FPM Configuration

; Copyright 2019 Bitnami.com All Rights Reserved

;

; Note: This file will be modified on server size changes

;

pm.max_children=25

pm.start_servers=4

pm.min_spare_servers=4

pm.max_spare_servers=10

pm.max_requests=5000



각각의 옵션 설명은 아래 설정 파일의 설명을 참고하면 된다. 


; Choose how the process manager will control the number of child processes.

; Possible Values:

;   static  - a fixed number (pm.max_children) of child processes;

;   dynamic - the number of child processes are set dynamically based on the

;             following directives. With this process management, there will be

;             always at least 1 children.

;             pm.max_children      - the maximum number of children that can

;                                    be alive at the same time.

;             pm.start_servers     - the number of children created on startup.

;             pm.min_spare_servers - the minimum number of children in 'idle'

;                                    state (waiting to process). If the number

;                                    of 'idle' processes is less than this

;                                    number then some children will be created.

;             pm.max_spare_servers - the maximum number of children in 'idle'

;                                    state (waiting to process). If the number

;                                    of 'idle' processes is greater than this

;                                    number then some children will be killed.

;  ondemand - no children are created at startup. Children will be forked when

;             new requests will connect. The following parameter are used:

;             pm.max_children           - the maximum number of children that

;                                         can be alive at the same time.

;             pm.process_idle_timeout   - The number of seconds after which

;                                         an idle process will be killed.

; Note: This value is mandatory.





pm.max_children 수를 50으로 증가시키고 php-fpm을 재시작해서 변경 사항을 반영한다. 


sudo /opt/bitnami/ctlscript.sh restart php-fpm







그런데 이 포스팅을 작성하면서 cpu 사용량이 조금씩 상승하고 있어서 아파치 접속 로그를 확인해봤더니 

아래와 같이 존재하지 않는 경로를 요청하는 수상한 ip들이 보여서 검색해보니 최근에 스팸 등으로 블랙리스트에 신고된 아이피였다. 


https://cleantalk.org/blacklists/173.212.194.14



bitnami$ tail -f /opt/bitnami/apache2/logs/access_log


173.212.194.14 - - [17/May/2020:00:22:28 +0900] "GET /product-tag/med/?product_orderby=popularity&product_count=24&product_view=grid&product_order=asc HTTP/1.1" 301 -

173.212.194.14 - - [17/May/2020:00:22:47 +0900] "GET /?product_orderby=popularity&product_count=24&product_view=grid&product_order=asc HTTP/1.1" 200 22179




이것만은 원인은 아니겠지만 그래도 원인 중 하나일 것 같아서 워드프레스 보안 플러그인 설정 변경도 해야겠다. 




php-fpm status 쉽게 확인하는 법




; By default the status page output is formatted as text/plain. Passing either

; 'html', 'xml' or 'json' in the query string will return the corresponding

; output syntax. Example:

;   http://www.foo.bar/status

;   http://www.foo.bar/status?json

;   http://www.foo.bar/status?html

;   http://www.foo.bar/status?xml

;

; By default the status page only outputs short status. Passing 'full' in the

; query string will also return status for each pool process.

; Example:

;   http://www.foo.bar/status?full

;   http://www.foo.bar/status?json&full

;   http://www.foo.bar/status?html&full

;   http://www.foo.bar/status?xml&full

; The Full status returns for each process:

;   pid                  - the PID of the process;

;   state                - the state of the process (Idle, Running, ...);

;   start time           - the date and time the process has started;

;   start since          - the number of seconds since the process has started;

;   requests             - the number of requests the process has served;

;   request duration     - the duration in µs of the requests;

;   request method       - the request method (GET, POST, ...);

;   request URI          - the request URI with the query string;

;   content length       - the content length of the request (only with POST);

;   user                 - the user (PHP_AUTH_USER) (or '-' if not set);

;   script               - the main script called (or '-' if not set);

;   last request cpu     - the %cpu the last request consumed

;                          it's always 0 if the process is not in Idle state

;                          because CPU calculation is done when the request

;                          processing has terminated;

;   last request memory  - the max amount of memory the last request consumed

;                          it's always 0 if the process is not in Idle state

;                          because memory calculation is done when the request

;                          processing has terminated;

; If the process is in Idle state, then informations are related to the

; last request the process has served. Otherwise informations are related to

; the current request being served.

; Example output:

;   ************************

;   pid:                  31330

;   state:                Running

;   start time:           01/Jul/2011:17:53:49 +0200

;   start since:          63087

;   requests:             12808

;   request duration:     1250261

;   request method:       GET

;   request URI:          /test_mem.php?N=10000

;   content length:       0

;   user:                 -

;   script:               /home/fat/web/docs/php/test_mem.php

;   last request cpu:     0.00

;   last request memory:  0

;

; Note: There is a real-time FPM status monitoring sample web page available

;       It's available in: /bitnami/lamp73stack-linux-x64/output/php/share/php/fpm/status.html

;

; Note: The value must start with a leading slash (/). The value can be

;       anything, but it may not be a good idea to use the .php extension or it

;       may conflict with a real PHP file.

; Default Value: not set

;pm.status_path = /status






그리고 응답이 늦어지는 상황을 좀 더 모니터링 하기 위해 slowlog를 설정한다


; The log file for slow requests

; Default Value: not set

; Note: slowlog is mandatory if request_slowlog_timeout is set

;slowlog = log/$pool.log.slow


; The timeout for serving a single request after which a PHP backtrace will be

; dumped to the 'slowlog' file. A value of '0s' means 'off'.

; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)

; Default Value: 0

;request_slowlog_timeout = 0



최종 변경사항들은 아래와 같다. 


pm.status_path = /status

slowlog = log/$pool.log.slow

request_slowlog_timeout = 0




참고: https://www.tecmint.com/enable-monitor-php-fpm-status-in-nginx/


https://stackoverflow.com/questions/15465333/php-fpm-processes-monitoring-profiling


https://docs.bitnami.com/aws/apps/wordpress-pro/configuration/configure-phpfpm-processes/


https://www.php.net/manual/en/install.fpm.configuration.php