PHPCBA-4: Server Limits may prevent a full check from being completed

Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: PHPChangedBinaries (PHPCBA)
Resolution: Fixed (2015-01-30 12:36:35)
Components: PHP Client ,

Created: 2015-01-30 10:52:09
Time Spent Working


Description
One of my deployments has a wrapper script which regenerates additional_files.cfg on each run. The average size of this file is 4.5m (roughly 48,000 files/directories).

Runs occasionally fail due to the webhost's tweaking of php.ini .

- If safe_mode is enabled files then file_exists may return false
- The PHP memory limit may be hit
- Max execution time may also be hit

The first can only really be addressed by editing php.ini - although there are potential workarounds, they all feel a bit hacky.

The latter two can be addressed, so a bugfix should be developed.


Toggle State Changes

Activity


2015-01-30 11:01:49
Have made sure each file checked can have at least 60 seconds execution time and have removed the additional_files.cfg contents from memory when no longer required.

The overall memory usage could also probably be reduced by removing files from changedbinariesmain::files as they're checked.

Running a test update/check with the current changeset at the moment.

2015-01-30 11:03:14

Repo: PHPChangedBinaries
Commit: b24cca41626594eaa1b598fabe4eccf0c8a6a111
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 10:54:47 2015 +0000
Commit Message: Unloaded the additional_files config file from memory when no longer required. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php




Webhook User-Agent 

GitHub-Hookshot/afa9a7c


View Commit


2015-01-30 11:03:15

Repo: PHPChangedBinaries
Commit: 307dd0d8cc0d25670d579c545fe226c618d36e22
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 10:59:22 2015 +0000
Commit Message: Extended time limit on a per file basis. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php




Webhook User-Agent 

GitHub-Hookshot/afa9a7c


View Commit


2015-01-30 11:13:14

Repo: PHPChangedBinaries
Commit: 195c33e8bb6ff565215d43f3ec2e651a98b308d6
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 11:07:39 2015 +0000
Commit Message: Removed file/directory paths from memory once checked. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php




Webhook User-Agent 

GitHub-Hookshot/afa9a7c


View Commit


2015-01-30 11:21:16
The test run appears to have completed successfully. Pulling in the changes in commits 1995c3e -> c5efbc8 and re-running

2015-01-30 11:42:32
New test run is underway.

The changes in 1995c3e have had the expected effect - memory usage still climbs as the script progresses (it is, after all, still buffering notifications etc) but not as sharply as it did before.

The host does, however, have quite a low memory limit set for PHP (32M!) so commit f890c98 uses ini_set to up this limit to 256M (which should be more than enough!).

2015-01-30 11:43:14

Repo: PHPChangedBinaries
Commit: f890c98bdbbeb9323b351c96583cca7c11b25918
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 11:41:59 2015 +0000
Commit Message: Set memory limit to 256M. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php




Webhook User-Agent 

GitHub-Hookshot/afa9a7c


View Commit


2015-01-30 11:45:00
For reference, the settings that host has used are as follows 

# php-cli -r "phpinfo();" | egrep 'memory_limit|max_exec|safe_'
max_execution_time => 0 => 0
memory_limit => 32M => 32M
safe_mode => On => On
safe_mode_exec_dir => /usr/local/php/bin => /usr/local/php/bin
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sql.safe_mode => Off => Off
filter.default => unsafe_raw => unsafe_raw
safe_mode_allowed_env_vars => PHP_ => PHP_
safe_mode_protected_env_vars => LD_LIBRARY_PATH => LD_LIBRARY_PATH


Safe mode has since been disabled.

2015-01-30 12:02:13
Commit 20fda47 prevents messages from being buffered if they won't eventually be emailed out (due to configuration settings). Messages will still be printed to stdout though.

2015-01-30 12:03:14

Repo: PHPChangedBinaries
Commit: 20fda4761a5d3c7b90bf56fde20ffc55f4016d1b
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 12:01:25 2015 +0000
Commit Message: Prevented buffering of messages which will not be emailed. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/lib/notify.class.php




Webhook User-Agent 

GitHub-Hookshot/afa9a7c


View Commit


2015-01-30 12:18:53
Took some metrics before and after applying 20fda47 to the test-run.

Generated with 
root@serv01 
Unassigned
# for i in {1..90}; do ps aux | grep changed_binaries >> usage.txt; sleep 30; done



Before 
root@serv01 
Unassigned
# grep "changed_binaries.php" usage.txt 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     29086  0.8  1.0 441584 42344 pts/3    S+   05:40   0:02 php-cli changed_binaries.php --upd
root     29086  0.8  1.1 442352 42932 pts/3    S+   05:40   0:02 php-cli changed_binaries.php --upd
root     29086  0.8  1.1 442864 43488 pts/3    S+   05:40   0:03 php-cli changed_binaries.php --upd
root     29086  0.8  1.1 443888 44396 pts/3    S+   05:40   0:03 php-cli changed_binaries.php --upd
root     29086  0.8  1.1 444656 44956 pts/3    S+   05:40   0:03 php-cli changed_binaries.php --upd
root     29086  0.8  1.1 445168 45448 pts/3    S+   05:40   0:03 php-cli changed_binaries.php --upd
root     29086  0.8  1.1 445936 46044 pts/3    S+   05:40   0:04 php-cli changed_binaries.php --upd
root     29086  0.8  1.2 446192 46548 pts/3    S+   05:40   0:04 php-cli changed_binaries.php --upd
root     29086  0.7  1.2 446704 47020 pts/3    S+   05:40   0:04 php-cli changed_binaries.php --upd
root     29086  0.7  1.2 447472 47592 pts/3    S+   05:40   0:04 php-cli changed_binaries.php --upd
root     29086  0.7  1.2 447984 48216 pts/3    R+   05:40   0:04 php-cli changed_binaries.php --upd
root     29086  0.7  1.2 448496 48820 pts/3    S+   05:40   0:05 php-cli changed_binaries.php --upd
root     29086  0.7  1.2 449008 49336 pts/3    S+   05:40   0:05 php-cli changed_binaries.php --upd
root     29086  0.7  1.2 449520 49872 pts/3    S+   05:40   0:05 php-cli changed_binaries.php --upd
root     29086  0.7  1.3 450032 50456 pts/3    S+   05:40   0:05 php-cli changed_binaries.php --upd
root     29086  0.7  1.3 451828 51908 pts/3    S+   05:40   0:06 php-cli changed_binaries.php --upd
root     29086  0.7  1.3 452084 52180 pts/3    S+   05:40   0:06 php-cli changed_binaries.php --upd
root     29086  0.7  1.3 452596 52692 pts/3    S+   05:40   0:06 php-cli changed_binaries.php --upd
root     29086  0.7  1.3 453364 53232 pts/3    S+   05:40   0:06 php-cli changed_binaries.php --upd
root     29086  0.8  1.3 453620 53704 pts/3    S+   05:40   0:07 php-cli changed_binaries.php --upd
root     29086  0.8  1.4 454388 54292 pts/3    S+   05:40   0:07 php-cli changed_binaries.php --upd
root     29086  0.8  1.4 454900 54864 pts/3    S+   05:40   0:08 php-cli changed_binaries.php --upd
root     29086  0.8  1.4 455412 55376 pts/3    S+   05:40   0:08 php-cli changed_binaries.php --upd
root     29086  0.8  1.4 455668 55860 pts/3    S+   05:40   0:08 php-cli changed_binaries.php --upd
root     29086  0.8  1.4 456436 56396 pts/3    S+   05:40   0:09 php-cli changed_binaries.php --upd
root     29086  0.9  1.4 456692 56808 pts/3    S+   05:40   0:09 php-cli changed_binaries.php --upd
root     29086  0.9  1.4 457460 57452 pts/3    S+   05:40   0:10 php-cli changed_binaries.php --upd
root     29086  0.9  1.5 457972 57924 pts/3    S+   05:40   0:10 php-cli changed_binaries.php --upd
root     29086  0.9  1.4 457456 57656 pts/3    S+   05:40   0:11 php-cli changed_binaries.php --upd
root     29086  1.0  1.4 457456 57656 pts/3    S+   05:40   0:13 php-cli changed_binaries.php --upd
root     29086  1.2  1.4 457456 57668 pts/3    R+   05:40   0:15 php-cli changed_binaries.php --upd
root     29086  1.3  1.5 457456 57776 pts/3    R+   05:40   0:17 php-cli changed_binaries.php --upd


After 
root@serv01 
Unassigned
# grep "changed_binaries.php" usage.txt 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     30501  3.4  1.0 441584 42316 pts/3    S+   06:03   0:00 php-cli changed_binaries.php --upd
root     30501  2.5  1.0 441584 42316 pts/3    S+   06:03   0:01 php-cli changed_binaries.php --upd
root     30501  2.1  1.0 441584 42316 pts/3    S+   06:03   0:01 php-cli changed_binaries.php --upd
root     30501  2.0  1.0 441584 42316 pts/3    S+   06:03   0:02 php-cli changed_binaries.php --upd
root     30501  1.9  1.1 441584 42356 pts/3    S+   06:03   0:02 php-cli changed_binaries.php --upd
root     30501  1.8  1.1 441584 42356 pts/3    S+   06:03   0:03 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:03 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:04 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:04 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:05 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:05 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:06 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:06 php-cli changed_binaries.php --upd
root     30501  1.6  1.1 441584 42356 pts/3    S+   06:03   0:06 php-cli changed_binaries.php --upd
root     30501  1.6  1.1 441584 42356 pts/3    S+   06:03   0:07 php-cli changed_binaries.php --upd
root     30501  1.6  1.1 441584 42356 pts/3    S+   06:03   0:08 php-cli changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:08 php-cli changed_binaries.php --upd
root     30501  1.7  1.0 440812 41848 pts/3    S+   06:03   0:09 php-cli changed_binaries.php --upd
root     30501  2.6  1.0 440812 41852 pts/3    R+   06:03   0:14 php-cli changed_binaries.php --upd


So memory usage has been reduced considerably - the run also completed in a shorter time. For reference, that run processed 52,278 hashes.

2015-01-30 12:25:14

Repo: PHPChangedBinaries
Commit: 622e7312940aecdd85c0471c5c17c911aaefa207
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 12:24:17 2015 +0000
Commit Message: Adjusted notification class to ensure counts are correct even if sending disabled. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/lib/notify.class.php




Webhook User-Agent 

GitHub-Hookshot/afa9a7c


View Commit


2015-01-30 12:25:21
Commit 20fda47 does, however, introduce an issue with the emailed report - the counts provided are based on counting the arrays (which are no longer being populated). Commit 622e731 implements a counter for each notification type.

2015-01-30 12:36:35
Test run completed successfully and notification count is correct. Closing this issue as Fixed.

2015-01-30 12:36:35
btasker changed status from 'Open' to 'Resolved'

2015-01-30 12:36:35
btasker added 'Fixed' to resolution

2015-01-30 12:36:39
btasker changed status from 'Resolved' to 'Closed'