##########################################################################################

       PHPCBA-4: Server Limits may prevent a full check from being completed

##########################################################################################


Issue Type: Bug 
-----------------------------------------------------------------------------------------

Issue Information
====================

Priority: Major					Status:      Closed
Resolution:  Fixed (2015-01-30 12:36:35)
Project: PHPChangedBinaries (PHPCBA)


Reported By: btasker					
Assigned To: btasker

Components: 
	- PHP Client 




Time Estimate: 0 minutes
Time Logged:   0 minutes


-----------------------------------------------------------------------------------------

Issue Description
==================

One of my deployments has a wrapper script which regenerates _additional\_files.cfg_ on
each run. The average size of this file is 4.5m (roughly 48,000 files/directories).

Runs occasionally fail due to the webhost's tweaking of _php.ini_. 

- If _safe\_mode_ is enabled files then _file\_exists_ may return false
- The PHP memory limit may be hit
- Max execution time may also be hit

The first can only really be addressed by editing _php.ini_  - although there are
potential workarounds, they all feel a bit hacky.

The latter two can be addressed, so a bugfix should be developed.

-----------------------------------------------------------------------------------------

Activity
==========

	
-----------------------------------------------------------------------------------------
2015-01-30 11:01:49              btasker
-----------------------------------------------------------------------------------------

Have made sure each file checked can have at least 60 seconds execution time and have
removed the _additional\_files.cfg_ contents from memory when no longer required.

The overall memory usage could also probably be reduced by removing files from
_changedbinariesmain::files_ as they're checked.

Running a test update/check with the current changeset at the moment.
	
-----------------------------------------------------------------------------------------
2015-01-30 11:03:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: PHPChangedBinaries
Commit: b24cca41626594eaa1b598fabe4eccf0c8a6a111
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 10:54:47 2015 +0000
Commit Message: Unloaded the additional_files config file from memory when no longer
required. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/afa9a7c
 -- END SNIPPET --

https://github.com/bentasker/PHPChangedBinaries/commit/b24cca41626594eaa1b598fabe4eccf0c8a6a111


	
-----------------------------------------------------------------------------------------
2015-01-30 11:03:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: PHPChangedBinaries
Commit: 307dd0d8cc0d25670d579c545fe226c618d36e22
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 10:59:22 2015 +0000
Commit Message: Extended time limit on a per file basis. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/afa9a7c
 -- END SNIPPET --

https://github.com/bentasker/PHPChangedBinaries/commit/307dd0d8cc0d25670d579c545fe226c618d36e22


	
-----------------------------------------------------------------------------------------
2015-01-30 11:13:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: PHPChangedBinaries
Commit: 195c33e8bb6ff565215d43f3ec2e651a98b308d6
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 11:07:39 2015 +0000
Commit Message: Removed file/directory paths from memory once checked. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/afa9a7c
 -- END SNIPPET --

https://github.com/bentasker/PHPChangedBinaries/commit/195c33e8bb6ff565215d43f3ec2e651a98b308d6


	
-----------------------------------------------------------------------------------------
2015-01-30 11:21:16              btasker
-----------------------------------------------------------------------------------------

The test run appears to have completed successfully. Pulling in the changes in commits
1995c3e -> c5efbc8 and re-running
	
-----------------------------------------------------------------------------------------
2015-01-30 11:42:32              btasker
-----------------------------------------------------------------------------------------

New test run is underway.

The changes in 1995c3e have had the expected effect - memory usage still climbs as the
script progresses (it is, after all, still buffering notifications etc) but not as sharply
as it did before.

The host does, however, have quite a low memory limit set for PHP (32M!) so commit f890c98
uses ini_set to up this limit to 256M (which should be more than enough!).
	
-----------------------------------------------------------------------------------------
2015-01-30 11:43:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: PHPChangedBinaries
Commit: f890c98bdbbeb9323b351c96583cca7c11b25918
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 11:41:59 2015 +0000
Commit Message: Set memory limit to 256M. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/bin/changed_binaries.php



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/afa9a7c
 -- END SNIPPET --

https://github.com/bentasker/PHPChangedBinaries/commit/f890c98bdbbeb9323b351c96583cca7c11b25918


	
-----------------------------------------------------------------------------------------
2015-01-30 11:45:00              btasker
-----------------------------------------------------------------------------------------

For reference, the settings that host has used are as follows


-- BEGIN SNIPPET --

# php-cli -r "phpinfo();" | egrep 'memory_limit|max_exec|safe_'
max_execution_time => 0 => 0
memory_limit => 32M => 32M
safe_mode => On => On
safe_mode_exec_dir => /usr/local/php/bin => /usr/local/php/bin
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sql.safe_mode => Off => Off
filter.default => unsafe_raw => unsafe_raw
safe_mode_allowed_env_vars => PHP_ => PHP_
safe_mode_protected_env_vars => LD_LIBRARY_PATH => LD_LIBRARY_PATH

 -- END SNIPPET --

Safe mode has since been disabled.
	
-----------------------------------------------------------------------------------------
2015-01-30 12:02:13              btasker
-----------------------------------------------------------------------------------------

Commit _20fda47_ prevents messages from being buffered if they won't eventually be emailed
out (due to configuration settings). Messages will still be printed to stdout though.
	
-----------------------------------------------------------------------------------------
2015-01-30 12:03:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: PHPChangedBinaries
Commit: 20fda4761a5d3c7b90bf56fde20ffc55f4016d1b
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 12:01:25 2015 +0000
Commit Message: Prevented buffering of messages which will not be emailed. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/lib/notify.class.php



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/afa9a7c
 -- END SNIPPET --

https://github.com/bentasker/PHPChangedBinaries/commit/20fda4761a5d3c7b90bf56fde20ffc55f4016d1b


	
-----------------------------------------------------------------------------------------
2015-01-30 12:18:53              btasker
-----------------------------------------------------------------------------------------

Took some metrics before and after applying _20fda47_ to the test-run.

Generated with

-- BEGIN SNIPPET --

root@serv01 [~]# for i in {1..90}; do ps aux | grep changed_binaries >> usage.txt; sleep
30; done

 -- END SNIPPET --


*Before*

-- BEGIN SNIPPET --

root@serv01 [~]# grep "changed_binaries.php" usage.txt 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     29086  0.8  1.0 441584 42344 pts/3    S+   05:40   0:02 php-cli
changed_binaries.php --upd
root     29086  0.8  1.1 442352 42932 pts/3    S+   05:40   0:02 php-cli
changed_binaries.php --upd
root     29086  0.8  1.1 442864 43488 pts/3    S+   05:40   0:03 php-cli
changed_binaries.php --upd
root     29086  0.8  1.1 443888 44396 pts/3    S+   05:40   0:03 php-cli
changed_binaries.php --upd
root     29086  0.8  1.1 444656 44956 pts/3    S+   05:40   0:03 php-cli
changed_binaries.php --upd
root     29086  0.8  1.1 445168 45448 pts/3    S+   05:40   0:03 php-cli
changed_binaries.php --upd
root     29086  0.8  1.1 445936 46044 pts/3    S+   05:40   0:04 php-cli
changed_binaries.php --upd
root     29086  0.8  1.2 446192 46548 pts/3    S+   05:40   0:04 php-cli
changed_binaries.php --upd
root     29086  0.7  1.2 446704 47020 pts/3    S+   05:40   0:04 php-cli
changed_binaries.php --upd
root     29086  0.7  1.2 447472 47592 pts/3    S+   05:40   0:04 php-cli
changed_binaries.php --upd
root     29086  0.7  1.2 447984 48216 pts/3    R+   05:40   0:04 php-cli
changed_binaries.php --upd
root     29086  0.7  1.2 448496 48820 pts/3    S+   05:40   0:05 php-cli
changed_binaries.php --upd
root     29086  0.7  1.2 449008 49336 pts/3    S+   05:40   0:05 php-cli
changed_binaries.php --upd
root     29086  0.7  1.2 449520 49872 pts/3    S+   05:40   0:05 php-cli
changed_binaries.php --upd
root     29086  0.7  1.3 450032 50456 pts/3    S+   05:40   0:05 php-cli
changed_binaries.php --upd
root     29086  0.7  1.3 451828 51908 pts/3    S+   05:40   0:06 php-cli
changed_binaries.php --upd
root     29086  0.7  1.3 452084 52180 pts/3    S+   05:40   0:06 php-cli
changed_binaries.php --upd
root     29086  0.7  1.3 452596 52692 pts/3    S+   05:40   0:06 php-cli
changed_binaries.php --upd
root     29086  0.7  1.3 453364 53232 pts/3    S+   05:40   0:06 php-cli
changed_binaries.php --upd
root     29086  0.8  1.3 453620 53704 pts/3    S+   05:40   0:07 php-cli
changed_binaries.php --upd
root     29086  0.8  1.4 454388 54292 pts/3    S+   05:40   0:07 php-cli
changed_binaries.php --upd
root     29086  0.8  1.4 454900 54864 pts/3    S+   05:40   0:08 php-cli
changed_binaries.php --upd
root     29086  0.8  1.4 455412 55376 pts/3    S+   05:40   0:08 php-cli
changed_binaries.php --upd
root     29086  0.8  1.4 455668 55860 pts/3    S+   05:40   0:08 php-cli
changed_binaries.php --upd
root     29086  0.8  1.4 456436 56396 pts/3    S+   05:40   0:09 php-cli
changed_binaries.php --upd
root     29086  0.9  1.4 456692 56808 pts/3    S+   05:40   0:09 php-cli
changed_binaries.php --upd
root     29086  0.9  1.4 457460 57452 pts/3    S+   05:40   0:10 php-cli
changed_binaries.php --upd
root     29086  0.9  1.5 457972 57924 pts/3    S+   05:40   0:10 php-cli
changed_binaries.php --upd
root     29086  0.9  1.4 457456 57656 pts/3    S+   05:40   0:11 php-cli
changed_binaries.php --upd
root     29086  1.0  1.4 457456 57656 pts/3    S+   05:40   0:13 php-cli
changed_binaries.php --upd
root     29086  1.2  1.4 457456 57668 pts/3    R+   05:40   0:15 php-cli
changed_binaries.php --upd
root     29086  1.3  1.5 457456 57776 pts/3    R+   05:40   0:17 php-cli
changed_binaries.php --upd

 -- END SNIPPET --

*After*

-- BEGIN SNIPPET --

root@serv01 [~]# grep "changed_binaries.php" usage.txt 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     30501  3.4  1.0 441584 42316 pts/3    S+   06:03   0:00 php-cli
changed_binaries.php --upd
root     30501  2.5  1.0 441584 42316 pts/3    S+   06:03   0:01 php-cli
changed_binaries.php --upd
root     30501  2.1  1.0 441584 42316 pts/3    S+   06:03   0:01 php-cli
changed_binaries.php --upd
root     30501  2.0  1.0 441584 42316 pts/3    S+   06:03   0:02 php-cli
changed_binaries.php --upd
root     30501  1.9  1.1 441584 42356 pts/3    S+   06:03   0:02 php-cli
changed_binaries.php --upd
root     30501  1.8  1.1 441584 42356 pts/3    S+   06:03   0:03 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:03 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:04 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:04 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:05 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:05 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:06 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:06 php-cli
changed_binaries.php --upd
root     30501  1.6  1.1 441584 42356 pts/3    S+   06:03   0:06 php-cli
changed_binaries.php --upd
root     30501  1.6  1.1 441584 42356 pts/3    S+   06:03   0:07 php-cli
changed_binaries.php --upd
root     30501  1.6  1.1 441584 42356 pts/3    S+   06:03   0:08 php-cli
changed_binaries.php --upd
root     30501  1.7  1.1 441584 42356 pts/3    S+   06:03   0:08 php-cli
changed_binaries.php --upd
root     30501  1.7  1.0 440812 41848 pts/3    S+   06:03   0:09 php-cli
changed_binaries.php --upd
root     30501  2.6  1.0 440812 41852 pts/3    R+   06:03   0:14 php-cli
changed_binaries.php --upd

 -- END SNIPPET --

So memory usage has been reduced considerably - the run also completed in a shorter time.
For reference, that run processed 52,278 hashes.
	
-----------------------------------------------------------------------------------------
2015-01-30 12:25:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: PHPChangedBinaries
Commit: 622e7312940aecdd85c0471c5c17c911aaefa207
Author: Ben Tasker <github@<Domain Hidden>>

Date: Fri Jan 30 12:24:17 2015 +0000
Commit Message: Adjusted notification class to ensure counts are correct even if sending
disabled. See PHPCBA-4



Modified (-)(+)
-------
changed_binaries/lib/notify.class.php



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/afa9a7c
 -- END SNIPPET --

https://github.com/bentasker/PHPChangedBinaries/commit/622e7312940aecdd85c0471c5c17c911aaefa207


	
-----------------------------------------------------------------------------------------
2015-01-30 12:25:21              btasker
-----------------------------------------------------------------------------------------

Commit 20fda47 does, however, introduce an issue with the emailed report - the counts
provided are based on counting the arrays (which are no longer being populated). Commit
622e731 implements a counter for each notification type.
	
-----------------------------------------------------------------------------------------
2015-01-30 12:36:35              btasker
-----------------------------------------------------------------------------------------

Test run completed successfully and notification count is correct. Closing this issue as
Fixed.
	
-----------------------------------------------------------------------------------------
2015-01-30 12:36:35              
-----------------------------------------------------------------------------------------

btasker changed status from 'Open' to 'Resolved'
	
-----------------------------------------------------------------------------------------
2015-01-30 12:36:35              
-----------------------------------------------------------------------------------------

btasker added 'Fixed' to resolution
	
-----------------------------------------------------------------------------------------
2015-01-30 12:36:39              
-----------------------------------------------------------------------------------------

btasker changed status from 'Resolved' to 'Closed'