/contrib/famzah

Enthusiasm never stops


Leave a comment

Achieving Zero-Downtime PHP-FPM Restarts and Atomic Updates

The Challenge with PHP-FPM Restarts

PHP-FPM (FastCGI Process Manager) is a powerful solution for managing PHP processes, but it poses challenges when updating PHP applications or configurations without impacting active requests. A key setting in PHP-FPM is process_control_timeout, which dictates how long the master process waits for a child to finish before forcefully terminating it on graceful restart. If a slow child lingers, incoming requests queue up, causing delays for the duration of the timeout. This delay can lead to significant downtime, especially for high-traffic applications.

The Solution: Zero-Downtime Restarts and Atomic Deployments

These challenges are addressed through a proof-of-concept designed to enable zero-downtime PHP-FPM restarts and atomic source code updates, ensuring seamless service continuity. Key components of the solution include:

  1. Redundant PHP-FPM Pools with Replicas:
    • PHP-FPM instances are managed in redundant pools with at least two replicas.
    • While one replica restarts, the other remains active, ensuring no downtime.
  2. Load Balancer Management:
    • A lightweight load balancer dynamically toggles traffic between replicas during restarts, making the switch invisible to users.
  3. Atomic Code Deployment:
    • Instead of directly using the “release” symlink as the PHP-FPM working directory, the release target is mounted in an isolated user namespace before starting the PHP-FPM master processes.
    • This ensures that changes to the symlink location can be made immediately for new deployments, while running PHP-FPM masters continue to use their isolated view of the release directory until they are restarted at a convenient time.

Important Considerations

  • Static Content and Atomicity:
    • In this setup, Apache serves static content (files not ending in “.php") directly by following the “release” symlink. This means that updates to the “release” symlink immediately impact Apache-served static files, diverging from the atomic deployment of PHP sources.
    • To achieve true atomic deployment of both static content and PHP files, the setup must be reworked. This could involve putting both PHP and static file serving behind a single backend and managing traffic between two such backends in the same way that traffic is currently managed between PHP-FPM backends.
  • Temporary Capacity Reduction:
    • During the restart of one replica (PHP-FPM master), only half of the capacity is available. This capacity reduction should be considered in the context of the expected traffic load during deployments or restarts.
  • Increased Memory Usage:
    • Running two (or more) identical PHP-FPM masters introduces higher memory consumption, as each master maintains its own independent OPcache. This redundancy ensures reliability and atomicity but comes at the cost of increased resource usage.

Demo, documentation and source code

The full code and setup guide are available on GitHub. Contributions and feedback are welcome!


Leave a comment

HTTP Keep-Alive timeout values used by popular websites

Here is the command to test the Keep-Alive timeout of a website:

VHOST="www.google.bg"; time openssl s_client -ign_eof -connect "$VHOST:443" <<< "$( echo -ne "GET / HTTP/1.1\r\nHost: $VHOST\r\nConnection: Keep-Alive\r\n\r\n" )"

And here are the today’s results for some popular websites:

slashdot.org: 0s
LWN.net: 5s
snag.gy: 5s
yahoo.com: 10s
readthedocs.org: 10s
www.superhosting.bg: 15s
httpd.apache.org: 30s
nginx.org: 1m
en.wikipedia.org: 1m
famzah.wordpress.com: 1m15s
aws.amazon.com: 2m50s
www.facebook.com: 3m
www.google.bg: 4m
cloudplatform.googleblog.com: 4m
www.cloudflare.com: 6m40s
www.mozilla.org: 6m40s
www.tagesschau.de: 8m
access.redhat.com: 8m20s
stackoverflow.com: 10m
www.timeanddate.com: 10m
www.dreamhost.com: 10m
www.reddit.com: 10m
twitter.com: 15m


Leave a comment

Enable Forward Secrecy in Apache 2.4

The Heartbleed bug once again raised a very serious question — what happens if someone steals our SSL private key. The future actions are all clear — we revoke the key, change it on the server and we are secure. However any recorded past SSL encrypted sessions which the attacker posses can be decrypted to plain text using the stolen SSL private key.

The article “SSL: Intercepted today, decrypted tomorrow” explains the problem in great details. It also suggests who may take advantage of your stolen SSL private key.

Fortunately there is a solution to this attack vector. It is called Forward Secrecy and solves the problem by using a different private key to encrypt each new SSL session. If an attacker wanted to decrypt all your SSL sessions, the attacker would need to brute-force the private keys of each of your SSL sessions. While this attack vector still exists, current computing power is too small to solve such a task in a reasonable time. Note that Forward Secrecy is not new at all and was invented in 1992, pre-dating the SSL protocol by two years, as stated in the “SSL: Intercepted today, decrypted tomorrow” article.

Many websites and blogs recommend their own cipher list which needs to be used, in order to enable Perfect Secrecy, and at the same time to not be vulnerable to the BEAST attack. Trusting such long lists of ciphers without understanding all of them makes me a bit insecure. Luckily the OpenSSL vendors have created a special string which we can use to identify the TLS v1.2 specific ciphers — “TLSv1.2”. This makes your Apache configuration very readable and leaves the control in the OpenSSL developers who can update the ciphers accordingly:

SSLProtocol -ALL -SSLv2 +SSLv3 +TLSv1 +TLSv1.1 +TLSv1.2
SSLHonorCipherOrder on
SSLCipherSuite TLSv1.2:RC4:HIGH:!aNULL:!eNULL:!MD5
SSLCompression off

TraceEnable Off

The effect of this configuration is that for TLS v1.2 connections you will prefer the strong TLS v1.2 ciphers. For all other connections, like SSLv3 or TLS v1.0, the RC4 ciphers will be used which are required to protect against the BEAST attack. Note that TLS v1.2 does not suffer from the BEAST attack.

This configuration won’t give you Forward Secrecy for Internet Explorer. The explanation here says:
“Internet Explorer, in all versions, does not support the ECDHE and RC4 combination (which has the benefit of supporting Forward Secrecy and being resistant to BEAST). But IE has long patched the BEAST vulnerability and so we shouldn’t worry about it.”

It’s worth mentioning that using those strong TLS v1.2 ciphers may increase the CPU time required to establish a new SSL connection 3 times.


Real-time SSL test:

Resources:

Solutions: