Centos 7 – Part 4 – HAProxy Standby with SSL support combined with NGINX Load Balancing

These instructions expand on the previous post. The previous post shows how to implement HAPROXY with SSL in front of two NGINX load balancers with NGINX servers having Fail Over enabled.  This post will show how to create add another HAPROXY server in order to have fail over enabled,

As explained on the previous post, HAPROXY and keepalived needs to be installed.

HAProxy

Configure HAProxy.

The configuration file for server HAPROXY2 is the same as with the configuration file with server HAPROXY1, minus of course the IP address that we bind. Important: You must copy the ssl certificate files from HAPROXY1 to HAPROXY2 server under the directory specified in the config file.

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

frontend https_frontend
  bind 10.0.0.53:443 ssl crt /etc/ssl/haproxy1.sfentona.lol/haproxy1.pem
  mode http
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend web_server

backend web_server
  mode http
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server wordpressvirtip 10.0.0.44:80

Configure Keepalived

Keep in mind that we already have keepalived running for the two NGINX load balancers. We have designated them in the keepalived.cfg as virtual_router_id 51. For the HAproxy servers we are going to assign them a different id. Servers HAPROXY1 and HAPROXY2 will now be designated as virtual_router_id 52 .

Keepalived config file for HAPROXY1

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
script "killall -0 haproxy"     # cheaper than pidof
interval 2                      # check every 2 seconds
weight 2                        # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface ens192
state MASTER
virtual_router_id 52
priority 101                    # 101 on master, 100 on backup
virtual_ipaddress {
10.0.0.54
}
track_script {
chk_haproxy
}
}

Keepalived config file for HAPROXY2

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
script "killall -0 haproxy"     # cheaper than pidof
interval 2                      # check every 2 seconds
weight 2                        # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface ens192
state MASTER
virtual_router_id 52
priority 100                    # 101 on master, 100 on backup
virtual_ipaddress {
10.0.0.54
}
track_script {
chk_haproxy
}
}

Configure your Firewall
The following IPTABLE rules should be running on both HAPROXY1 and HAPROXY2. Copy and paste the following rules in a text file and import them to your firewall table.

# Generated by iptables-save v1.4.21 on Thu Oct  8 15:18:59 2015
*filter
:INPUT ACCEPT [26988:2784395]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [35111:2263400]
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p vrrp -j ACCEPT
-A INPUT -d 224.0.0.0/8 -j ACCEPT
COMMIT
# Completed on Thu Oct  8 15:18:59 2015
iptables-restore < /root/firewall.fw
Share Button

Centos 7 – Part 3 – HAProxy with SSL support combined with NGINX Load Balancing

In the previous post instructions were given on how to create a HAProxy combined with NGINX Load Balancing. However that particular setup did not support SSL. These instructions will implement SSL support at the HAProxy server. The NGINX servers receive SSL traffic but the connection between the NGINX servers and the Apache web servers are with out SSL. It should be noted the ability to login the actual IPs of visitors is not lost with the implementation of SSL.

The difference here is that in the haproxy config file we specify the use of SSL and we no longer use the “listen” section like we did with out the use of SSL in the previous post.

nginx-HA

Generate the SSL certificate.

openssl genrsa -out haproxy1.key  1024
openssl  req -new -key haproxy1.key  -out haproxy1.csr
openssl ca -policy policy_anything -in haproxy1.csr  -out haproxy1.crt
openssl x509 -req -days 365 -in  haproxy1.csr  -signkey haproxy1.key  -out haproxy1.cr
cat haproxy1.crt haproxy1.key |   tee haproxy1.pem

 

Configure HAPROXY

vi /etc/haproxy/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

frontend https_frontend
  bind 10.0.0.52:443 ssl crt /etc/ssl/haproxy1.sfentona.lol/haproxy1.pem
  mode http
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend web_server

backend web_server
  mode http
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server wordpressvirtip 10.0.0.44:80
Share Button

Centos 7 – Part 2 – HAProxy combined with NGINX Load Balancing.

In this previous post instructions were written on how to setup a Round Robin Load Balancer by using NGINX and a virtual IP that would pass requests to the Apache Web Servers.

In this post we will use the very same setup but we place a HAProxy server in front of the Virtual IP the NGINX servers created. This server will use the Round Robin protocol as well and it will pass the requests to the NGINX servers which will in return will pass the web requests to the Apache web servers. SSL is not yet implemented.

The IP address of the HAProxy is 10.0.0.52 and the IP address of the virtual IP we created is 10.0.0.44 with a DNS entry “wordpressvirtip”

nginx-HA

 

 

 

 

 

 

 

 

 

 

 

 

 

 

yum install haproxy
vi /etc/haproxy/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

# [HTTP Site Configuration]
listen  http_web 10.0.0.52:80
        mode http
        balance roundrobin  # Load Balancing algorithm
        option httpchk
        option forwardfor
        server wordpressvirtip 10.0.0.44:80 weight 1 maxconn 512 check
        #server server2 10.0.0.40:80 weight 1 maxconn 512 check

# [HTTPS Site Configuration]
#listen  https_web 192.168.10.10:443
#        mode tcp
#        balance source# Load Balancing algorithm
#        reqadd X-Forwarded-Proto:\ http
#        server server1 192.168.10.100:443 weight 1 maxconn 512 check
#        server server2 192.168.10.101:443 weight 1 maxconn 512 check
 system start haproxy
Share Button

Centos 7 – Part 1 – NGINX – Apache – Load Balancing High Availability

These instructions show how to setup a web Load Balancer by using two NGINX servers as the Load Balancers and two Apache servers.

2 – Centos 7  servers running NGINX will be used as Load Balancers.

2 – Cent0s 7 serves running Apache will be used to serve web pages via virtual hosts.

The NGINX servers will:
– Determine the appropriate destination service based on the method chosen; in this case it will be Round Robin, which is the default option.
– Will use KeepAlive in order to create a virtual IP address. The IP address will based on the already NICs of the Load Balancers.

The Apache servers will:
– Act as your normal every day web servers with virtual hosts.
– Will log the IP of the actual client.

nginx-HA

Setting up the Apache web Servers.

  • Create the directories where the content of you virtual hosts will be placed.
mkdir /etc/httpd/vhosts.d
mkdir -p /sites/wordpress/
chown -R apache:apache /sites/wordpress/
chmod 755 /sites/wordpress/

 

  • Instruct Apache to look into the directory you created for your virtual hosts.
vim /etc/httpd/conf.d/vhosts.conf
IncludeOptional vhosts.d/*.conf

 

  • Create the config file for the corresponding web site.
vim /etc/httpd/vhosts.d/wordpress.sfentona.lol.conf
ServerAdmin webmaster@dummy-host.example.com
DocumentRoot /sites/wordpress/
ServerName wordpress.sfentona.lol
ServerAlias www.wordpress.sfentona.lol

Directory "/sites/wordpress";
DirectoryIndex index.html index.php
Options FollowSymLinks
AllowOverride All
Require all granted

 

  • Set up log x-fowarded-for in order to get the the IP of the actual client who visited the site and not the IP of the load balancer in your logs.
vi /etc/httpd/conf/httpd.conf
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded

 

Configure NGINX as a load balancer (view lines 34-38)

vi /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    upstream wordpress {
    server 10.0.0.42:80;
    server 10.0.0.43:80;
}


        server {
        listen       80;
        server_name  www.wordpress.sfentona.lol;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
                proxy_pass http://wordpress;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

Allow NGINX to bind to a non-local shared ip

vi /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1
sysctl -p

Set up your firewall in order for Multicast and VRRP to work correctly.

iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
iptables -I INPUT -p vrrp -j ACCEPT

 

Configure Keep Alive.

vi/etc/keepalived/keepalived.conf

This is for the Master Load Banacer LB1

 notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb1@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.44
    }
}

This is for the Master Load Banacer LB2. Priority on the slave is a lower number.

global_defs {
   notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb2@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.44
    }
}
Share Button

Puppet Enterprise – Manifest for Centos7 to create a yum repo for Centos 7.

This is my first puppet manifest. It can pick up between Debian and RedHat/Centos operating systems. It installs apache, creates the config files and directories for the virtual hosts to exist. It also creates the directories for where the RPMs will be downloaded in order for the repo to be created. In addition it also creates a cron job to sync the RPMs. The command on the cronjob section can be changed. The one currently listed is not optimal at all.

Note:

class yumrepos {
        include ::yumrepos::params
        include ::yumrepos::install
        include ::yumrepos::file
}

class yumrepos::install {
        package { 'apache2':
                ensure  => installed,
                name    => $yumrepos::params::packagename,
                }
}

class yumrepos::file {
        file {$yumrepos::params::centorepodirs:
                ensure  => directory,
                owner   => 'apache',
                group   => 'apache',
                require => Package['apache2'],

                }

        file {$yumrepos::params::vhostdirconf:
                ensure  => directory,
                require => Package['apache2'],
                }

        file {$yumrepos::params::httpd_conf:
                ensure  => file,
                require => Package[$yumrepos::params::packagename],
                content => template('yumrepos/httpd.conf.erb'),
                }

        file {$yumrepos::params::repoVhost:
                ensure  => file,
                require => Package[$yumrepos::params::packagename],
                content => template('yumrepos/rpmsrepo.sfentona.lol.conf.erb'),
                }
        file {$yumrepos::params::vhostconfig:
                ensure  => file,
                require => File[$yumrepos::params::vhostdirconf],
                content => template('yumrepos/vhosts.conf.erb'),
                }
        file {$yumrepos::params::disableWelcome:
                ensure  => file,
                require =>  Package['apache2'],
                content => template('yumrepos/welcome.conf.erb'),
                }
        #FOR TESTING 
        #file {$yumrepos::params::repoindex:
        #       ensure  => file,
        #       require => File[$yumrepos::params::centorepodirs],
        #       content => template('yumrepos/index.html.erb'),
        #       }

        cron { 'update_cron':
                ensure          => 'present',
                command         => 'rsync -vaH --exclude-from=${EXCLUDES} --numeric-ids --delete --delete-after --delay-updates rsync://eu-msync.centos.org/CentOS/ /repo/Centos06/centos/ & 2>> /var/log/centosrsync',
                user            => 'root',
                monthday        => [1],
                hour            => [23],
                require         => File[$yumrepos::params::centorepodirs],
                }

}

class yumrepos::params{

        $packagename = $::osfamily ? {
                /(?i:debian)/ => 'apache2',
                /(?i:redhat)/ => 'httpd',
                        }
#Directories
$centorepodirs  = ["/repo/", "/repo/Centos06", "/repo/Centos06/centos/"]
$vhostdirconf   = "/etc/httpd/vhosts.d"

#Apache config files
$repoVhost      = "/etc/httpd/vhosts.d/rpmsrepo.sfentona.lol.conf"
$httpd_conf     = "/etc/httpd/conf/httpd.conf"
$vhostconfig    = "/etc/httpd/conf.d/vhosts.conf"
$disableWelcome = "/etc/httpd/conf.d/welcome.conf"

#HTML file FOR TESTING
#$repoindex      = "/repo/Centos06/centos/index.html"

}
Share Button

Apache2.4.6 (CentOS 7) – Create Virtual Hosts

The instructions below show how to create virtual hosts in Centos 7. In this case the virtual hosts will be running from a custom location in the system and not from directories created in the /home directory.

Edit/Create your Virtual Host file
Create/edit the following file by entering the following line. This will tell httpd to read the configuration for the actual virtual hosts, provided of course that the main httpd.conf file is instructed (usually by default) to read the files in ‘conf.d/*.conf’ directory.

vim /etc/httpd/conf.d/vhosts.conf
IncludeOptional vhosts.d/*.conf

Create the directory where the config files for the Virtual Hosts will reside.

mkdir /etc/httpd/vhosts.d

Create the config file for your first Virtual Host.

vim /etc/httpd/vhosts.d/rpmsrepo.sfentona.lol.conf

Include the following lines in the config file.

<VirtualHost *:80>
 ServerAdmin webmaster@dummy-host.example.com
 DocumentRoot /repo/Centos06/centos/
 ServerName rpmsrepo.sfentona.lol
 ServerAlias www.rpmsrepo.sfentona.lol

<Directory "/repo/Centos06/centos/">
 DirectoryIndex index.html index.php
 Options FollowSymLinks
 AllowOverride All
 Require all granted
 </Directory>
</VirtualHost>

Create the directories where your virtual hosts will be running from
mkdir /repo/Centos06/centos/
chown -R apache:apache /repo/Centos06/centos/
chmod 755  /repo/Centos06/centos/

Check your httpd configuration and start httpd

apachectl configtest
systemctl restart httpd
systemctl status httpd
Share Button

Apache/2.4.6 (CentOS 7) – Determine which MPM module is the default and which one is being used

Find which MPM module is the default of your Apache install.

httpd -V

Output

Server MPM: prefork


Find which MPM module is actually being used.

– Make sure the mod_info.so module is enabled in your configuration file (httpd.conf or in /etc/httpd/conf.modules.d/00-base.conf). Also include into your httpd.conf file the following.

SetHandler server-info

– Then execute the following command below to find which MPM module is currently being used.

curl localhost/server-info | grep 'MPM Name'

Change which MPM Module apache is using.

vim /etc/httpd/conf.modules.d/00-mpm.conf

Select which MPM module you wish to use.

#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
#LoadModule mpm_worker_module modules/mod_mpm_worker.so
#LoadModule mpm_event_module modules/mod_mpm_event.so
Share Button

Apache2 – Parameters, StartServers, MinSpareServers, etc

  • TimeOut Directive: Amount of time the server will wait for certain events before failing a request

The TimeOut directive defines the length of time Apache will wait for I/O in various circumstances:

  • When reading data from the client, the length of time to wait for a TCP packet to arrive if the read buffer is empty.
  • When writing data to the client, the length of time to wait for an acknowledgement of a packet if the send buffer is full.
  • In mod_cgi, the length of time to wait for output from a CGI script.
  • In mod_ext_filter, the length of time to wait for output from a filtering process.
  • In mod_proxy, the default timeout value if ProxyTimeout is not configured.
  • On virtualized servers like VPS servers, lowering this value to 100 can help improve performance.
  • KeepAlive: Enables HTTP persistent connections. High volume and/or load balanced servers should have this setting disabled (Off) to increase connection throughput.
  • MaxKeepAliveRequests: This setting works only if KeepAlive is enabled as well. This setting limits the number of requests allowed per persistent connection when KeepAlive is on. If it is set to 0, unlimited requests will be allowed.
  • KeepAliveTimeout: The number of seconds Apache will wait for another request before closing the connection. Setting this to a high value may cause performance problems in heavily loaded servers. The higher the timeout, the more server processes will be kept occupied waiting on connections with idle clients.
  • StartServers: is the number of Apache (httpd) processes that will be created upon startup
  • MinSpareServers: Minimum number of Servers kept on IDLE
  • MaxSpareServers: Max number of Server kept on IDLE.

 

Share Button

Apache2 – Access based on IP – http password access

How to allow/deny access based on IP address. Also how to use the htpasswd file

1) Access based on IP – See lines 9 to 11


    Options Indexes FollowSymLinks
    AllowOverride AuthConfig
    AuthType Basic
    AuthName "Authentication Required"
    Require valid-user
    AuthUserFile "/var/www/html/.htpasswd"
    #Require all granted
    Order deny,allow
    Deny from all
    Allow from 10.0.0.29


2) – http password access – See lines 3 to 7
The .htpasswd file will need to be told which users it should accept. You do this issuing the following command:

1
htpasswd -c .htpasswd gmastrokostas

    Options Indexes FollowSymLinks
    AllowOverride AuthConfig
    AuthType Basic
    AuthName "Authentication Required"
    Require valid-user
    AuthUserFile "/var/www/html/.htpasswd"
    #Require all granted
    Order deny,allow
    Deny from all
    Allow from 10.0.0.29


Share Button

Python – Find RAM usage for highest HTTP process and calculate MAXClients setting.

The script examines the /proc directory and imports the numerical directories in a list and distinguishes which process is for HTTP. It will then calculate the amount of RAM it uses. This script actually can pickup other process as well, if told so and report the amount of RAM it consumes. The script will then calculate how much RAM it uses

#!/usr/bin/python
import os
import psutil
import subprocess
import time

'''
if os.exists("/etc/redhat-release"):
    process = "httpd"
else:
    process = "apache2"
'''

pids = [int(pid) for pid in os.listdir('/proc') if pid.isdigit()]
http_proccess = []
for elements in pids:
    p = psutil.Process(elements)
    p.name()
    p.memory_info_ex()[0]
    if p.name() == 'apache2':
        #print p.name(), p.memory_info_ex()[0]
        http_proccess.append(p.memory_info()[0])
        process = p.name()
        apache_status = True
    elif p.name() == 'httpd':
        #print p.name(), p.memory_info_ex()[0]
        http_proccess.append(p.memory_info()[0])
        process = p.name()
        apache_status = True
    else:
        pass

print "--------------------------------------------------"
http_largest_hog = float((max(http_proccess) / 1000000.0))
print "Largest ", process, " process is ",  round(http_largest_hog, 10), "MBs"

free_memory_proc = float(psutil.virtual_memory().free) / 1000000.0
print "Free memory with ", process, " running",round(free_memory_proc,10), "MBs"
if apache_status == True:
    subprocess.call('service apache2 stop', shell=True )
    time.sleep(10)
    print "Processing...."
else:
    pass

free_memory_Noproc = float(psutil.virtual_memory().free) / 1000000.0
print "Free memory with out ", process, " running",round(free_memory_Noproc,10), "MBs"

ram_gained =  free_memory_proc - free_memory_Noproc
print "RAM Gained after ", process, "terminated: ",ram_gained, " MBs"

apache_maxClients = ram_gained / http_largest_hog
print "Adjust Apache MAXCLIENTS parameter at ", apache_maxClients
Share Button