Python 2.7 – Find Location of IP with the geoip2 database.

The script collects the IP addresses from the apache log file. It then uses the geoip2 database in order to find the geographical location of the IP. More information for the geoip2 database can be found at http://dev.maxmind.com/geoip/geoip2/downloadable/

The module used to capture the IPs from the apache log file requires a CustomLog format. It needs to be specified in the apache config file and in the script. The string used is

("%h <<%P>> %t %Dus \"%r\" %>s %b  \"%{Referer}i\" \"%{User-Agent}i\" %l %u"
import geoip2.database
import apache_log_parser

#specify the log file we will capture the IP from.
dir     = "/var/log/apache2/"
file    = "access.log"
apache_logfile = dir+file


#Create a connection to the mmdb file with all the IP geo-location data.
reader = geoip2.database.Reader("GeoLite2-City.mmdb")

#In case we cannot open the file throw an error message
try:
        f_open = open(apache_logfile, "rb")
except Exception as e:
        print e

#As required by the apache_log_parser module
line_parser = apache_log_parser.make_parser("%h <<%P>> %t %Dus \"%r\" %>s %b  \"%{Referer}i\" \"%{User-Agent}i\" %l %u")

#This is the list we will put in IPs.
ip_list = []

for loop in f_open:  #We are going through the file specified
    log_line_data = line_parser(loop) #We are using the apache parser as specified above
    remote_ip  =  log_line_data['remote_host'] #The apache parser returns a dictionary. We just want the remote_host key.
    for ip  in remote_ip:
        ip_list.append(remote_ip)  #We are appending the IPs to the list we created above.


unique_ip_list = set(ip_list)  # We delete the duplicate IP entries from our list.
for ips in unique_ip_list:
        try: #In case the IP is not recognized by the geoip2 database
                locate_ip = reader.city(ips) # we are using the geoip2 module here with the IPs from our list
                print ips, locate_ip.country.name
        except Exception as e:
                print e

Share Button

HAproxy – KeepAlived important notes.

  • Multicast must be enabled for fail over to work. Otherwise, If the haproxy service stops but the keepalived service is still running, the VirtualIP will still be pingable however any requests made to that IP will not be forwarded via HAproxy simply because HAproxy is no longer running. The keepalived will switch to the standby server only when the keepalived service it self stops from ServerA. Then ServerB will automatically take over.  In other words, failure of HAproxy does not mean Keepalive will automatically switch to the standby server.
  • If for whatever reason you wish to run a HA service with out SSL then your HAproxy config files should listen to the virtual IP address.
Share Button

Centos 7 – Part 4 – HAProxy Standby with SSL support combined with NGINX Load Balancing

These instructions expand on the previous post. The previous post shows how to implement HAPROXY with SSL in front of two NGINX load balancers with NGINX servers having Fail Over enabled.  This post will show how to create add another HAPROXY server in order to have fail over enabled,

As explained on the previous post, HAPROXY and keepalived needs to be installed.

HAProxy

Configure HAProxy.

The configuration file for server HAPROXY2 is the same as with the configuration file with server HAPROXY1, minus of course the IP address that we bind. Important: You must copy the ssl certificate files from HAPROXY1 to HAPROXY2 server under the directory specified in the config file.

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

frontend https_frontend
  bind 10.0.0.53:443 ssl crt /etc/ssl/haproxy1.sfentona.lol/haproxy1.pem
  mode http
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend web_server

backend web_server
  mode http
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server wordpressvirtip 10.0.0.44:80

Configure Keepalived

Keep in mind that we already have keepalived running for the two NGINX load balancers. We have designated them in the keepalived.cfg as virtual_router_id 51. For the HAproxy servers we are going to assign them a different id. Servers HAPROXY1 and HAPROXY2 will now be designated as virtual_router_id 52 .

Keepalived config file for HAPROXY1

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
script "killall -0 haproxy"     # cheaper than pidof
interval 2                      # check every 2 seconds
weight 2                        # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface ens192
state MASTER
virtual_router_id 52
priority 101                    # 101 on master, 100 on backup
virtual_ipaddress {
10.0.0.54
}
track_script {
chk_haproxy
}
}

Keepalived config file for HAPROXY2

vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
script "killall -0 haproxy"     # cheaper than pidof
interval 2                      # check every 2 seconds
weight 2                        # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface ens192
state MASTER
virtual_router_id 52
priority 100                    # 101 on master, 100 on backup
virtual_ipaddress {
10.0.0.54
}
track_script {
chk_haproxy
}
}

Configure your Firewall
The following IPTABLE rules should be running on both HAPROXY1 and HAPROXY2. Copy and paste the following rules in a text file and import them to your firewall table.

# Generated by iptables-save v1.4.21 on Thu Oct  8 15:18:59 2015
*filter
:INPUT ACCEPT [26988:2784395]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [35111:2263400]
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p vrrp -j ACCEPT
-A INPUT -d 224.0.0.0/8 -j ACCEPT
COMMIT
# Completed on Thu Oct  8 15:18:59 2015
iptables-restore < /root/firewall.fw
Share Button

Centos 7 – Part 3 – HAProxy with SSL support combined with NGINX Load Balancing

In the previous post instructions were given on how to create a HAProxy combined with NGINX Load Balancing. However that particular setup did not support SSL. These instructions will implement SSL support at the HAProxy server. The NGINX servers receive SSL traffic but the connection between the NGINX servers and the Apache web servers are with out SSL. It should be noted the ability to login the actual IPs of visitors is not lost with the implementation of SSL.

The difference here is that in the haproxy config file we specify the use of SSL and we no longer use the “listen” section like we did with out the use of SSL in the previous post.

nginx-HA

Generate the SSL certificate.

openssl genrsa -out haproxy1.key  1024
openssl  req -new -key haproxy1.key  -out haproxy1.csr
openssl ca -policy policy_anything -in haproxy1.csr  -out haproxy1.crt
openssl x509 -req -days 365 -in  haproxy1.csr  -signkey haproxy1.key  -out haproxy1.cr
cat haproxy1.crt haproxy1.key |   tee haproxy1.pem

 

Configure HAPROXY

vi /etc/haproxy/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

frontend https_frontend
  bind 10.0.0.52:443 ssl crt /etc/ssl/haproxy1.sfentona.lol/haproxy1.pem
  mode http
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend web_server

backend web_server
  mode http
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server wordpressvirtip 10.0.0.44:80
Share Button

Centos 7 – Part 2 – HAProxy combined with NGINX Load Balancing.

In this previous post instructions were written on how to setup a Round Robin Load Balancer by using NGINX and a virtual IP that would pass requests to the Apache Web Servers.

In this post we will use the very same setup but we place a HAProxy server in front of the Virtual IP the NGINX servers created. This server will use the Round Robin protocol as well and it will pass the requests to the NGINX servers which will in return will pass the web requests to the Apache web servers. SSL is not yet implemented.

The IP address of the HAProxy is 10.0.0.52 and the IP address of the virtual IP we created is 10.0.0.44 with a DNS entry “wordpressvirtip”

nginx-HA

 

 

 

 

 

 

 

 

 

 

 

 

 

 

yum install haproxy
vi /etc/haproxy/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

# [HTTP Site Configuration]
listen  http_web 10.0.0.52:80
        mode http
        balance roundrobin  # Load Balancing algorithm
        option httpchk
        option forwardfor
        server wordpressvirtip 10.0.0.44:80 weight 1 maxconn 512 check
        #server server2 10.0.0.40:80 weight 1 maxconn 512 check

# [HTTPS Site Configuration]
#listen  https_web 192.168.10.10:443
#        mode tcp
#        balance source# Load Balancing algorithm
#        reqadd X-Forwarded-Proto:\ http
#        server server1 192.168.10.100:443 weight 1 maxconn 512 check
#        server server2 192.168.10.101:443 weight 1 maxconn 512 check
 system start haproxy
Share Button

Centos 7 – Part 1 – NGINX – Apache – Load Balancing High Availability

These instructions show how to setup a web Load Balancer by using two NGINX servers as the Load Balancers and two Apache servers.

2 – Centos 7  servers running NGINX will be used as Load Balancers.

2 – Cent0s 7 serves running Apache will be used to serve web pages via virtual hosts.

The NGINX servers will:
– Determine the appropriate destination service based on the method chosen; in this case it will be Round Robin, which is the default option.
– Will use KeepAlive in order to create a virtual IP address. The IP address will based on the already NICs of the Load Balancers.

The Apache servers will:
– Act as your normal every day web servers with virtual hosts.
– Will log the IP of the actual client.

nginx-HA

Setting up the Apache web Servers.

  • Create the directories where the content of you virtual hosts will be placed.
mkdir /etc/httpd/vhosts.d
mkdir -p /sites/wordpress/
chown -R apache:apache /sites/wordpress/
chmod 755 /sites/wordpress/

 

  • Instruct Apache to look into the directory you created for your virtual hosts.
vim /etc/httpd/conf.d/vhosts.conf
IncludeOptional vhosts.d/*.conf

 

  • Create the config file for the corresponding web site.
vim /etc/httpd/vhosts.d/wordpress.sfentona.lol.conf
ServerAdmin webmaster@dummy-host.example.com
DocumentRoot /sites/wordpress/
ServerName wordpress.sfentona.lol
ServerAlias www.wordpress.sfentona.lol

Directory "/sites/wordpress";
DirectoryIndex index.html index.php
Options FollowSymLinks
AllowOverride All
Require all granted

 

  • Set up log x-fowarded-for in order to get the the IP of the actual client who visited the site and not the IP of the load balancer in your logs.
vi /etc/httpd/conf/httpd.conf
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded

 

Configure NGINX as a load balancer (view lines 34-38)

vi /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    upstream wordpress {
    server 10.0.0.42:80;
    server 10.0.0.43:80;
}


        server {
        listen       80;
        server_name  www.wordpress.sfentona.lol;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
                proxy_pass http://wordpress;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

Allow NGINX to bind to a non-local shared ip

vi /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1
sysctl -p

Set up your firewall in order for Multicast and VRRP to work correctly.

iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
iptables -I INPUT -p vrrp -j ACCEPT

 

Configure Keep Alive.

vi/etc/keepalived/keepalived.conf

This is for the Master Load Banacer LB1

 notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb1@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.44
    }
}

This is for the Master Load Banacer LB2. Priority on the slave is a lower number.

global_defs {
   notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb2@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.44
    }
}
Share Button

Apache/2.4.6 (CentOS 7) – Determine which MPM module is the default and which one is being used

Find which MPM module is the default of your Apache install.

httpd -V

Output

Server MPM: prefork


Find which MPM module is actually being used.

– Make sure the mod_info.so module is enabled in your configuration file (httpd.conf or in /etc/httpd/conf.modules.d/00-base.conf). Also include into your httpd.conf file the following.

SetHandler server-info

– Then execute the following command below to find which MPM module is currently being used.

curl localhost/server-info | grep 'MPM Name'

Change which MPM Module apache is using.

vim /etc/httpd/conf.modules.d/00-mpm.conf

Select which MPM module you wish to use.

#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
#LoadModule mpm_worker_module modules/mod_mpm_worker.so
#LoadModule mpm_event_module modules/mod_mpm_event.so
Share Button

CentOS: MySQL-VSFTP authentication – Plus use this FTP directory as your YUM FTP repository.

This how to shows how to use VSFTPD with virtual users that authenticate against a MySQL database. This solution does not scale well because as far as I can tell you cannot have the MySQL database on an isolated server while other several FTP servers use it to authenticate user logins. My understanding is that both VSFTP and MySQL need to reside on the same machine in order for this work.If I am wrong please let me know.  In addition this how-to shows you how to setup this same VSFTP server as a FTP YUM repository.

Continue reading

Share Button

Centos 6.5 – Create a YUM repository for your private network.

This tutorial shows how to create a YUM repository which you can use on your own private network. You may want this if you do not want your production servers to reach on the Internet thus keeping them protected. These servers will get their RPM packages by a server that holds the always up to date RPM packages. This is called the repository server and it will need access to the Internet in order to get the latest RPM packages.

 

Requirements:

1) Find an official CentOS site that supports RSYNC. List of Links here.

2) Have a spare server to use as your Repo server. This server will need to have Apache installed.

3) Enter an DNS entry for the local web site you will create that will host the RPM packages.

4) Setup yum on your servers to get the new packages from your Repo Server.

Continue reading

Share Button