Centos 7 – Part 1 – NGINX – Apache – Load Balancing High Availability

These instructions show how to setup a web Load Balancer by using two NGINX servers as the Load Balancers and two Apache servers.

2 – Centos 7  servers running NGINX will be used as Load Balancers.

2 – Cent0s 7 serves running Apache will be used to serve web pages via virtual hosts.

The NGINX servers will:
– Determine the appropriate destination service based on the method chosen; in this case it will be Round Robin, which is the default option.
– Will use KeepAlive in order to create a virtual IP address. The IP address will based on the already NICs of the Load Balancers.

The Apache servers will:
– Act as your normal every day web servers with virtual hosts.
– Will log the IP of the actual client.

nginx-HA

Setting up the Apache web Servers.

  • Create the directories where the content of you virtual hosts will be placed.
mkdir /etc/httpd/vhosts.d
mkdir -p /sites/wordpress/
chown -R apache:apache /sites/wordpress/
chmod 755 /sites/wordpress/

 

  • Instruct Apache to look into the directory you created for your virtual hosts.
vim /etc/httpd/conf.d/vhosts.conf
IncludeOptional vhosts.d/*.conf

 

  • Create the config file for the corresponding web site.
vim /etc/httpd/vhosts.d/wordpress.sfentona.lol.conf
ServerAdmin webmaster@dummy-host.example.com
DocumentRoot /sites/wordpress/
ServerName wordpress.sfentona.lol
ServerAlias www.wordpress.sfentona.lol

Directory "/sites/wordpress";
DirectoryIndex index.html index.php
Options FollowSymLinks
AllowOverride All
Require all granted

 

  • Set up log x-fowarded-for in order to get the the IP of the actual client who visited the site and not the IP of the load balancer in your logs.
vi /etc/httpd/conf/httpd.conf
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded

 

Configure NGINX as a load balancer (view lines 34-38)

vi /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    upstream wordpress {
    server 10.0.0.42:80;
    server 10.0.0.43:80;
}


        server {
        listen       80;
        server_name  www.wordpress.sfentona.lol;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
                proxy_pass http://wordpress;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

Allow NGINX to bind to a non-local shared ip

vi /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind=1
sysctl -p

Set up your firewall in order for Multicast and VRRP to work correctly.

iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
iptables -I INPUT -p vrrp -j ACCEPT

 

Configure Keep Alive.

vi/etc/keepalived/keepalived.conf

This is for the Master Load Banacer LB1

 notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb1@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.44
    }
}

This is for the Master Load Banacer LB2. Priority on the slave is a lower number.

global_defs {
   notification_email {
     sysadmin@mydomain.com
     support@mydomain.com
   }
   notification_email_from lb2@mydomain.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_instance VI_1 {
    state MASTER
    interface ens192
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.44
    }
}
Share Button

Centos6.5-Win2012R2 – Setup Windows as your Master DNS and Bind as your Slave DNS.

This tutorial show how to setup Windows 2012-R2 as a Master DNS and how to set up Centos 6 as a slave DNS.

PRIMARY DNS NAME AND IP: AD1.SFENTONA.LOL  / 10.0.0.6
SLAVE   DNS NAME AND IP: DNS1.SFENTONA.LOL /10.0.0.10

Centos DNS CONFIG STEPS

———————————————————————————————————-
The following config files have been used in order to get DNS services up and running in Centos 6.

vi /etc/resolv.conf
nameserver 127.0.0.1
search sfentona.lol
vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=dns1.sfentona.lol
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:0C:29:CA:90:ED
TYPE=Ethernet
UUID=49076518-17fb-4416-be14-de64aa36843a
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPADDR=10.0.0.10
NETMASK=255.255.255.192
GATEWAY=10.0.0.1
DNS1=127.0.0.1
DOMAIN="sfentona.lol"
vi /etc/named.conf

Under Options you will have to specify the IP address of your Centos DNS server and from which network you will accept queries.

listen-on port 53 { 127.0.0.1; 10.0.0.10; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query { localhost; 10.0.0.10/26; };
//allow-transfer { 10.0.0.0/26; };
recursion yes;

Under Zones you will have to create your forward lookup and reverse lookup zones. Between the sections LOGGING and ZONE include the following lines. We are essentially telling to our Centos DNS service that it is of type slave and the IP of the master DNS. We are also telling where the location of the zone files will be located

/var/named/slaves

sfentona.lol.zone

zone "sfentona.lol" IN {
type slave;
masters { 10.0.0.6; };
allow-query { any; };
file "slaves/sfentona.lol.zone";
};

sfentona.lol.rr.zone

zone "0.0.10.in-addr.arpa" IN {
type slave;
masters { 10.0.0.6; };
allow-query { any; };
file "slaves/sfentona.lol.rr.zone";
};

 

 

Windows DNS CONFIG STEPS

———————————————————————————————————-

  • On your maind DNS properties settings tree check “Enable Bind Secondaries”
  • You will have enter as Name Server your Linux Server for both your Forward and Reverse Lookup zones.
  • On your DNS Zone (in this case sfentona.lol) under properties settings enable “Zone Transfer”. Specify your slave DNS or you can opt to update all available DNS servers. Make to do this for both your Forward and Reverse lookup zones for your Domain.
Share Button

Python – Install Python 2.7 alongside 2.6 in Centos 6

These instructions were taken from toomuchdata.com.  The instructions on the site show how to install Python 3 as well.  For my own purposes however, I only copied and pasted the instructions for Python 2.7. The instructions were tested on a Centos 6 machine and they worked with no issues right off the gate.

 

yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
wget http://python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
tar xf Python-2.7.6.tar.xz
cd Python-2.7.6
./configure --prefix=/usr/local --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && make altinstall
cd -
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
python2.7 ez_setup.py
easy_install-2.7 pip

In case you receive an error regarding ZLIB after you try to execute the east_install command, execute the following command.

cp /usr/lib64/python2.6/lib-dynload/zlibmodule.so /usr/local/lib/python2.7/lib-dynload/zlibmodule.so
Share Button

Python – RegEx examples

Searching for text in multiple files within a directory OR one file in a directory

This script searches the yum directory for all the repo files and searches for the URLs each repo goes to.

Search for text in multiple files

src_dict = ("/etc/yum.repos.d/") #Specify base directory
pattern = re.compile ('http\S+') #CPatter to search for

for yum_files in os.listdir(src_dict): # obtain list of files in directory
    files = os.path.join(src_dict, yum_files) #join the full path with the names of the files.
    strng = open(files) #We need to open the files
    for lines in strng.readlines(): #We then need to read the files
        if re.search(pattern, lines): #If we find the pattern we are looking for
            print re.split(r'=', lines)[1] #We split using as a delimeter the = sign.
            #INSERT WHATEVER CODE YOU WANT

Search for text in one file in a directory

file = 'file.csv'
drc = '/home/gmastrokostas/PycharmProjects/learning'
f = open( os.path.join(drc, file) )
for lines in f.readlines():
    searpatt = re.search(pattern, lines)
    if re.search(pattern,lines):
        print (searpatt.group())

Change one specific extension type of files in a directory that contains multiple types of extensions

import os
import re

src_drct='/home/gmastrokostas/tmp'

for files in os.listdir(src_drct):

    if files.endswith('.txt'):
        oldF = os.path.join(src_drct, files)
        #midF = re.split(r'\.', files)#This works too.
        midF = os.path.splitext(files) #It creates a list and splits the name of the file from the extension.
        newF = oldF.replace('.txt', '.py')#Simple replacement.
        out = os.rename(oldF, newF)

Search for specific types of files and rename the files or part of the files

The script finds files with a specific file extension and renames parts of the files it found.

import os
import re
 
src_drct='/home/gmastrokostas/tmp'
 
for files in os.listdir(src_drct):
 
    if files.endswith('.txt'): #Select all files with the .txt ext
        oldF = os.path.join(src_drct, files) #Join full path with files found
        midF = re.split(r'\_', files) #split files that contain underscore
        newF = oldF.replace('_01012015', '_01012014') #create the replacement var
        out = os.rename(oldF, newF) #start renaming.

Replace a string in multiple files with in a given directory

The script below will search for a string in files with in a directory and subdirectories. Before it starts replacing any text it will create a backup of the original file(s). Note the script does not into account possible duplicate files when copying the files in /tmp.

import re
import os
import shutil

drc = '/root/tmp'
backup = '/tmp'
pattern = re.compile('PYTHON')
oldstr = 'PYTHON'
newstr = 'Python'

for dirpath, dirname, filename in os.walk(drc):#Getting a list of the full paths of files
    for fname in filename:
        path = os.path.join(dirpath, fname) #Joining dirpath and filenames
        strg = open(path).read() #Opening the files for reading only
        if re.search(pattern, strg):#If we find the pattern ....
            #print path, strg
            shutil.copy2(path, backup) #we will create a backup of it
            strg = strg.replace(oldstr, newstr) #We will create the replacement condistion
            f = open(path, 'w') #We open the files with the WRITE option
            f.write(strg) # We are writing the the changes to the files
            f.close() #Closing the files

Replace a string in One file with in a given directory

 
oldstr = 'Time'
newstr = 'TIME'
file_path = '/home/gmastrokostas/PycharmProjects/learning/file.csv'
with open(file_path, 'r') as f:
   fread = f.read()
   strg  = fread.replace(oldstr, newstr)
   fopen = open(file_path, 'w')
   fopen.write(strg)
f.close()

Write text from file temp1.txt to named.txt (use the ‘a’ option if you want to append instead)

 
#!/usr/bin/python
import os
import re
with open("named.txt", "wt") as fout:
    with open("templ.txt", "rt") as fin:
        for line in fin:
            fout.write(line)

Search for a string in a file and display the Result

 
import re
text="XXX"
file ='templ.txt'
for line in open(file):
    if text in line:
        print line

Search for a string by using compile in a file and display the Result

 
import re
text = "XXX"
pattern = re.compile(text)
file = 'templ.txt'
of = open(file)
for line in of.readlines():
    if re.search(pattern,  line):
        print line

Append the contents of a file to another file

import re
import os
fo = open("templ.txt","r")
fi = open("named.txt","a")
for line in fo:
    fi.write(line)
    #print line
fi.close()
fo.close()

Isolating text field from a variable

text = '11:47:55.045 -T- MFS_TEST10 Received FIX Message 3  8=FIX.4.2|9=146|35=D|34=3|49=MFS_TEST10|52=20120726-15:47:55|56=CCG|115=YYYC|11=AAA 0981/07262012|54=1|38=100000|55=ACL|40=1|47=A|60=20120726-15:47:55|21=1|207=N|10=044|'
print re.split(r'\s', text)[2]
Output
MFS_TEST10

Isolating text fields from a file.

 import re
#!/usr/bin/python
import re
import os
import fnmatch
fo = open("templ.txt", "r")
for line in fo:
        print line
        print re.split(r'\s', line, re.I|re.M)[1]
Provided the text with in the file was "Road 123 Str NY USA 11214"
Output 123

Searching for specific type files.

 

filenames = ['httpd.conf', 'samba.conf', 'header.h', 'socket.net']

[loop for loop in filenames if loop.endswith(('.h'))]
for loop in filenames:
    if loop.endswith('.h') or loop.startswith('httpd'):
        print loop
Output
httpd.conf
header.h

Searching through a List

 
#Great use with os.dirlist
addresses = [
    '5412 N CLARK ST',
    '1060 W ADDISON ST',
    '1039 W GRANVILLE AVE',
    '2122 N CLARK ST',
    '4802 N BROADWAY',
]

for loop in addresses:
    if fnmatch.fnmatchcase(loop, '* ST'):
        print loop
Output
5412 N CLARK ST
1060 W ADDISON ST
 2122 N CLARK ST

Print specific fields from a string

 
text = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
pattern = re.compile('Today\s\w+\s\d+\W\d+\W\d+', re.IGNORECASE)
searpatt = re.search(pattern, text)
print (searpatt.group())

Output

Today is 11/27/2012

Replace specific fields of text

print text
#!/usr/bin/python
import re

text = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
newt = "Yesterday"
pattern = re.compile('Today\s\w+\s\d+\W\d+\W\d+', re.IGNORECASE)
searpatt = re.search(pattern, text)
print re.sub(pattern, newt, text)

Output

Yesterday. PyCon starts 3/13/2013.

Print a specific element of the split you created

#!/usr/bin/python
import re

text = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
newt = "Yesterday"
pattern = re.compile('Today\s\w+\s\d+\W\d+\W\d+', re.IGNORECASE)
searpatt = re.search(pattern, text)
print text
print re.split(pattern, text )[1]

PyCon starts 3/13/2013.

Replace strings in a file

#!/usr/bin/python
import os
import re
text_ip = "#listen_addresses = 'localhost'"
        subtext_ip = "listen_addresses = '10.0.0.27'"
        text_port = "#port = 5432"
        subtext_port = "port = 5432"

        s = open("/home/postgres/postgres94/postgresql.conf").read()
        s = s.replace(text_ip, subtext_ip)
        s = s.replace(text_port, subtext_port)
        f = open("/home/postgres/postgres94/postgresql.conf", 'w')
        f.write(s)
        f.close()
Share Button

Python/Postgresql – Query Database and present data in a plot

How to query a database using a Python script and present the data on a plot.

import pandas.io.data
import matplotlib.pyplot as plt
import pandas as pd
from pandas import DataFrame
import datetime
import pandas.io.data
import psycopg2
import re
import os


conn = psycopg2.connect("host='172.31.98.161' dbname='servers' user='seeker'")
cur = conn.cursor() #Create the cursor
cur.execute("select total_ram_raw, used_ram_raw, time_captured  from server_perf  where hostname='localhost.localdomain'")
rows = cur.fetchall()
conn.close()

df = pd.DataFrame(rows, columns=['TotalRam','UsedRAM', 'Time'])

df.to_csv("file.csv")
df.TotalRam = df.TotalRam.astype(float)
df.UsedRAM  = df.UsedRAM.astype(float)

df = pd.read_csv('file.csv', index_col='Time')
df[['TotalRam', 'UsedRAM']].plot()
plt.show()
Share Button

Python /PostgresqSQL 9.4 – Server Performance Data Capture V.2

In this second version, the scripts which capture data for the servers are using classes. In addition the the script that captures dynamic data from the remote servers captures additional data. In specific it captures RAM, HD usage in raw numbers and not only “humanized” format. The humanized fields were not taken out. As a result the appropriate tables had to be modified, which means the schema has changed as well. Also, the database and the scripts are now being installed via puppet. The Puppet manifests are far from polished. They need more work but they do work.

These scripts gather static and dynamic information from servers and insert that data into a PostgresSQL database. The static information is information that unless a major upgrade takes place it never changes. They Dynamic data is performance data of the servers. The purpose of the static data is to be able to query for dynamic information which being inserted to the database every X amount of minutes via cron.

These scripts work only on Linux/Unix based machines.

The PUPPET modules used to install the Database and the scripts are located here

The static information for the remote servers are as follows :

hostname
iface
ipaddress
OS
OSRel
OSKern
total_M
brand
Hz
cores
arch

The dynamic information for the remote servers are as follows :

hostname
iface
ipaddress
total_ram_hum
used_ram_hum
total_ram_raw
used_ram_raw
used_ram_perc
total_HD_hum
used_HD_hum
total_HD_raw
used_HD_raw
used_HD_perc
cpu_use_perc
swap_used_hum
swap_total_hum
swap_perc
swap_used_raw
swap_total_raw

The static.py script will need to be run only one time on the remote servers or when a major upgrade occurs that might change configuration regarding RAM, Partitions, IP, Operating System (even an upgrade), CPU, NIC replacement.

The dynamic.py script will be run on the remote servers via cron. It is the script that captures information which is being constantly changed, like memory, storage, swap usage. All this data is sent for insertion to the remote database. The script executes it self via cron and then sent to be inserted into a PostgreSQL database.

In both scripts data is entered into a dictionary and then a connection to the database is created in order to insert the data.

The Static.py script

#!/usr/bin/python
import psutil
import os
import math
import platform
import subprocess
import socket
import psycopg2
import netifaces as ni
import humanize
from cpuinfo import cpuinfo


class Static():
    def __init__(self):
        #NIC INFO
        self.hostname   = socket.gethostname()
        self.iface      = ni.interfaces()[1]
        self.ipaddress  = ni.ifaddresses(self.iface)[2][0]['addr']
        #---OS INFO
        #For Linux (RH-Debian) Operating Systems
        self.distro  = platform.dist()[0]
        self.release = platform.dist()[1]
        self.kernel  = platform.release()
        #For Windows Operating Systems
        self.osinfo_2_os    = platform.uname()[0]
        self.osinfo_2_ver   = platform.uname()[2]
        self.osinfo_2_rel   = platform.uname()[3]
        #----RAM INFO
        raw_totalM = psutil.virtual_memory().total
        self.total_M    = humanize.naturalsize(raw_totalM)
        #----CPU INFO
        self.info       = cpuinfo.get_cpu_info()
        self.brand      = self.info['brand']
        self.Hz         = self.info['hz_advertised']
        self.cores        = self.info['count']
        self.arch       = self.info['bits']

    def get_OS_make(self):
       if platform.system()  =="Linux":
           return self.distro, self.release, self.kernel
       elif platform.system()     =="Windows":
           return self.osinfo_2_os, self.osinfo_2_ver, self.osinfo_2_rel

info = Static()



hostname  = info.hostname
iface     = info.iface
ipaddress = info.ipaddress
OS        = info.get_OS_make()[0]
OSRel     = info.get_OS_make()[1]
OSKern    = info.get_OS_make()[2]
total_M   = info.total_M
brand     = info.brand
Hz        = info.Hz
cores     = info.cores
arch      = info.arch




#Create the Database PostgreSQL 9.4 connection.
conn = psycopg2.connect("host='172.31.98.161' dbname='servers' user='seeker'")
cur = conn.cursor() #Create the cursor
#Create a Dictionary to pass the value of each function.
server_info = {'hostname': hostname, 'iface':iface, 'ipaddress': ipaddress, 'OS': OS, 'OSRel': OSRel, 'OSKern': OSKern, 'total_M': total_M, 'brand': brand, 'Hz':Hz, 'cores': cores, 'arch': arch}
cur.execute("INSERT INTO servers(hostname, iface, ipaddress, OS, OSRel, OSKern, total_M, brand, Hz, cores, arch) VALUES ('%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s')" % (hostname, iface, ipaddress, OS, OSRel, OSKern, total_M, brand, Hz, cores, arch))
#If this is not present the changes will not get commited.
conn.commit()

The Dynamic.py srcipt

#THIS FILE IS MANAGED BY PUPPET
#!/usr/bin/python
import psutil
import os
import math
import platform
import subprocess
import socket
import psycopg2
import netifaces as ni
import humanize
from cpuinfo import cpuinfo

class Dynamic():
    def __init__(self):
        #NIC INFO
        self.hostname   = socket.gethostname()
        self.iface      = ni.interfaces()[1]
        self.ipaddress  = ni.ifaddresses(self.iface)[2][0]['addr']
        #RAM USAGE INFO-------------------------------------------------------------------
        self.total_ram_hum       = humanize.naturalsize(psutil.virtual_memory().total)
        self.used_ram_hum        = humanize.naturalsize(psutil.virtual_memory().used)
        #---------Raw info
        self.total_ram_raw       = (psutil.virtual_memory().total)
        self.used_ram_raw        = (psutil.virtual_memory().used)
        self.used_ram_perc       = psutil.virtual_memory().percent
        #HD USAGE INFO-------------------------------------------------------------------
        self.total_HD_hum        = humanize.naturalsize(psutil.disk_usage('/').total)
        self.used_HD_hum         = humanize.naturalsize(psutil.disk_usage('/').used)
        #---------Raw info
        self.total_HD_raw        =(psutil.disk_usage('/').total)
        self.used_HD_raw         =(psutil.disk_usage('/').used)
        self.used_HD_perc        = psutil.disk_usage('/').percent
        #CPU USAGE INFO-------------------------------------------------------------------
        self.cpu_use_perc        = psutil.cpu_percent()
        #SWAP USAGE INFO-------------------------------------------------------------------
        self.swap_used_hum           = humanize.naturalsize(psutil.swap_memory().used)
        self.swap_total_hum          = humanize.naturalsize(psutil.swap_memory().total)
        self.swap_perc               = psutil.swap_memory()[3]
        #---------Raw info
        self.swap_used_raw           = (psutil.swap_memory().used)
        self.swap_total_raw          = (psutil.swap_memory().total)
    def export_to_csv(self):
        print self.hostname
info = Dynamic()

hostname            = info.hostname
iface               = info.iface
ipaddress           = info.ipaddress
total_ram_hum       = info.total_ram_hum
used_ram_hum        = info.used_ram_hum
total_ram_raw       = info.total_ram_raw
used_ram_raw        = info.used_ram_raw
used_ram_perc       = info.used_ram_perc
total_HD_hum        = info.total_HD_hum
used_HD_hum         = info.used_HD_hum
total_HD_raw        = info.total_HD_raw
used_HD_raw         = info.used_HD_raw
used_HD_perc        = info.used_HD_perc
cpu_use_perc        = info.cpu_use_perc
swap_used_hum       = info.swap_used_hum
swap_total_hum      = info.swap_total_hum
swap_perc           = info.swap_perc
swap_used_raw       = info.swap_used_raw
swap_total_raw      = info.swap_total_raw


conn = psycopg2.connect("host='172.31.98.161' dbname='servers' user='seeker'")
cur = conn.cursor() #Create the cursor
#Create a Dictionary to pass the value of each function.
server_info = {'hostname':hostname, 'iface': iface,'ipaddress': ipaddress, 'total_ram_hum': total_ram_hum, 'used_ram_hum': used_ram_hum, 'total_ram_raw': total_ram_raw, 'used_ram_raw':used_ram_raw,'used_ram_perc': used_ram_perc, 'HD_hum': total_HD_hum, 'used_HD_hum': used_HD_hum, 'total_HD_raw': total_HD_raw, 'used_HD_raw':used_HD_raw, 'used_HD_perc': used_HD_perc, 'cpu_use_perc': cpu_use_perc,'swap_used_hum':swap_used_hum, 'swap_total_hum': swap_total_hum, 'swap_perc': swap_perc, 'swap_used_raw': swap_used_raw, 'swap_total_raw': swap_total_raw}
cur.execute("INSERT INTO SERVER_PERF(hostname, iface, ipaddress, total_ram_hum, used_ram_hum, total_ram_raw, used_ram_raw, used_ram_perc, total_HD_hum, used_HD_hum, total_HD_raw,used_HD_raw,used_HD_perc, cpu_use_perc,swap_used_hum, swap_total_hum, swap_perc, swap_used_raw, swap_total_raw) VALUES ('%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s')" % (hostname, iface, ipaddress, total_ram_hum, used_ram_hum, total_ram_raw, used_ram_raw,used_ram_perc, total_HD_hum, used_HD_hum, total_HD_raw, used_HD_raw, used_HD_perc, cpu_use_perc, swap_used_hum, swap_total_hum, swap_perc, swap_used_raw, swap_total_raw))
#If this is not present the changes will not get commited.
conn.commit()

The new Database schema is as follows:

pg_dump -U seeker -d servers -s -h 172.31.98.161 > servers_db_schema

--
CREATE DATABASE SERVERS;
CREATE ROLE seeker WITH PASSWORD 'Password!';
ALTER DATABASE SERVERS OWNER TO seeker;
ALTER ROLE seeker WITH LOGIN;
GRANT ALL PRIVILEGES ON DATABASE SERVERS to seeker;
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;

--
-- Name: plpgsql; Type: EXTENSION; Schema: -; Owner:
--

CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;

--
-- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner:
--

COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';

SET search_path = public, pg_catalog;

SET default_tablespace = '';

SET default_with_oids = false;

--
-- Name: server_perf; Type: TABLE; Schema: public; Owner: seeker; Tablespace:
--

CREATE TABLE server_perf (
hostname text NOT NULL,
iface text,
ipaddress inet NOT NULL,
total_ram_hum text,
used_ram_hum text,
total_ram_raw numeric(30,2),
used_ram_raw numeric(30,2),
used_ram_perc text,
total_hd_hum text,
used_hd_hum text,
total_hd_raw numeric(30,2),
used_hd_raw numeric(30,2),
used_hd_perc text,
cpu_use_perc text,
swap_used_hum text,
swap_total_hum text,
swap_perc text,
swap_used_raw numeric(30,2),
swap_total_raw numeric(30,2),
time_captured timestamp without time zone DEFAULT now()
);

ALTER TABLE server_perf OWNER TO seeker;

--
-- Name: servers; Type: TABLE; Schema: public; Owner: seeker; Tablespace:
--

CREATE TABLE servers (
hostname text NOT NULL,
iface text,
ipaddress inet NOT NULL,
os text,
osrel text,
oskern text,
total_m text,
brand text,
hz text,
cores numeric(4,1),
arch text
);

ALTER TABLE servers OWNER TO seeker;

--
-- Name: pk_hostname; Type: CONSTRAINT; Schema: public; Owner: seeker; Tablespace:
--

ALTER TABLE ONLY servers
ADD CONSTRAINT pk_hostname PRIMARY KEY (hostname);

--
-- Name: server_perf_hostname_fkey; Type: FK CONSTRAINT; Schema: public; Owner: seeker
--

ALTER TABLE ONLY server_perf
ADD CONSTRAINT server_perf_hostname_fkey FOREIGN KEY (hostname) REFERENCES servers(hostname);

--
-- Name: public; Type: ACL; Schema: -; Owner: postgres
--

REVOKE ALL ON SCHEMA public FROM PUBLIC;
REVOKE ALL ON SCHEMA public FROM postgres;
GRANT ALL ON SCHEMA public TO postgres;
GRANT ALL ON SCHEMA public TO PUBLIC;

--
-- PostgreSQL database dump complete
--
Share Button

Python – Compare modification dates between files and find time difference

The script searches in a specific location for files and it gets in POSIX the time of modification of all files. It also creates a temp file (and write text in it) within the same directory (which gets deleted once the script exits). This temp file is used to get today’s modification date. Then all POSIX dates are converted into human readable format and a comparison is done between the temp file and the files we are examining. If the files we are examining are three months or older, then …. you can enter what ever custom action you want.

#!/usr/bin/python
#The script searches in a specific location for files and it gets in POSIX the time of modification of all files.
#It also creates a temp file (and write text in it)  within the same directory (which gets deleted once the script exits)
#This temp file is used to get today's modification date. Then all POSIX dates are converted into human readable format
#and a comparison is done between the temp file and the files we are examining. If the files we are examining are
#three months or older, then .... you can enter what ever custom action you want.

import os.path
import tempfile
import datetime
import dateutil.relativedelta
import dateutil

#Path of where we are looking for the files
drc = '/root/Music'

#We create a temp file so we can get today's date. File is deleted after script is done executing
tempF = tempfile.NamedTemporaryFile(dir=drc)#, delete=False)
tempF.write('something')#We put in some text in so we are sure to get a time modified.
fileT =  os.stat(tempF.name)[8] #We are getting the mtime aka time of modification
fileT_human = datetime.datetime.fromtimestamp(fileT)#We are converting it to human readeable

for dirpath, dirname, filename in os.walk(drc):  #we are going huntinf for...
        for fname in filename:                   #...files
                path = os.path.join(dirpath, fname) #full path of files
                mtime = os.stat(path)[8] #We are getting time of modification of all the files
                mtime_human  = datetime.datetime.fromtimestamp(mtime) #We humanize the date

                #We are comparing each file with the fileT_human in order to get the difference in date when it comes to modification time
                diff = dateutil.relativedelta.relativedelta (fileT_human, mtime_human)

                #Just a sample output. You can use years, months, days to do the comparison between files.
                #print "%d years, %d months, %d days, %d hours, %d minutes and %d seconds" % (diff.years, diff.months, diff.days, diff.hours, diff.minutes, diff.seconds)
                months = diff.months
                if  months > 3:
                        print "Deleting file ", path
                        #Put whatever action you want here.
                        os.remove(path) #it will delete files only
                else:
                        exit;
Share Button

Centos 7 PostgresSQL 9.4 – Install database and use a non default PGDATA location, auto start on Boot (systemd)

These instructions show how to install postgresSQL 9.4 on Centos7. It also shows how to use a non default location from where to run your database engine. This is done by altering the PGDATA variable which dictates where the database engine is running from. It should be noted that when installing Postgres, a user Postgres is also created. This user account has environmental variables that deviate from the other accounts. You can cat the /etc/passwd file and see the default location for the home directory. As a result in order to change the environmental variable PGDATA you will have to alter the .bash_profile in that location in order for the change to take effect.

 

Step 1) Install the yum Repo for  PostgresSQL 9.4

yum -y localinstall http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-1.noarch.rpm

Step 2) Download the RPMS and install the database engine

yum -y install   postgresql94-contrib  postgresql94-server

Step 3) Create the directories from where the database will be running from. In this case it will be from /home/postgres

mkdir /home/postgres; chown -R postgres:postgres /home/postgres

Step 4) Initialize the database. 

cd  /usr/pgsql-9.4/bin/; ./initdb -D /home/postgres

Step 5)  Alter the default location from where your database will be running from.

vim /var/lib/pgsql/.bash_profile
#Alter the PGDATA as follows. See sample below
#PGDATA=/var/lib/pgsql/9.4/data
PGDATA=/home/postgres
export PGDATA

6) Configure systemd to start the database on boot-up

 vi /usr/lib/systemd/system/postgresql-9.4.service
# Note: avoid inserting whitespace in these Environment= lines, or you may
# break postgresql-setup.
# Location of database directory
#Environment=PGDATA=/var/lib/pgsql/9.4/data/
Environment=PGDATA=/home/postgres/
systemctl daemon-reload
systemctl enable  postgresql-9.4.service
systemctl start  postgresql-9.4.service
Share Button

Puppet Enterprise – Manifest for Centos7 to create a yum repo for Centos 7.

This is my first puppet manifest. It can pick up between Debian and RedHat/Centos operating systems. It installs apache, creates the config files and directories for the virtual hosts to exist. It also creates the directories for where the RPMs will be downloaded in order for the repo to be created. In addition it also creates a cron job to sync the RPMs. The command on the cronjob section can be changed. The one currently listed is not optimal at all.

Note:

class yumrepos {
        include ::yumrepos::params
        include ::yumrepos::install
        include ::yumrepos::file
}

class yumrepos::install {
        package { 'apache2':
                ensure  => installed,
                name    => $yumrepos::params::packagename,
                }
}

class yumrepos::file {
        file {$yumrepos::params::centorepodirs:
                ensure  => directory,
                owner   => 'apache',
                group   => 'apache',
                require => Package['apache2'],

                }

        file {$yumrepos::params::vhostdirconf:
                ensure  => directory,
                require => Package['apache2'],
                }

        file {$yumrepos::params::httpd_conf:
                ensure  => file,
                require => Package[$yumrepos::params::packagename],
                content => template('yumrepos/httpd.conf.erb'),
                }

        file {$yumrepos::params::repoVhost:
                ensure  => file,
                require => Package[$yumrepos::params::packagename],
                content => template('yumrepos/rpmsrepo.sfentona.lol.conf.erb'),
                }
        file {$yumrepos::params::vhostconfig:
                ensure  => file,
                require => File[$yumrepos::params::vhostdirconf],
                content => template('yumrepos/vhosts.conf.erb'),
                }
        file {$yumrepos::params::disableWelcome:
                ensure  => file,
                require =>  Package['apache2'],
                content => template('yumrepos/welcome.conf.erb'),
                }
        #FOR TESTING 
        #file {$yumrepos::params::repoindex:
        #       ensure  => file,
        #       require => File[$yumrepos::params::centorepodirs],
        #       content => template('yumrepos/index.html.erb'),
        #       }

        cron { 'update_cron':
                ensure          => 'present',
                command         => 'rsync -vaH --exclude-from=${EXCLUDES} --numeric-ids --delete --delete-after --delay-updates rsync://eu-msync.centos.org/CentOS/ /repo/Centos06/centos/ & 2>> /var/log/centosrsync',
                user            => 'root',
                monthday        => [1],
                hour            => [23],
                require         => File[$yumrepos::params::centorepodirs],
                }

}

class yumrepos::params{

        $packagename = $::osfamily ? {
                /(?i:debian)/ => 'apache2',
                /(?i:redhat)/ => 'httpd',
                        }
#Directories
$centorepodirs  = ["/repo/", "/repo/Centos06", "/repo/Centos06/centos/"]
$vhostdirconf   = "/etc/httpd/vhosts.d"

#Apache config files
$repoVhost      = "/etc/httpd/vhosts.d/rpmsrepo.sfentona.lol.conf"
$httpd_conf     = "/etc/httpd/conf/httpd.conf"
$vhostconfig    = "/etc/httpd/conf.d/vhosts.conf"
$disableWelcome = "/etc/httpd/conf.d/welcome.conf"

#HTML file FOR TESTING
#$repoindex      = "/repo/Centos06/centos/index.html"

}
Share Button

Apache2.4.6 (CentOS 7) – Create Virtual Hosts

The instructions below show how to create virtual hosts in Centos 7. In this case the virtual hosts will be running from a custom location in the system and not from directories created in the /home directory.

Edit/Create your Virtual Host file
Create/edit the following file by entering the following line. This will tell httpd to read the configuration for the actual virtual hosts, provided of course that the main httpd.conf file is instructed (usually by default) to read the files in ‘conf.d/*.conf’ directory.

vim /etc/httpd/conf.d/vhosts.conf
IncludeOptional vhosts.d/*.conf

Create the directory where the config files for the Virtual Hosts will reside.

mkdir /etc/httpd/vhosts.d

Create the config file for your first Virtual Host.

vim /etc/httpd/vhosts.d/rpmsrepo.sfentona.lol.conf

Include the following lines in the config file.

<VirtualHost *:80>
 ServerAdmin webmaster@dummy-host.example.com
 DocumentRoot /repo/Centos06/centos/
 ServerName rpmsrepo.sfentona.lol
 ServerAlias www.rpmsrepo.sfentona.lol

<Directory "/repo/Centos06/centos/">
 DirectoryIndex index.html index.php
 Options FollowSymLinks
 AllowOverride All
 Require all granted
 </Directory>
</VirtualHost>

Create the directories where your virtual hosts will be running from
mkdir /repo/Centos06/centos/
chown -R apache:apache /repo/Centos06/centos/
chmod 755  /repo/Centos06/centos/

Check your httpd configuration and start httpd

apachectl configtest
systemctl restart httpd
systemctl status httpd
Share Button