A note to visitors

This web site is used for me to store information/skills I acquire in order to retain it.  These instructions are not meant for productions environments. They serve as a mere blueprint on how to do certain things.

Thank you.

Share Button

Python 2.7 – Generate GET requests from web server

This script generates simple GET requests from a web server. Threading is used in order to generate multiple GET requests at the same time. Note that this script cannot create significant stress on a Web Server.

import requests
import threading


def get_thrasher():

    req_get  = requests.get('http://192.168.56.101/')

threads = []

for counter_1 in range(10):
    thrd = threading.Thread(target=get_thrasher())
    thrd.start()
    threads.append(thrd)

for counter_2 in threads:
    counter_2.join()
Share Button

Python 2.7 – Find Location of IP with the geoip2 database.

The script collects the IP addresses from the apache log file. It then uses the geoip2 database in order to find the geographical location of the IP. More information for the geoip2 database can be found at http://dev.maxmind.com/geoip/geoip2/downloadable/

The module used to capture the IPs from the apache log file requires a CustomLog format. It needs to be specified in the apache config file and in the script. The string used is

("%h <<%P>> %t %Dus \"%r\" %>s %b  \"%{Referer}i\" \"%{User-Agent}i\" %l %u"
import geoip2.database
import apache_log_parser

#specify the log file we will capture the IP from.
dir     = "/var/log/apache2/"
file    = "access.log"
apache_logfile = dir+file


#Create a connection to the mmdb file with all the IP geo-location data.
reader = geoip2.database.Reader("GeoLite2-City.mmdb")

#In case we cannot open the file throw an error message
try:
        f_open = open(apache_logfile, "rb")
except Exception as e:
        print e

#As required by the apache_log_parser module
line_parser = apache_log_parser.make_parser("%h <<%P>> %t %Dus \"%r\" %>s %b  \"%{Referer}i\" \"%{User-Agent}i\" %l %u")

#This is the list we will put in IPs.
ip_list = []

for loop in f_open:  #We are going through the file specified
    log_line_data = line_parser(loop) #We are using the apache parser as specified above
    remote_ip  =  log_line_data['remote_host'] #The apache parser returns a dictionary. We just want the remote_host key.
    for ip  in remote_ip:
        ip_list.append(remote_ip)  #We are appending the IPs to the list we created above.


unique_ip_list = set(ip_list)  # We delete the duplicate IP entries from our list.
for ips in unique_ip_list:
        try: #In case the IP is not recognized by the geoip2 database
                locate_ip = reader.city(ips) # we are using the geoip2 module here with the IPs from our list
                print ips, locate_ip.country.name
        except Exception as e:
                print e

Share Button

Types of Replication general info

Asynchronous

 Data is replicated after it has been commited. Data loss can exist
if the master server crushes before the changes are commited by the remote server.

Synchronous

System has to ensure the data written by the transaction will be present
to at least two servers when the transaction commits. Confirmation
from the remote server is needed and this creates an overhead.


 

Single Master Replication

The master server replicates the data to the slave server.
The writes go to the Master server and these changes are disturbuted
to the slaves servers.

Multi Master

In this server writes are allowed to all the servers in the cluster.
This means alot of writes can go to many nodes at the same time.


 

Logical Replication

This replication distributes the changes at a logical level. It
does not concern it self with physical layout of the data structure
of the database it self.

Physical Replication

Is the type of replication where data are moved as is. The replication
of data is done at a binary level.

Share Button

PostgreSQL 9.4 – Database dumps and Restores

Create a Compressed dump file of a Remote Database and Restore it

  • Create a highly compressed data dump
pg_dump -h 10.0.0.19 -U seeker -C -c -v -F c -d servers -f data_dump.dump
  • Restore the data dump you created
pg_restore -C -v -U seeker -d servers -h 10.0.0.32  data_dump.dump

Create SQL data dumps and Restore them

  • Create a SQL Schema only dump
pg_dump -U  seeker  -d servers   -h 10.0.0.32 --schema-only -f schema_only.sql
  • Create a SQL Data only dump
pg_dump -U  seeker  -d servers   -h 10.0.0.32 --data-only -f data_only.sql
  • Restore SQL data dump using psql
\i "file_name".sql
     Important Notes about Backing up and Restoring

The database, schema and the role must be first created.

Back up and Restore specific tables

  • Backup a specific table
 pg_dump -Fc -v -U seeker -d servers -t a_table > /tmp/a_table.pgdump
  • View the tables of a data dump
pg_restore -l  data_dump_file.dump
  • Extract a table from a dump file
pg_restore -U seeker --data-only --table=a_table database.pgdump > a_table.pg
  • Upload the data of this table to a table in the database
 psql -U user_name -d database_name < name_of_your_dump.pg
Share Button

pg_dump incompatibility issue between servers

You may have to do pg_dump from server that has pg_dump version 9.4 to remote server that has pg_dump version 9.3. An error like this may occur.

 pg_dump: server version: 9.4.4; pg_dump version: 9.3.10 

You will have to install the 9.4 version of postgres and then adjust the soft link /usr/bin/pg_dump to point to /usr/pgsql-9.4/bin/pg_dump

sudo ln -s /usr/pgsql-9.4/bin/pg_dump  /usr/bin/pg_dump
Share Button

Postgresql 9.4: Create a template database.

In postgres template1 is the default source database name for CREATE DATABASE. The template0 database is used after we specify it. More information at the official documentation located here.

The template0 database we create a user database that "contains none
of the site-local additions in template1. This is particularly handy
when restoring a pg_dump dump: the dump script should be restored
in a virgin database to ensure that one recreates the correct
contents of the dumped database. Another common reason for
copying template0 instead of template1 is that new encoding and
locale settings can be specified when copying template0, whereas a
copy of template1 must use the same settings it does. This is because
template1 might contain encoding-specific or locale-specific data,
while template0 is known not to."

You can create your own custom Template to be used in the creation of databases. For example, assume you have the database with the name ‘servers’. You can create a template out of that database with the following steps.

Make sure to kill all connections from the database and disallow any new connections.

update pg_database 
set datallowconn = false 
where datname = 'servers'

Kill any other live connections

select pg_terminate_backend(pid)
from pg_stat_activity
where datname = 'servers'

Create your template

create database servers_2 template servers;

Make sure to activate the template flag in your template database in order to avoid having the template being altered.

update servers_2 
set datistemplate = TRUE 
where datname = 'servers';

Remember to allow connections to the database you used to create the template.

update pg_database 
set datallowconn = true 
where datname = 'servers'
Share Button

Postgresql 9.4: System tables ‘pg_stat_activity’ and ‘pg_database’, Cancel/Kill/Stop connections queries to a database.

You can view the status of connections to the postgresql database and the status of the queries. This info can be sorted by different fields.

      Column      | Modifiers
------------------+-----------
 datid            |
 datname          | The database name
 pid              | The pid number
 usesysid         |
 usename          | The username
 application_name |
 client_addr      | The IP address connected
 client_hostname  |
 client_port      | The port used
 backend_start    | The time at which the database was accessed by the client
 xact_start       | The time at which the transaction started to run. Should always have occurred before the query_start time 
 query_start      | The time the actual query started 
 state_change     |
 waiting          |
 state            |
 backend_xid      |
 backend_xmin     |
 query            | The actual query

View queries for specific user

select 
datname, pid, usename, client_addr, query_start, query
from pg_stat_activity where usename = 'seeker';
datname     | servers
pid         | 21694
usename     | seeker
client_addr | 10.0.0.19
query_start | 2015-10-16 15:59:08.234508-04
query       | INSERT INTO IP_DNS_SCAN(hostname, ipaddress, remote_ip, pingStatus, hostDNScheck) VALUES ('superman.sfentona.lol','10.0.
0.19','10.0.0.54','UP','wordpress.sfentona.lol.')

You can cancel or Kill queries by using pg_terminate_backend or pg_cancel_backend

Cancel or Kill the queries of a specific pid

select pg_terminate_backend(pid) 
from pg_stat_activity 
where pid = 24297;

Cancel or Kill the queries of a specific user

select pg_terminate_backend(pid)
from pg_stat_activity 
where usename = 'gmastrokostas';

Kill all queries besides your own

select pg_terminate_backend(pid)
from pg_stat_activity
where usename <> 'gmastrokostas';

Disallow all new connections to a specific Database

update pg_database 
set datallowconn = FALSE 
wnere datname = 'name_your_database';
Share Button

PostgreSQL 9.4: System Table – ‘pg_settings” – Queries that display Postgresql configuration settings

Obtaining Configuration settings.

Get Version of Database

select version();

Show Data directory

show data_directory;

Show config files

select name, setting 
from pg_settings
where category = 'File Locations';

 


Query your postgresql.conf file via pg_settings

name      - Name of Setting
context   - Does it require a hard or soft restart (postgres yes - user no)
unit      - MBs, KBs etc
setting   - Current value
boot_val  - Default setting
reset_val - New value once posgres service is restarted

Example

select 
name, context, unit, setting, boot_val, reset_val
from pg_settings
where name in 
('listen_addresses', 'max_connections', 'shared_buffers');
       name       |  context   | unit | setting | boot_val  | reset_val
------------------+------------+------+---------+-----------+-----------
 listen_addresses | postmaster |      | *       | localhost | *
 max_connections  | postmaster |      | 100     | 100       | 100
 shared_buffers   | postmaster | 8kB  | 16384   | 1024      | 16384
(3 rows)

All pg_settings fields you can query
All pg_settings fields

name | allow_system_table_mods
name | application_name
name | archive_command
name | archive_mode
name | archive_timeout
name | array_nulls
name | authentication_timeout
name | autovacuum
name | autovacuum_analyze_scale_factor
name | autovacuum_analyze_threshold
name | autovacuum_freeze_max_age
name | autovacuum_max_workers
name | autovacuum_multixact_freeze_max_age
name | autovacuum_naptime
name | autovacuum_vacuum_cost_delay
name | autovacuum_vacuum_cost_limit
name | autovacuum_vacuum_scale_factor
name | autovacuum_vacuum_threshold
name | autovacuum_work_mem
name | backslash_quote
name | bgwriter_delay
name | bgwriter_lru_maxpages
name | bgwriter_lru_multiplier
name | block_size
name | bonjour
name | bonjour_name
name | bytea_output
name | check_function_bodies
name | checkpoint_completion_target
name | checkpoint_segments
name | checkpoint_timeout
name | checkpoint_warning
name | client_encoding
name | client_min_messages
name | commit_delay
name | commit_siblings
name | config_file
name | constraint_exclusion
name | cpu_index_tuple_cost
name | cpu_operator_cost
name | cpu_tuple_cost
name | cursor_tuple_fraction
name | data_checksums
name | data_directory
name | DateStyle
name | db_user_namespace
name | deadlock_timeout
name | debug_assertions
name | debug_pretty_print
name | debug_print_parse
name | debug_print_plan
name | debug_print_rewritten
name | default_statistics_target
name | default_tablespace
name | default_text_search_config
name | default_transaction_deferrable
name | default_transaction_isolation
name | default_transaction_read_only
name | default_with_oids
name | dynamic_library_path
name | dynamic_shared_memory_type
name | effective_cache_size
name | effective_io_concurrency
name | enable_bitmapscan
name | enable_hashagg
name | enable_hashjoin
name | enable_indexonlyscan
name | enable_indexscan
name | enable_material
name | enable_mergejoin
name | enable_nestloop
name | enable_seqscan
name | enable_sort
name | enable_tidscan
name | escape_string_warning
name | event_source
name | exit_on_error
name | external_pid_file
name | extra_float_digits
name | from_collapse_limit
name | fsync
name | full_page_writes
name | geqo
name | geqo_effort
name | geqo_generations
name | geqo_pool_size
name | geqo_seed
name | geqo_selection_bias
name | geqo_threshold
name | gin_fuzzy_search_limit
name | hba_file
name | hot_standby
name | hot_standby_feedback
name | huge_pages
name | ident_file
name | ignore_checksum_failure
name | ignore_system_indexes
name | integer_datetimes
name | IntervalStyle
name | join_collapse_limit
name | krb_caseins_users
name | krb_server_keyfile
name | lc_collate
name | lc_ctype
name | lc_messages
name | lc_monetary
name | lc_numeric
name | lc_time
name | listen_addresses
name | lo_compat_privileges
name | local_preload_libraries
name | lock_timeout
name | log_autovacuum_min_duration
name | log_checkpoints
name | log_connections
name | log_destination
name | log_directory
name | log_disconnections
name | log_duration
name | log_error_verbosity
name | log_executor_stats
name | log_file_mode
name | log_filename
name | log_hostname
name | log_line_prefix
name | log_lock_waits
name | log_min_duration_statement
name | log_min_error_statement
name | log_min_messages
name | log_parser_stats
name | log_planner_stats
name | log_rotation_age
name | log_rotation_size
name | log_statement
name | log_statement_stats
name | log_temp_files
name | log_timezone
name | log_truncate_on_rotation
name | logging_collector
name | maintenance_work_mem
name | max_connections
name | max_files_per_process
name | max_function_args
name | max_identifier_length
name | max_index_keys
name | max_locks_per_transaction
name | max_pred_locks_per_transaction
name | max_prepared_transactions
name | max_replication_slots
name | max_stack_depth
name | max_standby_archive_delay
name | max_standby_streaming_delay
name | max_wal_senders
name | max_worker_processes
name | password_encryption
name | port
name | post_auth_delay
name | pre_auth_delay
name | quote_all_identifiers
name | random_page_cost
name | restart_after_crash
name | search_path
name | segment_size
name | seq_page_cost
name | server_encoding
name | server_version
name | server_version_num
name | session_preload_libraries
name | session_replication_role
name | shared_buffers
name | shared_preload_libraries
name | sql_inheritance
name | ssl
name | ssl_ca_file
name | ssl_cert_file
name | ssl_ciphers
name | ssl_crl_file
name | ssl_ecdh_curve
name | ssl_key_file
name | ssl_prefer_server_ciphers
name | ssl_renegotiation_limit
name | standard_conforming_strings
name | statement_timeout
name | stats_temp_directory
name | superuser_reserved_connections
name | synchronize_seqscans
name | synchronous_commit
name | synchronous_standby_names
name | syslog_facility
name | syslog_ident
name | tcp_keepalives_count
name | tcp_keepalives_idle
name | tcp_keepalives_interval
name | temp_buffers
name | temp_file_limit
name | temp_tablespaces
name | TimeZone
name | timezone_abbreviations
name | trace_notify
name | trace_recovery_messages
name | trace_sort
name | track_activities
name | track_activity_query_size
name | track_counts
name | track_functions
name | track_io_timing
name | transaction_deferrable
name | transaction_isolation
name | transaction_read_only
name | transform_null_equals
name | unix_socket_directories
name | unix_socket_group
name | unix_socket_permissions
name | update_process_title
name | vacuum_cost_delay
name | vacuum_cost_limit
name | vacuum_cost_page_dirty
name | vacuum_cost_page_hit
name | vacuum_cost_page_miss
name | vacuum_defer_cleanup_age
name | vacuum_freeze_min_age
name | vacuum_freeze_table_age
name | vacuum_multixact_freeze_min_age
name | vacuum_multixact_freeze_table_age
name | wal_block_size
name | wal_buffers
name | wal_keep_segments
name | wal_level
name | wal_log_hints
name | wal_receiver_status_interval
name | wal_receiver_timeout
name | wal_segment_size
name | wal_sender_timeout
name | wal_sync_method
name | wal_writer_delay
name | work_mem
name | xmlbinary
name | xmloption
name | zero_damaged_pages

Share Button

PostgreSQL 9.4 – System Tables – Get Database/Table number, size and other general info for your database

Get information about your existing databases

select * from pg_database;

Show uptime of Postgres server (Not uptime of OS)

select date_trunc ('second', current_timestamp - pg_postmaster_start_time());

View how much space a database consumes

select pg_size_pretty(pg_database_size(current_database()));

View the largest tables in your database

\x
select relname, relpages
from pg_class 
order by relpages desc;

view how large a table is

select pg_size_pretty(pg_relation_size('name_of_table'));

or

\dt+ name_of_table

view how many tables a user owns

select  count(*)  from pg_tables where tableowner = 'seeker';

View how many tables a schema of your database has.

select count(*) from information_schema.tables where table_schema not in ('pg_catalog', 'information_schema');

View how many databases a server has.

select count(*)  from pg_database;

View size of all databases

select sum(pg_database_size(datname)) from pg_database;

List all roles

select * from information_schema.enabled_roles;

List all schemas

select schema_name from information_schema.schemata;
Share Button

Centos 7 – Part 5 – HAproxy systemctl script

vi /etc/systemd/system/haproxy
[Service]
ExecStart=/usr/sbin/haproxy
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=haproxy
User=root
Group=root
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target
systemctl enable haproxy
Share Button