18 August 2020

Rundeck - Runbook Automation tool

Rundeck is an opensource tool that helps you automate & schedule your operational jobs. It provides number of features like scheduling jobs, automating execution of ansible playbooks, notifying about the status of your job in form of sending emails is my favourite.

Configuring rundeck is straight forward, you can install rundeck as a service in your linux host or use it as a docker image as well.

quick setup
$ vi /etc/yum.repos.d/rundeck.repo [rundeck] name=rundeck baseurl=https://packages.rundeck.com/pagerduty/rundeck/rpm_any/rpm_any/$basearch repo_gpgcheck=1 gpgcheck=1 enabled=1 gpgkey=https://packages.rundeck.com/pagerduty/rundeck/gpgkey,https://docs.rundeck.com/keys/BUILD-GPG-KEY-20230105.key sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt metadata_expire=300
$ yum install rundeck java $ service rundeckd start
$ service rundeckd status
 rundeckd.service - SYSV: rundeckd, providing rundeckd
   Loaded: loaded (/etc/rc.d/init.d/rundeckd; bad; vendor preset: disabled)
   Active: active (running) since Mon 2020-08-17 13:23:14 BST; 20h ago

$ tail -f /var/log/rundeck/service.log
[2020-08-14T09:02:28,539] INFO  rundeckapp.BootStrap - Rundeck is ACTIVE: executions can be run.
[2020-08-14T09:02:28,635] WARN  rundeckapp.BootStrap - [Development Mode] Usage of H2 database is recommended only for development and testing
[2020-08-14T09:02:28,899] INFO  rundeckapp.BootStrap - Rundeck startup finished in 646ms
[2020-08-14T09:02:28,991] INFO  rundeckapp.Application - Started Application in 25.616 seconds (JVM running for 28.068)
Grails application running at http://localhost:4440 in environment: production


quick setup as a docker Image and config customization

$ docker pull rundeck/rundeck

# Update the default port if it is blocked (4440) & localhost to DNS, in below three files
$ vi /etc/rundeck/profile //only port
$ vi /etc/rundeck/framework.properties // IP & PORT
$ vi /etc/rundeck/rundeck-config.properties // IP & PORT

# changing the default password of rundeck
$ cd /etc/rundeck/
edit realm.properties and change the admin values to something new

# adding a new user
$ cd /etc/rundeck/
$ sudo vi realm.properties
(add following lines next to admin:admin,user,admin line)
        user1: user1pass,user,admin,architect,deploy,build
   where user,admin,architect,deploy,build are different roles we can assign to user1


now login to rundeck console with admin access and navigate to 

settings > Access Control  > + Create ACL Policy

add following two scopes in order to give read access as an example to user user1

# Project scope
descriptionuser1 with read access to projects.
context:
  project'.*'
for:
  resource:
    - equals:
        kindjob
      allow: [read# allow to read jobs
    - equals:
        kindnode
      allow: [read# allow to read node sources
    - equals:
        kindevent
      allow: [read]
  job:
    - allow: [read# allow read of all jobs
  adhoc:
    - deny: [run# don't allow adhoc execution
  node:
    - allow: [run# allow run on nodes with the tag 'mytag'
    
by:
  groupadmin

---
# Application scope
descriptionapplication level ACL.
context:
  application'rundeck'
for:
  resource:
    - equals:
        kindproject
      allow: [read]
    - equals:
        kindsystem
      allow: [read]
    - equals:
        kindsystem_acl
      allow: [read]
    - equals:
        kinduser
      allow: [admin]
  project:
    - match:
        name'.*'
      allow: [read]

by:
  groupadmin

happy rundecking!

11 July 2017

ANSIBLE: Configuration management tool for Infra automation

https://www.middlewareandme.tech/search/label/ansible
Ansible is an agent-less task execution engine, used for configuring, managing and installing software onto clients and nodes without any downtime and without an agent installed on them.
It uses SSH to communicate with clients.
provided all the nodes should have python installed in them + every step should not be carried with root user instead with ansible user.
Ansible needs:
  • SSH connection
  • a user
  • Python 2.4+
  • It works on the principle of 'PUSH Based', which means it pushes modules from VCS to servers directly without the intervention of any intermediate client/agent
  • it contains Playbooks that are written in YAML code ( YAML ain't markup language)






Hierarchy of Ansible roles -

- create <roles> directory
    - inside roles create roles <roleDir>
    - inside <roleDir> -> create taskDir, defaultDir, templateDir
        - inside <taskDir> -> add all the ymls.
        - inside <defaultDir> -> add main.yml
        - inside <templateDir> -> add any template file if you are calling in any playbook

- last at <roles> directory create a yaml and call  your <roleDir> as shown below -


- name: NameYourPlay
  hosts: "{{ target_hosts | default ('none') }}"
  become: yes
  become_method: sudo
  roles:
    - roleDir
 


Prototype of ansible-playbook



Ansible Contain Playbooks 
Playbook has a number of plays
Play contains tasks
Task calls core or custom modules
Task can use templates
Handlers triggers from notify
executed at the end and only once

- Ansible contains more than 750 modules and can be customized and turned into custom modules.
- Modules gets executed when you run Playbook onto your 1..n nodes.
- For connectivity it uses
  • SSH password less connection by generating Public key install on all your nodes
  • Connection Plugins
  • export ANSIBLE_HOST_KEY_CHECKING=False

 Quickly start with Ansible, try using my docker image
 pull : docker pull punitporwal07/ansible:2.6
 run : docker run -it punitporwal07/ansible:2.6
 test: ./runansibletest.sh
 
 By default ansible package is not available in some yum repositories, 
 so you need to enable/add EPEL(extra package for Enterprise Linux) repository 
 which is maintained over at Fedora Project
 
 $ rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
 $ yum install ansible -y
 
 Installing Ansible on Ubuntu
 $ yum update -y
 $ apt-get update
 $ apt-get install ansible
 or
 $ sudo yum install ansible -y
 $ ansible --version

 Installing Ansible & boto on/for amazon EC2 instance
 $ amazon-linux-extras install epel 
 $ sudo yum update -y
 $ sudo yum install ansible -y
 $ sudo yum install python-boto python-boto3 -y
 
 Install boto from source, if unable to find package
 $ git clone git://github.com/boto/boto.git
 $ cd boto && python setup.py install
 $ sudo apt-get install -y python3-botocore
  
 SETTING SSH COMMUNICATION BETWEEN SERVERS
 prepare SSH key for remote hosts
 switch to ansible user
 $ ssh-keygen -t rsa              --> enter enter enter
 $ ssh-copy-id -i ansible@nodes   --> enter passwords
                            or
 $ export ANSIBLE_HOST_KEY_CHECKING=False
   //which will bypass the host-key-check
 
 test using
 $ ansible -m ping all
 
 sudo pass command
 $ ansible-playbook --ask-sudo-pass
   //it will prompt for sudo password 

 — If no passwordless ssh setup is there, then hardcode the cred. as below:

 linux1 ansible_host=10.20.192.1 ansible_connection=ssh ansible_user=username ansible_ssh_pass="pass"
 linux2 ansible_host=10.20.192.3 ansible_connection=ssh ansible_user=username ansible_ssh_pass="pass"
linux3 ansible_host=10.20.192.3 ansible_connection=ssh ansible_user=username ansible_ssh_pass="pass" linux1 ansible_host=172.20.192.1 ansible_become_pass="rootpass" ansible_ssh_user=usertossh ansible_ssh_pass="pass" linux2 ansible_host=172.20.192.3 ansible_become_pass="rootpass" ansible_ssh_user=usertossh ansible_ssh_pass="pass" linux3 ansible_host=172.20.192.3 ansible_become_pass="rootpass" ansible_ssh_user=usertossh ansible_ssh_pass="pass"


HOST INVENTORY


 $ /etc/ansible/hosts //default Location
 
 provide list of target IP address which can be grouped as
 
 [local]
 localhost ansible_connection=local
 
 [appserver]
 1.2.3.4
 2.3.4.5
 
 [dbserver]
 3.4.5.6
 

alternatively, you can design your own inventory and place it anywhere

- Inventory is the expression of your environment
- Hostnames, groups, vars are for YOUR use, they have to make sense to YOU
- Ansible cares about hosts and tasks, everything else is in support of that
- Select a single source of truth. or try to minimize duplication of data
- Normally, there is a simpler way to do it
- Ansible makes it easy to switch approaches, don't be afraid to test and try
- Mistakes are not failures

How ansible commands are structured

ansible host-group  + module    +   argument to module
ansible  localhost      -m yum    +  -a "name=nginx state=latest"
ansible  allserver       + -m shell   +   -a 'uptime'
ansible  appserver     -m user    +   -a "name=red group=oracle shell=bin/bash/"

Ad-hoc commands
ansible all -a 'uptime' (determine the uptime of all machines)
ansible -m ping all (test connection with all the host defined in host_inventory)

You can find some sample playbook's from my git repository
apache.yaml

Ansible vault

 
 We can use this utility to secure your sensitive data like password, keys etc
 
 some useful commands of ansible-vault
 
 $ ansible-vault encrypt  //encrypt any file
 $ ansible-vault edit     //edit encrypted file
 $ ansible-vault view     //view encrypted file
 $ ansible-vault rekey    //change the pass of encrypted file
 $ ansible-playbook -i inv.ini playbook.yml --ask-vault-pass 
   // this will ask for vault pass while running playbook

 Using Ansible-vault without inline commands
 
 Create ansible inventory using command
 $ ansible-vault
 
 Define your inventory pass in some hidden text file 
 $ echo mypass > .my-pass.txt 
 
 Add password file in ansible.cfg
 vault_password_file = /path/to_your_file/.my-pass.txt
  

 Now next time when you run any playbook to use an encrypted inventory file it will pick up your inventory pass from the txt file defined in ansible.cfg

      Rundeck also uses the same mechanism when you define your ansible.cfg as a configuration file.