INDEX
########################################################### 2024-02-15 17:30 2024-02-16 14:20 2024-02-17 09:10 ########################################################### Jeff Geerling... Ansible 101 https://www.youtube.com/playlist?list=PL2_OBreMn7FqZkvMYt6ATmgC0KAGGJNAN ############ Episode 1: Intro Can install ansible with pip - installs playbook commands etc. pip3 install ansible Agentless by needs python installed on servers - uses ssh in background Inventory files default to ini format # inventory [example] # Define a group of servers www.example.com # Reference server by DNS 107.20.106.183 # Reference server by IP asible -i inventory example -m ping -u centos # Ping all servers in "example" # "-u centos" means use the user "centos" on the server # ansible.cfg # Define the main config file [defaults] # Overwrite the "defaults" group INVENTORY = inventory # Now you don't need to specify the inventory file ansible example -m command -a "date" -u centos # "-m command" module is set by default # Can do password auth with -K or --ask-pass Vagrant is a good tool for making VMs - uses vagrant files to config servers vagrant init geerlingguy/centos7 # Creates a vagrant file - this one using centos7 # vagrantcloud.com is a repository for OS vagrant files vagrant up # Start VM vagrant ssh # Connect over ssh to VM vagrant ssh-config # Gives an ssh config for connecting to the VM # Default creates private keypair but runs on 127.0.0.1:2222 - not on separate subnet vagrant halt # Stop VM vagrant destroy # Delete the VM # Vagrantfile Vagrant.configure("2") do [config] config.vm.box = "geerlingguy/centos7" # Give OS image to run config.vm.provision "ansible" do [ansible] # Auto run ansible playbook ansible.playbook = "playbook.yml" end end # playbook.yml - hosts: all # Run on all defined servers become: yes # Run as root user (uses sudo by default) tasks: - name: Ensure NTP is installed yum: name=ntp state=present # Use yum module to install ntp - name: Ensure NTP is running service: name=ntpd state=started enabled=yes # Use service module to start/enable vagrant provision # Runs any provisions defined (ansible playbook linked) # Ansible is idempotent so only applies if there are actual changes - command = yum install -y ntp # Technically different to yum module as does not check state - shell: | if ! rpm -qa | grep -qw ntp; then yum install -y ntp; fi # Alt do this with state checks ############ Episode 2: Ad hoc Can configure many VMs at the same time # Vagrantfile Vagrant.configure("2") do [config] config.vm.box = "geerlingguy/centos7" config.ssh.insert_key = false # Prevent generation of random keys for each VM config.vm.synced_folder ".", "/vagrant", disable: true # Disable default mount option config.vm.provider :virtualbox do |v| v.memory = 256 # Cap memory usage to 256MB each v.linked_clone = true # Faster to start up as cloned VMs share some data end config.vm.define "app1" do |app| app.vm.hostname = "arc.app1.test" # .dev and .local used by some services already app.vm.network :private_network, ip: "192.168.60.4" # Set IP of server end ... # Repeat for app2 and db (on IPs .5 and .6) end vagrant up # Build all VMs Define servers in groups for ansible # inventory (typically start with ini and go to yaml if it gets too complex) [app] 192.168.60.4 192.168.60.5 [db] 192.168.60.6 [multi:children] # Define a group of (children) groups app db [multi:vars] # Define variables for this group ansible_ssh_user=vagrant ansible_ssh_private_key_file=~/.vagrant.d/insecure_private_key. # Use global default key ansible multi -i inventory -a "hostname" # Check hostname of all servers in "multi" group # By default ansible forks 5 times to connect - can set "-f 1" to connect 1 at a time ansible -i inventory db -m setup # Show all ansible info about a server Can run adhoc commands with 1-line commands ansible -i inventory multi -b -m yum "name=ntp state=present" # (-b=become) install ntpd ansible -i inventory multi -b -m service -a "name=ntpd state=started enabled=yes" # run ntpd ansible -i invenotry multi -b -a "ntpdate -q 0.rhel.pool.ntp.org" # Check ntp offset ansible -i inventory db -b -m mysql_user \ -a "name=django host=% password=123 priv=*.*:ALL state=present" # Set up DB user "django" # "--limit" to only run on a single server (can use filtered arguments and wildcards) ############ Episode 3: Intro to Playbooks Can run jobs and poll them for progress - big tasks to run in the background ansible -i inventory multi -b -B 3600 -P 0 -a "yum -y update" # Run a job for 1 hour # Creates a result file with job-IDs - can be used to cehck status ansible -i inventory multi -b -m async_status -a "jid=ID" # Gives output of a job -m git -a "repo=URL dest=DIR update=yes version=1.2.4" # Update a repo in a folder Ansible by default has ssh pipelining off (allows reuse of ssh connections) # ansible.cfg [ssh_connection] pipelining = True Ansible playbooks allow you to script actions to run on servers Also checks state in layers (| = combine all lines, > = combine all lines as 1 line) ansible-doc yum # Check documentation for yum module # Version 1 - name: Install Apache command: yum install --quiet -y httpd httpd-devel - name: Copy config command: cp httpd.conf /etc/httpd/conf/httpd.conf - name: Make sure service started command: service httpd start # Version 2 - name: Install Apache yum: name: - httpd - httpd-devel state: present # Make sure installed on system become: true # Elevate user only for this task - name: Copy config copy: src: "{{ item.src }}" # Reference each item's src - OR {{ item['src'] }} dest: "{{ item.dest }}" owner: root group: root mode: 0644 with_items: # Creates a list indexed by "item" - src: httpd.conf dest: /etc/httpd/conf/httpd.conf - name: Make sure service started service: name: httpd state: started enabled: true # Can also true/yes/1 and false/no/0 ansible-playbook -i inventory playbook.yml ansible-inventory --list -i inventory # List all items in inventory Ideally you want a conditional task flow so service is not restarted if files not copied etc. ############ Episode 4: Real World Playbook Want to run Apache Solr to local search/indexing system # inventory [solr] 54.210.120.211 ansible_user=ubuntu # vars.yml (set specific variables) download_dir: /tmp solr_dir: /opt/solr solr_version: 8.5.0 solr_checksum: ... # main.yml (can name it anything) --- - hosts: solr become: true vars_files: - vars.yml pre_tasks: # Do before everything else (just a setup item - preference) - name: Update apt cache if needed apt: update_cache=true cache_valid_time=3600 handlers: # Like a function for restarting a service - name: Restart solr service: name=solr state=restarted tasks: - name: Install Java apt: name=openjdk-8.jdk state=present # Can use "package" but this one only works for apt - name: Download solr get_url: url: ".../solr/{{ solr_version }}/solr-{{ solr_version }}.tgz" dest: "{{ download_dir }}/solr-{{ solr_version }}.tgz" checksum: "{{ solr_checksum }}" - name: Expand solr unarchive: src: "{{ download_dir }}/solr-{{ solr_version }}.tgz" # Could also use URL dest: "{{ download_dir }}" remote_src: true # By default it copies to ansible server an unarchives there creates: "{{ download_dir }}/solr-{{ solr_version }}/README # Declare state to track - name: Run Solr installation script command: > {{ download_dir }}/solr-{{ solr_version }}/bin/install_solr_service.sh {{ download_dir }}/solr-{{ solr_version }}.tgz -i /opt -d /var/solr -u solr -s solr -p 8983 creates={{ solr_dir }}/bin/solr - name: Ensure solr is started and enabled at boot. service: name=solr state=started enabled=yes --- ansible-playbook -i inventory.yml main.yml --syntax-check ############ Episode 5: Playbook handlers A handler is just a task you can call # apache.yml (part of) handlers: # Can define these anywhere in the file - order independent - name: restart apache service: name=httpd state=restarted tasks: - name: Copy test config file copy: src: files/test.conf dest: /etc/httpd/conf.d/test.conf notify: restart apache # Calls handler by name at END of playbook - can be a list - name: Make sure handlers are flushed immediately meta: flush_handlers # Calls all handlers # Handlers won't run if playbok fails ansible-playbook -i inventory apache.yml --force-handlers # Force run even on fail Handlers can call other handlers using the same "notify" property Can drop in environment variables using a task # apache.yml (part of) tasks: - name: Add an env variable to user's shell lineinfile: dest: "~/.bash_profile" # Look at this file regexp: '^ENV_VAR=' # Checks for existence of variable in file line: 'ENV_VAR=value' become: false # Do not run as root - name: Get value of of environment variable shell: 'source ~/.bash_profile && echo $ENV_VAR' # Export a variable to screen register: foo # Save shell object to "foo" variable - debug: msg="The variable is {{ foo.stdout }}" # Extract variable contents # Can set variables inline with tasks if needed - name: Download a file get_url: url: ... dest: /tmp environment: http_proxy: http://example-proxy:80 # Or set them in a "vars" section (outside of tasks) and reference in a task vars: proxy_vars: http_proxy: http://example-proxy:80 ... tasks: - name: Download a file ... environment: proxy_vars # Or set environment variables for all tasks environment: http_proxy: http://example-proxy:80 # Can also store externally in a separate file Can also use variables to install different packages for different systems (eg apache_package) # apache_defaults.yml apache_package: apache2 # apache_RedHat.yml (has to be RedHat not CentOS) apache_package: httpd # apache.yml (part of) vars_files: - "apache_defaults.yml" # Load the default variables - "apache_{{ ansible_os_family }}.yml" # Load the OS dependent variables # alt version using a pretask pre_tasks: - name: Load variable files include_vars: "{{ item }}" with_first_found: # Loads the first file it detects - "vars/apache_{{ ansible_os_family }}.yml" - "vars/apache_default.yml" # Can restrict system from setting magic variables by scanning servers gather_facts: false # Debug arbitrary variables - debug: var=foo # Debug the "foo" variable ############ Episode 6: Ansible Vault and Roles Use a key to unlock ansible vault passwords - should not store passwords in code ansible-vault encrypt FILE # Encrypt, say, a file with an API key - modifies that file # Can equally do "decrypt" to reverse ansible-playbook main.yml # Will fail if depends on encrypted file ansible-playbook main.yml --ask-vault-pass # Works as asks for password - but inconvenient ... --vault-password-file FILE # Pass in a file containing the decryption password ansible-vault edit FILE # Edits the encrypted file - "rekey" to change password Can have conditionals on tasks: when, changed_when and failed_when to set tests and alerts ignore_errors: true # Ignores any failures on a task tags: # Apply tags to a task - api - echo ansible-playbook main.yml --tags=api # Runs only "api" tags - use sparingly If playbook is 80-100 lines probably start thinking about splitting into into pieces Can store handlers in another file and "include" them handlers: - import_tasks: handlers/apache.yml # Can similarly do for tasks - just use "import_tasks" in the tasks section tasks: - import_tasks: tasks/apache.yml vars: apache_package: apache3 # Can explicitly pass variables to an imported task # Tasks are imported statically so it's like pasting the import in that line include_tasks: tasks/ansible.yml # Imports dynamically - avoids some variable conflicts # Can import whole playbooks to the end of another playbook - import_playbook: app.yml # Run tasks conditionally - name: Start example Node.js app # Runs only if not already running command: "forever start {{ node_apps_location }}/app/app.js" when: "forever_list.stdot.find(node_apps_location + '/app/app.js') == -1" Roles are used to package up tasks to be reused in different places # roles/nodesjs/tasks/main.yml (here you store all the tasks) # roles/nodesjs/meta/main.yml (list dependencies) dependencies: [] # Import in the main script roles: # Runs after pre_tasks. Can add a task with "include_role: nodejs" to bypass - nodejs # Now prefixes all tasks with this "nodejs" label # Ansible galaxy is a repo of roles you can use ansible-galaxy role init test # Creates a structure of folders for a "test" role # Means you can run code entirely from role imports (roles: geerlingguy.java) ############ Episode 7: Molecule testing Can install roles globally or locally to system with ansible-galaxy ansible-galaxy install geerlingguy.homebrew # Installs globally # ansible.cfg [defaults] nocows = True roles_path = ./roles # requirements.yml roles: - name: geerlingguy.homebrew version: 3.1.0 ansible-galaxy install -r requirements.yml # Install required roles to ./roles folder Want to be able to automatically test ansible code (valid yml, indempotent, compare to prod) yamllint # Check yaml validity ansible-playbook --syntax-check # Check ansible yaml validity ansible-lint # Check best practices of ansible molecule test # Tests in a fresh environment ansible-playbook --check # Compare to prod to see if things break # Parallel infra - test everything on a separate infrastructure Can do conditional processes and tests in ansible with "when" and "assert" # Debug uptime: - name: Register uptime command: uptime register: system_uptime # Store as a variable - name: Print uptime debug: # Prints something to the screen var: system_uptime.stdout # Prints some variable called "system_uptime" - name: Print message if results change debug: msg: "Command resulted in a change!" when: system_uptime is changed # Can make a task fail based on a variable - name: Fail based on variable=true/false fail: msg: "There was a failure" when: should_fail_via_fail - name: Fail based on a variable test assert: that: "should_fail_via_assert != true" # Can make list a list of statements Sometimes yaml can fail from spacing issues yamllint . # Prints all yaml warnings/errors - install with pip ansible-playbook main.yml --syntax-check # Checks if file is interpretable by ansible # Can only check static imports - anything done at runtime (include) can't be checked ansible-lint main.yml # Checks best practices like naming tasks - install with pip # Molecule creates a test environment to test everything you do in ansible - install with pip molecule init role myrole # Creates role folder structure (similar to galaxy + /molecule) # molecule.yml defines the test environment (e.g. centos7 inside docker) # converge.yml just imports the role - use this to run molecule # verify.yml executes the tests molecule test # Run this within the role folder - builds and tests molecule converge # Runs test but does not destroy environment molecule login # Alt use "docker exec -it ID /bin/bash" - explore test environment molecule destroy # Delete test environment ############ Episode 8: Testing with Molecule and Github Actions Systemd running in a container often fails - so molecule with docker can have issues # Docker is kept running by giving it an initial command that runs forever command: bash -c "while true; do sleep 10000 done" # Loops doing nothing # But this runs as ID=1 - which should be systemd for systemd to work # molecule.yml dependency: name: galaxy driver: name: docker platform: - name: instance image: gerrlingguy/docker-centos8-ansible:latest # defaults to centos8 with python command: "" # Overwrite default "run forever" command volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro # Mount systemd cgroup privileged: true # Run container with minimal security (only use to test on known images) pre_build_image: true provisioner: name: ansible verifier: name: ansible # But how do you run this in debian? Use environment variables with defaults image: gerrlingguy/docker-${MOLECULE_DISTRO:-centos8}-ansible:latest MOLECULE_DISTRO=debian10 molecule test # Run test with specific OS Use verify playbook to see if apache is running properly # verify.yml - name: Verify hosts: all tasks: - name: Verify apache is serving web requests uri: uri: http://localhost status_code: 200 # Check if web server returns status 200 # Can also run this directly in the main playbook - depends what makes sense # By default linting is disabled # (part of) molecule.yml lint: | # Runs both linting tests as part of "molecule test" set -e yamllint . ansibl-lint Can add a github action for linting # .github/workflows/ci.yml (similar to ansible) name: CI # Workflow name 'on': # Run on all pull requests and any pushes to master pull_requests: push: branches: - master jobs: test: name: Molecule runs-on: ubuntu-latest strategy: matrix: # Defines a variable "distro" distro: - centos8 - debian10 steps: - name: Check out the codebase uses: actions/checkout@v2 # Use builtin for checking out code - name: Set up python3 uses: actions/setup-python@v2 with: python-version: '3.x' - name: Install test dependencies run: pip3 install molecule docker yamllint ansible-lint - name: Run Molecule tests run: molecule test env: # Force use colours on output PY_COLORS: '1' ANSIBLE_FORCE_COLOR: '1' MOLECULE_DISTRO: ${{ matrix.distro }} # Pass in distro variable # Commit and push this - it should run automatically ############ Episode 9: First 5 min server security Basic security measures: - Use secure and encrypted comms - Disable root login and use sudo - use the principle of least privilege - Remove unused software - open only required ports - use properly configured firewall - Update the OS and installed software - Make sure log files are populated and rotated - Monitor logins and block suspect IP addresses - Use SELinux Secure SSH using key-based auth - disable password auth - can also change ssh port "mosh" is a version of ssh for low quality connections # main.yml - hosts: centos become: true handlers: - name: restart ssh service: name: sshd state: restarted tasks: - name: Tell SELinux about SSH new port # Report to selinux when you do changes seport: ports: 2849 proto: tcp setype: ssh_port_t state: present - name: Make sure SSH is more secure # Set sshd config lineinfile: dest: /etc/ssh/sshd_config regexp: "{{ item.regexp }}" # Adds the line if it doesn't already exist line: "{{ item.line }}" state: present validate: 'sshd -T -f %s' # Runs a command to validate the config with_items: # Can also use "loop" - regexp: "^PasswordAuthentication" line: "PasswordAuthentication no" - regexp: "^PermitRootLogin" line: "PermitRootLogin no" - regexp: "^Port" line: "Port 2849" # Need to also change this in ansible config notify: restart ssh - name: Add user johndoe. user: # Could add group here too name: johndoe - name: Add sudo gorup rights from deploy user lineinfile: dest: /etc/sudoers regexp: '^johndoe' line: 'johndoe ALL=(ALL) NOPASSWD: ALL' state: present validate: 'visudo -cf %s' - name: Remove unused packages package: name: - httpd - nano - mailutils state: absent # Makes sure they are not installed purge: yes # Uninstalls them - name: Edit file permissions file: path: /etc/ssh/sshd_config mode: 0600 # Ansible needs octal - don't use 777 - name: Make sure everything is up to date # Probably best to separate this task apt: upgrade: dist update_cache: yes - name: Set up autoupdates yum: # Can use package - but depends on command you are running name: yum-cron state: present - name: Ensure autoupdates are running # For debian use "unattended upgrades" service: name: yum-cron # Runs nightly - can set it to reboot as well state: started enabled: yes - name: Install firewalld package: name: firewalld state: present - name: Configure ports on firewall firewalld: # Disables all non-explicit ports state: "{{ item.state }}" port: "{{ item.port }}" zone: external immediate: yes permanent: yes with_items: - { state: 'enabled', port: 2849/tcp } # For SSH - { state: 'enabled', port: 123/udp } # For NTP - name: Ensure firewalld is running (run this AFTER setting rules to not be locked out) service: name: firewalld state: started enabled: yes To ensure you don't lock yourself out, always have a session open when applying changes Permission 0644 is rw-r-r - r=4,w=2,x=1 Can use geerlingguy.securty role to automatically do most security policies Fail2Ban uses log file to check if someone is trying to hack - blocks IPs ############ Episode 10: Ansible Tower and AWX Ansible Tower to AWX is like Redhat to Fedora - use tower operator (in minikube) to install minikube start --memory 8g --cpus 4 # Needs lots of power minikube addons enable ingress minikube ip # Add this to /etc/hosts as "example-tower.test" # Download and extract tower-operator - go into that folder molecule test -s test-minikube # Builds basic test environment in minikube # Go to example-tower.test - takes some time to get started - gives a dashboard of hosts # Run the demo job - to test if things are running properly Can store roles in collections: USER.COLLECTION.ROLE ansible-galaxy install -r requirements.yml # Go into Credentials in AWX - add Machine credential with ssh key using sudo # In Inventories - add Inventory - add Groups and servers similar to inventory file # In Project - add Git project # In Templates - add project: playbooks, webhooks, gitops etc. # For older versions might have to move roles/collections to separate foldersa Tower gives fail/pass rates and graphs of jobs - has clear logging/scheduling ############ Episode 11: Dynamic Inventory Can define groups in "inventory/group_vars/all.yml" instead of in the inventory file ansible-inventory -i inventory/hosts.ini --list # See internal view of ansible inventory Can define inventory in a script file (returns a json file) # Say you have inventory.php (+x) which can both get and set the inventory inventory/inventory.php --list # Gives json output of inventory ansible-invetory -i inventory/inventory.php --list # Imports by running php file ... --graph # Shows a clearer hierarchy of hosts ... --list -y # Gives a yaml output of servers (can compare dynamic to static) If you're using amazon servers you could query from amazon to get data or use ansible builtins Look on ansible docs for source plugins # demo_aws_ec2.yml plugin: aws_ec2 boto_profile: jeffgeerling # For different AWS logins regions: - us-east-1 keyed_groups: # Can automatically make groups by region - key: tags-group separator: '' hostnames: - network-interface.association.public-ip # Use IP address instead of public DNS # Could use "private-ip" to connect to internal IP addresses Inside Tower you use a "source" and link to AWS - same as the aws config file ############ Episode 12: Real World Ansible Playbooks Can make sure something always runs no matter the selected tag with "tags: ['always']" ############ Episode 13: Ansible Collections Collections are distributions of modules separate from Ansible core A test plugin is the easiest thing to make # main.yml - hosts: all vars: color_choice: blue tasks: - name: "Verify {{ color_choice }} is a form of blue" assert: that: "{{ color_choice }} == 'blue'" ansible-playbook main.yml -c local -i "localhost," # Force run on localhost But what if you have multiple blues? Ansible is not for programming Make a plugin to define what "blue" is - look in ansible plugin development documentation # test_plugins/blue.py def is_blue(string): blue_values = ['blue', '#0000ff', '#00f', 'rgb(0,0,255)', rgb(0%,0%,100%)'] return string in blue_values class TestModule(objec): def tests(self): return { 'blue': is_blue } # Have to give a namespace and a path - creates a galaxy file ansible-galaxy collection init local.colors --init-path ./collections/ansible_collections # galaxy.yml namespace: local name: colors version: 1.0.0 readme: README.md authors: - your name <example@domain.com> # Move blue.py to colors/plugins/test # Have to refer fully qualified collection name to reference it assert: that: color_choice is local.colors.blue When you import collections in requirements.yml they don't go to local folder # ansible.cfg [defaults] collections_paths = ./collections # Force install them into local folder ansible-galaxy install -r requirements.yml # may have to do "collection install" ############ Episode 14: Ansible and Windows To get ansible working in WSL you need to install virtual box on windows Can use ansible to configure Windows - need openssh installed # hosts.ini [windows] 10.0.100.112 ansible_connection=ssh ansible_shell_type=cmd # Can set to "powershell" ansible windows -m win_ping --ask-pass # Ping windows server