Updated: Fri 10 Jul 2020 04:48:53 PM EDT (output from date command 😁 )

This is an aggregation (so I don't loose them 😅️) of different shell tricks I have learned.

TOC

Other blog posts

Other blog posts - Mine

bash

bash - General

bash - General - disk space

sudo du -s ./* 2>/dev/null | sort -n 

bash - AWS

bash - AWS - aliases

bash - AWS - SSM

SSM stands for Simple Systems Management

alias aws_ssm_list="aws ec2 describe-instances --query \"Reservations[].Instances[].[InstanceId,Tags[?Key=='Name'].Value]\" --filter 'Name=instance-state-name,Values=running' | jq -r '.[][]'| grep -A 2 '^i' | sed 's/^/ /' | grep -vE '\[|--'"
  • this is a command that will list out all your running aws ec2 instances (for whatever region you specified in your aws cli config), and output their id above their name

then chain that with this:

alias aws_ssm_join="aws ssm start-session --target "
  • this will (once you install the ssm plugin aws cli: here) will let you create and join an ssm session with whatever ec2 instance id you specify afterwards (i.e. aws_ssm_join i-447e3abcef9123455

bash - AWS - CloudFront

distri_id='<distribution_id_for_cloudfront>' ; invalidation_id="$(aws cloudfront create-invalidation --distribution-id "${distri_id}" --paths "/*" | jq -r '.Invalidation.Id')" ; watch -n 1 aws cloudfront get-invalidation --distribution-id "${distri_id}" --id  "${invalidation_id}"
  • explaination
  • pre-reqs:
    • jq
    • aws cli
  • the only thing you have to input is the Cloudfront distribution id at the beginning of the oneline, and then it will invalidate it and also continuously check the status of it's invalidation

bash - AWS - CloudFormation

bash - AWS - CloudFormation - parameter overrides
# pull secrets from AWS secrets manager service by the name
infra_params=$( aws secretsmanager get-secret-value --secret-id "${secret_name}" )

# parse our secrets format (json) and add it to paramz array
mapfile -t paramz < <(jq -r '.SecretString' <<< "${infra_params}" | jq -r '.params[] | "\(.ParameterKey)=\(.ParameterValue)"')

# deploy cloudformation stack w/expanding the paramz array
aws cloudformation deploy --stack-name "${stack_name}" --template <template_name> --parameter-overrides "${paramz[@]}"
  • pre-reqs:
    • jq
    • aws cli
    • bash >= 4.4
  • We use this at work, Secure Ideas, in a bash script to take a few arguments, but I have stripped them out and left in the main logic of the script.
    • We use this to dynamically grab a secret from secrets manager (so we aren't hard coding anything into our source code)
    • then create an array and parse it with the mapfile builtin bash command to handle quoting for key/values with spaces (so you don't have to escape the spaces from the jq results i.e. how people appear to have solved it in the past).
    • which passes the properly quoted commands to the aws cloudformation deploy command, when we expand the array that was created by mapfile.
  • special thanks to @OchaunM and @84d93r for working with me on this.

bash - AWS - EC2

bash - AWS - EC2 - Security Group IP formatting
grep -oP '\d+\.\d+\.\d+\.\d+(\/\d+|)' <<'EOF' | sed 's,$,&/32,g' | tr '\n' ','
8.8.8.8 - personal home IP
1.1.1.1 - alice's ip address
EOF
  • Using the logic from this shell trick, we can format the ip address we get to the proper format that is needed for AWS security groups (SG). Logic breakdown:
    • grepping out IP addresses
    • reading them in from heredoc
    • pipe to sed and append the /32 to every line
    • and then comma delimit them so we can just paste them into the SG's asking for IP addresses for each service.

bash - Scripting

bash - Scripting - misc

mapfile -t env_vars <<< "$(env | cut -d '=' -f 1 | grep -vE '^(PATH|SHELL|HOME|PWD|USER)')"
unset "${env_vars[@]}"

so...this command is something I thought of to try and ensure not to have colliding variables (yes, I know in theory I shouldn't have environment variables colliding with script variables because all environment variables should be all caps while all scripts should be lowercase), but it got me thinking...

How could I wipe out all my environment variables except ones that are expected by a script...

so, that is what the code snippet above does...wipes out all environemnt variables that are stored in the array that is created ( read more as to why I chose mapfile here). Which the array is excluding all environment variables that are listed in that grep command (checkout the command that makes the array here (at explainshell)).

bash - Scripting - bash's "main"

function main(){
  your_function_here
}

if [[ "${0}" = "${BASH_SOURCE[0]}" ]] ; then
  main "${@}"
fi

This is the bash's equivilent to python's main function. This is really helpful if you want to source the shell script and use it's functions inside of another script, or you just want to run the script directly. If you try to source the script without this, then either one of two situations will happen:

  1. everything will just execute instead of allowing you to re-use functions
  2. you won't be able to execute the script directly because there is nothing initiating all the functions inside of it.

checkout bash - Scripting - source - set safety for more info on how to do this without affecting your shell evironment.

credit

bash - Scripting - source

bash - Scripting - source - set safety
source <(grep -v '^set' <scripts_file.sh>)

so, this is how you can source a script without it affecting your current shell settings you have configured with set.

bash - Scripting - grep

bash - Scripting - grep - ip addresses
grep -oP '\d+\.\d+\.\d+\.\d+(\/\d+|)' file_with_ips
bash - Scripting - grep - heredoc w/ip addresses
grep -oP '\d+\.\d+\.\d+\.\d+(\/\d+|)' <<'EOF'
8.8.8.8 - personal home IP
1.1.1.1 - alice's ip address
EOF

bash - Scripting - nmap

bash - Scripting - nmap - Cidr calculations...
nmap -iL ip_ranges -sL -n | grep report | awk '{print $5}' | tee ip_addresses
  • explaination

    • so, I like automation, and if you have ever been handed a list of ip ranges by a client to do testing on, and they say

    Hit all those ranges except these subsets...

    then you know the struggle of trying to exlude ip addresses...well no more...

    • nmap is awesome! as you can see from the explainshell link above there is an option to do a -sL (List Scan), which essentially just lists out all the ip addresses that you would scan.
    • so, the command above by itself is awesome for outputting ip addresses without you have to calculate weird cidrs, but the following command can auto exclude ip address that you provide from a file.
      nmap -iL ip_ranges -sL -n --excludefile exclude.txt | grep report | awk '{print $5}'
      

bash - Scripting - zfs

bash - Scripting - zfs - deleting zfs-auto-snapshot snapshots

I am sure there is a better way to do this, and I will figure that out eventually, but for now when I need to make sure I clear out enough of the zfs auto snapshots I run this oneliner/script:

for pool in $(zpool list -Hg -o name) ; do
  log_file="${HOME}/deletion-${pool}.log"
  rm -f "${log_file}" &&
    for i in $(zfs list -t snapshot | grep "^${pool}" | grep 'zfs-auto-snap'| awk '{print $1}') ; do
      sudo zfs destroy -v "$i" | tee -a "${log_file}"
    done
done

bash - Scripting - Network Manager

If you like this section you should checkout my other blog post that enables random wifi mac addresses through Network Manager: https://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.sitehttps://blog.elreydetoda.site/ubuntu-install/#networkmanager

bash - Scripting - Network Manager - Disable autoconnect

Read more here about what device probing is, but the following script is used to disable auto connect for all your current wireless network profiles that are in Network Manager.

NOTE: If you are running zsh (i.e. oh-my-zsh) then you need to drop down to a bash shell for running this script. even running emulate -L sh doesn't properly recognize the mapfile command.

mapfile -t networks_array < <(nmcli -t -e no -f NAME,TYPE connection show | grep '802-11-wireless' | rev |cut -d ':' -f 2- | rev) &&
  for connection in "${networks_array[@]}" ; do
    printf 'currently modifying this profile: %s\n' "${connection}" && 
      sudo nmcli connection modify "${connection}" connection.autoconnect false
  done &&
    sudo systemctl restart NetworkManager

bash - Scripting - Modified script "hardening"

set -${-//[sc]/}eu${DEBUG+xv}o pipefail

sed

sed - parsing for urls

curl -fsSL "${current_terraform_url}" | sed -n '/href=".*linux_amd64.zip"/p' | awk -F'["]' '{print $10}'

Vagrant

Vagrant - Aliases

If you have used vagrant, or know how I love to automate things 😀️, at all then you know typing vagrant up, vagrant ssh, and vagrant destroy can get a little cumbersome...heck I got tired of typing it while typing the blog post up...🙃️

So, I created a few shell aliases for chaining vagrant commands together, and I even have an alias for printing out my aliases 😀️. One of the other notable aliases is the v-config alias, which is used to grep out all comments and blank lines of a Vagrantfile so that way you can see what your current Vagrantfile's configuration is.

alias v-newz_conn='vagrant destroy -f && vagrant up && vagrant ssh'
alias v-newz_up_conn='vagrant destroy -f && vagrant box update && vagrant up && vagrant ssh'
alias v-newz_up_snap_conn='vagrant destroy -f && vagrant box update && vagrant up && vagrant halt && vagrant snapshot push && vagrant up && vagrant ssh'
alias v-newz_snap_conn='vagrant destroy -f && vagrant up && vagrant halt && vagrant snapshot push && vagrant up && vagrant ssh'
alias v-snap_conn='vagrant halt && vagrant snapshot push && vagrant up && vagrant ssh'
alias v-reboot='vagrant halt && vagrant up'
alias v-reboot_conn='vagrant halt && vagrant up && vagrant ssh'
alias v-connect='vagrant up && vagrant ssh'
alias v-revert_conn='vagrant snapshot pop --no-delete && vagrant ssh'
alias v-revert_prov_conn='vagrant snapshot pop --no-delete --provision && vagrant ssh'
alias v-config="grep -vP '^\s+#|^#|^$' Vagrantfile"
alias v-aliases="grep '^alias v-' ~/.zshrc"

another trick to help chaining even my aliases together is that if you see where I have an alias that ends with conn or connect (i.e. when I do vagrant ssh) then you can pass -c exit to keep the command going instead of ssh'ing into the vagrant box.

A great example of this is when I revert a box to a snapshot (i.e. v-revert_conn and I want to then provision it with a specific provisioner I have declared in the Vagrantfile. Like this command here:

v-revert_conn -c exit && vagrant provision --provision-with static-analysis

Vagrant - Functions

Sometimes when you want to do something a little more complex than an alias you have to put the command in a shell function inside your rc file.

So, I currently have this vagrant function that I use to iterate over all my running vagrant boxes to stop all of them. This is really useful for two situation:

  1. I am getting ready to reboot/turn off my machine, and while I know that the vms should be fine, the sysadmin side of me says turn them off...it doesn't hurt
  2. If I want to start using a different hypervisor. Since I run linux I have both Virtualbox and also I use QEMU/Libvirt (combined with virt-manager), which is a more linux native hypervisor, but you can't run both those hypervisors at the same time. So, I will stop all my vms with virtualbox to then switch over to libvirt.
v-stop_all () {
	for i in $(vagrant global-status | grep -oP '\srunning\s+/.*' | cut -d ' ' -f 4-)
	do
		pushd $i || return 1 && vagrant halt && popd
	done
}

ansible-playbooks

I promise this has to do with shell scripting 😁️, it is using bash scripting to help extend your ansible playbooks by giving you more enrichment to your playbooks through bash/shel scripting.

ansible-playbooks - semver sorting

here is the best explaination for what the below code does and why: https://github.com/diodonfrost/ansible-role-vagrant/pull/1

an ansible playbook that does semantic version sorting, more here: https://github.com/diodonfrost/ansible-role-vagrant/pull/1
an ansible playbook that does semantic version sorting, more here: https://github.com/diodonfrost/ansible-role-vagrant/pull/1 - ansible-semver.yml

recording example here ( full screen to make it look right )