AWS CLI: 10 useful commands you may not know

The AWS console is certainly very well laid out and, with time, becomes very easy to use. However if you are not using the AWS CLI (Command Line Interface) from your local terminal, you may be missing out on whole lot of great functionality and speed. If you are not yet comfortable with the AWS Command Line Interface, there’s a great course on the subject available right now on Cloud Academy.

Even if you are used to the AWS CLI, I encourage you to take a look at the commands below, as you may not be completely aware of the power of the AWS CLI, and you might just end up saving yourself a whole lot of time. One important note: the precise syntax of some commands can vary between versions and packages.

1. Delete an S3 bucket and all its contents with just one command

Sometimes you may end up with a bucket full of hundreds or thousands of files that you no longer need. If you have ever had to delete a substantial number of items in S3, you know that this can be a little time consuming. The following command will delete a bucket and all of its content including directories:

aws s3 rb s3://bucket-name –force

2. Recursively copy a directory and its subfolders from your PC to Amazon S3

If you have used the S3 Console, at some stage, you’ve probably found yourself having to copy a ton of files to a bucket from your PC. It can be a little clunky at times, especially if you have multiple directory levels that need to be copied. The following AWS CLI command will make the process a little easier, as it will copy a directory and all of its sub folders from your PC to Amazon S3 to a specified region.

aws s3 cp MyFolder s3://bucket-name — recursive [–region us-west-2]

3. Display subsets of all available ec2 images

The following will display all available ec2 images, filtered to include only those built on Ubuntu (assuming, of course, that you’re working from a terminal on a Linux or Mac machine).

aws ec2 describe-images | grep ubuntu

Warning: this may take a few minutes.

4. List users in a different format

Sometimes, depending on the output format you chose as default, when you invoke long lists – like a large set of users – the display format can be a little hard to read. Including the –output parameter with, say, the table argument, will display a nice, easy-to-read table this one time without having to change your default.

aws iam list-users –output table

5. List the sizes of an S3 bucket and its contents

The following command uses JSON output to list the size of a bucket and the items stored within. This might come in handy when auditing what is taking up all your S3 storage.

1
aws s3api list-objects –bucket BUCKETNAME –output json –query “[sum(Contents[].Size), length(Contents[])]”

6. Move S3 bucket to different location

If you need to quickly move an S3 bucket to a different location, then this command just might save you a ton of time.

1
aws s3 sync s3://oldbucket s3://newbucket –source-region us-west-1 –region us-west-2

7. List users by ARN

“jq” is like sed for JSON data – you can use it to slice, filter, map, and transform structured data with the same ease that sed, awk, grep and friends let you play with non-JSON text.

Armed with that knowledge, we can now nicely list all our users, but only show their ARNs.

aws iam list-users –output json | jq -r .Users[].Arn

Note: jq, might not be installed on your system by default. On Debian-based systems (including Ubuntu), use sudo apt-get install jq

8. List all of your instances that are currently stopped, and the reason for the stop

Here’s another use of the JSON output parameter. This one will list all of your stopped instances and, best of all, show the reason that they were stopped:

1
aws ec2 describe-instances –filters Name=instance-state-name,Values=stopped –region eu-west-1 –output json | jq -r .Reservations[].Instances[].StateReason.Message
9. Test one of your public CloudFormation templates

If you have written a Cloud Formation Template and need to validate it before launching, you can do it from the CLI using the following format:

1
aws cloudformation validate-template –region eu-west-1 –template-url https://s3-eu-west-1.amazonaws.com/ca/ca.cftemplate
10. Other ways to pass input parameters to the AWS CLI with JSON:

You can pass all sorts of input parameters to the AWS CLI. Here’s an example of how to do it:

1
aws iam put-user-policy –user-name AWS-Cli-Test –policy-name Power-Access –policy-document ‘{ “Statement”: [ { “Effect”: “Allow”, “NotAction”: “iam:*”, “Resource”: “*” } ] }’

.so: cannot open shared object file: Permission denied

When I start the application in debug mode (sh -x) it works fine after taking few seconds while loading the libraries but without debug mode it failes with .so : cannot open shared object file: Permission denied even though it has proper read permission as well as proper read permission to all parent folders.

Then I figured out that its due to selinux in enforcing state.

Change : /etc/selinux/config from SELINUX=enforcing ## or permissive to SELINUX=disabled

Vagrant and Ansible: Automate wordpress blog site. Post a blog using a file by python script

Prerequisites:

1. You need to have Virtualization software installed on your desktop. You can try Oracle Vm VirtualBox.

2. You need to have Vagrant installed your computer.

Instruction for using this.
Steps:

1. Setup Linux Vagrant VM. I have used CentOS here as mentioned in below command.

#mkdir -p /path/to/your/project

#vagrant init chef/centos-6.5 https://atlas.hashicorp.com/chef/boxes/centos-6.5

– Setup the Vagrantfile provided in this repository and replace it here. Make sure to change the port on line no 45 if the
host port 8888 is already used on your computer.

2. Start the Virtual machine Vagrant box.

#vagrant up

– You are now running with wordpress site and you can verify that by accessing http://localhost:8888/ ( The port no is the
one that you have specified as host port no in above step)

– During this installation it installs epel repo, git and ansible locally

– It also runs ansible locally here. But you can leverage ansible script to run remotely.

– It also make sure that sshe keys are setup so that it wont prompt for login promp and key authentication during ansible play book run.

– It downloads the current git repo as per Vagrant files config.vm.provision script.

– It runs the ansible playbook to install all require software for wordpress.

– Here is the command that you have use in order to apply ansible-playbook manually.

– ansible-playbook -i hosts site.yml
3. Make sure to change the permission for blog_post.py as executable and while executing that script pass file that you want
to post it.

#chmod +x blog_post.py

#./blog_post.py filename.txt

– This adds the first line of filename.txt as a title of the post and rest of the line as contents of the post.

– Make sure to provide correct site URL here. on line no 7 of the script wp_site = “http://localhost/” or wp_site = “http://localhost:8888/” from whereever you are testing the site.

– This scripts uses automation user login credentails which were created during mysql db creation/import processs from
ansible’s wordpress role.

Documentation for Ansible configuration.

1. hosts:

– In this file make sure to add the correct IP or hostname at the bottom of the file. Right now it had localhost IP. but
if you are running it remotely make sure to add the IP or hostname under [wordpress-server]

2. site.yml
– Thi is site configuration and will apply only for all machines mentioned in [wordpress-server] and applied those by
roles.

3. roles
– This folder has different roles which has details about different components require for wordpress installation.
– common: You can add common software here. I have added epel-release repo and git there. So if you are using the ansible config from seperate ansible it will install require software on local macine if you are not installing it from vagrant provision script.

– apache: Installs httpd apache. Make sure to update. Custom config file for it is at templates/vhost.conf.j2

– mysql: Installs mysql related sofrware and adds the configuration for it. my.cnf file is specified in
templates/my.cnf.j2
– wordpress: Downloads and installs wordpress. Custom config file for it is at templates/wp-config.php.j2 as as well as
sql dump file. This mysqldump has been created after insall process (seperately) in order to get plain site page to
take out UI steps of going through install process after wordpress installation.

Configurations and credentails:

Configuations are mentioned in : /group_vars/all

wordpress console login :

Admin User : admin

Password: q1w2e3r4

Api script login :

Username: automation

Password: q1w2e3r4

Why Nginx faster then apache?

The main difference is its architecture. How it was designed to handle HTTP request.
Apache:

Apache creates processes and threads to handle additional connections.
We can configure the server to control the maximum number of allowable processes. But if this is creased outside of server capacity, too many processes exhaust memory and can cause the machine to swap memory to disk, severely degrading performance. Plus, when the limit of processes is reached, Apache refuses additional connections.

Nginx:
Nginx handles the HTTP request asynchroneously and its event based.
It also do not create new process per request instead it has no of process defined (.e.g 4 total nginx processes) during start time and it creates threads out of it. Each of these processes is single-threaded. Each worker can handle thousands of concurrent connections. It does this asynchronously with one thread, rather than using multi-threaded programming.

Linux find cpu information in details:

Script to get cpu info in details.

 

PHYSICAL_CPU=`grep ‘physical id’ /proc/cpuinfo | sort | uniq | wc -l`
VIRTUAL_CPU=`grep ^processor /proc/cpuinfo | wc -l`
CORES=`grep ‘cpu cores’ /proc/cpuinfo | head -1 | cut -f2 -d “:”`
MEMORY=`cat /proc/meminfo  | head -1 | awk ‘{printf “%.0f”,$2/(1024*1024)}’`
CPU_SPEED=`grep “^cpu MHz” /proc/cpuinfo | head -1 | awk -F”:” ‘{printf “%0.2f”,($2/1000)}’`
CPU_CACHE_SIZE=`grep “^cache size” /proc/cpuinfo| head -1 | awk -F”:” ‘{print $2}’`
KERNEL=`uname -m`

ARCHS=`grep flags /proc/cpuinfo | uniq | egrep -o -w “tm|lm” | wc -l`

if [ ${ARCHS} -eq 2 ]
then
SUPPORTED_ARCH=”x86_64,x86″
else
SUPPORTED_ARCH=”x86″
fi

echo “Hostname: $HOSTNAME”
echo -n “Physical Processors: ”
echo ${PHYSICAL_CPU}

echo -n “Virtual Processors: ”
echo ${VIRTUAL_CPU}

echo -n “CPU Cores: ”
echo ${CORES}

echo -n “CPU Speed: ”
echo “${CPU_SPEED} GHz”

echo -n “Cache Size: ”
echo “${CPU_CACHE_SIZE}”

echo -e “Memory: ${MEMORY}G”

echo “Kernel Arch: ${KERNEL}”

echo “CPU Arch: ${SUPPORTED_ARCH}”

echo “Notes:”
if [ ${CORES} -eq 1 -a ${VIRTUAL_CPU} -gt ${PHYSICAL_CPU} ]
then
echo -e “\tCPU is Hyperthreading”
fi

if [ ${ARCHS} -eq 2 -a `echo ${SUPPORTED_ARCH} | grep -c ${KERNEL}` -eq 0 ]
then
echo -e “\tHardware is 64-bit while installed kernel is 32-bit”
fi

Difference between prefork and worker apache mpm module

MPM stands for Multi Processing Module which extends apache’s capability to implement hybrid multi processing multi threading in apache web server.

Default mpm can be checked with httpd -l or apachectl -l

The default MPM for Unix is the Prefork module.
The Worker MPM was introduced in Apache2.

Before I explain the difference between Prefork and worker its necessary to understand how it works. So let’s see how it works.

  • Prefork MPM

Working operation : – A single control process is responsible for launching child processes which listen for connections and serve them when they arrive. Apache always tries to maintain several spare or idle server processes, which stand ready to serve incoming requests. In this way, clients do not need to wait for a new child processes to be forked before their requests can be served.
We can adjust this spare process through the apche conf. For a normal server which is having 256 simultaneous connections can use the default prefork settings.

Perfork is the default module given by apache.

# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild directive sets the limit on the number of requests that an individual child server process will handle. After MaxRequestsPerChild requests, the child process will die. If MaxRequestsPerChild is 0, then the process will never expire

As the name suggests this will pre fork necessary child process while starting apache. It is suitable for websites which avoids threading for compatibility for non-thread-safe libraries . It is also known as the best mpm for isolating each request.

  • Worker MPM