Common Issues When Running the AWS CLI using CRON & Bash

This post will show some of the common issues experienced when running the AWS CLI via Bash using CRON on Linux. The issues are typically related to either a Bash script error, AWS CLI command error or AWS CLI credentials/config error.

Why would you run the AWS CLI via Bash using CRON?

There are quite a few reasons! The AWS CLI complements your CRON jobs allowing you to do some amazing things. For example, your Bash scripts can AWS SNS notifications. You can download your latest maintenance scripts from AWS S3. Perhaps you want to determine the status of another AWS EC2 instance? The opportunities are endless.

Debugging Your Scripts

You can easily debug the Bash and AWS CLI scripts!

To debug your Bash script –

Enable Bash to debug mode by adding “-x” to the Bash shebang. So you change “#!/bin/bash” and add a ” -x”.

Example:

#!/bin/bash -x
...
...

The Bash script will get verbose and show the execution steps.

To debug your AWS CLI command –

Set the AWS CLI to debug mode requires the following command “–debug“.

Example using the AWS CLI to copy files from S3:

sudo aws s3 cp s3://server-bash-library/maintenance /home/user/scripts/maintenance --recursive --debug

The AWS CLI will get verbose and show the execution steps.

AWS Credentials and Config Configuration

Check your configuration of the AWS credentials and config files. These files are available under ‘/home/yourUser/.aws’. You can use the ‘ls –altr‘ command to view the hidden folder. The ‘
/home/yourUser‘ portion of this folder is represented by a ‘~‘.

The ‘.aws’ folder must contain two configuration files:

~/.aws/credentials should look like this:

[default] 
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

~/.aws/config should look like this:

[default] 
region=us-west-2
output=json

You can configure the files when you run the ‘aws configure’ command.

AWS Credentials and Config Accessibility

If your Bash script works when you run it directly, but not from CRON, then there is probably something different in the environment. One such different – The AWS CLI stores the credentials and config under the installer’s profile (example: ‘/home/yourUser/.aws’).

Remember, that CRON does not execute as your user!

The CRON user’s home folder will not be the same as yours. As such, the CRON user will not able to read the ‘.aws’ folder.

Luckily the AWS CLI supports environment variables!

The AWS CLI supports the following environment variables:

  • AWS_SHARED_CREDENTIALS_FILE – Specifies the location of the file that the AWS CLI uses to store access keys.
  • AWS_CONFIG_FILE – Specifies the location of the file that the AWS CLI uses to store configuration profiles.

The ‘printenv’ command will show you the current environment variables.
The ‘eval echo ~$USER’ command will show you your current home folder. My home folder is ‘/home/ubuntu’.

The following commands will allow you to add the required environment credentials and config to your environment variables. Change the script to use your home folder. BACKUP your /etc/environment file before making changes!

echo 'AWS_SHARED_CREDENTIALS_FILE="/home/ubuntu/.aws/credentials"' >> /etc/environment    
echo 'AWS_CONFIG_FILE="/home/ubuntu/.aws/config"' >> /etc/environment

The ‘>>’ redirects the output of the command on its left-hand side to the end of the file on the right-hand side. The ‘>>’ replaced by ‘>’, will replace everything! We do not want to do this.

Lastly, check your syslog for any errors from your CRON job running your Bash script. You can use this handy command ‘sudo grep —colori cron /var/log/syslog’, to help you along!

Need More Help

Click here to see the lessons learned by this a fellow blogger. The blog by Adeel Mufti shows you how to specify the path to the CLI in Bash. As an added bonus – read the blog post by Kevin Dees about setting up server level CRON backups. Lastly, putorius.net has a fantastic article about enhancing your Bash scripts with Exit Traps!

Getting the AWS CLI to run via CRON can be tough, but it’s worthwhile! Post your feedback and let others know how you did this. Sharing is caring!

I hope you liked my post! Click here to see more great posts. Anto will love to hear from you, so feel free to post a comment or like this post. Share this post, and you shall receive millions of years of good Karma. Alternatively, Why not click here to follow me on Twitter?



About the Authors

Each member of Anto's editorial team is a Cloud expert in their own right. Anto Online takes great pride in helping fellow Cloud enthusiasts. Let us know if you have an excellent idea for the next topic!

Support the Cause

Support Anto Online and buy us a coffee. Anything is possible with coffee and code.

Buy me a coffee



Leave a Reply

Your email address will not be published. Required fields are marked *