I frequently fire up a temporary Ubuntu server on Amazon EC2 to test out some package feature, installation process, or other capability where I’m willing to pay a few pennies for a clean install and spare CPU.
I occasionally forget that I started an instance and leave it running for longer than I intended, turning my decision to spend ten cents into a cost of dollars. In one case, I ended up paying several hundred dollars for a super-sized instance I forgot I had running. Yes, ouch.
Because of this pain, I have a habit now of pre-scheduling the termination of my temporary instances immediately after creating them. I used to do this on the instance itself with a command like:
However, this only terminates the instance if its root disk is instance-store (S3 based AMI). I generally run EBS boot instances now, and a shutdown or halt only “stops” an EBS boot instance by default which leaves you paying for the EBS boot volume at, say, $1.50/month.
So, my common practice these days is to pre-schedule an instance termination call, generally from my local laptop, using a command like:
The
at utility runs the commands on stdin with the exact same environment ($PATH, EC2 keys, current working directory, umask, etc.) as are in the current shell. There are a number of different ways to specify different times for the schedule and little documentation, but it will notify you when it plans to run the commands so you can check it.
After the command is run,
at returns the output through an email. This gives you an indication of whether or not the terminate succeeded or if you will need to follow up manually.
Here’s an example email I got from the above
at command:
I already have a personal custom command which starts an instance, waits for it to move to running, waits for the ssh server to accept connections, then connects with ssh. I think I’ll add a
--temporary option to do this termination scheduling for me as well.
You can get a list of the currently scheduled
at jobs with
You can see the commands in a specific job id with:
If you decide along the way that the instance should not be temporary and you want to cancel the scheduled termination, you can delete a given
at job with a command like:
I’ve been thinking of writing something simple that would regularly monitor my AWS/EC2 resources (instances, EBS volumes, EBS snapshots, AMIs, etc.) and alert me if it detects patterns that may indicate I am spending money where I may not have intended to.
How do you monitor and clean up temporary resources on Amazon AWS/EC2?
EC2 Scheduler has been superseded by AWS Instance Scheduler.
In 2016, the EC2 Scheduler was launched to help AWS customers easily configure custom start and stop schedules for their Amazon Elastic Compute Cloud (Amazon EC2) instances. In 2018, AWS launched AWS Instance Scheduler, a new and improved scheduling solution that enables customers to schedule Amazon EC2 instances, Amazon Relational Database Service (Amazon RDS) instances, and more. We encourage customers to migrate to AWS Instance Scheduler for future updates and new features.
Legacy templates, scripts, and documentation for EC2 Scheduler are available in this GitHub repository.
EC2 Scheduler is a simple AWS-provided solution that enables customers to easily configure custom start and stop schedules for their Amazon EC2 instances. The solution is easy to deploy and can help reduce operational costs for both development and production environments.
Source code for the AWS solution 'EC2 Scheduler'.
Cloudformation templates
Lambda source code
Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Amazon Software License (the 'License'). You may not use this file except in compliance with the License. A copy of the License is located at
or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License.
Have you ever forgotten to shut down your test instances that you just used for few days and got a surprise when the monthly AWS bill came? Do you want to stop your development instances automatically at night and weekends when you don’t use them? If your answers are yes, you will want to continue reading this post.
It is fairly easy to set up a schedule to start and stop your EC2 instances automatically to control your EC2 usage. There are quite a few 3rd party providers offering this service. If you search for “EC2 schedule” or “auto start stop EC2” in google or AWS forums, you will be able to see what is available. I list few of them in alphabetical order for your reference in case you are interested:
Backup Aws Ec2 Instance
Of course, there is a monthly subscription fee associated with these 3rd party services, and you also need to give out your access key and secret key to the 3rd party service provider. These service providers offer more feature than just auto start and stop EC2 instances. It is probably worth using them if you are planning to use other features as well. Otherwise, it may be hard to justify when you can do it on your own without revealing your credentials to a 3rd party provider.
In fact, AWS has provided all the APIs you need to manipulate EC2 instances. That’s how these 3rd party service providers are able to manage AWS resources. It is indeed pretty straight forward. All you really have to do is to put together a simple Python script to leverage these APIs with Boto, which are well documented and supported.
In this post, I will show you how to use a micro instance with an IAM role to start and stop your EC2 instances automatically within a region. Here are the 3 simple steps to get started:
STEP 1: Create an IAM Role
On the IAM Management Console, go to Roles and click Create New Roles to open a wizard to start the process to create a new role. In the screenshots below, I called the IAM role as ec2-operator. You can call it whatever you want as long as you can identify it.
Type a self-explanatory name to the Role Name field. It is used for identifying your IAM role. You will need to use the name later when you create a micro EC2 instance.
On the next screen, select AWS Service Roles and then Amazon EC2.
On the next screen, select Custom Policy.
Type an arbitrary name to the Policy Name field.
Paste the following JSON block to the Policy Document field to indicate the IAM role is allowed to perform describe instances, start instances and stop instances action.
Review the role information and click Create Role to finish. The new role should be created immediately.
STEP 2: Provision a micro EC2 instance
I recommend to use a micro EC2 instance. I think it is sufficient to handle what we need here to set up a cron job to run a Python script to process auto-start and auto-stop requests, but feel free to use a different instance type.
On the EC2 Management Console, click Launch Instance to start the process to provision a micro EC2 instance. Let’s call this instance as an EC2 operator.
I recommend to select the latest Amazon LinuxAMI, which has AWS tools, python and boto pre-installed.
You can pretty much take the default value for each step except on Step 3: Configure Instance Details and Step 6: Configure Security Group.
On the Configure Instance Details screen, you need to make sure you select the IAM role you created and assign it to the instance. In my case, I selected ec2-operator as the IAM role. You will not be able to change the IAM role for the instance once it is launched.
You can cut and past the following shell script into the User Data field. It will install few Python libraries and modules, create a Python script, and set up a cron job to execute it every 5 minutes. The shell script is set to operate instances for all regions. In other words, the script will go through each region to start and stop instances.
On the Configure Security Group screen, make sure you are not open the instance to the world (0.0.0.0/0) for the security group. In fact, you can get rid of the security rule. You don’t really need to SSH to the instance.
STEP 3: Tag your instances
The last step is go to EC2 Dashboard Console to tag your instances that you want to start and stop automatically at certain time. The EC2 operator instance looks for the auto:start and auto:stop tags to determine when to start and stop instances. The name of the tags are case-sensitive. The value of the tags should be in a cron format. For instance, 0 14 * * * is 2 pm everyday. The time is based on the OS time of the EC2 operator instance. If you have not changed the default time zone of the EC2 operator instance, it should be in UTC. It will ignore any instances that do not have the tags set or the format is invalid. The auto:start tag holds the start schedule, and the auto:stop tag holds the stop schedule. Do not tag the EC2 operator instance, otherwise it will get stopped by itself.
The script may not start or stop instances at the requested time exactly. For the start operation, it will start the instance between 30 minutes prior to the start time and the requested time. For the stop operation, it will stop the instance between the requested time and 30 minutes after the requested time. The EC2 operator instance is set to check every 5 minutes. It will process any auto-start and auto-stop requests few times if they have not started or stopped during the 30-minute window in case the operation does not go well.
I have been using the script for a while, and it has been performing pretty well. I am able to get all my development instances stopped during non-business hours. I am even able to check the operations with CloudTrail to audit if the instances were indeed started and stopped within few minutes of the requested time.
Advertisements
Is there an easy way to start and stop AWS EC2 instances at a given time each day? This could save me quite a lot of money for my development and test servers.
TimTim
19k44 gold badges1919 silver badges5151 bronze badges
2 Answers
Update
AWS have released a tool called the 'Instance Scheduler', including a full configuration guide which is linked from that page. It looks to be an enhancement of the EC2 Scheduler I describe below, with a few more features, but it's essentially the same thing.
The guide below will still work, but it's probably better to look at the instance scheduler for new installations.
Original Post
AWS have a tool called EC2 Scheduler that gives you very flexible control over starting and stopping EC2 instances.
The tool allows you to define default start and stop times when you set the tool up, which you can change later. You can choose which instances get controlled, you and you can specify different start and stop times for each instance using tags.
While it's a great tool the documentation is somewhat vague and confusing. It's like the documentation has been written by an engineer who wrote the tool and knows everything about it, rather than a technical writer.
Note : if you have feedback or corrections comments are appreciated. If you have a question based on this please start your own question.
What is EC2 Scheduler
This tool is a Lambda Function that works with Cloudwatch Events and DynamoDB. It's deployed using a Cloudformation template, which also sets up the necessary IAM roles and policies. You can read about the architecture here.
Deployment
Start by going to this page and clicking 'launch solution'. Right now the direct link is here, but it could change.
Select the region you want the resources deployed to at the top of the console. The script controls EC2 instances in any region, but it runs in one region.
Tagging EC2 Instances
This is covered in the documentation here, but it's not as simple as it could be.
You control which instances are started and stopped by tagging your instances.
The simplest case requires you to tag each EC2 instance you want to be started and stopped according to the schedule. To do this find your EC2 instance in the console, click tags, and create this tag
To enable copy and paste:
If you want to have specific instance started and stopped on a different schedule you append additional information to the tag key and value. For example if you want an instance to start at 1500 UTC and stop at 2400 UTC on Tuesday, Thursday and Friday you enter the following.
Key: scheduler:ec2-startstop:lateValue: 1500;2400;utc;tue,thu,fri
Note that the word 'late' can be any string, 'late' has no special meaning.
You can convert UTC to your local time using this tool.
You can use the tag editor to bulk tag instances. That could more easily let you set up bulk tagging, which could be useful for having different settings for dev, test, and production. I doubt you'd use this on production though.
CloudFormation Parameters
Auto Start Stop Ec2
When you run the CloudFormation template you have to enter a lot of parameters. Most you can leave at default. Here's some of the most important parameters
Permissions, Policies, and Roles
The Permissions / IAM role section of the CloudFormation template is a red herring - ie it's largely irrelevant. It specifies only the role used to run the CloudFormation script, it makes no difference to the resources created or the role used when the lambda function runs. In retrospect this is obvious, but it wasn't obvious to me when I started.
Whatever role you run this script as the same role and inline permissions are created within IAM. The Lambda function runs using an 'ec2 scheduler role' that the script creates.
I've included my policies below in case they're helpful for anyone.
CloudWatch Events and Metrics
If you want to see logs from your Lambda Function go into Cloudwatch Events. The logging is quite good. There are metrics as well, so you can see when it runs, time it runs for, etc.
Additional
The code for the lambda function is available on Github.
Policies
These aren't generally necessary, but could be for someone so I'll include them.
Policy for IAM Role
Trust policy for IAM role
TimTim
19k44 gold badges1919 silver badges5151 bronze badges
If you just want to start and stop instances, here's another take on this which also make use of the Lambda service. It assumes you want to control a specific instance id. You can control multiple instances by adding more ids separated by a comma. (ex: 'i-3453453','i-45656745'). You can find the id of your instance in the AWS console Instances section.
Code below
Code below
Code below
So if you’re a procrastinator, or a sleeper-inner, pick it up – and see how quickly you can morph those habits into habitual success. Do you want to know why Febreze became a bestselling product? When you get around to it, obviously.” ( Stylist)“In this fascinating book, Charles Duhigg reveals the myriad ways in which our habits shape our lives. The power of habit pdf indonesia.
Here you will create a CloudWatch Event that will trigger your Lambda function at night
Note: Cron expressions are evaluated in UTC. Be sure to adjust the expression for your preferred time zone. Here is an example that will run the function every day at 08:00 GMT/UTC):
To restart your instances in the morning, repeat these steps and use your preferred start time.If you want to send a mail message whenever the functions fail, you can set up an SNS topic and configure the sending of that message under Debugging in the Lmbda Function Creation Window.
The source of all this can be found here: AWS documentation
netfednetfed
Not the answer you're looking for? Browse other questions tagged amazon-ec2amazon-web-services or ask your own question.Comments are closed.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |