hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 295d ago [edited]

Thanks Russell! The AWS docs don't always have all the sample code you'd like to see, especially when it comes to filters. This bit of code helped me with the 'tag:key' filters which I was having trouble generating based on boto3 documentation. Nice Job :)

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 224d ago [edited]

Russell,

As with the other poster thank you for the examples, they helped a lot. However I am looking for one other function... and I may have to split it up... but since these commands have to traverse the internet before producing output, they are very costly and I would like to use it only once.

That being said... my goal is to use describe_instances to display the configuration of multiple instances where I know the instance-id of approximately 8 and only specific tags of 2-3... I was trying to do something like this:

client=boto3.client("ec2")
filter1={"Name":"tag:Name","Values":["my_tag"]}
filter2={"Name":"instance-id","Values":["i-xxxxxxxxxxxxxxxxx"]}
client.describe_instances(Filters=[filter1,filter2])["Reservations"])

Unfortunately it did not work as planned. I believe what is happening is that "and" logic is applied when using the two filters this way. What I am looking to accomplish is an "or" statement so that I can get what I need without running such an expensive command more than once.

Do you know of such a way?

Thanks,

Scott

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 224d ago

How many instances do you have in this AWS account and region?

To solve this, I think I would:

  • request all instance descriptions
  • write my own filter function / algorithm using Python

I also cannot seem to find anything in the docs which describe an "or" functionality.

In the docs, Collection methods are chainable which seems to lends itself to an "and" behavior.

hide preview

What's next? verify your email address for reply notifications!

russell 7y, 224d ago

I suppose I would like to hear more about your goal you are attempting to solve and less about this specific problem.

remark link parent
hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 224d ago

I am trying to write a backup script that... is largely unnecessary for a customer to back up their 8 auto-scaling instances... (although I keep telling them that there is no benefit to backing up these instances.) I only need to backup one instance from each group, since they are all the same.

I can get one instance ID for each group from the describe_auto_scaling_groups() function. that is why I know the Instance ID's of some instances.

However (and comparatively more importantly) their 2-3 staging instances, which may or may not be destroyed and recreated from time to time (causing their IDs to change) needs to be identified via their tag. Once I have these instance IDs I will use them to get their attached EBS volumes (likely only the staging instances) and store them together in a dictionary. Then I plan to take that dictionary to make snapshots for every instance.

I ended up coming to the same conclusion you did as well, and created my own function to parse the output of the describe_instances() function.

sometimes I have trouble translating my English to Japanese, and that may be the cause for the customer not understanding that there is no need to backup the auto-scaling instances... but since we are still in the design phase I'm sure that the requirements will change.

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 224d ago [edited]

I'll share the code I've written so far.

I only began coding recently so there may be inefficiencies but feel free to have a look.

http://pastebin.com/HNGvZ8j7

remark link parent
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 222d ago [edited]

I think you are on the right track.

Personally I prefer to use the Boto3 "resource" connection over the lower level "client" connection, where possible.

With the ec2 resource you can ask for every instance object in the aws account / region. To know if an instance is autoscaling, you can look for the 'aws:autoscaling:groupName' tag/key. The value of this key is the autoscaling group name.

This will simplify your calls to AWS API.

  1. get all instance objects: EC2.ServiceResource.instances
  2. iterate over objects collecting one from each autoscaling group (store in a list)
  3. iterate over list of instance objects needing a backup, access their volumes property and fire a snapshot
  4. ???
  5. profit
remark link parent

load more (2 remarks)
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 222d ago [edited]
>>> for instance in ec2.instances.all():
...     print(instance.id, instance.tags, instance.volumes)
remark link parent
hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 217d ago

I like your code there it is much simpler than navigating through the dictionaries.

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 217d ago [edited]

And point 4. is:

  • Integrate the new snapshots(AMIs) with their auto-scaling group's AMI
  • Delete Old AMI's

... Although unless the customer has a valid reason for wanting to backup auto-scaling instances... I'm pretty sure I'll be convincing them to only backup their staging instances.

I'm think going to re-write the code using the resource connection. It sounds much easier to read, write, and understand.

remark link parent
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 217d ago

perfect, glad I could help!

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 215d ago [edited]

how would you make the filter case insensitive?

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 210d ago

Personally, I would fix my tags to lowercase (except for the Name tag). I would fix / normalize the data.

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 210d ago [edited]

Can I exclude certain tags? Say I want to exclude instances which have a certain tag (which might, for example, indicate that they are to be scaled down), how would I do it?

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 210d ago [edited]

I don't think this is currently possible using Boto3 (https://github.com/boto/boto3/issues/173). There might be a exclusion filter for Boto2. My suggestion is to fetch instances and then do the filtering yourself using Python code if you really want to use Boto3.

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 201d ago [edited]

Thank you so much, this took me forever to find this blog post, but now my code works!

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 200d ago

You are very welcome!

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 189d ago

Thank you ! Unfortunately as a newbie to coding the Boto3 documentation is very poor and confusing. Once again thank you for some clarity

Mario

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 162d ago

Do you have incantations to return the value of a particular tag for a particular EC2 instance queried by name or instance-id? I promise I'm not asking you to answer my homework question. ;)

Right now I'm just using describe_instances with a Filter= on the tag:Name but that gives me the entire set of instance metadata when I really only care about one particular tag.

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 92d ago [edited]

Hi, I want to filter the particular instance & want to check whether particular instance is Running or not in AWS by using python boto3. You have any idea about this.?

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 7y, 91d ago [edited]

Boto3 has an Instance object which has a method called state: EC2.Instance.state

You can use normal Python to test if it is running.

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 54d ago

How can I Tag an instance, while creating with boto3? I am doing this way.

instances = ec2.create_instances( ImageId=image_dict[image_name], MinCount=1, MaxCount=1, SecurityGroupIds=[securitygroup_dict[security_group]], SubnetId=subnet_dict[subnet_name], InstanceType="t2.micro" ) I want to add a tag to the instance

hide preview

What's next? verify your email address for reply notifications!

unverified 7y, 49d ago

How can I get all details with out filtering like Group Name, Description etc. in Security group?

hide preview

What's next? verify your email address for reply notifications!

unverified 6y, 308d ago

Hi As I see to get the instance metadata i need to use the filter function. but can it be used in a lambda function. Means I am using ec2 = boto3.resource('ec2', region_name='us-west-2') instances=ec2.instances.filter(Filters=[{'Name':'instance-state-name','Values':['running']}]) for instance in instances: print(instance.id,instance.instance_type) I am not getting any result in my lambda function

hide preview

What's next? verify your email address for reply notifications!

unverified 6y, 244d ago

Thanks Russell! It does so much help!!

hide preview

What's next? verify your email address for reply notifications!

unverified 5y, 354d ago [edited]

I want to list snapshots whose start time is less than 'somedate'. How do I mention <(less than) in filters of describe_snapshots

ec2.describe_snapshots(

         Filters=[         
     {
         'Name': 'start-time'}]
remark link
hide preview

What's next? verify your email address for reply notifications!

russell 5y, 354d ago

You can't do it with a Boto3 filter, but you can simply request all snapshots and filter the description documents yourself using Python.

hide preview

What's next? verify your email address for reply notifications!

S9Vo4NAP 5y, 314d ago

Hi Rusell,

I am using lambda with max 512 MB of memory. Lets say when i describe ecs services, is there a way I can filter result. Because I don't want entire description of service. Probably running count and pending count is all good for me. Yes I can write my own function to cascade with results, but I want to save compute time on lambda. Any suggestions ?

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 5y, 314d ago [edited]

As far as I can tell the boto3 client for ECS does not support the ability to trim down the response document.

I think if I wanted to speed up execution I would have a separate service to query and then cache the results. I would then query the cache instead of working directly with real time data. I'm not sure if your problem can deal with slightly stale data.

My questions for you:

  • How long is your AWS Lambda execution time right now?
  • How certain are you that waiting for the ECS service descriptions is the slowest part of the current implementation?
  • How often does your AWS Lambda run?
  • If you could have instant ECS service descriptions, how much would you really save?
  • Is this really the best cost saving problem you could be working on?

I'm naive of why you want to speed up execution times. If you want to save on cost, I personally wouldn't bother. Engineering a robust solution to speed up execution time is likely going to cost more than what your Lambda bill will be.

If it's to speed up a long pipeline of dependent tasks, I think I would try engineering some sort of caching service instead of working directly with AWS API responses.

hide preview

What's next? verify your email address for reply notifications!

unverified 5y, 47d ago

excellent !

hide preview

What's next? verify your email address for reply notifications!

unverified 4y, 332d ago

is it possible to filter a tag name whose value might be 'webapp1' and 'webapp2' but not 'webapp3'(!= 'webapp3') how can we add a not in the filter statement?

remark link
hide preview

What's next? verify your email address for reply notifications!

russell 4y, 260d ago

I would ask the API for webapp* and filter the result using Python.

hide preview

What's next? verify your email address for reply notifications!

unverified 4y, 260d ago

Hi Russell,

Is there any way we can get all the openshfit cluster using boto3 from AWS.

remark link
hide preview

What's next? verify your email address for reply notifications!

unverified 4y, 260d ago

or you can give any link where using AWS CLI or python i can get all openshift cluster from AWS, My motto is to know cluster creation date and want to calculate cost based on that

hide preview

What's next? verify your email address for reply notifications!

unverified 4y, 115d ago

Hi Russell,

How do i filter particular fields from a response.

What if i want to print particular fields from a describe_image response.

response = client.describe_images(ImageIds=[ami]) So reponse being a dictionary which captures the response, what if i want extract specific fields like imageid, imagelocation what shall i do

Thanks in advance

hide preview

What's next? verify your email address for reply notifications!

unverified 3y, 305d ago

How can I filter all ec2 instances in all regions based on the ec2 tag?

hide preview

What's next? verify your email address for reply notifications!

unverified 3y, 41d ago

I want to use two filter first will search name of the instances whose name started by ram and then if it is stopped , is this right ? ec2_result = ec2.describe_instances(Filters=[ {'Name': 'tag:Name', 'Values': ['?ram*'], 'Name': 'instance-state-name', 'Values': [ 'stopped', 'running' ] }, ])

hide preview

What's next? verify your email address for reply notifications!