Dynamic instance type information

[cristim]

Github issue

Issue type

• Feature Idea

Summary

Currently AutoSpotting ships with a static instance type database originating from ec2instances.info. This is very efficient but it only supports commercially available regions, may get stale over time and requires software updates to get support for newer instance types.

Ideally this data should be fetched periodically and cached by the Lambda function in its scratch area.

One option could be to just fetch it from somewhere at startup and update it daily, or alternatively query this data directly from the AWS pricing API.

[gjmveloso]

The new (last year) API DescribeInstanceTypes along with Query API for AWS Pricing allow us to fetch prices and instance attributes in realtime.

[cristim]

Thanks @gjmveloso, I'm well aware of those APIs, it's just that last time I tried they werr relatively slow and resource-intensive, not very well suited for using them from a Lambda function, unless we can somehow cache the data between runs assuming the Lambda function is kept warm.

The data we get from ec2instances.info is generated using those APIs pretty frequently but it's much more space and time efficient for running from a Lambda function.

The current drawback is the fact that data and code are currently coupled so we need to upgrade the code when upgrading the instance type data. The next thing on my to-do list is to decouple them and potentially offer a data upgrade service to my paying subscribers if they express significant interest in this.

[gjmveloso]

Never heard about https://github.com/banzaicloud/cloudinfo before. It may be a potential prior art work that could be used here

[cristim]

Thanks,I'll have a look at this when I get to work on this issue

[cristim]

I've started working on this functionality a while back and added support for updating instance type data in the https://github.com/LeanerCloud/ec2-instances-info library we use.

I also looked into the Banzaicloud solution but at a first glance it seemed tightly coupled with their Pipeline setup so decided not to use it, or at least in the first iteration.