Amazon Q: Putting Amazon’s new AI assistant to the test
One of the new services that dropped at this year's re:Invent was Amazon Q, a new type of assistant powered by generative AI. Based on Amazon Bedrock, the new conversational assistant has been given the strapline "Transforming the way you build, optimize, and operate applications and workloads on AWS", which is a pretty bold claim for such a new tool.
Amazon Q has been popping up all over the AWS landscape, including the Management Console, the documentation, and even your favourite IDE. It has got us all wondering what it's capable of, and if it can make our lives any easier. In this blog post, we're going to dive a little deeper into Amazon's newest foray into AI and see what it can do, as well as where it falls short.
“Q” are you?
Amazon Q aims to be a cloud companion who can answer natural language questions about a wide variety of topics. The headline questions in the promotional material range from "What are the ways to build a web app on AWS?" to "Why can't I connect my EC2 instance using SSH?". The former of these questions feels like the kind of thing ChatGPT could answer, drawing on a hearty wealth of AWS documentation and slicing and dicing an answer for you into a neat little bullet pointed list. However, the latter piqued my interest, as it suggests Amazon Q is running a little deeper than just a cursory Google search. If Amazon Q has eyes on your AWS account and the things in it, it suddenly becomes a very powerful tool.
Colour me "Q-rious"
I started with its suggested question of not being able to reach my EC2 instance. Interestingly it was smart enough to know that I only have one EC2 instance in this account and quickly got to work running Reachability Analyzer for me. This was not just the usual Reachability Analyzer interface - it was instead wrapped up in a nice Amazon Q chat window.
It ran the tests for me, stepping through each of the controls that sit between me and my instance, and correctly concluded that my instance is in fact reachable on port 22. It would've been nice to see it offer some other debugging tips seeing as I had originally told it I couldn't reach my instance. Even a simple "Are you sure you're using the right SSH key?" might've helped, but I digress, AI Cardboard Programmers must be harder to create than I thought!
As slick as this interaction with Amazon Q was, it didn't offer anything new - it simply connected me to the right tool faster. In fact, Reachability Analyzer has been around for over 3 years now and a new page in the documentation even lists out all the parts of Reachability Analyzer that Amazon Q doesn't support yet. As helpful as it was to have Amazon Q hold my hand through the process, I wanted to see what else it had up its virtual sleeve.
"Se-Q-rity" in the Cloud
As best practices go, enabling MFA for your root user is one of the easiest and most fundamental ways you can help secure your AWS account, and I wanted to see if Amazon Q could check this for me. After a quick hop into the console of the account I wanted to check, I was soon presented with a list of helpful suggestions on how I could go about verifying this.
However, none of these got me to the boolean 'Yes' or 'No' answer I sought, so I pressed Amazon Q a little harder.
And there we have it, straight from the (virtual) horse's mouth. With no access to my account, Amazon Q isn't able to check the status of my root user's MFA. I imagine the security implications of giving Amazon Q access to each and every AWS account are massive, but the promotional material hasn't been particularly explicit with what its limitations are so I expect we’re not the only people prodding Amazon Q with these types of requests.
"Q"ing up the questions
The obvious competitor to Amazon Q (aside from your friendly neighbourhood cloud experts at Cloudsoft!) is ChatGPT, which has taken the world by storm in all sorts of unusual places. Whilst Amazon Q claims to have been trained on 17 years of AWS documentation, ChatGPT is AWS Certified, so this got us wondering who gives better advice. We posed a variety of questions to the two AI tools to see what their responses were.
I want to save money on my EC2 instances. What are the top three ways to achieve this?
Amazon Q:
- Use Reserved Instances
- Leverage Savings Plans
- Optimize configurations
ChatGPT:
- Use Reserved Instances
- Use Spot Instances
- Implement Auto Scaling
Whilst they're all good suggestions, ChatGPT has a lean towards "use less" whilst Amazon Q is suggesting we "commit more". Both are valid strategies, but I'd say Amazon Q's suggestions are easier to implement and would involve less code changes, so I'd give it the point here.
Winner: Amazon Q
How can I block traffic to my EC2 instance from a specific IP address?
Amazon Q:
- Use Security Groups
ChatGPT:
- Use Security Groups
- Use NACLs
Both AI models have fallen into the common trap of thinking Security Groups can deny traffic! Really poor answer from Amazon Q here - it even gave me a CLI command to add a rule to allow IP address 1.2.3.4, with a rule description of "Deny IP 1.2.3.4". If I’d blindly followed this I would’ve been in trouble! However, ChatGPT does expand on its answer and suggests NACLs as another option, so it gets the point on a technicality.
Winner: ChatGPT
How can I monitor COPY commands to my Redshift cluster?
Amazon Q:
- Check the SYS_LOAD_HISTORY view in the cluster
- Check the SYS_LOAD_ERROR_DETAIL view in the cluster
ChatGPT:
- Enable audit logging
- Check CloudWatch logs
- Integrate with other tools
ChatGPT really waffled on this one, giving me a lot of general monitoring and logging tips but not quite getting to the root of my question. Amazon Q on the other hand knew exactly what I wanted and even gave me the SQL to query these views. Top marks!
Winner: Amazon Q
How can I make my bastion host HA in AWS?
Amazon Q:
- “It looks like you need help with network connectivity issues. Amazon Q works with VPC Reachability Analyzer to provide an interactive generative AI experience for troubleshooting network connectivity issues."
ChatGPT:
- Deploy a bastion host in each AZ
- Use a load balancer to route traffic between them
Whilst there were some minor technicalities in ChatGPT's answer here, it at least made a good stab at the question and gave me a pretty sensible answer. Amazon Q on the other hand got a bit too excited to show me Reachability Analyzer again...
Winner: ChatGPT
How long should my function timeout be on a Lambda that is querying an SQS queue?
Amazon Q:
- Set your Lambda function timeout to a value shorter than the SQS queue's visibility timeout
ChatGPT:
- Ensure that the Lambda function timeout is longer than the visibility timeout
It all boils down to this! The AWS documentation on integrating Lambdas and SQS queues says:
To allow your function time to process each batch of records, set the source queue's visibility timeout to at least six times the timeout that you configure on your function. The extra time allows for Lambda to retry if your function is throttled while processing a previous batch.
which means Amazon Q is our winner!
Conclusion
Amazon Q is an interesting new tool with plenty of advice up its sleeve, and whilst it is mostly repeating AWS Documentation back to you, it does cut down on the amount of hunting you'd have to do to find your answer. The universal integration of Amazon Q in the Management Console and the documentation pages means it’s always available right where you need it, but having it baked into the browser window means the whole chat history vanishes with each page refresh. Having a step-by-step guide on how to fix your problem is great, but not so much when the first click wipes the remaining steps from your screen.
We also can't excuse some of Amazon Q’s rookie mistakes and must take its answers with the same pinch of salt given to all AI assistants at the moment. I will say that it was a little too heavy handed with the Reachability Analyzer on any question that mentioned 'networking' or 'host' and would hope to see this fixed before it comes out of preview.
It also would have been nice if Amazon Q could give me advice tailored to my account and specific setup, e.g. “Do any of my instances have open ports?” or “Are any of my volumes unencrypted?”, but who knows what wizardry AWS has in the pipeline.
Got some feedback?
As ever, we would be interested to hear about your experience with Amazon Q. Please email info@cloudsoft.io with your comments and suggestions or book a free session with one of our cloud experts.