Skeddly Blog

Skeddly news and announcements...

Partitioning Your Data With Amazon Athena

Amazon Athena pricing is based on the bytes scanned. Anything you can do to reduce the amount of data that’s being scanned will help reduce your Amazon Athena query costs.

In our previous article, Getting Started with Amazon Athena, JSON Edition, we stored JSON data in Amazon S3, then used Athena to query that data.

In this article, we will partition the data, and compare the results.


Using Compressed JSON Data With Amazon Athena

Amazon Athena pricing is based on the bytes scanned. That is a little ambiguous. To clarify, it’s based on the bytes read from S3. It’s not based on the bytes loaded into Athena.

So, you can reduce the costs of your Athena queries by storing your data in Amazon S3 in a compressed format.

In our previous article, Getting Started with Amazon Athena, JSON Edition, we stored JSON data in Amazon S3, then used Athena to query that data.

In this article, we will compress the JSON data, and compare the results.


Getting Started with Amazon Athena, JSON Edition

At AWS re:Invent 2016, Amazon announced Amazon Athena, a query service allowing you to execute SQL queries on your data stored in Amazon S3. Athena can query against CSV files, JSON data, or row data parsed by regular expressions. Using Amazon Athena, you don’t need to extract and load your data into a database to perform queries against your data.

Amazon Athena is not a full CRUD database system. It can only query data. So you basically can only perform SELECT queries on your data in S3. You cannot execute any INSERT, UPDATE, or DELETE queries.

We wanted to do some experimenting with Athena to see what it can do.


Create CloudTrail Trails Action

Whether you are a small AWS user or a large enterprise, security and auditing should be front-of-mind when it comes to your AWS account.

AWS CloudTrail provides a way to record all AWS API calls. Logs are saved to an Amazon S3 bucket for you. You can read and process the logs yourself, or you can use one of the many third-party tools and services available to you.

Today, we’re happy to announce a new action available to you: Create CloudTrail Trails.


AWS re:Invent Lanyard Colours

This week is AWS re:Invent. It’s bigger than ever, spanning 3 hotels: The Venetian, The Mirange, and The Wynn & Encore. So make sure you have some good walking shoes.

As you’re walking around AWS re:Invent this year, you’ll notice different coloured lanyards holding everyone’s badges. Ever wondered what the different colours mean? Here’s the list.


Joining Elastic Beanstalk to AWS Directory Service

AWS Directory Service is a managed Microsoft Active Directory solution. Using this service, you can manage the identities for your organization, and by joining EC2 instances to your directory domain, authorize access to your AWS resources.

AWS Elastic Beanstalk is a deployment service for your web applications. By configuring your Elastic Beanstalk environment, AWS can manage the scaling and high-availability for your web application. Supported environments include NodeJS, Ruby, and .NET, among others.

If you have an existing .NET application, or are building one that utilizes “Windows Authentication”, you can deploy your application using Elastic Beanstalk, and use Directory Service as your Windows authentication directory.

Ideally, we would implement this combination using off-the-shelf Elastic Beanstalk AMI images. However, due to an issue with Elastic Beanstalk hostnames being re-used, we’ll need to create a custom AMI image to be used by our Elastic Beanstalk applications.

The custom AMI will only address the hostname issue. This way, once AWS fixes it on their end, the custom AMI can be done away with.

We have employed these techniques in our own internal applications and we wanted to share them with anyone else who wanted to do the same.

This article will walk you through these steps:

  1. Setting up the domain.
  2. Creating the SSM document used to join EC2 instances to the domain.
  3. Configuring your ASP.NET application for Windows Authentication, and installing Windows Authentication on your EC2 instances.
  4. Configuring your ASP.NET application to join your domain.
  5. Creating an IAM Role for your EC2 instances.
  6. Creating a custom AMI.
  7. Creating your Elastic Beanstalk application.


Action Exclusions

Today, we are excited to unveil Action Exclusions. Action Exclusions are a way to exclude certain AWS resources from being acted upon by Skeddly actions. This allows you to take a subset of your AWS resources out of Skeddly control for a short or long period of time.

For example, you can exclude an EC2 instance from being stopped by Skeddly for the next 12 hours.


Identity Federation Using SAML

Identity Federation is the ability to access Service Providers (such as Skeddly) using an existing Identity Provider (such as Active Directory Federation Services). It uses your existing users when accessing the Service’s resources, and eliminates the need to provision new user credentials for the Service Provider.

Today, we’re happy to announce that you can now use your existing SAML 2.0 compliant Identity Providers to access your Skeddly accounts.


Visit Skeddly at AWS re:Invent 2016

For the second year, we’re excited to be sponsoring AWS re:Invent in Las Vegas.

Where: AWS re:Invent 2016, Booth #919

When: November 28 - December 2, 2016

This year’s event is expected to be bigger and better than last year’s. There is an entire extra day of sessions, a bigger expo, and the return of “reserved seating” (ooooooooh).

Skeddly will be at booth #919 in the expo hall not far from the main AWS booth. Stop by the booth to:

  • see a demonstration of Skeddly,
  • get some one-on-one training,
  • get personal assistance setting up your Skeddly account,
  • have your questions answered about Skeddly and AWS in general,
  • have friendly conversation, and
  • pick up some cool Skeddly stickers.

Delete EBS Snapshots Enhancements

Skeddly includes an action called “Delete EBS Snapshots” which can be used to remove old EBS snapshots based on matching criteria and minimum age. It can also ensure that a minimum number of EBS snapshots are preserved.

Today, we’ve made two new enhancements to this action available:

  • We have improved the user interface for this action so that it is clearer as to what will be happening.
  • We have added a new delete scheme called “Grandfather-Father-Son” which is a more advanced selection process for deciding which snapshots will be preserved.