Tidal Workload Automation Adapter for Amazon EC2 and S3 

Adapter Overview

The Tidal Workload Automation Adapter for Amazon – both AWS EC2 and AWS S3 – gives you the capability to automate public cloud resources.

Managed Amazon Elastic Compute Cloud (EC2) resources include start, stop, and delete instances and Amazon Machine Images (AMIs) and Elastic Block Store (EBS) volumes. Tidal also allows the management of Amazon Simple Storage Service (S3) storage buckets and data copy and move processes.

Amazon S3

The Tidal Amazon Adapter automates the execution of AmazonS3 jobs as a part of Tidal’s managed process. It is an API-level integration solution. As a platform independent solution, the Adapter can run on any platform where the Tidal master runs.

An AmazonS3 Adapter job can be used to create buckets that can store large volumes of data. It also provides authentication mechanisms to ensure that data is kept secure from unauthorized access, objects can be made private or public, and rights can be granted to specific users.

The Adapter provides the following features.

  • Connection/Availability – reports the connectivity status.

  • Data organization – allows user to create a bucket, move/copy/rename/restore objects from one bucket to another bucket or within the same bucket.

  • Data security for bucket – allows user to configure ACL permission to a bucket.

  • Data storage and retrieval – can be integrated with Data Mover Job to allow user to upload/download objects(s).

  • Data protection – allows you to specify server-side encryption while uploading/ downloading object(s) from/to the Amazon S3 Bucket Amazon S3 service. APIs allow the AmazonS3 Adapter to make requests to Amazon S3 service.

  • Data security for object – allows user to configure ACL permission to an object. User can set ACL permission for an object.

  • Free/cleanup data storage (objects) – allows user to delete objects and bucket.

The Tidal Workload Automation Adapter for AmazonS3 uses the AWS SDK, which is a set of application programming interfaces (APIs) that allow interaction with the Amazon S3 service. The APIs allow the AmazonsS3 Adapter to read, write and delete objects, manage lifecycle of buckets and objects, configure or update bucket and object properties, and manage files within Amazon S3 buckets. 

Amazon EC2

Amazon EC2 is an Amazon Web service that provides scalable compute-capacity in the cloud. Specifically, EC2 provides the ability to run virtual machine (VM) instances on Amazon's cloud infrastructure. EC2 is elastic, which means instances can be spun up or down on the fly. EC2 is also configurable. You can choose from a large set of VM images that come pre-installed with the latest operating systems and applications. Or, you can upload your own custom VM images to EC2. You can also choose the exact CPU, memory, and storage configurations to suit your computing needs.

The Tidal Amazon Adapter, in conjunction with Tidal Workload Automation, can be used to define, launch, control, and monitor jobs submitted to EC2 on a scheduled basis. The Adapter integrates seamlessly in an enterprise scheduling environment. Tidal's Amazon Adapter includes the following features: 

  • Connection management to monitor system status with a live connection to the EC2 Server.
  • EC2 job management with following: 
    • Scheduling and monitoring of jobs from a centralized work console with Tidal Workload Automation.
    • Defined dependencies and events with Tidal Workload Automation for scheduling control.