Glue

  •  AWS Glue is a serverless, fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores and data streams. 
  • AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog.
  • AWS Glue is designed to work with semi-structured data.
  • You can use AWS Glue to organize, cleanse, validate, and format data for storage in a data warehouse or data lake. 
  • Glue discovers and catalogs metadata about your data stores into a central catalog. 
  • Populates the AWS Glue Data Catalog with table definitions from scheduled crawler programs.
  • AWS Glue can catalog your Amazon Simple Storage Service (Amazon S3) data, making it available for querying with Amazon Athena and Amazon Redshift Spectrum.
  • You can run your ETL jobs as soon as new data becomes available in Amazon S3 by invoking your AWS Glue ETL jobs from an AWS Lambda function.
  • Glue supports data sources - S3, RDS, Dynamo, Mongo, Document DB, 3rd party JDBC accessible DBs. Crawlers can pull data from sources and update Data Catalog with metadata. AWS Glue can generate a script to transform your data. 
  • Glue can access Kafka or Kinesis data streams.
  • You can run your job on demand, or you can set it up to start when a specified trigger occurs. The trigger can be a time-based schedule or an event.
  • Tables and databases in AWS Glue are objects in the AWS Glue Data Catalog. They contain metadata; they don't contain data from a data store. Similar to Hive metastore.
  • Each AWS account has one AWS Glue Data Catalog per region.
  • A data store is a repository for persistently storing your data. Examples include Amazon S3 buckets and relational databases. A data target is a data store that a process or transform writes to.


















  • The AWS Glue Schema Registry is a new feature that allows you to centrally discover, control, and evolve data stream schemas. A schema defines the structure and format of a data record. 

Comments

Post a Comment

Popular posts from this blog

Key Concepts

A  resource group  is a collection of AWS resources that are all in the same AWS Region, and that match the criteria specified in the group's query.  Tag-based queries include lists of resources and tags.  In an AWS CloudFormation stack-based query, you choose an AWS CloudFormation stack in your account in the current region, and then choose resource types within the stack that you want to be in the group.   You can use  resource groups  to organize your AWS resources. AWS Resource Groups is the service that lets you manage and automate tasks on large numbers of resources at one time.  With Resource Groups, you can create a custom console that organizes and consolidates information based on criteria specified in tags, or the resources in an AWS CloudFormation stack.  Transfer Family - Supports connecting to S3 or EFS through FTPS/SFTP/FTP protocols.  The hostname for the endpoint can also configured with R53.  The transfer family ca...
Read more

Linear Algebra Concepts

Vector Spaces Rank of matrix is the number of non zero rows in REF form. Alternatively rank of matrix is = total number of columns - total number of dependent columns. In the REF form, If a column of a matrix can be expressed in terms of other columns then it is a dependent vector. The vectors on the left most side with leading 1 at the col number position ( 1 at 1st row 1st col, 1 at 2nd row 2nd col, likewise) then such columns are linearly independent columns. For linerarly independence a1v1+a2v2+a3v3=0 where v1,v2,v3 are vectors and a1,a2,a3 are scalars. Span of a vector space is all the possible vectors which can be formed using the basis vectors. Basis vectors are linearly independent vectors which can span the vector space. Column space is all possible set of vectors formed by the column vectors. Row space is all possible set of vectors formed by the row vectors. Dimension of a row space or column space is equal to its rank. Null space or kernel of a matrix is any vector x which ...
Read more

Cryptography

 Handling Secure Communications between Client and Server The client or application that wants to access your service will need an  API Key  and a  Secret Key  from you as the service owner. These keys are usually randomly generated strings and is given to the client beforehand. API Keys are unique to each client/application.  Both the client and server will hold the API Key and Secret Key. When the client makes a call to the API, the message content is hashed, using the secret key on the client, to generate a HMAC signature. This value, along with the original message and the API Key is then passed to the server’s API. Using the client API key, the equivalent Private key stored on the server is retrieved and the server also generates a HMAC signature which is compared with the one from client. Request is authorized when both match. Digital Signature The client holds the  Private Key  used to sign the message. The client will provide its public ke...
Read more