Glue

  •  AWS Glue is a serverless, fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores and data streams. 
  • AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog.
  • AWS Glue is designed to work with semi-structured data.
  • You can use AWS Glue to organize, cleanse, validate, and format data for storage in a data warehouse or data lake. 
  • Glue discovers and catalogs metadata about your data stores into a central catalog. 
  • Populates the AWS Glue Data Catalog with table definitions from scheduled crawler programs.
  • AWS Glue can catalog your Amazon Simple Storage Service (Amazon S3) data, making it available for querying with Amazon Athena and Amazon Redshift Spectrum.
  • You can run your ETL jobs as soon as new data becomes available in Amazon S3 by invoking your AWS Glue ETL jobs from an AWS Lambda function.
  • Glue supports data sources - S3, RDS, Dynamo, Mongo, Document DB, 3rd party JDBC accessible DBs. Crawlers can pull data from sources and update Data Catalog with metadata. AWS Glue can generate a script to transform your data. 
  • Glue can access Kafka or Kinesis data streams.
  • You can run your job on demand, or you can set it up to start when a specified trigger occurs. The trigger can be a time-based schedule or an event.
  • Tables and databases in AWS Glue are objects in the AWS Glue Data Catalog. They contain metadata; they don't contain data from a data store. Similar to Hive metastore.
  • Each AWS account has one AWS Glue Data Catalog per region.
  • A data store is a repository for persistently storing your data. Examples include Amazon S3 buckets and relational databases. A data target is a data store that a process or transform writes to.


















  • The AWS Glue Schema Registry is a new feature that allows you to centrally discover, control, and evolve data stream schemas. A schema defines the structure and format of a data record. 

Comments

Post a Comment

Popular posts from this blog

AWS Organizations, IAM

Image
Active Directory AD Connector is a directory gateway with which you can redirect directory requests to your on-premises Microsoft Active Directory without caching any information in the cloud. AD Connector comes in two sizes, small and large. You can spread application loads across multiple AD Connectors to scale to your performance needs.  AD Connector is your best choice when you want to use your existing on-premises directory with AWS services. AD Connectors and your on-premises AD domains have a 1-to-1 relationship.  Your end users and IT administrators can use their existing corporate credentials to log on to AWS applications such as WorkSpaces, Amazon WorkDocs, or Amazon WorkMail. Simple AD is a standalone managed directory.  Small - Supports up to 500 users,  Large - Supports up to 5,000 users.  Simple AD does not support features such as multi-factor authentication (MFA), trust relationships with other domains.  AWS Directory Service lets you run Microsoft Active Directory (AD)
Read more

Key Concepts

A  resource group  is a collection of AWS resources that are all in the same AWS Region, and that match the criteria specified in the group's query.  Tag-based queries include lists of resources and tags.  In an AWS CloudFormation stack-based query, you choose an AWS CloudFormation stack in your account in the current region, and then choose resource types within the stack that you want to be in the group.   You can use  resource groups  to organize your AWS resources. AWS Resource Groups is the service that lets you manage and automate tasks on large numbers of resources at one time.  With Resource Groups, you can create a custom console that organizes and consolidates information based on criteria specified in tags, or the resources in an AWS CloudFormation stack.  Transfer Family - Supports connecting to S3 or EFS through FTPS/SFTP/FTP protocols.  The hostname for the endpoint can also configured with R53.  The transfer family can be associated with Elastic IP. Security group pe
Read more

Linear Algebra Concepts

Vector Spaces Rank of matrix is the number of non zero rows in REF form. Alternatively rank of matrix is = total number of columns - total number of dependent columns. In the REF form, If a column of a matrix can be expressed in terms of other columns then it is a dependent vector. The vectors on the left most side with leading 1 at the col number position ( 1 at 1st row 1st col, 1 at 2nd row 2nd col, likewise) then such columns are linearly independent columns. For linerarly independence a1v1+a2v2+a3v3=0 where v1,v2,v3 are vectors and a1,a2,a3 are scalars. Span of a vector space is all the possible vectors which can be formed using the basis vectors. Basis vectors are linearly independent vectors which can span the vector space. Column space is all possible set of vectors formed by the column vectors. Row space is all possible set of vectors formed by the row vectors. Dimension of a row space or column space is equal to its rank. Null space or kernel of a matrix is any vector x which
Read more