Glue

  •  AWS Glue is a serverless, fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores and data streams. 
  • AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog.
  • AWS Glue is designed to work with semi-structured data.
  • You can use AWS Glue to organize, cleanse, validate, and format data for storage in a data warehouse or data lake. 
  • Glue discovers and catalogs metadata about your data stores into a central catalog. 
  • Populates the AWS Glue Data Catalog with table definitions from scheduled crawler programs.
  • AWS Glue can catalog your Amazon Simple Storage Service (Amazon S3) data, making it available for querying with Amazon Athena and Amazon Redshift Spectrum.
  • You can run your ETL jobs as soon as new data becomes available in Amazon S3 by invoking your AWS Glue ETL jobs from an AWS Lambda function.
  • Glue supports data sources - S3, RDS, Dynamo, Mongo, Document DB, 3rd party JDBC accessible DBs. Crawlers can pull data from sources and update Data Catalog with metadata. AWS Glue can generate a script to transform your data. 
  • Glue can access Kafka or Kinesis data streams.
  • You can run your job on demand, or you can set it up to start when a specified trigger occurs. The trigger can be a time-based schedule or an event.
  • Tables and databases in AWS Glue are objects in the AWS Glue Data Catalog. They contain metadata; they don't contain data from a data store. Similar to Hive metastore.
  • Each AWS account has one AWS Glue Data Catalog per region.
  • A data store is a repository for persistently storing your data. Examples include Amazon S3 buckets and relational databases. A data target is a data store that a process or transform writes to.


















  • The AWS Glue Schema Registry is a new feature that allows you to centrally discover, control, and evolve data stream schemas. A schema defines the structure and format of a data record. 

Comments

Post a Comment

Popular posts from this blog

Linear Algebra Concepts

Vector Spaces Rank of matrix is the number of non zero rows in REF form. Alternatively rank of matrix is = total number of columns - total number of dependent columns. In the REF form, If a column of a matrix can be expressed in terms of other columns then it is a dependent vector. The vectors on the left most side with leading 1 at the col number position ( 1 at 1st row 1st col, 1 at 2nd row 2nd col, likewise) then such columns are linearly independent columns. For linerarly independence a1v1+a2v2+a3v3=0 where v1,v2,v3 are vectors and a1,a2,a3 are scalars. Span of a vector space is all the possible vectors which can be formed using the basis vectors. Basis vectors are linearly independent vectors which can span the vector space. Column space is all possible set of vectors formed by the column vectors. Row space is all possible set of vectors formed by the row vectors. Dimension of a row space or column space is equal to its rank. Null space or kernel of a matrix is any vector x which ...
Read more

Artificial Computation Intelligence

  Components of Problem Formulation Initial State - Is the starting configuration of system. Actions - The permissible moves or operations that can be performed in the problem space. Transition Model - Describes the result of applying specific action in a given state. Goal State - Desired configuration that indicates problem has been solved. Cost function - numerical cost for each action. Task is to setup an automated traffic monitoring system that should be equipped with 10 cameras at each intersection to detect traffic violoations and detect license plates to cross verify with stolen vehicle database. What is the problem formulation for this, provide PEAS description, which agent architecture will fit this ? PEAS Description: Performance Measure: The system's performance is measured by its accuracy in detecting traffic violations, identifying license plates, and cross-verifying them with the stolen vehicle database. Additionally, the efficiency of the system in processing and ana...
Read more

Serverless - Lambda functions

 Lambda All operational activities of applying patch, capacity provisioning, monitoring fleet health, code deployment etc is taken care by Lamda. Best suited for use cases which are event based, such as changes to S3 buckets,  serverless processing of streaming data, updates to Dynamo DB table or custom event generated by apps/devices. All code uploaded to Lambda is encrypted. Each Lambda has built-in ephemeral storage of 512 MB. In-bound network connections are blocked by Lambda. Total Metrics, Account and Function level concurrency usage, latency and throttled requests are metrics shared by Lambda through Cloud Watch. Each Lambda function has a log group which can be traced in Cloudwatch logs. We can define the memory to be set for the function. Lambda function max execution time is 15 mins. Lambda can do polling of resources that do not publish events. Or it can be triggered basis events. S3 Integration - The S3 Bucket notifications can be sent to Lambda functions. Lambda c...
Read more