Glue

  •  AWS Glue is a serverless, fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores and data streams. 
  • AWS Glue consists of a central metadata repository known as the AWS Glue Data Catalog.
  • AWS Glue is designed to work with semi-structured data.
  • You can use AWS Glue to organize, cleanse, validate, and format data for storage in a data warehouse or data lake. 
  • Glue discovers and catalogs metadata about your data stores into a central catalog. 
  • Populates the AWS Glue Data Catalog with table definitions from scheduled crawler programs.
  • AWS Glue can catalog your Amazon Simple Storage Service (Amazon S3) data, making it available for querying with Amazon Athena and Amazon Redshift Spectrum.
  • You can run your ETL jobs as soon as new data becomes available in Amazon S3 by invoking your AWS Glue ETL jobs from an AWS Lambda function.
  • Glue supports data sources - S3, RDS, Dynamo, Mongo, Document DB, 3rd party JDBC accessible DBs. Crawlers can pull data from sources and update Data Catalog with metadata. AWS Glue can generate a script to transform your data. 
  • Glue can access Kafka or Kinesis data streams.
  • You can run your job on demand, or you can set it up to start when a specified trigger occurs. The trigger can be a time-based schedule or an event.
  • Tables and databases in AWS Glue are objects in the AWS Glue Data Catalog. They contain metadata; they don't contain data from a data store. Similar to Hive metastore.
  • Each AWS account has one AWS Glue Data Catalog per region.
  • A data store is a repository for persistently storing your data. Examples include Amazon S3 buckets and relational databases. A data target is a data store that a process or transform writes to.


















  • The AWS Glue Schema Registry is a new feature that allows you to centrally discover, control, and evolve data stream schemas. A schema defines the structure and format of a data record. 

Comments

Post a Comment

Popular posts from this blog

Machine Learning

Image
Classification problem Features Labels Training set Term frequency representation - Getting frequency of words based on a sentence  Output is a categorical value. Naive Bayes Sentiment Analysis Classification Setup = Take a comment classify as positive or negative P = Positive, 1-P = Negative Probability of word being positive = Sum of frequency of the word in positive comments / sum frequency of the data in entire corpus   Drop connective words (stop words) when processing Then multiply the positivity of the other words which are part of sentence to get overall score * overall probability of the number being positive. Sentiment labelled sentences data set - IMDB, Yelp and Amazon reviews. Scikit-Learn module in Python  Can be used to classify any number of categories SVM Ad detection - Internet advertisement dataset Can be used only for binary classification Regression Problem Examples Finding stock price returns on a given day CAPM model/formula for finding future price ...
Read more

API Gateway

 API gateway manages creation, publishing, monitoring and securing of Http, Rest and Websocket APIs. Helps create Restful with lower latency and cost that RESt APIs. Helps create Websocket API using which client and server can communicate continuously. API endpoint form - api-id.execute-api.region.amazonaws.com Proxy Integration - can be HTTP Proxy or Lambda Proxy integration. In HTTP Proxy integration, API g/w passes entire request/response between frontend and backend. In Lambda Proxy integration, API g/w passes request to Lambda function and the output of Lambda function is returned to frontend. Lambda, IAM, Cognito User Pools, OAuth 2.0 Authorizers are supported for Http API. For Rest except OAuth rest of them are supported. Lambda authorizers are Lambda functions that control access of REST or Http API using bearer token authentication or using request parameters like headers, paths, query string, etc. Cognito User Pools helps control who can access REST or Http API. The ...
Read more

IOT Core

AWS IoT Core  is a managed cloud service that enables connected devices to securely interact with cloud applications and other devices.  AWS IoT provides device software that can help you integrate your IoT devices into AWS IoT-based solutions.  AWS IoT Core supports these protocols: MQTT MQTT WSS - Secure Web Sockets HTTPS Long range WAN - LoRaWAN AWS IoT Greengrass  extends AWS IoT to edge devices so they can act locally on the data they generate and use the cloud for management, analytics, and durable storage.  FreeRTOS  is an open source, real-time operating system for microcontrollers that lets you include small, low-power edge devices in your IoT solution. FreeRTOS includes a kernel and a growing set of software libraries that support many applications. FreeRTOS systems can securely connect your small, low-power devices to  AWS IoT  and support more powerful edge devices running  AWS IoT Greengrass .    AWS IoT Things Graph pr...
Read more