AWS - Moving data and Migrations

 

  1. AWS Data Pipeline 
    1. To enable running activities using on-premise resources, AWS Data Pipeline does the following: It supply a Task Runner package that can be installed on your on- premise hosts. 
    2. This package continuously polls the AWS Data Pipeline service for work to perform. When it's time to run a particular activity on your on-premise resources, it will issue the appropriate command to the Task Runner. 
    3. Data Pipeline service does not support data streams.
    4. Task runners call PollForTask to receive a task to perform from AWS Data Pipeline. If tasks are ready in the work queue, PollForTask returns a response immediately. If no tasks are available in the queue, PollForTask uses long-polling and holds on to a poll connection for up to 90 seconds, during which time any newly scheduled tasks are handed to the task agent. 
  2. Migration
    1. CloudEndure Migration is a block-level replication tool that simplifies the process of migrating applications from physical, virtual, and cloud-based servers to AWS. 
      1. If your source environment includes bare metal servers, and you can install agents.
      2. CloudEndure Migration continuously replicates your source servers to your AWS account.
    2. AWS SMS supports migrations of virtual machines from VMware, Hyper-V, or Microsoft Azure. AWS SMS doesn’t currently support physical infrastructure or other public cloud providers except for Microsoft Azure.
    3. It is recommended by AWS to use the Server Migration Service (SMS) to migrate VMs from a vCenter environment to AWS. SMS automates the migration process by replicating on-premises VMs incrementally and converting them to Amazon machine images (AMIs). You can continue using your on-premises VMs while the migration is in progress. The Server Migration Connector is a FreeBSD VM that you install in your on-premises virtualization environment.
    4. Use AWS Server Migration Service (SMS) to create and launch a replication job for each tier of the application. Launch instances from the AMIs created by AWS SMS. After initial testing, perform a final replication and create new instances from the updated AMIs.
    5. During the replication process, AWS SMS creates an Amazon S3 bucket in the Region on your behalf, with server-side encryption enabled. AWS SMS replicates server volumes from your environment to this bucket and then creates EBS snapshots from the volumes.
    6. AWS Server Migration Service supports the automated migration of multi-server application stacks from your on-premises data center to Amazon EC2. Where server migration is accomplished by replicating a single server as an Amazon Machine Image (AMI), application migration replicates all of the servers in an application as AMIs and generates an AWS CloudFormation template to launch them in a coordinated fashion. Applications can be further subdivided into groups that allow you to launch tiers of servers in a defined order. 
    7. DMS - AWS DMS is a server in the AWS Cloud that runs replication software. You create a source and target connection to tell AWS DMS where to extract from and load to. Then you schedule a task that runs on this server to move your data. AWS DMS creates the tables and associated primary keys if they don't exist on the target. 
      1. AWS DMS doesn't automatically create secondary indexes, foreign keys, user accounts, and so on, in the target database.  Indexes, triggers, and referential integrity constraints can affect your migration performance and cause your migration to fail. 
      2. When migrating to an Amazon RDS database, it's a good idea to turn off backups and Multi-AZ on the target until you're ready to cut over. 
      3. AWS DMS doesn't perform schema or code conversion - you can use the AWS Schema Conversion Tool (AWS SCT).
      4. When you plan your migration, we recommend that you run diagnostic support scripts. A premigration assessment evaluates specified components of a database migration task to help identify any problems that might prevent a migration task from running as expected. 
      5. AWS DMS uses some resources on your source database- reduce the number of tasks or tables for each task for your migration.
      6. With Amazon RDS databases, it's a good idea to turn off backups and Multi-AZ until the cutover.
      7. If you turn on data validation for a task, AWS DMS begins comparing the source and target data immediately after a full load is performed for a table.
      8. A premigration assessment evaluates specified components of a database migration task to help identify any problems that might prevent a migration task from running as expected. This assessment gives you a chance to identify issues before you run a new or modified task.

Comments

Post a Comment

Popular posts from this blog

AWS Organizations, IAM

Image
Active Directory AD Connector is a directory gateway with which you can redirect directory requests to your on-premises Microsoft Active Directory without caching any information in the cloud. AD Connector comes in two sizes, small and large. You can spread application loads across multiple AD Connectors to scale to your performance needs.  AD Connector is your best choice when you want to use your existing on-premises directory with AWS services. AD Connectors and your on-premises AD domains have a 1-to-1 relationship.  Your end users and IT administrators can use their existing corporate credentials to log on to AWS applications such as WorkSpaces, Amazon WorkDocs, or Amazon WorkMail. Simple AD is a standalone managed directory.  Small - Supports up to 500 users,  Large - Supports up to 5,000 users.  Simple AD does not support features such as multi-factor authentication (MFA), trust relationships with other domains.  AWS Directory Service lets you run Microsoft Active Directory (AD)
Read more

Key Concepts

A  resource group  is a collection of AWS resources that are all in the same AWS Region, and that match the criteria specified in the group's query.  Tag-based queries include lists of resources and tags.  In an AWS CloudFormation stack-based query, you choose an AWS CloudFormation stack in your account in the current region, and then choose resource types within the stack that you want to be in the group.   You can use  resource groups  to organize your AWS resources. AWS Resource Groups is the service that lets you manage and automate tasks on large numbers of resources at one time.  With Resource Groups, you can create a custom console that organizes and consolidates information based on criteria specified in tags, or the resources in an AWS CloudFormation stack.  Transfer Family - Supports connecting to S3 or EFS through FTPS/SFTP/FTP protocols.  The hostname for the endpoint can also configured with R53.  The transfer family can be associated with Elastic IP. Security group pe
Read more

Linear Algebra Concepts

Vector Spaces Rank of matrix is the number of non zero rows in REF form. Alternatively rank of matrix is = total number of columns - total number of dependent columns. In the REF form, If a column of a matrix can be expressed in terms of other columns then it is a dependent vector. The vectors on the left most side with leading 1 at the col number position ( 1 at 1st row 1st col, 1 at 2nd row 2nd col, likewise) then such columns are linearly independent columns. For linerarly independence a1v1+a2v2+a3v3=0 where v1,v2,v3 are vectors and a1,a2,a3 are scalars. Span of a vector space is all the possible vectors which can be formed using the basis vectors. Basis vectors are linearly independent vectors which can span the vector space. Column space is all possible set of vectors formed by the column vectors. Row space is all possible set of vectors formed by the row vectors. Dimension of a row space or column space is equal to its rank. Null space or kernel of a matrix is any vector x which
Read more