Tohuwabohu excorcism

User Tools

Site Tools

Trace: • cdk • iot_core • aws • apigateway
cloud:aws

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
cloud:aws [2023/11/01 07:13] – removed - external edit (Unknown date) 127.0.0.1cloud:aws [2024/01/30 14:52] (current) – skipidar
Line 1: Line 1:
 +===== AWS =====
 +
 +== Service catalogue ==
 +
 +Human friendly overview of all services
 +
 +https://docs.aws.amazon.com/?nc2=h_ql_doc_do
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/J6ES5jwMpO.png}}
 +
 +
 +
 +== Prescriptive-guidance ==
 +
 +Prescriptive Guidance provides time-tested strategies, guides, and patterns to help accelerate your cloud migration, modernization, and optimization projects. 
 +
 +https://aws.amazon.com/de/prescriptive-guidance/
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/qrsyAETPou.png}}
 +
 +== Library ==
 +
 +
 +  * Build applications patterns: https://aws.amazon.com/builders-library/
 +  * Infrastructure patterns: https://aws.amazon.com/architecture/well-architected/
 +
 +
 +===== Glossary =====
 +
 +\\
 +\\
 +\\
 +=====Regions and Availability-Zones=====
 +The Amazon Cloud Servers can be user in different Regions. \\
 +Each Region is separated in availability zones.
 +
 +
 +
 +The Regions are quite separated from each other. The Availability Zones are more interconnected, they are chosen each time when an instance is launched.
 +The Data can easily transfered inside one Availability-one but not inside one region.
 +
 +\\
 +
 +==Regions==
 +US-East, US-West, EU-West, Asia-Pacific... \\
 +
 +\\
 +
 +==Availability zones== 
 +e.g.: eu-west-1a, eu-west-1b, eu-west-1c \\
 +
 +\\
 +\\
 +\\
 +
 +=====VPC Virtual Private Cloud =====
 +
 +
 +==Data Traffic pricing==
 +
 +|Traffic within "the same Availability Zone is free" - free| https://aws.amazon.com/ec2/pricing/on-demand/|
 +|Traffic "across Availability Zones or VPC Peering connections in the same AWS Region is charged at $0.01/GB"|https://aws.amazon.com/ec2/pricing/on-demand/|
 +|Traffic "transferred across Inter-Region VPC Peering connections is charged at the standard inter-region data transfer rates." ($0.01/GB - $0.02/GB)|<WRAP>
 + https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-support-for-inter-region-vpc-peering/
 + Prices, see "Data Transfer OUT From Amazon EC2 To" https://aws.amazon.com/ec2/pricing/on-demand/
 +</WRAP>|
 +
 +
 +==Network segment ==
 +http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html#YourVPC
 +
 +
 +==CIDR Masks do separate segments==
 +https://ru.wikipedia.org/wiki/%D0%9C%D0%B0%D1%81%D0%BA%D0%B0_%D0%BF%D0%BE%D0%B4%D1%81%D0%B5%D1%82%D0%B8
 +
 +
 +Subnet calculator for CIDR blocks, which is able to calculate asymmetric blocks
 +  * http://vlsmcalc.net/
 +
 +
 +== AWS ACL (Access Control List) ==
 +
 +The ACLs are stateless, so they don't remember the traffic, which left 
 +and do not automatically allow traffic back.
 +(contrary to security-groups)
 +
 +
 +The **Inbound** ports - must be the **ports, under which a service is running** within the VPC-subnet. (like 22, 80 ...)
 +
 +  * To allow **Outbound traffic** back from subnet to the client 
 +  * **as port range** you must **NOT use the Service-Port-Range** (22, 80, ..) 
 +  * Instead, to communicate from **within the subnet back to client** the "Ephemeral Port Range" is used  
 +    * which **("Ephemeral Port Range") is chosen by the client-OS**
 +
 +See examples of port ranges by OS:
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/XMI0dVWHHU.png?600x300}}
 +
 +
 +Compare Security Groups and ACLs:
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/23iNK1RMay.png?700x250}}
 +
 +
 +=====AWS Command Line Interface=====
 +Commands Reference
 +http://docs.aws.amazon.com/cli/latest/reference/ec2/index.html#cli-aws-ec2
 +
 +
 +
 +== VPC architecture ==
 +
 +The architecture inside the VPC.
 +  * connecting on premises and be able to scale?
 +  * maintain resources in a shared VPC?
 +  * Transit Gateway
 +
 +{{youtube>ar6sLmJ45xs?medium}}
 +[[ https://www.youtube.com/watch?v=ar6sLmJ45xs&feature=youtu.be&t=1361 ]]
 +
 +
 +
 +
 +
 +
 +
 +
 +==== FIlters ====
 +There are predefined filters, which may be applied for several commands.\\
 +Filters **DO NOT MATCH THE NAME OF THE PROPERTY!** E.g. instance-id in filter and instanceId in JSON
 +
 +^Command^ List of FIlters^
 +|describe-instances|http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html|
 +|describe-volumes|http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-volumes.html|
 +
 +<code>
 +# volume ids for given instance
 +aws ec2 describe-volumes   --filters Name=attachment.instance-id,Values='i-44abd0cc' --query Volumes[].VolumeId --output text
 +
 +# root devices name for given instance 
 +aws ec2 describe-instances --filters Name='instance-id',Values='i-44abd0cc' --query 'Reservations[0].Instances[0].RootDeviceName'
 +</code>
 +
 +===== Query =====
 +
 +http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html#controlling-output-filter
 +
 +Specific tag filtering
 +<code>
 +--query 'Reservations[0].Instances[0].Tags[?Key==`Name`].Value'
 +</code>
 +
 +
 +===== Automatic Backups =====
 +
 +=== Existing solution ===
 +Here is an instance which can be executed and contains an existing Asigra solution
 +https://aws.amazon.com/marketplace/pp/B01FWRFHLM?qid=1478774150478&sr=0-2&ref_=srh_res_product_title
 +
 +
 +=== Own Script ===
 +
 +The AWS CLI is required for this script. How to configure it:
 +http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
 +
 +Script to make automatic snapshots and clean up:
 +<code>
 +#!/bin/bash
 +export PATH=$PATH:/usr/local/bin/:/usr/bin
 +
 +# Safety feature: exit script if error is returned, or if variables not set.
 +# Exit if a pipeline results in an error.
 +set -ue
 +set -o pipefail
 +
 +## Automatic EBS Volume Snapshot Creation & Clean-Up Script
 +#
 +# Written by Casey Labs Inc. (https://www.caseylabs.com)
 +# Contact us for all your Amazon Web Services Consulting needs!
 +# Script Github repo: https://github.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash
 +#
 +# Additonal credits: Log function by Alan Franzoni; Pre-req check by Colin Johnson
 +#
 +# PURPOSE: This Bash script can be used to take automatic snapshots of your Linux EC2 instance. Script process:
 +# - Determine the instance ID of the EC2 server on which the script runs
 +# - Gather a list of all volume IDs attached to that instance
 +# - Take a snapshot of each attached volume
 +# - The script will then delete all associated snapshots taken by the script that are older than 7 days
 +#
 +# DISCLAIMER: This script deletes snapshots (though only the ones that it creates). 
 +# Make sure that you understand how the script works. No responsibility accepted in event of accidental data loss.
 +#
 +
 +
 +## Variable Declartions ##
 +
 +# Get Instance Details
 +instance_id=$(wget -q -O- http://169.254.169.254/latest/meta-data/instance-id)
 +region=$(wget -q -O- http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/\([1-9]\).$/\1/g')
 +
 +# Set Logging Options
 +logfile="/var/log/ebs-snapshot.log"
 +logfile_max_lines="5000"
 +
 +# How many days do you wish to retain backups for? Default: 7 days
 +retention_days="7"
 +retention_date_in_seconds=$(date +%s --date "$retention_days days ago")
 +
 +
 +## Function Declarations ##
 +
 +# Function: Setup logfile and redirect stdout/stderr.
 +log_setup() {
 +    # Check if logfile exists and is writable.
 +    ( [ -e "$logfile" ] || touch "$logfile" ) && [ ! -w "$logfile" ] && echo "ERROR: Cannot write to $logfile. Check permissions or sudo access." && exit 1
 +
 +    tmplog=$(tail -n $logfile_max_lines $logfile 2>/dev/null) && echo "${tmplog}" > $logfile
 +    exec > >(tee -a $logfile)
 +    exec 2>&1
 +}
 +
 +# Function: Log an event.
 +log() {
 +    echo "[$(date +"%Y-%m-%d"+"%T")]: $*"
 +}
 +
 +# Function: Confirm that the AWS CLI and related tools are installed.
 +prerequisite_check() {
 + for prerequisite in aws wget; do
 + hash $prerequisite &> /dev/null
 + if [[ $? == 1 ]]; then
 + echo "In order to use this script, the executable \"$prerequisite\" must be installed." 1>&2; exit 70
 + fi
 + done
 +}
 +
 +# Function: Snapshot all volumes attached to this instance.
 +snapshot_volumes() {
 + for volume_id in $volume_list; do
 + log "Volume ID is $volume_id"
 +
 + # Get the attched device name to add to the description so we can easily tell which volume this is.
 + device_name=$(aws ec2 describe-volumes --region $region --output=text --volume-ids $volume_id --query 'Volumes[0].{Devices:Attachments[0].Device}')
 +
 + # Take a snapshot of the current volume, and capture the resulting snapshot ID
 + snapshot_description="$(hostname)-$device_name-backup-$(date +%Y-%m-%d)"
 +
 + snapshot_id=$(aws ec2 create-snapshot --region $region --output=text --description $snapshot_description --volume-id $volume_id --query SnapshotId)
 + log "New snapshot is $snapshot_id"
 +  
 + # Add a "CreatedBy:AutomatedBackup" tag to the resulting snapshot.
 + # Why? Because we only want to purge snapshots taken by the script later, and not delete snapshots manually taken.
 + aws ec2 create-tags --region $region --resource $snapshot_id --tags Key=CreatedBy,Value=AutomatedBackup
 + done
 +}
 +
 +# Function: Cleanup all snapshots associated with this instance that are older than $retention_days
 +cleanup_snapshots() {
 + for volume_id in $volume_list; do
 + snapshot_list=$(aws ec2 describe-snapshots --region $region --output=text --filters "Name=volume-id,Values=$volume_id" "Name=tag:CreatedBy,Values=AutomatedBackup" --query Snapshots[].SnapshotId)
 + for snapshot in $snapshot_list; do
 + log "Checking $snapshot..."
 + # Check age of snapshot
 + snapshot_date=$(aws ec2 describe-snapshots --region $region --output=text --snapshot-ids $snapshot --query Snapshots[].StartTime | awk -F "T" '{printf "%s\n", $1}')
 + snapshot_date_in_seconds=$(date "--date=$snapshot_date" +%s)
 + snapshot_description=$(aws ec2 describe-snapshots --snapshot-id $snapshot --region $region --query Snapshots[].Description)
 +
 + if (( $snapshot_date_in_seconds <= $retention_date_in_seconds )); then
 + log "DELETING snapshot $snapshot. Description: $snapshot_description ..."
 + aws ec2 delete-snapshot --region $region --snapshot-id $snapshot
 + else
 + log "Not deleting snapshot $snapshot. Description: $snapshot_description ..."
 + fi
 + done
 + done
 +}
 +
 +
 +## SCRIPT COMMANDS ##
 +
 +log_setup
 +prerequisite_check
 +
 +# Grab all volume IDs attached to this instance
 +volume_list=$(aws ec2 describe-volumes --region $region --filters Name=attachment.instance-id,Values=$instance_id --query Volumes[].VolumeId --output text)
 +
 +snapshot_volumes
 +cleanup_snapshots
 +</code>
 +
 +The script an be used in crontab as following:
 +<code>
 +# Minute   Hour   Day of Month       Month          Day of Week        Command
 + 
 +# makes SHORT term backups every day. Deletes short term snapshots older than 7 days.
 +5 2 * * *  /home/username/backupscript.sh 7 shortterm
 + 
 +# makes LONG term backups every Saturday night days. Deletes long term snapshots older than 30 days
 +30 2 * * 6  /home/username/backupscript.sh 30 longterm
 + 
 +</code>
 +
 +===== IAM Domains and Supporting Services =====
 +
 +Which services do you need for identity management?
 +https://www2.deloitte.com/content/dam/Deloitte/us/Documents/risk/us-cloud-and-identity-and-access-management.pdf
 +
 +  * Identification
 +  * Authentification Authorization
 +  * Access GOvernance
 +  * Accountability
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/n44sYxxTVG.png}}
 +
 +===== Policies =====
 +Via policies, assinged to Groups, assigned to Users 
 +one can configure which resources are reachable for some particular user.
 +
 +
 +== Write access to a S3 bucket ==
 +<code>
 +{
 +    "Version": "2012-10-17",
 +    "Statement": [
 +        {
 +            "Effect": "Allow",
 +            "Action": [
 +                "s3:ListBucket",
 +                "s3:GetBucketLocation",
 +                "s3:ListBucketMultipartUploads",
 +                "s3:AbortMultipartUpload",
 +                "s3:DeleteObject",
 +                "s3:DeleteObjectVersion",
 +                "s3:GetObject",
 +                "s3:GetObjectAcl",
 +                "s3:GetObjectVersion",
 +                "s3:GetObjectVersionAcl",
 +                "s3:PutObject",
 +                "s3:PutObjectAcl",
 +                "s3:PutObjectAclVersion"
 +            ],
 +            "Resource": [
 +                "arn:aws:s3:::aws-migration-landingzone/toaws/artifacts/*",
 +                "arn:aws:s3:::aws-migration-landingzone/toaws/svn/*",
 +                "arn:aws:s3:::aws-migration-landingzone/fromaws/artifacts/*",
 +                "arn:aws:s3:::aws-migration-landingzone/fromaws/svn/*"
 +            ]
 +        }
 +    ]
 +}
 +</code>
 +
 +
 +===== Route tables =====
 +
 +Route tables are defining rules for how traffic leaves the subnet. \\
 +Traffic destined for the **Destination** will be sent to the **Target**.
 +
 +{{https://i.stack.imgur.com/5Nprn.png}}
 +
 +
 +===== Lambdas =====
 +
 +|||
 +|Programming model|http://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html|
 +|||
 +
 +Configuring functions:
 +https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html
 +
 +== Python ==
 +
 +Use the python API boto3.
 +From boto3 Use the client API, dont use the resource API.
 +
 +Here is the documentation for the available services: http://boto3.readthedocs.io/en/latest/reference/services/index.html
 +
 +== Java - custom runtime ==
 +
 +Build a custom Java runtime for AWS Lambda
 +
 +https://aws.amazon.com/de/blogs/compute/build-a-custom-java-runtime-for-aws-lambda/
 +
 +===== Organisations =====
 +
 +http://docs.aws.amazon.com/de_de/organizations/latest/userguide/organizations-userguide.pdf
 +
 +
 +
 +==== SCP ====
 +
 +THe policies are applied on an organizational level.
 +
 +This is an example, how all services, but sts, s3, iam are denied
 +
 +<sxh>
 +{
 +  "Version": "2012-10-17",
 +  "Statement": [
 +    {
 +      "Sid": "AllowSTS",
 +      "Effect": "Deny",
 +      "NotAction": [
 +        "sts:*",
 +        "s3:*",
 +        "iam:*"
 +      ],
 +      "Resource": "*"
 +    }
 +  ]
 +}
 +</sxh>
 +
 +This is how policies are evaluated.
 +https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html
 +
 +{{https://docs.aws.amazon.com/images/IAM/latest/UserGuide/images/PolicyEvaluationHorizontal111621.png}}
 +===== CloudFormation =====
 +
 +Debugging can be done, via following the logs:
 +
 +Connect via SSH to the instance, which is currently modified and follow the logs:
 +
 +<code>
 +tail -f /var/logs/cloud-inig-output.log
 +</code>
 +
 +
 +===== ELB =====
 +For experimenting with ELB use the following script, when provisioning EC2 instances, to deploy a webserver with a simple index.html
 +
 +It will have a default site with the name of the instance in it.
 +
 +<code>
 +#!/bin/bash
 +yum update -y
 +yum install -y httpd24 php56 mysql55-server php56-mysqlnd
 +service httpd start
 +chkconfig httpd on
 +groupadd www
 +usermod -a -G www ec2-user
 +chown -R root:www /var/www
 +chmod 2775 /var/www
 +find /var/www -type d -exec chmod 2775 {} +
 +find /var/www -type f -exec chmod 0664 {} +
 +echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php
 +uname -n > /var/www/html/index.html
 +</code>
 +
 +
 +===== STS =====
 +
 +You can give users a valid STS-token using this service.
 +The users must authenticate themselves against something else, like corporate app. 
 +Then they can use the token to identify themselves against AWS apps.
 +You can then programmatically check, whether the STS token is valid.
 +
 +{{https://i.imgur.com/G173DJp.png}}
 +
 +Details: https://www.slideshare.net/cloudITbetter/aws-security-token-service
 +
 +===== S3 =====
 +
 +Storage classes
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/Gcxu7v80fC.png?500x300}}
 +
 +
 +
 +===== Guard Duty =====
 +
 +  * analyzes logs (VPC Flow Logs, CloudTrail )
 +  * identifies findings with severity
 +
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/Vf3NW6LJAl.png}}
 +
 +
 +===== Shield =====
 +
 +
 +AWS Shield protects the OSI model’s infrastructure layers (Layer 3 Network, Layer 4 Transport)
 +
 +AWS **Shield is a managed Distributed Denial of Service (DDoS) protection service**, 
 +whereas AWS WAF is an application-layer firewall that controls access via Web ACL’s.
 +
 +See https://cmakkaya.medium.com/how-to-secure-our-resources-from-doos-attacks-with-aws-waf-shield-5307c85cb476
 +
 +{{https://miro.medium.com/v2/resize:fit:720/format:webp/1*t6FzPmxrCszis2rRLtUEyw.png}}
 +
 +Shield **"Simple"** - AWS reacts on DDoS attacks
 +
 +Shield **"Advanced"** - AWS reacts on DDoS attacks and provides a 24x7 team and reports.
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/KLvzXWSlgv.png?600x300}}
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/aOZMJUFzjB.png?600x300}}
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/DuuoS72NYf.png?600x300}}
 +
 +
 +===== WAF Web Application Firewall =====
 +
 +See https://medium.com/@comp87/deep-dive-into-the-aws-web-application-firewall-waf-14148ea0d3d
 +
 +
 +Network firewalls operate at Layer 3 (Network) and only understand the 
 +  * source IP Address, 
 +  * port, and 
 +  * protocol. 
 +
 +AWS **Security Group**s are a great example of this.
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/1bUVo0Mme3.png}}
 +
 +
 +WAF works on **OSI layer 7** (Application)
 +
 +means it understands higher-level protocols such as an 
 +  * HTTP(S) request, including its 
 +    * headers, 
 +    * body, 
 +    * method, and 
 +    * URL
 +
 +
 +  * WAF interacts with 
 +    * CloudFront distributions, 
 +    * application load balancers, 
 +    * AppSync GraphQL, 
 +    * APIs and 
 +    * API Gateway REST APIs. 
 +
 +A WAF can be configured to detect traffic from the following:
 +
 +  * specific IPs;
 +  * IP ranges or country of origin;
 +  * content patterns in request bodies, headers and cookies;
 +  * SQL injection attacks;
 +  * cross-site scripting; and
 +  * IPs exceeding rate-based rules
 +
 +When incoming traffic matches any of the configured rules, WAF can reject requests, return custom responses or simply create metrics to monitor applicable requests.
 +
 +
 +
 +
 +===== AWS API Gateway =====
 +Redirects HTTP calls, can modify the content. In both directions.
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/2GAhuEvU9y.png?600x400}}
 +
 +
 +
 +===== Authentication and Authorization =====
 +
 +  - https://aws.amazon.com/blogs/compute/nginx-reverse-proxy-sidecar-container-on-amazon-ecs/
 +  - https://openiam.atlassian.net/wiki/plugins/servlet/mobile?contentId=524591#content/view/524591
 +  - https://aws.amazon.com/answers/networking/controlling-vpc-egress-traffic/
 +  - https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-getting-started.html#select-load-balancer-type
 +  - https://aws.amazon.com/premiumsupport/knowledge-center/public-load-balancer-private-ec2/
 +
 +
 +
 +===== SSH via Ec2 instance connect =====
 +
 +One can ssh to the instance via the AWS instance connect feature.
 +This allows to push an own Amazon agent to the target machine using the permission "ec2-instance-connect:SendSSHPublicKey"
 +
 +To set it up read:
 +https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html
 +
 +
 +After that one can ssh to the machine.
 +
 +SSH Tunneling
 +<code>
 +ssh ec2-user@i-12342f8f3a2bc6367 -i ~/.ssh/ec2key.priv.openssh.ppk -fNTC -L 0.0.0.0:4440:localhost:4440
 +</code>
 +
 +
 +SSH connect
 +<code>
 +ssh ec2-user@i-09452f8f3a2bc6367 -i ~/.ssh/ec2key.priv.openssh.ppk
 +</code>
 +
 +
 +==== Boto3 ====
 +
 +
 +=== generate a signed URL ===
 +
 +Use the script to generate a signed URL
 +
 +<sxh python>
 +import argparse
 +import logging
 +import boto3
 +from botocore.exceptions import ClientError
 +import requests
 +import sys
 +
 +logger = logging.getLogger(__name__)
 +logging.basicConfig(stream=sys.stdout, level=logging.INFO)
 +
 +
 +def generate():
 +
 +    s3 = boto3.client('s3')
 +
 +    bucket_name='poc-arc-dev-signedurl'
 +    object_key = "test2.txt"
 +   
 +    response = s3.generate_presigned_post(
 +        Bucket=bucket_name,
 +        Key=object_key,
 +    )
 +    logger.info("Got presigned POST URL: %s", response['url'])
 +
 +    # formulate a CURL command now
 +
 +    response['fields']['file'] = '@{key}'.format(key=object_key)
 +
 +    form_values = "\n    ".join(["-F {key}={value} \\".format(key=key, value=value)
 +                            for key, value in response['fields'].items()])
 +
 +    print('curl command: \n')
 +    print('curl -v {form_values} \n    {url}'.format(form_values=form_values, url=response['url']))
 +
 +    print('-'*88)
 +
 +
 +
 +if __name__ == '__main__':
 +    generate()
 +
 +
 +</sxh>
 +
 +
 +===== CloudWatch agent =====
 +
 +The agent to send logs to cloudwatch.
 +
 +Done on Ubuntu 22
 +
 +
 +Give the ec2 role of that machine the ability to access cloudwatch and ec2 services,
 +so that it can read ec2 metadata.
 +
 +<sxh shell>
 +      {
 +        "Action": [
 +          "cloudwatch:Put*",
 +          "logs:Create*",
 +          "logs:Put*",
 +          "logs:Describe*",
 +          "ec2:Describe*",
 +          "ec2:Read*",
 +          "ec2:List*"
 +        ],
 +        "Effect": "Allow",
 +        "Resource": "*"
 +      }
 +</sxh>
 +
 +
 +Config files:
 +
 +file: "amazon-cloudwatch-agent.json"
 +
 +Commands to run as root, to get permissions to access log files.
 +
 +<sxh shell>
 +{
 +    "agent" : {
 +        "metrics_collection_interval": 60,
 +        "run_as_user": "root"
 +    },
 +    "logs": {
 +        "logs_collected": {
 +          "files": {
 +            "collect_list": [
 +              {
 +                "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
 +                "log_group_name": "alf-dev-webserv2-ec2instance",
 +                "log_stream_name": "amazon-cloudwatch-agent.log",
 +                "timezone": "UTC"
 +              },
 +              {
 +                "file_path": "/var/log/syslog",
 +                "log_group_name": "alf-dev-webserv2-ec2instance",
 +                "log_stream_name": "syslog",
 +                "timezone":"UTC"
 +              }
 +            ]
 +          }
 +        },
 +        "force_flush_interval" : 5
 +    },
 + "metrics": {
 + "append_dimensions": {
 + "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
 + "ImageId": "${aws:ImageId}",
 + "InstanceId": "${aws:InstanceId}",
 + "InstanceType": "${aws:InstanceType}"
 + },
 + "metrics_collected": {
 + "disk": {
 + "measurement": [
 + "used_percent"
 + ],
 + "metrics_collection_interval": 60,
 + "resources": [
 + "*"
 + ]
 + },
 + "mem": {
 + "measurement": [
 + "mem_used_percent"
 + ],
 + "metrics_collection_interval": 60
 + }
 + }
 + }
 +}
 +
 +
 +</sxh>
 +
 +
 +The ansible playbook to install the agent
 +
 +file: "configCloudWatchAgent.ansible.yaml"
 +<sxh ansible>
 +---
 +# Configuring syslog rotation
 +- hosts: "{{ host }}"
 +  become: yes
 +  become_method: sudo
 +  become_user: root
 +  tasks:
 +
 +  - name: Download awslogs-agent installer
 +    get_url: 
 +      url: https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
 +      dest: /tmp/amazon-cloudwatch-agent.deb
 +      owner: "root"
 +      group: "root"
 +      force: yes
 +      mode: 0777
 +
 +
 +  - name: Install deb
 +    apt:
 +      deb: /tmp/amazon-cloudwatch-agent.deb
 +    become: true
 +    register: result
 +
 +
 +  - debug:
 +      msg: "{{ result }}"
 +
 +
 +  - name: Remove the /tmp/amazon-cloudwatch-agent.deb
 +    file: 
 +      path: /tmp/amazon-cloudwatch-agent.deb
 +      state: absent
 +
 +
 +  - name: awslogs config folder
 +    file:
 +      path: /var/awslogs/etc/
 +      state: directory
 +      mode: "u=rwx,g=rwx,o=rwx"
 +      owner: root
 +      group: root
 +      recurse: yes
 +
 +
 +  - name: Copy a "amazon-cloudwatch-agent.json" file on the remote machine for editing. Overwrite if exists
 +    copy:
 +      src: "{{ playbook_dir }}/../config/amazon-cloudwatch-agent.json"
 +      dest: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
 +      mode: "u=rwx,g=r,o=r"
 +      remote_src: false
 +
 +
 +  - name: now fetch the config and run agent
 +    ansible.builtin.shell:
 +      cmd: amazon-cloudwatch-agent-ctl -s -a fetch-config -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
 +    become: true
 +</sxh>
 +
 +
 +Check status of service, should be "active (running)"
 +<sxh shell>
 +sudo systemctl status amazon-cloudwatch-agent.service
 +</sxh>
 +
 +
 +Check status of agent via ctl tool
 +<sxh shell>
 +$ amazon-cloudwatch-agent-ctl -a status
 +
 +
 +{
 +  "status": "running",
 +  "starttime": "2023-09-12T09:09:08+00:00",
 +  "configstatus": "configured",
 +  "version": "1.300026.3b189"
 +}
 +
 +</sxh>
 +
 +
 +
 +
 +
 +The consolidated configurations 
 +which the system collects from above manual config files 
 +one can check via:
 +
 +<sxh shell>
 +$ cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml
 +
 +
 +connectors: {}
 +exporters:
 +    awscloudwatch:
 +        force_flush_interval: 1m0s
 +        max_datums_per_call: 1000
 +        max_values_per_datum: 150
 +        namespace: CWAgent
 +        region: eu-central-1
 +        resource_to_telemetry_conversion:
 +            enabled: true
 +extensions: {}
 +processors:
 +    ec2tagger:
 +        ec2_instance_tag_keys:
 +            - AutoScalingGroupName
 +        ec2_metadata_tags:
 +            - ImageId
 +            - InstanceId
 +            - InstanceType
 +        refresh_interval_seconds: 0s
 +receivers:
 +    telegraf_disk:
 +        collection_interval: 1m0s
 +        initial_delay: 1s
 +    telegraf_mem:
 +        collection_interval: 1m0s
 +        initial_delay: 1s
 +service:
 +    extensions: []
 +    pipelines:
 +        metrics/host:
 +            exporters:
 +                - awscloudwatch
 +            processors:
 +                - ec2tagger
 +            receivers:
 +                - telegraf_mem
 +                - telegraf_disk
 +    telemetry:
 +        logs:
 +            development: false
 +            disable_caller: false
 +            disable_stacktrace: false
 +            encoding: console
 +            error_output_paths: []
 +            initial_fields: {}
 +            level: info
 +            output_paths:
 +                - /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
 +
 +</sxh>
 +
 +
 +Especially the agent logs are in:
 +
 +<sxh shell>
 +$ cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
 +</sxh>
 +
 +
 +==== Backup strategies ====
 +
 +Which patterns for high availability are available?
 +
 +How much do they cost?
 +
 +https://aws.amazon.com/de/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot-light-and-warm-standby/
 +
 +
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/V7lfR0RHwg.png}}
 +
 +
 +
 +===== Migration =====
 +
 +Prescriptive guideline describes pretty well how to organize it:
 +
 +https://docs.aws.amazon.com/prescriptive-guidance/latest/application-portfolio-assessment-guide/introduction.html
 +
 +{{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/BezztUtsuP.png}}
 +
 +
 +===== Domains =====
 +
 +==== Finance and credit card ====
 +
 +Real time credit card fraud evaluation:
 +
 +https://aws.amazon.com/blogs/machine-learning/use-streaming-ingestion-with-amazon-sagemaker-feature-store-and-amazon-msk-to-make-ml-backed-decisions-in-near-real-time/
 +
 +{{https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2023/08/30/ML-13533-image001-new.jpg}}
 +
 +
 +==== Goldman Sachs ==== 
 +
 +Achieving cross-regional availibility
 +
 +https://developer.gs.com/blog/posts/building-multi-region-resiliency-with-amazon-rds-and-amazon-aurora
 +
 +