INDEX
########################################################### 2024-01-04 17:00 ########################################################### Devops Toolkit... Amazon Lightsail Containers https://www.youtube.com/watch?v=CWXrW2rgego Amazon Lightsail is essentially a simple way of making VMs, services, containers etc. # Make a basic lightsail VM aws lightsail create-container-service --serice-name devops-toolkit \ --power micro --scale 1 --output yaml aws lightsail get-container-services --output yaml Find out structure of yaml file and make it aws lightsail create-container-service-deployment --generate-cli-skeleton yaml-input # devops-toolkit-2.7.0.yaml serviceName: devops-toolkit containers: devops-toolkit: image: vfarcic/devops-toolkit-series:2.7.0 ports: "80": HTTP publicEndpoint: containerName: devops-toolkit containerPort: 80 # Create container using yaml aws lightail create-container-service-deployment --output yaml \ --cli-input-yaml file://devops-toolkit-2.7.0.yaml # Delete container aws lightsail delete-container-service --service devops-toolkit --output yaml
Devops Toolkit... AWS Lambda Containers https://www.youtube.com/watch?v=DsQbBVr-GwU Just as a note, looking at case studies for companies is very interesting Go on https://aws.amazon.com/solutions/case-studies/ and scroll down AWS lambda runs functions packages as zip files - can it run containers as a service? So take a git repo with a Go function and a Dockerfile to build it # Dockerfile FROM public.ecr.aws/bitnami/goland:1.15 AS build # Builds it with lambda COPY main.go . RUN go get -d -v github.com/aws/aws-lambda-go/lambda RUN CGO_ENABLE=0 GOOS=linux go build -o app FROM public.ecr.aws/lambda/go:1 # Runs the code with lambda COPY --from=build /go/app ${LAMBDA_TASK_ROOT} CMD ["app"] Cannot run arbitrary images - only images that have lambda function capabilities Now you want to establish that it works locally docker image build --tag lambda-container-demo:0.0.1 $PWD docker container run --publish 9000:8080 lambda-container-demo:0.0.1 curl "http://localhsot:9000/2015-03-31/functions/function/invocations" \ -d '{"name": "Viktor"}' && echo # Sends data to same style endpoint as lambda Once this works, upload it to a repository (say, AWS ECR - elastic container registry) # Create the container registry aws ecr create-repository --repository-name lambda-container-demo \ --image-scanning-configuration scanOnPush=true export REPO_URI=... # Store the given repository URL, as rather complex # Need to authenticate docker for pushing iamges aws ecr get-login-password | docker login --username AWS --password-stdin $REPO_URI # Need to tag with name of the registry docker image tag lambda-container-demo:0.0.1 $REPO_URI:0.0.1 # Tag version docker image push $REPO_URI:0.0.1 # Push to AWS ECS Now test it to see if it works - go on AWS website, Lambda, New, From container image Then select the container and "Create function" Add a trigger: API Gateway -> HTTP API + Open. Now should be accessible publicly export PAYLOAD=$(echo '{"name": "Viktor"}' | base64) # Store payload in http supported form aws lambda invoke --function-name lambda-container-demo --payload $PAYLOAD response.json cat response.json; echo # Should show the actual output
########################################################### 2024-01-16 21:00 ########################################################### Data Science Garage... First AWS Lambda Function https://www.youtube.com/watch?v=SslOHC-qji4 Open AWS Console - Lambda - Create Function - From scratch - Runtime=python3.9 Execution role configures connectivity to other services - use defaults, for simple function # lambda_function import json def lambda_handler(event, context): # Generic structure of lambda functions print("This is my AWS lambda function") if event["planet"] == "Earth": return "Moon" elif event["planet"] == "Sun": return "Not a planet" else: return "We do not recognise your input" # Now press "deploy" - then "test" with: { "planet": "Earth" } (event argument)
Data Science Garage... Install Jenkins on AWS EC2 https://www.youtube.com/watch?v=lpB6ZB58fTY Go on EC2 - Launch - Linux 2Mi t2 micro - default config/storage - tag=Jenkins For security group, add HTTP TCP 80 and Custom TCP 8080 Launch and create keypair (to connect with terminal - download keypair) Wait till "Running" - Select and "connect" - opens a terminal page in browser yum update -y # Update all packages - likely need to elevate to root first wget -I JENKINS_REPO; rpm --import JENKINS_TRUSTKEY yum install java-1.8.0 -y # Install java amazon-linux-extras install epel # Install epel packages on system yum install jenkins -y; service jenkins start Open AWS S3 info age - copy "public IPv4 DNS" and open "URL:8080" to view Jenkins cat /var/lib/jenkins/secrets/initialAdminPassword # Get unlock password
########################################################### 2024-01-17 09:30 ########################################################### Travis Media... Develop AWS Lambda Functions Locally in VS Code https://www.youtube.com/watch?v=fEZE3rm8Ma8 AWS serverless app model (SAM) is serverless shorthand converted to CloudFormation code Insall SAM CLI, Docker desktop, authenticate AWS (can use MFA) sam init # Create quick template - queries for lambda config # template.yaml = main config generated by init - manages gateway etc but can simplify AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Resources: AWS:AddDebugConfiguration HelloWorldFunction: Type: AWS::Serverless::Function Properties: CodeUri: hello_workd/ Handler: app.lambda_handler Runtime: python3.9 # hello_workd/app.py import json def lambda_handler(event, context): return { "statusCode": 200, "body": json.dumps({ "message": "hello world", }), } # events/event.json (test event values you can run locally) { # Extract values in python with e.g. event["first_name"] "first_name": "Travis", "last_name": "Media", "message": "Good morning", } Can build contents of function locally (can zip this and upload to lambda) sam build # Creates build/ sam deploy --guided # Push up to AWS (Y to save args to config) When you deploy, it makes 2 cloudformation stacks: the function and "sam-cli-managed" "sam-cli-managed" is an s3 bucket for SAM artifacts - all deployments added here As you update locally it updates change sets upstream sam local invoke HelloWorldFunction -e events/event.json # Test function in Docker sam deploy # Shows what changes and updates app sam delete # Delete online resources sam delete-stack --stack-name sam-app # Should delete local resources # samconfig.toml (simplifies deploy process version = 0.1 [default] [default.deploy] [default.deploy.parameters] stack_name = "sam-app" s3_buckt = "aws-sam-cli-managed-default-samclisourcebucket-..." s3_prefix = "sam-app" region = "us-east-1" confirm_changeset = true capabilities = "CAPABILITY_IAM"
Travis Media... Using Python to Automate AWS Services https://www.youtube.com/watch?v=3DRiruDUhiA We need automation to tag new EC2 instances with whoever creates it EC2 made -> rule in eventbridge runs -> triggers lambda runction -> tags EC2 Start with lambda: create function "TagEC2Instance" in python - make blank test Running a test creates a log group (so now you have log details needed) Create cloud trail (for eventbridge): create "my-trail" and store logs in S3 (with logs) Create event bridge: create rule (EC2launch) and enable rule Event pattern = EC2 - "AWS API call via cloudtrail" ("EC2 state trail" lacks user info) Specific operation = "RunInstances" - sets up an event pattern for you If you make an EC2 instance - go in cloudtrail and you can see a "runinstances" log for this Target = "lambda function" - "TagEC2Instance" - create rule. Should add trigger on lambda Boto3 is the AWS SDK for python - in lambda functions this is already there # Lambda function: import boto3 ec2 = boto3.client("ec2") # Look at documentation for boto3 ec2 methods def lambda_handler(event, context): # event = all info cloudformation gives you user = event["detail"]["userIdentity"]["userName"] instanceId = event["detail"]["responseElements"]["instancesSet"]["items"][0]["instanceId"] ec2.create_tags( resources=[ instanceId ], Tags=[{ "Key": "Owner", "Value": user }] ) return # Deploy this and launch - now responds to EC2 instances being made If this doesn't work, the issue is permissions of the lambda function on EC2 Go in configuration - permissions - Role page - Edit policy - add permission - EC2 Then approve "tagging - create tags" on call resources
Travis Media... AWS Networking Basics https://www.youtube.com/watch?v=2doSoMN2xvI VPC (virtual private cloud) isolates resources inside it - Go to AWS VPC Create VPC - "VPC only" - IPv4 CIDR 10.0.0.0/16 (last 2 octets free) Within VPC you have subnets (ranges of IPs): public and private Create Subnet - VPC ID (from dropdown) - "PublicSubnet" and 10.0.0.0/24 (last octet free) Also make "PrivateSubnet" and 10.0.1.0/24 Create EC2 instance (launch) - t2micro - use keypair - edit net and use public subnet Auto assign IP - Set security group name (virtual firewall for port traffic) But subnet is not public yet - need gateway (just links networks - here linking to internet) Go in VPC settings - Create internet gateway - Attach to VPC - Now need to route subnet Route tables allow data out to public gateway (by default you have basic local routetable) Create route tables (pub/priv) - Edit associations and link each route tables to their subnet In route table - Routes - edit - add "0.0.0.0/0" with internet gateway - NOW EC2 public Now launch an EC2 instance in private subnet (with new security group) Can ssh from publicEC2 to privateEC2 (use ssh keys for each - just copy from local machine) ssh -i KEY ec2-user@10.0.1.68 # SSH from 10.0.0.0/24 -> 10.0.1.0/24 # But cannot do things like update packages as there is no internet access Can fix this by adding a NAT gateway (network address translation) Link private route table to NAT gateway, which is in the public subnet Subnets - Create NAT gateway - put in public - allocate IP (takes some time to run) In private route table - edit - add "0.0.0.0/0" and routetable Can now reach internet from inside private subnet, but not other way around NACL (network access control) protects subnet - stateless virtual firewall for subnet Most people keep this default, but can use it to block IPs Security groups protect EC2 instance - stateful firewall allowing 2-way traffic