计算 尽管函数计算和容器的快速崛起，EC2依然是AWS的业务焦点，
主要的新功能包括：
基于Nitro平台的针对HPC和机器学习的负载的实例
基于定制芯片Inferencia针对机器学习实例
标准实例支持100Gb网络带宽
网络 传输网关支持多播
加速的网络到网络的VPN链接
存储和数据分析 S3 Access Points和数据湖
ES搜索支持S3
联合查询支持关系数据库，REDSHIFT数据仓库，S3数据湖，而不需要移动数据
AQUA查询加速器
数据湖导出(REDSHIFT数据仓库查询结果能直接导出到S3，并以Parquet格式存放)
托管的Cassandra服务
函数计算 Provisioned Concurrency for Lambda Functions
EKS正式支持FARGATE
RDS PROXY
VPC支持更新
先锋基金IT迁移架构图 私有云架构包括4千万行的单体应用，hadoop数据仓库(20PB)和PaaS(2015年) 数据仓库和PaaS迁移到AWS PaaS实施基于EDA架构的改造 PaaS迁移到Fargate
aws sts get-caller-identity aws s3control list-access-points --account-id 46569194568 aws s3control create-access-point --name my-access-point --account-id 46569194568 --bucket wubigo aws s3control get-access-point --account-id "46569194568" --name my-access-point { "Name": "my-access-point", "PublicAccessBlockConfiguration": { "IgnorePublicAcls": true, "BlockPublicPolicy": true, "BlockPublicAcls": true, "RestrictPublicBuckets": true }, "CreationDate": "2019-12-04T14:24:38Z", "Bucket": "wubigo", "NetworkOrigin": "Internet" }
[TOC]
AWS领先的设计理念和强大的技术生态
使你身陷其中，学习你要用它，开发你要
用它，上线还要用它。 一年下来项目还没有
正式商用，已经有十几万的研发费用。
今天向你推荐 localstack（与openstack啥关系？私有云+公有云），
让你使用AWS免费，至少在项目POC或开发测试阶段免费。
有了它， 你不用再焦急的等待老板审批公有云的
计算，存储，数据库资源开发申请。
是不是这个项目听起来很激动？
那如何使用localstack呢？
安装localstack localstack是一个非常活跃的正在快速成长中的项目，
建议通过源代码安装
下载源代码
git clone
[email protected]:localstack/localstack.git git fetch --all git checkout tags/v0.10.3 -b v0.10.3 启用需要使用的AWS服务
修改配置文件，启用你需要使用的AWS服务:ec2,s3,iot,kafka等。
注意服务的名字必须来自服务名字列表， 否则不识别
启用服务就是修改下边的配置文件
localstack\docker-compose.yml
SERVICES=${SERVICES-ec2,ecs,stepfunctions,iam,lambda,dynamodb,apigateway,s3,sns} DATA_DIR=${DATA_DIR-/tmp/localstack/data } volumes: - "${TMPDIR:-d:/tmp/localstack}:/tmp/localstack" make sure driver D is shared in docker desktop daemon
docker-compose up localstack_1 | Starting mock S3 (http port 4572)... localstack_1 | Starting mock SNS (http port 4575).
创建函数 分配角色
zip function.zip index.js aws lambda create-function --function-name sns-db-function \ --zip-file fileb://function.zip --handler index.handler --runtime nodejs12.x \ --role arn:aws:iam::465691908928:role/fn-case-role
发布测试消息到SNS git clone https://github.com/wubigo/node-fn/blob/master/fn-case/sns_publishtotopic.js node sns_publishtotopic.js Message MESSAGE_TEXT send sent to the topic arn:aws:sns:ap-northeast-1:465691908928:func-topic MessageID is 8b5c90f2-0c74-5985-8a34-c676c0370f73 根据MessageID查看函数执行结果 The $ in [$LATEST] needs to be escaped…[\$LATEST].
aws logs describe-log-groups --query logGroups[*].logGroupName aws logs describe-log-streams --log-group-name '/aws/lambda/my-function' --query logStreams[*].logStreamName aws logs get-log-events --log-group-name '/aws/lambda/my-function' --log-stream-name '2019/12/31/[$LATEST]7467497f9cdb4078a876ab889797793c' { "ingestionTime": 1577764111252, "timestamp": 1577764096184, "message": "2019-12-31T03:48:16.183Z\tc01c9f5e-6c33-40a1-a6d9-c11ab248ab48\tINFO\tEVENT\n{\n \"Records\": [\n {\n \"EventSource\": \"aws:sns\",\n \"EventVersion\": \"1.0\",\n \"EventSubscriptionArn\": \"arn:aws:sns:ap-northeast-1:465691908928:func-topic:2e0e0d95-f1c8-47bd-90ff-40ca4129794b\",\n \"Sns\": {\n \"Type\": \"Notification\",\n \"MessageId\": \"5f80d26e-bdeb-579f-bc81-84ea7ad4e2ae\",\n \"TopicArn\": \"arn:aws:sns:ap-northeast-1:465691908928:func-topic\",\n \"Subject\": null,\n \"Message\": \"MESSAGE_TEXT\",\n \"Timestamp\": \"2019-12-31T03:48:15.
https://medium.com/people-ai-engineering/building-a-data-lake-in-aws-9c1fb3876e23
https://towardsdatascience.com/building-a-data-pipeline-from-scratch-on-aws-35f139420ebc
restore snapshot shell snapshot.sh
!#/bin/bash wget https://nodejs.org/dist/v12.13.1/node-v12.13.1-linux-x64.tar.xz tar xvf node-v12.13.1-linux-x64.tar.xz export PATH=/home/ubuntu/node-v12.13.1-linux-x64/bin:$PATH wget https://manning-content.s3.amazonaws.com/download/0/ddbbd36-251d-42ef-9934-55e5a881a336/FinalSourceCode.zip sudo apt update sudo apt install unzip unzip FinalSourceCode.zip mv Final\ Source\ Code/ sls sudo apt install python-pip pip install awscli which aws_completer cp ~/.bashrc ~/.bashrc_orig tee -a ~/.bashrc <<-'EOF' complete -C '/home/ubuntu/.local/bin/aws_completer' aws export PATH=/home/ubuntu/node-v12.13.1-linux-x64/bin:$PATH EOF aws configure npm install claudia -g claudia -v 5.11.0 cd chapter-03 npm install claudia create \ --region ap-northeast-1 \ --api-module api packaging files npm install -q --no-audit --production npm WARN pizza-api@1.
AWS leverages a standard JSON Identity and Access Management (IAM)
policy document format across many services to control authorization
to resources and API actions
terraform https://www.terraform.io/docs/providers/aws/r/iam_role_policy.html
resource "aws_iam_role_policy" "s3_policy" { name = "s3_policy" role = "${aws_iam_role.lambda_s3_role.id}" policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::bucket-name"] }, { "Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::bucket-name/*"] } ] } EOF } resource "aws_iam_role" "lambda_s3_role" { name = "lambda_s3_role" assume_role_policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "lambda.
Amazon EC2 networking doesn’t allow to use private ips in the containers
through bridges or macvlan. Dedicating a network interface to a
container makes it directly unreachable from the host.
docker network create -d macvlan --subnet 172.30.80.0/20 --gateway 172.30.80.1 -o parent=eth0 pub_net docker run -d --network pub_net --ip 172.30.80.10 busybox
VPC之间的连接方式 专线
VPN托管服务
VPN Peering
自建或第三方VPN
PrivateLink
VPC与私有网络之间的连接方式 VPN托管服务
专线
专线+VPN
自建或第三方VPN
Transit VPC
VPN Hub
Enabling Enhanced Networking on Ubuntu sudo apt-get update && sudo apt-get upgrade -y linux-aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide//sriov-networking.html#enhanced-networking-ubuntu
创建BUCKET 使用两种方式之一创建BUCKET
terraform
git clone https://github.com/wubigo/iaas cd s3 terraform apply awscli
aws s3 website s3://s.wubigo.com/ --index-document index.html --error-document 404.html aws s3api put-bucket-policy --bucket s.wubigo.com --policy file://policy.json 确认配置 aws s3api get-bucket-website --bucket s.wubigo.com { "IndexDocument": { "Suffix": "index.html" }, "ErrorDocument": { "Key": "404.html" } } 配置DNS C记录 查看S3 Website Endpoints: s.wubigo.com.s3-website-ap-northeast-1.amazonaws.com
CNAME Record s s.wubigo.com.s3-website-ap-northeast-1.amazonaws.com 上传站点内容 aws s3 cp wubigo.github.io s3://s.wubigo.com/ --recursive
AWS免费类型 首次注册后的12个月免费
永久免费
试用
aws永久免费的服务 计算服务 Lambda
1百万请求/月
400,000 GB-seconds of compute time per month
Step
4000/月
存储 DynamoDB
25GB
S3
The S3 free tier allows users to store 5 GB of data with standard storage, issue
20,000 GET requests and 2,000 PUT requests, and transfer 15 GB of data out each
month
Glacier
10GB
查看本月费用 The Cost Explorer API enables you to programmatically query your cost and usage data. You can query for aggregated data such as total monthly costs or total daily usage. You can also query for granular data, such as the number of daily write operations for Amazon DynamoDB database tables in your production environment Linux
aws ce get-cost-and-usage --time-period Start=$(date -u -d "$TODAY" '+%Y-%m-01'),End=$(date -u +"%Y-%m-%d" --date="+1 day") --granularity MONTHLY --metrics UnblendedCost --output text 注意：CE服务是按api调用次数收费
We can no longer assign a public IP address to your instance
The auto-assign public IP address feature for this instance is disabled because you specified multiple network interfaces. Public IPs can only be assigned to instances with one network interface. To re-enable the auto-assign public IP address feature, please specify only the eth0 network interface.
ip MAC=`curl http://169.254.169.254/latest/meta-data/mac` curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/${MAC}/local-ipv4s 配置第二块网卡 ip a | grep ^[[:digit:]] tee -a /etc/network/interfaces.
QUERY aws dynamodb scan --table-name "orders"
本文所说的新手是指还在12个月免费期内的用户
只要有一张信用卡，就可以注册一个AWS账号。
对于AWS新注册账号，有12个月的免费使用额度
具体额度如下：
服务
服务 额度 当月使用统计 服务器 750小时 t2.micro 硬盘 30GB 硬盘快照 1GB 网盘 5G 数据库 25G 函数计算 1百万次调用 下图我这个月的使用额度
计算和存储 从上面的图可以看出，AWS免费额度里面比较鸡肋的是
网盘快照的额度太少，走常规的操作系统镜像备份是要
花钱的，因为一个最小的ubuntu实例镜像就是8G，
如何做到保存自己的最新工作成果，而又额外使用快照从而
节省存储费用呢？
解决办法如下：
硬盘外挂 创建EC2
创建一块硬盘，小于20G即可，并把该硬盘外挂到EC2
在外挂的硬盘里面保存自己的操作数据
使用user_data初始化包括安装常用软件包，自动外挂硬盘等
使用完EC2销毁即可(卸载外挂硬盘千万不要销毁外挂的硬盘)
上面的操作可以使用基础设施配置工具(ansible, terraform, pupport等)
REMOVE ROLE delete policy before delete role
aws iam list-roles aws iam list-role-policies --role-name api-executor aws iam delete-role-policy --role-name api-executor -policy-name "log-writer" aws iam delete-role --role-name pizza-api-executor ADD ROLE POLICY aws iam put-role-policy \ --role-name pizza-api-executor \ --policy-name PizzaApiDynamoDB \ --policy-document file://./roles/dynamodb.json You need to provide a path to dynamodb.json with the file:// prefix. If you are providing an absolute path, keep in mind that you will have three slashes after file:.
NAT GATEWAY 数据备份S3 Infrequent Access Tier, All Storage / Month $0.0125 per GB
Archive 50 TB into Amazon S3 If you perform a one-time migration of 50 TB of 16 MB files into Amazon S3 in US East (Ohio), it costs you the following to use DataSync: (50 TB copied into S3 * 1024 GB * $0.0125/GB) + (1 S3 LIST request * $0.005 / 1000) + (50 TB / 16 MB S3 PUT requests * $0.
典型使用场景 单一公开子网 公开子网和私有子网 企业数据中心+公开子网 企业数据中心 公网网关 An Internet gateway is a fully managed AWS service that performs bi-direction source and destination network address translation for your EC2 instances. Optionally, a VPC may use a virtual private gateway to grant instances secure access to a user’s corporate network via VPN or direct connect links. Instances in a subnet can also be granted outbound only Internet access through a NAT gateway.