Posts

2 min read IT

version notes some only works on 1.13 kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-16T15:29:34Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Starting with Kubernetes 1.12, the K8S.gcr.io/kube-${ARCH}, K8S.gcr.io/etcd and K8S.gcr.io/pause images don’t require an -${ARCH} suffix get all Pending pods kubectl get pods --field-selector=status.phase=Pending images list kubeadm config images list -v 4 I0217 07:28:13.305268 14495 interface.go:384] Looking for default routes with IPv4 addresses I0217 07:28:13.307275 14495 interface.go:389] Default route transits interface "enp0s3" I0217 07:28:13.
1 min read

track http redirection http://wubigo.com/post -> http://wubigo.com/post/ -> https://wubigo.com/post/ curl -IL http://wubigo.com/post HTTP/1.1 301 Moved Permanently Location: https://wubigo.com/post Via: 1.1 varnish X-Cache: HIT X-Cache-Hits: 1 HTTP/1.1 200 OK Content-Length: 0 HTTP/1.1 301 Moved Permanently Strict-Transport-Security: max-age=31556952 Location: http://wubigo.com/post/ Access-Control-Allow-Origin: * X-Cache: HIT X-Cache-Hits: 1 HTTP/1.1 301 Moved Permanently Location: https://wubigo.com/post/ X-Cache: HIT X-Cache-Hits: 1 HTTP/1.1 200 OK Access-Control-Allow-Origin: * Cache-Control: max-age=600 X-Cache: HIT X-Cache-Hits: 1 main goal HTTP/2’s multiplexed connections, allowing multiple streams of data to reach all the endpoints independently.
1 min read

安装mc

https://dl.min.io/client/mc/release/windows-amd64/mc.exe

mc config host add b2 http://192.168.1.3:9000   B2_keyID B2_applicationKey

  • 本地文件同步到b2

    mc cp -r . b2/wubigo/

安装S3CMD

https://github.com/s3tools/s3cmd/releases/download/v2.0.2/s3cmd-2.0.2.tar.gz

sudo python setup.py install

~/.s3cfg

# Setup endpoint
host_base = http://192.168.1.3:9000
host_bucket = http://192.168.1.3:9000
bucket_location = us-east-1
use_https = True

# Setup access keys
access_key =  Q3AM3UQ867SPQQA43P2F
secret_key = zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

# Enable S3 v4 signature APIs
signature_v2 = False

  • 同步本地文件到B2

    s3cmd sync . s3://wubigo/

总结

在不进行任何优化的情况下，s3cmd比mc传输速度快好几倍

1 min read

检查域名使用的名字服务器

https://lookup.icann.org/

更改名字服务器

更改名字服务器需通过域名注册服务商进行更改

1 min read

端到端VXLAN(unicast)

ip a

ip link add vxlan0 type vxlan \
    id 42 \
    dstport 4789 \
    remote 10.12.0.172 \
    local 10.12.2.95 \
    dev eth0


ip -d link show dev vxlan0



ip addr add 192.168.8.101/24 dev vxlan0
ip link set vxlan0 up


ip r
default via 10.12.0.1 dev eth0
10.12.0.0/21 dev eth0  proto kernel  scope link  src 10.12.2.95
192.168.8.0/24 dev vxlan0  proto kernel  scope link  src 192.168.8.101


bridge fdb | grep vxlan0

ip neigh

多播vxlan(multicast)

1 min read 

git clone https://git.zx2c4.com/wireguard-go
1 min read

创建函数

  • 分配角色

    zip function.zip index.js

aws lambda create-function --function-name sns-db-function \
--zip-file fileb://function.zip --handler index.handler --runtime nodejs12.x \
--role arn:aws:iam::465691908928:role/fn-case-role
1 min read

发布测试消息到SNS git clone https://github.com/wubigo/node-fn/blob/master/fn-case/sns_publishtotopic.js node sns_publishtotopic.js Message MESSAGE_TEXT send sent to the topic arn:aws:sns:ap-northeast-1:465691908928:func-topic MessageID is 8b5c90f2-0c74-5985-8a34-c676c0370f73 根据MessageID查看函数执行结果 The $ in [$LATEST] needs to be escaped…[\$LATEST]. aws logs describe-log-groups --query logGroups[*].logGroupName aws logs describe-log-streams --log-group-name '/aws/lambda/my-function' --query logStreams[*].logStreamName aws logs get-log-events --log-group-name '/aws/lambda/my-function' --log-stream-name '2019/12/31/[$LATEST]7467497f9cdb4078a876ab889797793c' { "ingestionTime": 1577764111252, "timestamp": 1577764096184, "message": "2019-12-31T03:48:16.183Z\tc01c9f5e-6c33-40a1-a6d9-c11ab248ab48\tINFO\tEVENT\n{\n \"Records\": [\n {\n \"EventSource\": \"aws:sns\",\n \"EventVersion\": \"1.0\",\n \"EventSubscriptionArn\": \"arn:aws:sns:ap-northeast-1:465691908928:func-topic:2e0e0d95-f1c8-47bd-90ff-40ca4129794b\",\n \"Sns\": {\n \"Type\": \"Notification\",\n \"MessageId\": \"5f80d26e-bdeb-579f-bc81-84ea7ad4e2ae\",\n \"TopicArn\": \"arn:aws:sns:ap-northeast-1:465691908928:func-topic\",\n \"Subject\": null,\n \"Message\": \"MESSAGE_TEXT\",\n \"Timestamp\": \"2019-12-31T03:48:15.
1 min read

https://medium.com/people-ai-engineering/building-a-data-lake-in-aws-9c1fb3876e23

https://towardsdatascience.com/building-a-data-pipeline-from-scratch-on-aws-35f139420ebc

2 min read

restore snapshot shell snapshot.sh !#/bin/bash wget https://nodejs.org/dist/v12.13.1/node-v12.13.1-linux-x64.tar.xz tar xvf node-v12.13.1-linux-x64.tar.xz export PATH=/home/ubuntu/node-v12.13.1-linux-x64/bin:$PATH wget https://manning-content.s3.amazonaws.com/download/0/ddbbd36-251d-42ef-9934-55e5a881a336/FinalSourceCode.zip sudo apt update sudo apt install unzip unzip FinalSourceCode.zip mv Final\ Source\ Code/ sls sudo apt install python-pip pip install awscli which aws_completer cp ~/.bashrc ~/.bashrc_orig tee -a ~/.bashrc <<-'EOF' complete -C '/home/ubuntu/.local/bin/aws_completer' aws export PATH=/home/ubuntu/node-v12.13.1-linux-x64/bin:$PATH EOF aws configure npm install claudia -g claudia -v 5.11.0 cd chapter-03 npm install claudia create \ --region ap-northeast-1 \ --api-module api packaging files npm install -q --no-audit --production npm WARN pizza-api@1.
1 min read

Debugging the Build Process Gatsby’s build and develop steps run as a Node.js application which you can debug using standard tools for Node.js applications. Debugging with Node.js’ built-in console console.log(args) VS Code Debugger (Auto-Config) Preferences: Type node debug into the search bar. Make sure the Auto Attach option is set to on. launch.json launch.json { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes.
1 min read

Client Side Rendering(CSR) Rendering an app in a browser, generally using the DOM The initial HTML rendered by the server is a placeholder and the entire user interface and data rendered in the browser once all your scripts load. PROS Rich site interactions Fast rendering after the initial load Partial real-time updates Cheaper to host & scale CONS SEO and index issues Mostly initial bundle.js load duration Performance issues on old mobile devices/slow networks Social Media crawlers and sharing problems (SMO) Server Side Rendering(SSR) Server rendering generates the full HTML for a page
2 min read

安装两种方式 从最新源代码发布版安装 git设置代理 [user] email = [email protected] name = bigo [http] proxy = http://127.0.0.1:49210 sslverify = false 系统代理 set HTTP_PROXY=http://127.0.0.1:49210/ set HTTPS_PROXY=http://127.0.0.1:49210/ 安装 go get github.com/minio/minio MAKE mkdir -p $GOPATH/src/github.com/minio cd $GOPATH/src/github.com/minio git clone https://github.com/minio/minio.git cd minio/ git checkout RELEASE.2020-01-03T19-12-21Z make -n test go install -v mkdir -p /home/bigo/go/bin which golint 1>/dev/null || (echo "Installing golint" && GO111MODULE=off go get -u golang.org/x/lint/golint) which staticcheck 1>/dev/null || (echo "Installing staticcheck" && wget --quiet https://github.
1 min read

JavaScript Arrow Functions

https://zendev.com/2018/10/01/javascript-arrow-functions-how-why-when.html

1 min read

微服务安全要点

  • 通信链路加密
  • 灵活的服务访问控制，包括细粒度访问策略
  • 访问日志审计
  • 服务提供方可替代性(batteries included)和可集成性

基本概念

  • 安全标识

在K8S，安全标识(service account)代表一个用户，一个服务或一组服务。

  • 安全命名

安全命名定义可运行服务的安全标识

微服务认证

  • 传输层认证
  • 终端用户认证

每一个终端请求通过JWT(JSON Web Token)校验, 支持Auth0, Firebase。

https://medium.facilelogin.com/securing-microservices-with-oauth-2-0-jwt-and-xacml-d03770a9a838

1 min read

AWS leverages a standard JSON Identity and Access Management (IAM) policy document format across many services to control authorization to resources and API actions terraform https://www.terraform.io/docs/providers/aws/r/iam_role_policy.html resource "aws_iam_role_policy" "s3_policy" { name = "s3_policy" role = "${aws_iam_role.lambda_s3_role.id}" policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::bucket-name"] }, { "Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::bucket-name/*"] } ] } EOF } resource "aws_iam_role" "lambda_s3_role" { name = "lambda_s3_role" assume_role_policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "lambda.
2 min read

运行环境 terraform -v Terraform v0.12.16 + provider.aws v2.39.0 创建函数 main.js 'use strict' exports.handler = function(event, context, callback) { var response = { statusCode: 200, headers: { 'Content-Type': 'text/html; charset=utf-8' }, body: '<p>Hello world!</p>' } callback(null, response) } zip ../example.zip main.js 上传 awslocal s3api create-bucket --bucket=terraform-serverless-example awslocal s3 cp example.zip s3://terraform-serverless-example/v1.0.0/example.zip 创建资源 lambda.tf resource "aws_lambda_function" "example" { function_name = "ServerlessExample" # The bucket name as created earlier with "aws s3api create-bucket" s3_bucket = "terraform-serverless-example" s3_key = "v1.
1 min read

Normally, ${SNAP_DATA} points to /var/snap/microK8S/current. snap.microK8S.daemon-docker, is the docker daemon started using the arguments in ${SNAP_DATA}/args/dockerd

$snap start microK8S
$microK8S.docker pull registry.cn-beijing.aliyuncs.com/google_containers/pause:3.1
$microK8S.docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 K8S.gcr.io/pause:3.1

for resource under namespace kube-system all-namespaces don’t include kube-system

$microK8S.kubectl describe po calico-node-4sq5r --namespace=kube-system

https://events.static.linuxfound.org/sites/events/files/slides/2016%20-%20Linux%20Networking%20explained_0.pdf

1 min read

generate configuration file $jupyter notebook --generate-config Writing default config to: /home/bigo/.jupyter/jupyter_notebook_config.py $ diff jupyter_notebook_config.py jupyter_notebook_config.py.bak c.NotebookApp.allow_remote_access = True c.NotebookApp.ip = '0.0.0.0' c.NotebookApp.open_browser = False set or reset password $jupyter notebook password Enter password: Verify password: [NotebookPasswordApp] Wrote hashed password to /home/bigo/.jupyter/jupyter_notebook_config.json then restart notebook server Sharing notebooks When people talk of sharing their notebooks, there are generally two paradigms they may be considering. Most often, individuals share the end-result of their work which means sharing non-interactive, pre-rendered versions of their notebooks; however, it is also possible to collaborate on notebooks with the aid version control systems such as Git
1 min read

函数计算概论

函数计算就是事件驱动架构(EDA），目前函数计算支持的事件类型列表

函数计算事件列表

计费模式

  • 请求数

  • 执行时间

  • 内存分配

优劣势分析

  • 真正做到谁开发谁运维(who code it who run it)

  • 不需要提前做计算容量规划，服务器配置，负责均衡，扩容

代表性产品

  • DB: Aurora