REDIS Redis (Remote Dictionary Server)是一个流行的开源内存 提供高级键值抽象的键值存储。 Redis 是单线程的，它只处理一个命令 客户端在进程的主线程中一次。 不同于传统的KV 键是简单数据类型（通常是字符串）的系统，键 在 Redis 中可以用作复杂的数据类型，例如哈希、列表、 集和排序集。 此外，Redis 支持复杂的原子 对这些数据类型的操作（例如，从 一个列表，将具有给定分数的新值插入排序集等）。 Redis 抽象和高摄取速度已被证明特别重要 对于许多延迟敏感的任务很有用。 因此，Redis 已获得广泛采用，并被越来越多的人使用 生产环境中的公司. Redis 支持高可用性和持久性。 高可用性 是通过将数据从主节点复制到 从节点并同步它们。 当一个主进程失败时，它的 对应的从属进程已准备好接管后续进程 称为故障转移。 持久性可以通过以下任一方式配置 以下两个选项： 使用时间点快照文件 称为 RDB（Redis 数据库） 使用名为AOF（仅附加文件）。 注意这三种机制（AOF重写、RDB 快照和复制）依赖 fork 获取 进程内存的时间点快照并将其序列化 （而主进程继续为客户端命令提供服务） 项目简介 最近在做工业自动化(IIoT)项目，涉及到很多场景需要对一系列设备进行监控和信号处理。 该类场景对实时处理能力，系统稳定性，高可用性，容灾能力等等要求非常高。 其中几个核心的需求： 设备数据不能丢失 实时告警(毫秒级延迟) 设备数据必须优先在边缘节点处理，边缘节点的物理服务器只有两台 每个边缘节点接入的设备上行数据量大概6万点/秒，数据包小于1K 中心需要汇聚和分析所有边缘节点的设备数据 技术现状 实时数据处理 实时数据处理是一个自 1990 年代以来一直在研究的问题 。 产生的数据量增加了，加上越来越复杂的软件解决方案 开发，需要满足这些需求出现了流式应用程序，例如 欺诈检测、网络监控和电子交易依赖于实时数据处理 确保所提供的服务被认为是正确和可靠的。 绝大多数现代应用程序使用某种数据库管理系统 处理数据。当应用程序收集或生成数据时，它会被存储和索引 它可以在以后由应用程序查询。但是，对于具有更严格 的实时数据处理的要求，这不是一个合适的选项，而是流 处理开始发挥作用的地方。流处理在接收数据时直接处理数据。实时流 处理应用程序通常具有必须满足的某些关键要求。有的 输入和处理后的数据输出之间的延迟是实现实时的关键特征 应用。更传统的批处理方法需要以这样的方式收集数据 称为批次，其中处理只能在每个批次的最终数据块完成后开始 到达的。对于实时用例，这导致的延迟是不可接受的，因为这些实时的延迟 流应用程序最好在毫秒内。

虎年新春快乐

Maybe this year, we ought to walk through the rooms of our lives not looking for flaws, but looking for potential—Ellen Goodman

社会价值

个人的价值在于为别人和社会解决问题和提供帮助，在帮助别人的过程 中促进个人成长。读书不是目的，读书只是为大脑提供了输入，产生对 社会有用的输出才是读书的真正目的

思考

思考比行动重要。害怕思考，草率行动是懒惰的表现。思考就是抑制动物的 本性过程，思考是个人成长的最快途径

沟通

理解对方是沟通的前提条件。沟通就是让对方认识到自己提议的价值， 让对方受益。无论沟通的对象是朋友，家人，同事，不要首先想着 如何改变别人，重要的调整自己。

业务和技术

软件解决方案的出发点是解决客户问题。解决问题与技术的先进性与否无关， 很多情况下解决方案取决于公司的组织形式和资源条件。运维成本是大型 软件解决方案最主要的成本。监控的目的在于预防失败。 越早发现问题，解决成本越低。

知识积累

价值与风险是统一的，看待机会既要看到可能的价值，同时认识到风险。 巨大的价值必然伴随着巨大的风险。偶然发现的所谓的价值机会可能只是 一个无知陷阱。知识和财富不仅在于积累更在于传承。财富最重要的是 在于保值。

BUILD

mkdir -p $GOPATH/src/github.com/mailflux
git clone https://github.com/wubigo/mainflux.git
cd github.com/mailflux/mailflux/
make
make dockers_dev
make run

Provision the System

mainflux-cli provision test

GET TOKEN

curl -k https://172.21.53.253/tokens

{
  "email": "[email protected]",
  "password": "12345678"
}

Configure an MQTT Client Agent within the IoT Gateway Plug-In for KEPServerEX to send data to Ali IoT. The connection can be made using MQTT over Transmission Control Protocol (TCP) and MQTT over Transport Layer Security (TLS). set up Kepware KEPServerEX IoT Gateway on Windows to connect with the MQTT bridge of IoT Core to push streaming data to Cloud and send control messages from IoT Core back to KEPServerEX

Data catalogs solve the problem by tagging fields and data sets with consistent business terms and providing a shopping-type interface that allows the users to find data sets by describing what they are looking for using the business terms that they are used to, and to understand the data in those data sets through tags and descriptions that use business terms. Data lakes are the do-it-yourself version of a data warehouse, allowing data engineering teams to pick and choose the various metadata, storage, and compute technologies they want to use depending on the needs of their systems.

多租户的运行 不同的应用运行在同一个集群 相同应用的多个实例运行在同一个集群 敏感数据应用独享实例，非敏感数据共享实例 hybrid architectures are also possible, such as a SaaS provider using a combination of per-customer workloads for sensitive data, combined with multi-tenant shared services. 隔离方式 控制面隔离机制 名字空间 访问控制 资源配额 数据面隔离机制 网络隔离 Pod-to-pod communication can be controlled using Network Policies, which restrict communication between pods using namespace labels or IP address ranges. In a multi-tenant environment where strict network isolation between tenants is required, starting with a default policy that denies communication between pods is recommended with another rule that allows all pods to query the DNS server for name resolution

配置

’.azure/config`

[defaults]
location = westus

[cloud]
name = AzureCloud

[core]
output = table

az config set defaults.location=westus2 defaults.group=MyResourceGroup

az v2不支持config，直接修改配置文件

AZURE REGION az account list-locations --query "sort_by([].{DisplayName:displayName, Name:name}, &DisplayName)" --output table 创建函数APP #!/bin/bash # Function app and storage account names must be unique. storageName=mystorageaccount$RANDOM functionAppName=myserverlessfunc$RANDOM region=westeurope # Create a resource group. az group create --name myResourceGroup --location $region # Create an Azure storage account in the resource group. az storage account create \ --name $storageName \ --location $region \ --resource-group myResourceGroup \ --sku Standard_LRS # Create a serverless function app in the resource group.

安装FUNC npm i -D azure-functions-core-tools@3 export PATH=./ export CLI_DEBUG=1 func host start --verbose 安装playwright-chromium export PLAYWRIGHT_BROWSERS_PATH=0 npm install [email protected] 确认chrome的存放路径 node_modules/playwright-chromium/.local-browsers/chromium-792639 创建函数 /home/ubuntu/sls/azure-sls/node_modules/.bin/func init func new func start 本地测试 export CLI_DEBUG=1 func host start --verbose host.json { "version": "2.0", "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": true, "excludedTypes": "Request" } } }, "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle", "version": "[2.*, 3.0.0)" } } 如果遇到如下问题 Value cannot be null.

创建函数 npm install -g [email protected] sls -v Framework Core: 2.65.0 Plugin: 5.5.1 SDK: 4.3.0 Components: 3.17.2 sls create -t azure-nodejs -p azure-fn cd azure-fn npm install npm list |grep serverless-azure-functions └─┬ [email protected] 部署函数 set AZURE_SUBSCRIPTION_ID=02a23ad5- set AZURE_TENANT_ID=e9950462 set AZURE_CLIENT_ID=39258bc8 set AZURE_CLIENT_SECRET=hYdvD0 sls deploy --dryrun 测试 sls invoke -f hello -d '{"name": "Azure"}' 清理 empty.json { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { }, "variables": { }, "resources": [ ], "outputs": { } } az group deployment create --mode complete --template-file .

CQuasar App Flow In order to better understand how a boot file works and what it does, you need to understand how your website/app boots: Quasar is initialized (components, directives, plugins, Quasar i18n, Quasar icon sets) Quasar Extras get imported (Roboto font – if used, icons, animations, …) Quasar CSS & your app’s global CSS are imported App.vue is loaded (not yet being used) Store is imported (if using Vuex Store in src/store) Router is imported (in src/router) Boot files are imported Router default export function executed Boot files get their default export function executed (if on Electron mode) Electron is imported and injected into Vue prototype (if on Cordova mode) Listening for “deviceready” event and only then continuing with following steps Instantiating Vue with root component and attaching to DOM

计算时间 vs 请求数量 平均计算时长 427586 / 27386 = 15.6 (秒) 设置默认配置 aws configure list-profiles default us-east-1 us-east-2 us-west-2 us-west-1 eu eu-west-1 af-south-1 ap-east-1 ap-south-1 ap-northeast-3 ap-northeast-2 ap-southeast-1 ap-southeast-2 ca-central-1 eu-west-2 eu-south-1 eu-west-3 eu-north-1 me-south-1 sa-east-1 export AWS_PROFILE=us 下载部署包 aws lambda get-function --function-name webdriver "Code": { "RepositoryType": "S3", "Location": "https://awslambda-ap-ne-1-tasks.s3.ap-northeast-1.amazonaws.com/snapshots/webdriver-aeb2eb63-9baf-4d06-9d3a-79459b172200?versionId=a71tk2dwwmvW1lPNB5VHKq8SbGS3laqE&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHIaDmFwLW5vcnRoZWFzdC0xIkcwRQIhAMRkIxPh1Fkd2nlCzgiDbsrmnCZEVunHibw2Cm6cyRIUAiB5t60IO6iESPDeUsTuQEjGyLfI73QyMK1mJY9Al70yECqNBAj8%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAIaDDkxOTk4MDkyNTEzOSIMjVD0S8e1HGmJujr6KuEDO8SCL9OcolFOwL4IKMbE3euJLEtiGjSxH6c8jRPbnjp07Zf%2BxrOfJmWT2MORQs0RAQSLJV5nOFfRWTIPI4dSNhI3Q628XqklZ8%2BF1UktvA5vRdEU3LhDvOSsDCmL19k&X-Amz-Signature=7f876918ec5283db390a3037512e7ad62e434330ec3e406db18b25f25f3da0fe" 从Location下载部署包 PROF = "eu-central-1" aws lambda create-function --function-name webdriver --runtime nodejs12.x --zip-file fileb:///home/ubuntu/webdriver.zip --handler index.handler --role arn:aws:iam::762491489154:role/service-role/webdriver-role-3hxi35t5 --timeout 63 --memory-size 1024 --layers arn:aws:lambda:us-east-1:764866452798:layer:chrome-aws-lambda:25 --profile us 配置 role-policy.

aws客户端环境准备 git clone pip install awscli https://github.com/wubigo/API.git python API/python/aws/aws.py cp API/python/aws/cred.json ~/.aws/credentials cp API/python/aws/config ~/.aws/config 创建函数部署包 mkdir lambda_web wget https://github.com/wubigo/API/blob/master/nodejs/lambda/aws/index.js -P lambda_web zip -r webdriver.zip lambda_web/* 配置 policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": ["lambda.amazonaws.com", "s3.amazonaws.com"] }, "Action": "sts:AssumeRole" } ] } export ACCOUNT_ID=820934811997 aws iam create-role --role-name lambda-s3 --assume-role-policy-document file://policy.json aws iam attach-role-policy --role-name lambda-s3 --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole aws iam attach-role-policy --role-name lambda-s3 --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess 复制 aws lambda create-function --function-name webdriver --runtime nodejs12.

HDFS The maximum number of files in HDFS depends on two things: total storage space in the cluster the heap size of the NameNode. 1) You can find out what percentage of storage has been used from HDFS NameNode UI. 2) The basic rule of thumb is that 1 GB heap is needed for every million of files. Each file object and each block object takes about 150 bytes of the memory.

用户代理 mobile devices browsing the web often see a pared-down ver‐ sion of sites, lacking banner ads, Flash, and other distractions. If you try changing your User-Agent to something like the following, you might find that sites get a little easier to scrape! User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53 scrapy architecture The data flow in Scrapy is controlled by the execution engine, and goes like this:

Replication Method Replication Method - Replication method to use for extracting data from the database. STANDARD replication requires no setup on the DB side but will not be able to represent deletions incrementally. CDC uses the Binlog to detect inserts, updates, and deletes. This needs to be configured on the source database itself. S3 Support in Apache Hadoop Apache Hadoop ships with a connector to S3 called “S3A”, with the url prefix “s3a:“; its previous connectors “s3”, and “s3n” are deprecated and/or deleted from recent Hadoop versions.

打开阅读模式

Open a new tab and enter chrome://flags/#enable-reader-mode to jump directly to the Reader Mode Flag.

使用

Reader Mode in Chrome is mad-easy to use. When you’re on a page that you’d like to push into the reader view, click on the three-dot menu button in the upper right, and then choose “Distill page.”

升级YARN npm install -g [email protected] > [email protected] preinstall C:\local\node-v14.17.5-win-x64\node_modules\yarn > :; (node ./preinstall.js > /dev/null 2>&1 || true) C:\local\node-v14.17.5-win-x64\yarn -> C:\local\node-v14.17.5-win-x64\node_modules\yarn\bin\yarn.js C:\local\node-v14.17.5-win-x64\yarnpkg -> C:\local\node-v14.17.5-win-x64\node_modules\yarn\bin\yarn.js + [email protected] updated 1 package in 0.7s yarn link /home/ubuntu/.config/yarn/link/matrix-js-sdk ubuntu@ip-172-31-44-135:~/.config/yarn/link$ ll total 8 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 19 09:31 ./ drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 19 07:25 ../ lrwxrwxrwx 1 ubuntu ubuntu 22 Aug 19 09:31 matrix-js-sdk -> ../../../matrix-js-sdk/ lrwxrwxrwx 1 ubuntu ubuntu 25 Aug 19 07:25 matrix-react-sdk -> .

准备环境

node -v
v14.17.5
npm install -g [email protected]

链接SDK

git clone https://github.com/matrix-org/matrix-js-sdk.git
pushd matrix-js-sdk
yarn link
yarn install
popd


git clone https://github.com/matrix-org/matrix-react-sdk.git
pushd matrix-react-sdk
yarn link
yarn link matrix-js-sdk
yarn install
popd



git clone https://github.com/vector-im/element-web.git
cd element-web
yarn link matrix-js-sdk
yarn link matrix-react-sdk
yarn install
yarn reskindex
yarn start

启动

cp config.sample.json config.json

curl http://127.0.0.1:8080/

登录

homeserver：http://192.168.43.16:8008/