SHELL

Shadowsocks Ubuntu Client

install client pip install shadowsocks client.json { "server":"server-ip", "server_port":8000, "local_port":3050, "password":"your-password", "timeout":600, "method":"aes-256-cfb" } { "server":"your_server_ip", #ss服务器IP "server_port":your_server_port, #端口 "local_address": "127.0.0.1", #本地ip "local_port":1080, #本地端口 "password":"your_server_passwd",#连接ss密码 "timeout":300, #等待超时 "method":"rc4-md5", #加密方式 "fast_open": false, # true 或 false。如果你的服务器 Linux 内核在3.7+,可以开启 fast_open 以降低延迟。开启方法: echo 3 > /proc/sys/net/ipv4/tcp_fastopen 开启之后,将 fast_open 的配置设置为 true 即可 "workers": 1 # 工作线程数 } sudo apt-get install privoxy /etc/privoxy/config listen-address 127.0.0.1:8118 forward-socks5 / 127.0.0.1:1080 . systemctl restart privoxy.

Cassandra

特性 与hadoop集成,支持MR数据读取 二级索引 支持长行(最多20亿的列) 动态schema更改 bulk load 其他数据源如hadoop with sstableloader, CSV importing with cqlsh DTCS优化时序数据性能 DB CATEGORY BY CAP CA To primarily support consistency and availability means that you’re likely using two-phase commit for distributed transactions. It means that the system will block when a network partition occurs, so it may be that your system is limited to a single data center cluster in an attempt to mitigate this.

Spring Oauth2

http://localhost:8080/oauth/token curl -u eagleeye:thisissecret -i -H ‘Accept:application/json’ -d “grant_type=password&scope=webclient&username=will&password=pass” -H “Content-Type: application/x-www-form-urlencoded” -X POST http://localhost:8080/oauth/token access protected resource

Gitlab Notes

push to a mirror repository push to github at same time when a commit is pushed to gitlab Protected Branches By default, protected branches are designed to: prevent their creation, if not already created, from everybody except Maintainers prevent pushes from everybody except Maintainers prevent anyone from force pushing to the branch prevent anyone from deleting the branch Project members permissions NOTE: In GitLab 11.0, the Master role was renamed to Maintainer The following table depicts the various user permission levels in a project.

Kafka vs Rabbit

rabbit kafka 创建时间 2007 2011 开发语言 erlang scala AMQP SUPPORT NO AGENT SMART(broker-centric) keeps track of consumer state dumb(producer-centric) 存储空间 in-memory disk INGRESS VOLUME 20K messages/sec 100k/sec messages/sec CONSUMERS mostly online(balancing load to many consumer) online and batch consumer ROUTING exchange, binding simple history N/A replay(删除by size 或时间) 数据压缩 N Y SPRING SUPPORT weak strong 安全 RBAC backed by a built-in data store, LDAP JAAS role based access 管理 Web 和 CLI JMX 和 CLI

Mysql Tuning on Many Tables

mysqladmin status MySQL is multithreaded, so there may be many clients issuing queries for a given table simultaneously. To minimize the problem with multiple client sessions having different states on the same table, the table is opened independently by each concurrent session. This uses additional memory but normally increases performance The table_open_cache and max_connections system variables affect the maximum number of files the server keeps open. If you increase one or both of these values, you may run up against a limit imposed by your operating system on the per-process number of open file descriptors.

Mysql Tuning on Query

innodb-memcached-multiple-get-range-query native partitioning in-place APIs

Mysql Slave

[mysqld] server-id = 2 relay-log-index = slave-relay-bin.index relay-log = slave-relay-bin mysql>CHANGE MASTER TO MASTER_HOST = 'db2',MASTER_PORT = 3306, MASTER_USER = 'repl_user', MASTER_PASSWORD = 'xyzzy'; Connecting the Master mysql> START SLAVE;

Mysql Tuning on OS

Server and Operating System Kernel – vm.swappiness Disables swapping completely while 1 causes the kernel to perform the minimum amount of swapping # Set the swappiness value as root echo 1 > /proc/sys/vm/swappiness # Alternatively, using sysctl sysctl -w vm.swappiness=1 # Verify the change cat /proc/sys/vm/swappiness 1 # Alternatively, using sysctl sysctl vm.swappiness vm.swappiness = 1 Filesystems – XFS/ext4/ZFS FILE SIZE mount option EXT4 16TB noatime,data=writeback,barrier=0,nobh,errors=remount-ro XFS 8EiB defaults,nobarrier Disk Subsystem – I/O scheduler Most modern Linux distributions come with noop or deadline I/O schedulers by default, both providing better performance than the cfq and anticipatory ones

Mysql 5.7 InnoDB

index buffer Depends on Storage Engine MyISAM (Caches Index Pages From .MYI files) SELECT FLOOR(SUM(index_length)/POWER(1024,2)) IndexSizesMB FROM information_schema.tables WHERE engine='MyISAM' AND table_schema NOT IN ('information_schema','performance_schema','mysql'); Subtract that from key_buffer_size. InnoDB (Caches Data and Index Pages) SELECT FLOOR(SUM(data_length+index_length)/POWER(1024,2)) InnoDBSizeMB FROM information_schema.tables WHERE engine='InnoDB'; Subtract that from innodb_buffer_pool_size. convert all tables from InnoDB into MyISAM SET @DATABASE_NAME = 'guowang'; SELECT CONCAT('ALTER TABLE `', table_name, '` ENGINE=MyISAM;') AS sql_statements FROM information_schema.

Mysql5.7

mysql install ubuntu 16.04 install mysql 5.7 at default sudo apt-get update sudo apt-get install mysql-server Enable root remote connection mysql -u root -p mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '<password>' WITH GRANT OPTION; mysql>FLUSH PRIVILEGES; SHOW current setting mysql> SHOW VARIABLES WHERE Variable_name LIKE 'innodb%'; MySQL 5.7 has significantly better default values. the following variables are set by default: [mysqld] innodb_buffer_pool_instances=8 innodb_flush_method=O_DIRECT setting of mysql 5.

Mysql Partition

whether MySQL Server supports partitioning mysql -u root -p123456 -e "SHOW PLUGINS;" |grep partition

Mysql 5.7 SQL MODE

Disable ONLY_FULL_GROUP_BY SHOW VARIABLES WHERE Variable_name LIKE 'sql_mode'; +---------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | Variable_name | Value | +---------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | sql_mode | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | [mysqld] sql_mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION" mysql> SHOW CREATE TABLE <tablename>;

Mysql Master

[mysqld] log-bin = master-bin log-bin-index = master-bin.index server-id = 1 Grant the user to retrieve the binary log from the master mysql>CREATE USER repl_user; GRANT REPLICATION SLAVE ON *.* TO repl_user IDENTIFIED BY 'xyzzy';

Mysql 5.7 MyISAM

key_buffer_size the size of the index buffers held in memory, which affects the speed of index reads recommend: 25% or more of the available server memory A good way to determine whether to adjust the value is to compare the key_read_requests value, which is the total value of requests to read an index, and the key_reads values, the total number of requests that had to be read from disk.

shell script

push docker images to ali registry-mirrors https://cr.console.aliyun.com #!/usr/bin/env bash docker login --username=wubigo registry.cn-beijing.aliyuncs.com docker images | grep v1.13 | awk '{ print $1 }' | sed --expression=s'/K8S.gcr.io\///' | xargs -i -t docker tag K8S.gcr.io/{}:v1.13.3 registry.cn-beijing.aliyuncs.com/co1/{}:v1.13.3 docker images |grep "registry.cn-beijing.aliyuncs.com"| awk '{ print $1 }'| sed --expression=s'/registry.cn-beijing.aliyuncs.com\/co1\///' | xargs -i -t docker push registry.cn-beijing.aliyuncs.com/co1/{}:v1.13.3 docker push through cache #!/usr/bin/env bash if [ -z "$VM" ]; then VM = t1 echo "VAR VM is not set" exit fi tee daemon.

应该掌握的linux命令

LINUX shell常用工具提供强大的功能,在日常中熟练掌握能给我 带来不少动能 grep cat find head/tail wc awk shuf 查找 在logs目录下查找所有包含2010_05_02的日志文件 ls logs/ | grep 2010_05_02 pip freeze | grep scipy scipy==1.1.0 grep -oP "'[\w]+ == [\d.]+'" setup.py scipy == 1.1.0 # find . -name '..*swp' -delete awk head -n 1 data.csv | awk -F ',' '{print NF}' shuf 从数据集中随机取50个样本 cat big_csv.csv | shuf | head -n 50 > sample_csv.csv