kafka broker connected remotely by ip config/server.properties advertised.listeners=PLAINTEXT://172.16.16.5:9092 ############################# Log Retention Policy ############################# # The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens # from the end of the log. # The minimum age of a log file to be eligible for deletion due to age log.

	rabbit	kafka
创建时间	2007	2011
开发语言	erlang	scala
AMQP	SUPPORT	NO
AGENT	SMART(broker-centric) keeps track of consumer state	dumb(producer-centric)
存储空间	in-memory	disk
INGRESS VOLUME	20K messages/sec	100k/sec messages/sec
CONSUMERS	mostly online(balancing load to many consumer)	online and batch consumer
ROUTING	exchange, binding	simple
history	N/A	replay(删除by size 或时间)
数据压缩	N	Y
SPRING SUPPORT	weak	strong
安全	RBAC backed by a built-in data store, LDAP	JAAS role based access
管理	Web 和 CLI	JMX 和 CLI

阿里云Maven中央仓库为 阿里云云效 提供的公共代理仓库

打开 maven 的配置文件（ windows 机器一般在 maven 安装目录的 conf/settings.xml ），

在标签中添加 mirror 子节点:

<mirror>
  <id>aliyunmaven</id>
  <mirrorOf>*</mirrorOf>
  <name>阿里云公共仓库</name>
  <url>https://maven.aliyun.com/repository/public</url>
</mirror>

mysqladmin status MySQL is multithreaded, so there may be many clients issuing queries for a given table simultaneously. To minimize the problem with multiple client sessions having different states on the same table, the table is opened independently by each concurrent session. This uses additional memory but normally increases performance The table_open_cache and max_connections system variables affect the maximum number of files the server keeps open. If you increase one or both of these values, you may run up against a limit imposed by your operating system on the per-process number of open file descriptors.

innodb-memcached-multiple-get-range-query

native partitioning in-place APIs

install curl -O http://download.redis.io/redis-stable.tar.gz tar xzvf redis-stable.tar.gz cd redis-stable make make test sudo make install config sudo mkdir /etc/redis sudo cp redis-stable/redis.conf /etc/redis sudo adduser --system --group --no-create-home redis sudo mkdir /var/lib/redis sudo chown redis:redis /var/lib/redis sudo chmod 770 /var/lib/redis /etc/redis/redis.conf supervised systemd dir /var/lib/redis # bind localhost start redis-server /etc/redis/redis.conf shutdown redis-cli shutdown redis-cli 127.0.0.1:6379> shutdown run in docker docker run --name redis -network host -v /var/lib/redis:/data /etc/redis/redis.

[mysqld]
server-id = 2
relay-log-index = slave-relay-bin.index
relay-log = slave-relay-bin

mysql>CHANGE MASTER TO MASTER_HOST = 'db2',MASTER_PORT = 3306, MASTER_USER = 'repl_user', MASTER_PASSWORD = 'xyzzy';

Connecting the Master

mysql> START SLAVE;

Server and Operating System Kernel – vm.swappiness Disables swapping completely while 1 causes the kernel to perform the minimum amount of swapping # Set the swappiness value as root echo 1 > /proc/sys/vm/swappiness # Alternatively, using sysctl sysctl -w vm.swappiness=1 # Verify the change cat /proc/sys/vm/swappiness 1 # Alternatively, using sysctl sysctl vm.swappiness vm.swappiness = 1 Filesystems – XFS/ext4/ZFS FILE SIZE mount option EXT4 16TB noatime,data=writeback,barrier=0,nobh,errors=remount-ro XFS 8EiB defaults,nobarrier Disk Subsystem – I/O scheduler Most modern Linux distributions come with noop or deadline I/O schedulers by default, both providing better performance than the cfq and anticipatory ones

mysql 8 测试环境快速搭建(WSL/root远程访问) sudo apt install -y mysql-server mysql --version sudo mysql mysql>CREATE USER 'root'@'%' IDENTIFIED BY '123'; mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'; mysql>flush PRIVILEGES mysql.conf.d/mysqld.cnf :32:bind-address = 0.0.0.0 sudo service mysql restart Create the root user (yes, a new user because what exists is ‘root@localhost’ which is local access only) root用户本地登录 本地登录使用系统认证(auth_socket) sudo mysql root用户远程登录 mysql -u root -p'123' -h 192.168.168.128 mysql> select user,host,plugin from mysql.

mysql install ubuntu 16.04 install mysql 5.7 at default sudo apt-get update sudo apt-get install mysql-server Enable root remote connection mysql -u root -p mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '<password>' WITH GRANT OPTION; mysql>FLUSH PRIVILEGES; SHOW current setting mysql> SHOW VARIABLES WHERE Variable_name LIKE 'innodb%'; MySQL 5.7 has significantly better default values. the following variables are set by default: [mysqld] innodb_buffer_pool_instances=8 innodb_flush_method=O_DIRECT setting of mysql 5.

postgres in docker https://github.com/wubigo/docker-compose/blob/main/docker-compose-psql.yml 启用pg_stat_statements docker volume inspect root_postgresql_data |grep "Mountpoint" echo "shared_preload_libraries = 'pg_stat_statements'" > /var/lib/docker/volumes/root_postgresql_data/_data/pgdata/postgresql.conf INSTALL sudo apt-get install -y postgresql-client psql --version TEST psql -h cmp.clv6a9yh.ap-northeast-1.rds.amazonaws.com -U postgres postgres=> \list List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | rdsadmin | rdsadmin | UTF8 | en_US.UTF-8 | en_US.

• whether MySQL Server supports partitioning

  mysql -u root -p123456 -e "SHOW PLUGINS;" |grep partition
  

show the last queries

mysql>SET GLOBAL log_output = 'FILE';
mysql>SET GLOBAL general_log = 'ON';
mysql>SHOW VARIABLES WHERE Variable_name LIKE 'general_log_file'

Disable ONLY_FULL_GROUP_BY

SHOW VARIABLES WHERE Variable_name LIKE 'sql_mode';
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value                                                                                                                                     |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| sql_mode      | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION |

[mysqld]  
sql_mode = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

mysql> SHOW CREATE TABLE <tablename>;

[mysqld]

log-bin = master-bin
log-bin-index = master-bin.index
server-id = 1

Grant the user to retrieve the binary log from the master

mysql>CREATE USER repl_user;
GRANT REPLICATION SLAVE ON *.* TO repl_user IDENTIFIED BY 'xyzzy';

API design: Choosing between names and identifiers in URLs

API design: Choosing between names and identifiers in URLs

mapr downside with intermediate state “MapReduce’s approach of fully materializing intermediate state has downsides compared to Unix pipes: * A MapReduce job can only start when all tasks in the preceding jobs (that generate”“its inputs) have completed, whereas processes connected by a Unix pipe are started at the same time, with output being consumed as soon as it is produced. Skew or varying load on different machines means that a job often has a few straggler tasks that take much longer to complete than the others.

update python to 2.7.11 on ubuntu apt_repository need python >2.7.9 ansible_galaxy has bug on ansible1.9.4 on proxy https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes-python2.7 $sudo -E apt-add-repository ppa:fkrull/deadsnakes-python2.7 (deb http://ppa.launchpad.net/fkrull/deadsnakes-python2.7/ubuntu trusty main) $sudo -E apt-get install --only-upgrade python2.7 $python --version Python 2.7.11 Install $ sudo -E apt-get install software-properties-common $ sudo -E apt-add-repository ppa:ansible/ansible $ sudo -E apt-get update $ sudo -E apt-get install –only-upgrade ansible Install latest ansible $git clone git://github.com/ansible/ansible.git –recursive $virtualenv -p /home/whg/python2.

系统配置 /etc/sysctl.conf vm.swappiness = 1 vm.overcommit_memory = 1 改变数据目录 sudo install -o redis -g redis -d /mnt/redis-data > config get dir 1) "dir" 2) "/mnt/redis-data" /lib/systemd/system/redis-server.service [Service] ReadWriteDirectories=-/mnt/redis-data pidfile NOT FOUND FROM SYSTEMD /etc/redis/redis.conf pidfile /var/run/redis/redis-server.pid /lib/systemd/system/redis-server.service PIDFile=/run/redis/redis-server.pid 删除消费组 XGROUP DESTROY STREAM:TEST STRRAM:TEST:GROUP Redis latency problems troubleshooting Make sure you are not running slow commands that are blocking the server. Use the Redis Slow Log feature to check this.

B+树 vs. LSM树

RDBMS使用B+树专门针对磁盘存储而优化的N叉排序树 NoSQL使用LSM树

查看表状态 show table status FROM redis_db like 'point_value'; +-------------+--------+---------+------------+-----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+ | Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment | +-------------+--------+---------+------------+-----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+ | point_value | InnoDB | 10 | Dynamic | 316755485 | 143 | 45420118016 | 0 | 0 | 3145728 | NULL | 2022-01-30 18:55:44 | 2022-02-06 18:17:56 | NULL | utf8_general_ci | NULL | | | +-------------+--------+---------+------------+-----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+ 1 row in set (0.