# 集群部署 jetlinks以及所需中间件的集群部署 ::: tip 注意： 仅企业版以上支持。 本文档以172.16.4.13、172.16.4.12、172.16.4.11三台服务器来搭建集群。 ::: ## 集群架构图 ![集群架构图](./cluster.svg) ## redis集群安装配置 这里准备在一台服务器上搭建一个三主三从的redis集群。 1. 安装redis 先[下载redis](http://download.redis.io/releases/),本文档以redis5.0.5为例。 下载完成后进行解压、编译、安装。 ::: tip 注意： redis是由C语言编写的，它的运行需要C环境，所以编译前需安装 gcc。 ::: 2. 新建一个cluster文件夹，用来存放集群节点目录。 分别在172.16.4.11上创建7000、7003、7001、7004、7002、7005六文件夹，，这些节点分别使用7000、7003、7001、7004、7002、7005端口，以7000节点为例配置如下： ```bash port 7000 bind 172.16.4.11 cluster-enabled yes cluster-config-file nodes_7000.conf cluster-node-timeout 5000 appendonly no daemonize yes ``` 其他节点只需修改端口和文件名，依次按此进行配置即可，配置完成后启动节点。 ::: tip 注意： 建议appendonly在从节点开启，主节点关闭。 ::: 2. 主从节点分配 ```bash redis-cli --cluster create 172.16.4.11:7000 172.16.4.11:7001 172.16.4.11:7002 172.16.4.11:7003 172.16.4.11:7004 172.16.4.11:7005 --cluster -replicas 1 ``` ::: tip 注意： 　其中-replicas 1表示每个主节点1个从节点 ::: 3. 查看集群节点信息 ```shell script ops@jetlinks-server-3:~/data/redis-5.0.5$ redis-cli -h 172.16.4.11 -p 7000 cluster nodes 562558e5afa0575d1059c47b9531a37cd75a9190 172.16.4.11:7003@17003 slave 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 0 1594368796612 4 connected bd974dbdd2f8447b27375c832c4c9c99328f4487 172.16.4.11:7005@17005 slave 3ab893f4cdcfaa52254a8cece2b54b561de29990 0 1594368797514 6 connected 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 172.16.4.11:7002@17002 master - 0 1594368797615 3 connected 10923-16383 3ab893f4cdcfaa52254a8cece2b54b561de29990 172.16.4.11:7001@17001 master - 0 1594368797000 2 connected 5461-10922 238585a079196b0ab15ebb47ac681d42c083cdaf 172.16.4.11:7000@17000 myself,master - 0 1594368796000 1 connected 0-5460 d1ae6c623694c5f1fafc64d7abbeac4a9926ff49 172.16.4.11:7004@17004 slave 238585a079196b0ab15ebb47ac681d42c083cdaf 0 1594368796000 5 connected ``` redis集群搭建请参考[redis官方文档](https://redis.io/topics/cluster-tutorial) ## elasticsearch集群安装配置 1. 分别在三台服务器上安装elasticsearch 前往官网[下载](https://www.elastic.co/cn/downloads/elasticsearch)。 下载完成后安装es。 2. 配置elasticsearch.yml文件 ```shell script ops@jetlinks-server-3:~/elasticsearch/config$ sudo vi elasticsearch.yml # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: es-cluster # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: node-3 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 172.16.4.11 transport.tcp.port: 9300 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["172.16.4.13:9300", "172.16.4.12:19300","172.16.4.11:9300"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # discovery.zen.minimum_master_nodes: 1 # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true ``` 三台服务器不同的配置： ```shell script #172.16.4.11 node.name: node-3 network.host:172.16.4.11 ``` ```shell script #172.16.4.12 node.name: node-2 network.host:172.16.4.12 ``` ```shell script #172.16.4.13 node.name: node-1 network.host:172.16.4.13 ``` 3 在三台服务器上分别启动es ```shell script ./bin/elasticsearch ``` ## 数据库启动 本文档案例使用docker启动postgresql，可参考[docker-compose.yml](https://github.com/jetlinks/jetlinks-community/blob/master/docker/run-all/docker-compose.yml)文件中的postgresql配置。 ## jetlinks集群安装配置 1.修改application.yml,并将jetlinks打成jar包 application.yml文件： ```yaml server: port: 8844 max-http-header-size: 64KB spring: profiles: active: dev application: name: jetlinks-platform jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: Asia/Shanghai serialization: WRITE_DATES_AS_TIMESTAMPS: true default-property-inclusion: non_null resources: static-locations: file:./static,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/, classpath:/public/ redis: # redis集群配置 cluster: nodes: - 172.16.4.11:7000 - 172.16.4.11:7001 - 172.16.4.11:7002 - 172.16.4.11:7003 - 172.16.4.11:7004 - 172.16.4.11:7005 max-redirects: 3 lettuce: pool: max-active: 1024 timeout: 20s serializer: jdk # 设置fst时,redis key使用string序列化,value使用 fst序列化. database: 0 r2dbc: url: r2dbc:postgresql://172.16.4.12:5432/jetlinks #数据库地址 本文档数据库为docker启动 username: postgres password: jetlinks pool: max-size: 32 codec: max-in-memory-size: 50MB rabbitmq: # 开启了device.message.writer.rabbitmq.enabled=true时生效 host: localhost port: 5672 username: admin password: jetlinks kafka: # 开启了device.message.writer.kafka.enabled=true时生效 consumer: client-id: ${spring.application.name}-consumer:${server.port} group-id: ${spring.application.name} max-poll-records: 1000 producer: client-id: ${spring.application.name}-producer:${server.port} acks: 1 retries: 3 bootstrap-servers: ["127.0.0.1:9092"] reactor: debug-agent: enabled: true # 开启调试代理,在打印异常时将会生成调用踪栈信息 easyorm: default-schema: public # 数据库默认的schema dialect: postgres #数据库方言 elasticsearch: client: max-conn-total: 128 connect-timeout: 5000 socket-timeout: 5000 connection-request-timeout: 8000 hosts: # es集群配置 - http://172.16.4.13:9200 - http://172.16.4.12:19200 - http://172.16.4.11:9200 index: default-strategy: time-by-month #默认es的索引按月进行分表 settings: number-of-shards: 1 # es 分片数量 number-of-replicas: 0 # 副本数量 device: message: writer: time-series: enabled: true # 直接写出设备消息数据到时序数据库 kafka: enabled: false # 推送设备消息到kafka consumer: true # 从kafka订阅消息并写入到时序数据库 topic-name: device.message rabbitmq: enabled: false # 推送设备消息到rabbitMQ consumer: true # 从rabbitMQ订阅消息并写入到时序数据库 thread-size: 4 # 消费线程数 auto-ack: true # 自定应答,为true可能导致数据丢失,但是性能最高。 topic-name: device.message # exchange名称 ignore-message-types: include-message-types: hsweb: cors: enable: true configs: - path: /** allowed-headers: "*" allowed-methods: ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"] allowed-origins: ["*"] allow-credentials: true max-age: 1800 dict: enum-packages: org.jetlinks file: upload: static-file-path: ./static/upload static-location: http://demo2.jetlinks.org:8080/upload # 此地址为服务器域名，将通过nginx代理到固定服务器上 webflux: response-wrapper: enabled #开启响应包装器(将返回值包装为ResponseMessage) # auth: #默认的用户配置 # users: # admin: # username: admin # password: admin # name: 超级管理员 authorize: auto-parse: true cache: type: redis redis: local-cache-type: guava jetlinks: server-id: ${spring.application.name}:${server.port} #设备服务网关服务ID,不同服务请设置不同的ID logging: system: context: server: ${spring.application.name} protocol: spi: enabled: false # 为true时开启自动加载通过依赖引入的协议包 rule: engine: server-id: ${jetlinks.server-id} server-name: ${spring.application.name} logging: level: org.jetlinks: info rule.engine: info org.hswebframework: debug org.springframework.transaction: debug org.springframework.data.r2dbc.connectionfactory: debug io.micrometer: warn org.hswebframework.expands: error system: warn org.jetlinks.rule.engine: info org.jetlinks.gateway: info org.springframework: warn org.apache.kafka: warn org.jetlinks.pro.device.message.writer: debug # org.elasticsearch: error config: classpath:logback-spring.xml vertx: max-event-loop-execute-time-unit: seconds max-event-loop-execute-time: 30 max-worker-execute-time-unit: seconds max-worker-execute-time: 30 prefer-native-transport: true micrometer: time-series: tags: server: ${spring.application.name} metrics: default: step: 30s ``` ::: tip 注意： 在多台服务器上启动jetlinks，配置中的jetlinks.server-id必须互不相同。 ::: 在项目根目录执行： ```shell script mvn clean package -DskipTests ``` 2. 分别在三台服务器上启动jetlinks 文档以脚本方式启动，脚本如下： ```shell script #!/bin/bash nohup java -jar -Dspring.application.name=jetlinks-cluster-test-3 jetlinks-standalone.jar >jetlinks-pro.log 2>&1 & ``` ::: tip 注意： 三台服务启动的jetlinks.server-id应该互不相同，在application.yml中jetlinks.server-id引用了spring.application.name，所以此处的name应不相同。 ::: ## nginx配置 通过nginx来代理前后端。 1. 安装nginx 前往官网[下载](https://nginx.org/download/) 下载完成后解压、编译、安装。 2. 修改配置文件 配置文件如下： ```text upstream iotserver { server 172.16.4.11:8844; server 172.16.4.12:8844; server 172.16.4.13:8844; } upstream webserver { server 172.16.4.11:9000; } upstream fileserver { server 172.16.4.11:8844; #此处指定文件上传到该服务器上 } server { listen 8080; server_name demo2.jetlinks.cn; location ^~/upload/ { proxy_pass http://fileserver; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } location ^~/jetlinks/file/static { proxy_pass http://fileserver/file/static; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_send_timeout 30m; proxy_read_timeout 30m; client_max_body_size 100m; } location ^~/jetlinks/ { proxy_pass http://iotserver/; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_connect_timeout 1; proxy_buffering off; chunked_transfer_encoding off; proxy_cache off; proxy_send_timeout 30m; proxy_read_timeout 30m; client_max_body_size 100m; } location / { proxy_pass http://webserver/; } } ``` 3. 启动 ```shell script systemctl status nginx.service ``` ## tcp负载均衡配置方式 用于代理设备连接。此处使用nginx作为演示,也可以使用LVS,haProxy等方式。 配置文件： ```text load_module /usr/lib/nginx/modules/ngx_stream_module.so; user root; worker_processes 1; error_log /etc/nginx/log/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } stream { upstream tcp-test { hash $remote_addr consistent; server 172.16.4.13:1889 max_fails=3 fail_timeout=10s; #server 172.16.4.12:1889 max_fails=3 fail_timeout=10s; server 172.16.4.11:1889 max_fails=3 fail_timeout=10s; } server { listen 1884; proxy_pass tcp-test; proxy_connect_timeout 30s; proxy_timeout 30s; } } ``` ::: tip 注意： 此配置监听的MQTT的1884端口，代理到服务器上的1889。如需代理自定义端口需保持代理端口与jetlinks平台中网络组件开启的端口一致。 如此处平台中MQTT服务网络组件设置的端口应为1889。 :::