Kubernetes (K8S) 中安装部署APISIX

Kubernetes (K8S) 中安装部署APISIX

王先森2023-09-252023-09-25

APISIX

Apache APISIX 是一个基于 OpenResty 和 Etcd 实现的动态、实时、高性能、可扩展的微服务 API 网关,目前已经是 Apache 顶级项目。提供了丰富的流量管理功能,如负载均衡、动态路由、动态 upstream、A/B 测试、金丝雀发布、限速、熔断、防御恶意攻击、认证、监控指标、服务可观测性、服务治理等。可以使用 APISIX 来处理传统的南北流量以及服务之间的东西向流量。

APISIX 基于 Nginx 和 etcd,与传统 API 网关相比,APISIX 具有动态路由和热加载插件功能,避免了配置之后的 reload 操作,同时 APISIX 支持 HTTP(S)、HTTP2、Dubbo、QUIC、MQTT、TCP/UDP 等更多的协议。而且还内置了 Dashboard,提供强大而灵活的界面。同样也提供了丰富的插件支持功能,而且还可以让用户自定义插件。

APISIX 架构图

上图是 APISIX 的架构图,整体上分成数据面和控制面两个部分,控制面用来管理路由,主要通过 etcd 来实现配置中心,数据面用来处理客户端请求,通过 APISIX 自身来实现,会不断去 watch etcd 中的 route、upstream 等数据。

APISIX Ingress

同样作为一个 API 网关,APISIX 也支持作为 Kubernetes 的一个 Ingress 控制器进行使用。APISIX Ingress 在架构上分成了两部分,一部分是 APISIX Ingress Controller,作为控制面它将完成配置管理与分发。另一部分 APISIX(代理) 负责承载业务流量。

apisix-ingress-controller

当 Client 发起请求,到达 Apache APISIX 后,会直接把相应的业务流量传输到后端(如 Service Pod),从而完成转发过程。此过程不需要经过 Ingress Controller,这样做可以保证一旦有问题出现,或者是进行变更、扩缩容或者迁移处理等,都不会影响到用户和业务流量。

同时在配置端,用户通过 kubectl apply 创建资源,可将自定义 CRD 配置应用到 K8s 集群,Ingress Controller 会持续 watch 这些资源变更,来将相应配置应用到 Apache APISIX(通过 admin api)。

从上图可以看出 APISIX Ingress 采用了数据面与控制面的分离架构,所以用户可以选择将数据面部署在 K8s 集群内部或外部。但 Ingress Nginx 是将控制面和数据面放在了同一个 Pod 中,如果 Pod 或控制面出现一点闪失,整个 Pod 就会挂掉,进而影响到业务流量。这种架构分离,给用户提供了比较方便的部署选择,同时在业务架构调整场景下,也方便进行相关数据的迁移与使用。

APISIX Ingress 控制器目前支持的核心特性包括:

  • 全动态,支持高级路由匹配规则,可与 Apache APISIX 官方 50 多个插件 & 客户自定义插件进行扩展使用
  • 支持 CRD,更容易理解声明式配置
  • 兼容原生 Ingress 资源对象
  • 支持流量切分
  • 服务自动注册发现,无惧扩缩容
  • 更灵活的负载均衡策略,自带健康检查功能
  • 支持 gRPC plaintext 与 TCP 4 层代理

APISIX安装

Helm安装

我们这里在 Kubernetes 集群中来使用 APISIX,可以通过 Helm Chart 来进行安装,首先添加官方提供的 Helm Chart 仓库:

代码语言:javascript
复制
helm repo add apisix https://charts.apiseven.com
helm repo update
helm install apisix apisix/apisix --create-namespace  --namespace apisix
# 启动一些其他配置
helm repo add apisix https://charts.apiseven.com && helm repo update && helm upgrade --install apisix apisix/apisix --create-namespace  --namespace apisix --set dashboard.enabled=true --set ingress-controller.enabled=true --set ingress-controller.config.apisix.serviceNamespace=apisix

APISIX参考文档:https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix/README.md

APISIX-Ingress参考文档:https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/README.md

APISIX-Dashboard参考文档:https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-dashboard/README.md

资源配置清单安装

PAISIX

准备资源配置清单

ConfigMapDeploymentService

vim cm.yml

代码语言:javascript
复制
apiVersion: v1
kind: ConfigMap
metadata:
  name: apisix
  namespace: apisix
data:
  config.yaml: |-
    apisix:
      # node_listen: 9080          # APISIX listening port.
      node_listen:                 # APISIX listening ports.
        - 9080
      #   - port: 9081
      #     enable_http2: true     # If not set, default to `false`.
      #   - ip: 127.0.0.2          # If not set, default to `0.0.0.0`
      #     port: 9082
      #     enable_http2: true
      enable_admin: true           # Admin API
      enable_dev_mode: false       # If true, set nginx `worker_processes` to 1.
      enable_reuseport: true       # If true, enable nginx SO_REUSEPORT option.
      show_upstream_status_in_response_header: false  # If true, include the upstream HTTP status code in
                                                      # the response header `X-APISIX-Upstream-Status`.
                                                      # If false, show `X-APISIX-Upstream-Status` only if
                                                      # the upstream response code is 5xx.
      enable_ipv6: true
  enable_server_tokens: true           # If true, show APISIX version in the `Server` response header.
  extra_lua_path: ""                   # Extend lua_package_path to load third-party code.
  extra_lua_cpath: ""                  # Extend lua_package_cpath to load third-party code.
  # lua_module_hook: "my_project.my_hook"  # Hook module used to inject third-party code into APISIX.

  proxy_cache:      # Proxy Caching configuration
    cache_ttl: 10s  # The default caching time on disk if the upstream does not specify a caching time.
    zones:
      - name: disk_cache_one    # Name of the cache.
        memory_size: 50m        # Size of the memory to store the cache index.
        disk_size: 1G           # Size of the disk to store the cache data.
        disk_path: /tmp/disk_cache_one  # Path to the cache file for disk cache.
        cache_levels: 1:2               # Cache hierarchy levels of disk cache.
      # - name: disk_cache_two
      #  memory_size: 50m
      #  disk_size: 1G
      #  disk_path: "/tmp/disk_cache_two"
      #  cache_levels: "1:2"
      - name: memory_cache
        memory_size: 50m

  delete_uri_tail_slash: false        # Delete the '/' at the end of the URI
  normalize_uri_like_servlet: false   # If true, use the same path normalization rules as the Java
                                      # servlet specification. See https://github.com/jakartaee/servlet/blob/master/spec/src/main/asciidoc/servlet-spec-body.adoc#352-uri-path-canonicalization, which is used in Tomcat.

  router:
    http: radixtree_host_uri    # radixtree_host_uri: match route by host and URI
                                # radixtree_uri: match route by URI
                                # radixtree_uri_with_parameter: similar to radixtree_uri but match URI with parameters. See https://github.com/api7/lua-resty-radixtree/#parameters-in-path for more details.
    ssl: radixtree_sni          # radixtree_sni: match route by SNI

  # http is the default proxy mode. proxy_mode can be one of `http`, `stream`, or `http&stream`
  proxy_mode: http
  # stream_proxy:                 # TCP/UDP L4 proxy
  #   only: true                  # Enable L4 proxy only without L7 proxy.
  #   tcp:
  #     - addr: 9100              # Set the TCP proxy listening ports.
  #       tls: true
  #     - addr: "127.0.0.1:9101"
  #   udp:                        # Set the UDP proxy listening ports.
  #     - 9200
  #     - "127.0.0.1:9201"

  # dns_resolver:                 # If not set, read from `/etc/resolv.conf`
  #   - 1.1.1.1
  #   - 8.8.8.8
  # dns_resolver_valid: 30        # Override the default TTL of the DNS records.
  resolver_timeout: 5             # Set the time in seconds that the server will wait for a response from the
                                  # DNS resolver before timing out.
  enable_resolv_search_opt: true  # If true, use search option in the resolv.conf file in DNS lookups.

  ssl:
    enable: true
    listen:                                       # APISIX listening port for HTTPS traffic.
      - port: 9443
        enable_http2: true
      # - ip: 127.0.0.3                           # If not set, default to `0.0.0.0`.
      #   port: 9445
      #   enable_http2: true
    ssl_trusted_certificate: /usr/local/apisix/ssl/ca.pem   # 此处需要修改成连接etcd的ca证书位置,开启https连接必须配置
    ssl_protocols: TLSv1.2 TLSv1.3                # TLS versions supported.
    ssl_ciphers: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl_session_tickets: false  # If true, session tickets are used for SSL/TLS connections.
                                # Disabled by default because it renders Perfect Forward Secrecy (FPS)
                                # useless. See https://github.com/mozilla/server-side-tls/issues/135.

    key_encrypt_salt:           # This field is only used to encrypt the private key of SSL.
      - edd1c9f0985e76a2        # Set the encryption key for AES-128-CBC. It should be a
                                # hexadecimal string of length 16.
                                # If not set, APISIX saves the original data into etcd.
                                # CAUTION: If you would like to update the key, add the new key as the
                                # first item in the array and keep the older keys below the newly added
                                # key, so that data can be decrypted with the older keys and encrypted
                                # with the new key. Removing the old keys directly can render the data
                                # unrecoverable.

    # fallback_sni: "my.default.domain"      # Fallback SNI to be used if the client does not send SNI during
    #                                        # the handshake.

  enable_control: true  # Control API
  # control:
  #  ip: 127.0.0.1
  #  port: 9090

  disable_sync_configuration_during_start: false  # Safe exit. TO BE REMOVED.

  data_encryption:                # Encrypt fields specified in `encrypt_fields` in plugin schema.
    enable: false
    keyring:                      # Set the encryption key for AES-128-CBC. It should be a
      - qeddd145sfvddff3          # hexadecimal string of length 16.
                                  # If not set, APISIX saves the original data into etcd.
                                  # CAUTION: If you would like to update the key, add the new key as the
                                  # first item in the array and keep the older keys below the newly added
                                  # key, so that data can be decrypted with the older keys and encrypted
                                  # with the new key. Removing the old keys directly can render the data
                                  # unrecoverable.

nginx_config:                     # Config for render the template to generate nginx.conf
  # user: root                    # Set the execution user of the worker process. This is only
                                  # effective if the master process runs with super-user privileges.
  error_log: logs/error.log       # Location of the error log.
  error_log_level:  warn          # Logging level: info, debug, notice, warn, error, crit, alert, or emerg.
  worker_processes: auto          # Automatically determine the optimal number of worker processes based
                                  # on the available system resources.
                                  # If you want use multiple cores in container, you can inject the number of
                                  # CPU cores as environment variable "APISIX_WORKER_PROCESSES".
  enable_cpu_affinity: false      # Disable CPU affinity by default as worker_cpu_affinity affects the
                                  # behavior of APISIX in containers. For example, multiple instances could
                                  # be bound to one CPU core, which is not desirable.
                                  # If APISIX is deployed on a physical machine, CPU affinity can be enabled.
  worker_rlimit_nofile: 20480     # The number of files a worker process can open.
                                  # The value should be larger than worker_connections.
  worker_shutdown_timeout: 240s   # Timeout for a graceful shutdown of worker processes.

  max_pending_timers: 16384       # The maximum number of pending timers that can be active at any given time.
                                  # Error "too many pending timers" indicates the threshold is reached.
  max_running_timers: 4096        # The maximum number of running timers that can be active at any given time.
                                  # Error "lua_max_running_timers are not enough" error indicates the
                                  # threshold is reached.

  event:
    worker_connections: 10620

  # envs:                         # Get environment variables.
  #  - TEST_ENV

  meta:
    lua_shared_dict:              # Nginx Lua shared memory zone. Size units are m or k.
      prometheus-metrics: 15m

  stream:
    enable_access_log: false                 # Enable stream proxy access logging.
    access_log: logs/access_stream.log       # Location of the stream access log.
    access_log_format: "$remote_addr [$time_local] $protocol $status $bytes_sent $bytes_received $session_time" # Customize log format: http://nginx.org/en/docs/varindex.html
    access_log_format_escape: default        # Escape default or json characters in variables.
    lua_shared_dict:                         # Nginx Lua shared memory zone. Size units are m or k.
      etcd-cluster-health-check-stream: 10m
      lrucache-lock-stream: 10m
      plugin-limit-conn-stream: 10m
      worker-events-stream: 10m
      tars-stream: 1m

  # Add other custom Nginx configurations.
  # Users are responsible for validating the custom configurations
  # to ensure they are not in conflict with APISIX configurations.
  main_configuration_snippet: |
    # Add custom Nginx main configuration to nginx.conf.
    # The configuration should be well indented!
  http_configuration_snippet: |
    # Add custom Nginx http configuration to nginx.conf.
    # The configuration should be well indented!
  http_server_configuration_snippet: |
    # Add custom Nginx http server configuration to nginx.conf.
    # The configuration should be well indented!
  http_server_location_configuration_snippet: |
    # Add custom Nginx http server location configuration to nginx.conf.
    # The configuration should be well indented!
  http_admin_configuration_snippet: |
    # Add custom Nginx admin server configuration to nginx.conf.
    # The configuration should be well indented!
  http_end_configuration_snippet: |
    # Add custom Nginx http end configuration to nginx.conf.
    # The configuration should be well indented!
  stream_configuration_snippet: |
    # Add custom Nginx stream configuration to nginx.conf.
    # The configuration should be well indented!

  http:
    enable_access_log: true             # Enable HTTP proxy access logging.
    access_log: logs/access.log         # Location of the access log.
    access_log_buffer: 16384            # buffer size of access log.
    access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\""
    # Customize log format: http://nginx.org/en/docs/varindex.html
    access_log_format_escape: default   # Escape default or json characters in variables.
    keepalive_timeout: 60s              # Set the maximum time for which TCP connection keeps alive.
    client_header_timeout: 60s          # Set the maximum time waiting for client to send the entire HTTP
                                        # request header before closing the connection.
    client_body_timeout: 60s            # Set the maximum time waiting for client to send the request body.
    client_max_body_size: 0             # Set the maximum allowed size of the client request body.
                                        # Default to 0, unlimited.
                                        # Unlike Nginx, APISIX does not limit the body size by default.
                                        # If exceeded, the 413 (Request Entity Too Large) error is returned.
    send_timeout: 10s   # Set the maximum time for transmitting a response to the client before closing.
    underscores_in_headers: "on"  # Allow HTTP request headers to contain underscores in their names.
    real_ip_header: X-Real-IP     # https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_recursive: "off" # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive
    real_ip_from:            # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - "unix:"

    # custom_lua_shared_dict:     # Custom Nginx Lua shared memory zone for nginx.conf. Size units are m or k.
    #  ipc_shared_dict: 100m      # Custom shared cache, format: `cache-key: cache-size`

    proxy_ssl_server_name: true   # Send the server name in the SNI extension when establishing an SSL/TLS
                                  # connection with the upstream server, allowing the upstream server to
                                  # select the appropriate SSL/TLS certificate and configuration based on
                                  # the requested server name.

    upstream:
      keepalive: 320              # Set the maximum time of keep-alive connections to the upstream servers.
                                  # When the value is exceeded, the least recently used connection is closed.
      keepalive_requests: 1000    # Set the maximum number of requests that can be served through one
                                  # keep-alive connection.
                                  # After the maximum number of requests is made, the connection is closed.
      keepalive_timeout: 60s      # Set the maximum time for which TCP connection keeps alive.
    charset: utf-8                # Add the charset to the "Content-Type" response header field.
                                  # See http://nginx.org/en/docs/http/ngx_http_charset_module.html#charset
    variables_hash_max_size: 2048 # Set the maximum size of the variables hash table.

    lua_shared_dict:              # Nginx Lua shared memory zone. Size units are m or k.
      internal-status: 10m
      plugin-limit-req: 10m
      plugin-limit-count: 10m
      prometheus-metrics: 10m
      plugin-limit-conn: 10m
      upstream-healthcheck: 10m
      worker-events: 10m
      lrucache-lock: 10m
      balancer-ewma: 10m
      balancer-ewma-locks: 10m
      balancer-ewma-last-touched-at: 10m
      plugin-limit-count-redis-cluster-slot-lock: 1m
      tracing_buffer: 10m
      plugin-api-breaker: 10m
      etcd-cluster-health-check: 10m
      discovery: 1m
      jwks: 1m
      introspection: 10m
      access-tokens: 1m
      ext-plugin: 1m
      tars: 1m
      cas-auth: 10m

# discovery:                      # Service Discovery
#  dns:
#    servers:
#      - "127.0.0.1:8600"         # Replace with the address of your DNS server.
#    order:                       # Resolve DNS records this order.
#      - last                     # Try the latest successful type for a hostname.
#      - SRV
#      - A
#      - AAAA
#      - CNAME

graphql:
  max_size: 1048576                # Set the maximum size limitation of graphql in bytes. Default to 1MiB.

plugins:                           # plugin list (sorted by priority)
  - real-ip                        # priority: 23000
  - ai                             # priority: 22900
  - client-control                 # priority: 22000
  - proxy-control                  # priority: 21990
  - request-id                     # priority: 12015
  - zipkin                         # priority: 12011
  #- skywalking                    # priority: 12010
  #- opentelemetry                 # priority: 12009
  - ext-plugin-pre-req             # priority: 12000
  - fault-injection                # priority: 11000
  - mocking                        # priority: 10900
  - serverless-pre-function        # priority: 10000
  #- batch-requests                # priority: 4010
  - cors                           # priority: 4000
  - ip-restriction                 # priority: 3000
  - ua-restriction                 # priority: 2999
  - referer-restriction            # priority: 2990
  - csrf                           # priority: 2980
  - uri-blocker                    # priority: 2900
  - request-validation             # priority: 2800
  - chaitin-waf                    # priority: 2700
  - openid-connect                 # priority: 2599
  - cas-auth                       # priority: 2597
  - authz-casbin                   # priority: 2560
  - authz-casdoor                  # priority: 2559
  - wolf-rbac                      # priority: 2555
  - ldap-auth                      # priority: 2540
  - hmac-auth                      # priority: 2530
  - basic-auth                     # priority: 2520
  - jwt-auth                       # priority: 2510
  - key-auth                       # priority: 2500
  - consumer-restriction           # priority: 2400
  - forward-auth                   # priority: 2002
  - opa                            # priority: 2001
  - authz-keycloak                 # priority: 2000
  #- error-log-logger              # priority: 1091
  - proxy-cache                    # priority: 1085
  - body-transformer               # priority: 1080
  - proxy-mirror                   # priority: 1010
  - proxy-rewrite                  # priority: 1008
  - workflow                       # priority: 1006
  - api-breaker                    # priority: 1005
  - limit-conn                     # priority: 1003
  - limit-count                    # priority: 1002
  - limit-req                      # priority: 1001
  #- node-status                   # priority: 1000
  - gzip                           # priority: 995
  - server-info                    # priority: 990
  - traffic-split                  # priority: 966
  - redirect                       # priority: 900
  - response-rewrite               # priority: 899
  - degraphql                      # priority: 509
  - kafka-proxy                    # priority: 508
  #- dubbo-proxy                   # priority: 507
  - grpc-transcode                 # priority: 506
  - grpc-web                       # priority: 505
  - public-api                     # priority: 501
  - prometheus                     # priority: 500
  - datadog                        # priority: 495
  - loki-logger                    # priority: 414
  - elasticsearch-logger           # priority: 413
  - echo                           # priority: 412
  - loggly                         # priority: 411
  - http-logger                    # priority: 410
  - splunk-hec-logging             # priority: 409
  - skywalking-logger              # priority: 408
  - google-cloud-logging           # priority: 407
  - sls-logger                     # priority: 406
  - tcp-logger                     # priority: 405
  - kafka-logger                   # priority: 403
  - rocketmq-logger                # priority: 402
  - syslog                         # priority: 401
  - udp-logger                     # priority: 400
  - file-logger                    # priority: 399
  - clickhouse-logger              # priority: 398
  - tencent-cloud-cls              # priority: 397
  - inspect                        # priority: 200
  #- log-rotate                    # priority: 100
  # <- recommend to use priority (0, 100) for your custom plugins
  - example-plugin                 # priority: 0
  #- gm                            # priority: -43
  - aws-lambda                     # priority: -1899
  - azure-functions                # priority: -1900
  - openwhisk                      # priority: -1901
  - openfunction                   # priority: -1902
  - serverless-post-function       # priority: -2000
  - ext-plugin-post-req            # priority: -3000
  - ext-plugin-post-resp           # priority: -4000

stream_plugins:                    # stream plugin list (sorted by priority)
  - ip-restriction                 # priority: 3000
  - limit-conn                     # priority: 1003
  - mqtt-proxy                     # priority: 1000
  #- prometheus                    # priority: 500
  - syslog                         # priority: 401
  # <- recommend to use priority (0, 100) for your custom plugins

plugin_attr:          # Plugin attributes
  log-rotate:         # Plugin: log-rotate
    interval: 3600    # Set the log rotate interval in seconds.
    max_kept: 168     # Set the maximum number of log files to keep. If exceeded, historic logs are deleted.
    max_size: -1      # Set the maximum size of log files in bytes before a rotation.
                      # Skip size check if max_size is less than 0.
    enable_compression: false    # Enable log file compression (gzip).
  skywalking:                                     # Plugin: skywalking
    service_name: APISIX                          # Set the service name for SkyWalking reporter.
    service_instance_name: APISIX Instance Name   # Set the service instance name for SkyWalking reporter.
    endpoint_addr: http://127.0.0.1:12800         # Set the SkyWalking HTTP endpoint.
    report_interval: 3                            # Set the reporting interval in second.
  opentelemetry:      # Plugin: opentelemetry
    trace_id_source: x-request-id   # Specify the source of the trace ID for OpenTelemetry traces.
    resource:
      service.name: APISIX          # Set the service name for OpenTelemetry traces.
    collector:
      address: 127.0.0.1:4318       # Set the address of the OpenTelemetry collector to send traces to.
      request_timeout: 3            # Set the timeout for requests to the OpenTelemetry collector in seconds.
      request_headers:              # Set the headers to include in requests to the OpenTelemetry collector.
        Authorization: token        # Set the authorization header to include an access token.
    batch_span_processor:
      drop_on_queue_full: false     # Drop spans when the export queue is full.
      max_queue_size: 1024          # Set the maximum size of the span export queue.
      batch_timeout: 2              # Set the timeout for span batches to wait in the export queue before
                                    # being sent.
      inactive_timeout: 1           # Set the timeout for spans to wait in the export queue before being sent,
                                    # if the queue is not full.
      max_export_batch_size: 16     # Set the maximum number of spans to include in each batch sent to the
    set_ngx_var: false              # export opentelemetry variables to nginx variables
                                    # OpenTelemetry collector.
  prometheus:                               # Plugin: prometheus
    export_uri: /apisix/prometheus/metrics  # Set the URI for the Prometheus metrics endpoint.
    metric_prefix: apisix_                  # Set the prefix for Prometheus metrics generated by APISIX.
    enable_export_server: true              # Enable the Prometheus export server.
    export_addr:                            # Set the address for the Prometheus export server.
      ip: 127.0.0.1                         # Set the IP.
      port: 9091                            # Set the port.
    # metrics:    # Create extra labels from nginx variables: https://nginx.org/en/docs/varindex.html
    #  http_status:
    #    extra_labels:
    #      - upstream_addr: $upstream_addr
    #      - status: $upstream_status  # The label name does not need to be the same as the variable name.
    #  http_latency:
    #    extra_labels:
    #      - upstream_addr: $upstream_addr
    #  bandwidth:
    #    extra_labels:
    #      - upstream_addr: $upstream_addr
    # default_buckets:
    #   - 10
    #   - 50
    #   - 100
    #   - 200
    #   - 500
  server-info:                        # Plugin: server-info
    report_ttl: 60                    # Set the TTL in seconds for server info in etcd.
                                      # Maximum: 86400. Minimum: 3.
  dubbo-proxy:                        # Plugin: dubbo-proxy
    upstream_multiplex_count: 32      # Set the maximum number of connections that can be multiplexed over
                                      # a single network connection between the Dubbo Proxy and the upstream
                                      # Dubbo services.
  proxy-mirror:                       # Plugin: proxy-mirror
    timeout:                          # Set the timeout for mirrored requests.
      connect: 60s
      read: 60s
      send: 60s
  # redirect:                         # Plugin: redirect
  #   https_port: 8443                # Set the default port used to redirect HTTP to HTTPS.
  inspect:                            # Plugin: inspect
    delay: 3                          # Set the delay in seconds for the frequency of checking the hooks file.
    hooks_file: "/usr/local/apisix/plugin_inspect_hooks.lua"  # Set the path to the Lua file that defines
                                                              # hooks. Only administrators should have
                                                              # write access to this file for security.

deployment:                    # Deployment configurations
  role: traditional            # Set deployment mode: traditional, control_plane, or data_plane.
  role_traditional:
    config_provider: etcd      # Set the configuration center.

  #role_data_plane:            # Set data plane details if role is data_plane.
  #  config_provider: etcd     # Set the configuration center: etcd, xds, or yaml.

  #role_control_plane:         # Set control plane details if role is control_plane.
  #  config_provider: etcd     # Set the configuration center.

  admin:                       # Admin API
    admin_key_required: true   # Enable Admin API authentication by default for security.
    admin_key:
      -
        name: admin                             # admin: write access to configurations.
        key: edd1c9f034335f136f87ad84b625c8f1   # Set API key for the admin of Admin API.
        role: admin
      -
        name: viewer                            # viewer: read-only to configurations.
        key: 4054f7cf07e344346cd3f287985e76a2   # Set API key for the viewer of Admin API.
        role: viewer

    enable_admin_cors: true       # Enable Admin API CORS response header `Access-Control-Allow-Origin`.
    allow_admin:                  # Limit Admin API access by IP addresses.
      - 127.0.0.0/24              # If not set, any IP address is allowed.
      - 0.0.0.0/0
      # - "::/64"
    admin_listen:                 # Set the Admin API listening addresses.
      ip: 0.0.0.0                 # Set listening IP.
      port: 9180                  # Set listening port. Beware of port conflict with node_listen.

    # https_admin: true           # Enable SSL for Admin API on IP and port specified in admin_listen.
                                  # Use admin_api_mtls.admin_ssl_cert and admin_api_mtls.admin_ssl_cert_key.
    # admin_api_mtls:             # Set this if `https_admin` is true.
    #   admin_ssl_cert: ""        # Set path to SSL/TLS certificate.
    #   admin_ssl_cert_key: ""    # Set path to SSL/TLS key.
    #   admin_ssl_ca_cert: ""     # Set path to CA certificate used to sign client certificates.

    admin_api_version: v3         # Set the version of Admin API (latest: v3).

  etcd:
    host:
     # - "https://10.1.1.100:2379"
     # - "https://10.1.1.120:2379"
     # - "https://10.1.1.130:2379"
     - "https://etcd1.local.host:2379"
     - "https://etcd2.local.host:2379"
     - "https://etcd3.local.host:2379"
    prefix: "/apisix"     # apisix configurations prefix
    timeout: 30   # seconds
    tls:
      verify: true
      cert: "/etcd-ssl/tls.crt"
      key: "/etcd-ssl/tls.key"
      sni: &#34;*.local.host&#34;</code></pre></div></div><blockquote><p>vim dp.yml</p></blockquote><div class="rno-markdown-code"><div class="rno-markdown-code-toolbar"><div class="rno-markdown-code-toolbar-info"><div class="rno-markdown-code-toolbar-item is-type"><span class="is-m-hidden">代码语言:</span>javascript</div></div><div class="rno-markdown-code-toolbar-opt"><div class="rno-markdown-code-toolbar-copy"><i class="icon-copy"></i><span class="is-m-hidden">复制</span></div></div></div><div class="developer-code-block"><pre class="prism-token token line-numbers language-javascript"><code class="language-javascript" style="margin-left:0">apiVersion: apps/v1

kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
app.kubernetes.io/version: 3.5.0
name: apisix
namespace: apisix
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
template:
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
spec:
nodeSelector: # 这里选择标签为app=apisix节点。
app: apisix
containers:
- image: apache/apisix:3.5.0-debian
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 30
name: apisix
ports:
- containerPort: 9080
name: http
protocol: TCP
- containerPort: 9443
name: tls
protocol: TCP
- containerPort: 9180
name: admin
protocol: TCP
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 9080
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/local/apisix/conf/config.yaml
name: apisix-config
subPath: config.yaml
- mountPath: /etcd-ssl/ # etcd证书目录,与configMap一致。
name: etcd-tls
- mountPath: /usr/local/apisix/ssl/ca.pem # ca证书路径,与configMap一致。
name: apisix-ca
subPath: ca.pem
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: apisix
name: apisix-config
- secret:
secretName: etcd-tls # 存放etcd证书的Secret
name: etcd-tls
- secret:
secretName: apisix-ca # 存放ca证书的Secret
name: apisix-ca

vim svc.yml

代码语言:javascript
复制
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
app.kubernetes.io/service: apisix-admin
app.kubernetes.io/version: 3.5.0
name: apisix-admin
namespace: apisix
spec:
ports:

  • name: apisix-admin
    port: 9180
    protocol: TCP
    targetPort: 9180
    selector:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/name: apisix
    sessionAffinity: None
    type: ClusterIP

apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: apisix
app.kubernetes.io/service: apisix-gateway
app.kubernetes.io/version: 3.5.0
name: apisix-gateway
namespace: apisix
spec:
externalTrafficPolicy: Cluster
ipFamilyPolicy: SingleStack
ports:

  • name: apisix-gateway-http
    nodePort: 30080
    port: 80
    protocol: TCP
    targetPort: 9080
  • name: apisix-gateway-https
    nodePort: 30443
    port: 443
    protocol: TCP
    targetPort: 9443
    selector:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/name: apisix
    sessionAffinity: None
    type: NodePort
应用资源配置清单
代码语言:javascript
复制
# 创建名称空间
kubectl create ns apisix

创建所需要的secret

注意 etcd证书需要带有域名信息

kubectl create secret tls etcd-tls --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem -n apisix
kubectl create secret generic apisix-ca --from-file=/opt/etcd/ssl/ca.pem -n apisix

创建

kubectl apply -f cm.yml
kubectl apply -f dp.yml
kubectl apply -f svc.yml

APISIX-Ingress

准备资源配置清单

ConfigMapDeploymentRBACCRDService

vim cm.yml

代码语言:javascript
复制
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: ingress-controller
app.kubernetes.io/version: 1.6.0
name: apisix-configmap
namespace: apisix
data:
config.yaml: |-
# log options
log_level: "info"
log_output: "stderr"
cert_file: "/etc/webhook/certs/cert.pem"
key_file: "/etc/webhook/certs/key.pem"
http_listen: ":8080"
https_listen: ":8443"
ingress_publish_service: ""
enable_profiling: true
apisix-resource-sync-interval: 1h
kubernetes:
kubeconfig: ""
resync_interval: "6h"
namespace_selector:
- ""
election_id: "ingress-apisix-leader"
ingress_class: "apisix"
ingress_version: "networking/v1"
watch_endpointslices: false
apisix_route_version: "apisix.apache.org/v2"
enable_gateway_api: false
apisix_version: "apisix.apache.org/v2"
plugin_metadata_cm: ""
apisix:
admin_api_version: "v3"
default_cluster_base_url: http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin
default_cluster_admin_key: "edd1c9f034335f136f87ad84b625c8f1"
default_cluster_name: "default"

vim dp.yml

代码语言:javascript
复制
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: ingress-controller
app.kubernetes.io/version: 1.6.0
name: apisix-ingress-controller
namespace: apisix
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: ingress-controller
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: ingress-controller
spec:
containers:
- command:
- /ingress-apisix/apisix-ingress-controller
- ingress
- --config-path
- /ingress-apisix/conf/config.yaml
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: apache/apisix-ingress-controller:1.6.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8080
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: ingress-controller
ports:
- containerPort: 8080
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8080
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /ingress-apisix/conf
name: configuration
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- -c
- until nc -z apisix-admin.apisix.svc.cluster.local 9180 ; do echo
waiting for apisix; sleep 2; done;
image: busybox:1.28
imagePullPolicy: IfNotPresent
name: wait-apisix-admin
resources: {}
securityContext: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: apisix-ingress-controller
serviceAccountName: apisix-ingress-controller
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
items:
- key: config.yaml
path: config.yaml
name: apisix-configmap
name: configuration

vim rbac.yml

代码语言:javascript
复制
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: apisix-view-clusterrole
rules:

  • apiGroups:
    • ""
      resources:
    • events
      verbs:
    • "*"
  • apiGroups:
    • ""
      resources:
    • configmaps
    • endpoints
    • persistentvolumeclaims
    • pods
    • replicationcontrollers
    • replicationcontrollers/scale
    • serviceaccounts
    • services
    • secrets
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • ""
      resources:
    • bindings
    • limitranges
    • namespaces/status
    • pods/log
    • pods/status
    • replicationcontrollers/status
    • resourcequotas
    • resourcequotas/status
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • ""
      resources:
    • namespaces
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • apps
      resources:
    • controllerrevisions
    • daemonsets
    • deployments
    • deployments/scale
    • replicasets
    • replicasets/scale
    • statefulsets
    • statefulsets/scale
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • autoscaling
      resources:
    • horizontalpodautoscalers
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • batch
      resources:
    • cronjobs
    • jobs
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • extensions
      resources:
    • daemonsets
    • deployments
    • deployments/scale
    • ingresses
    • networkpolicies
    • replicasets
    • replicasets/scale
    • replicationcontrollers/scale
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • policy
      resources:
    • poddisruptionbudgets
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • networking.k8s.io
      resources:
    • ingresses
    • ingresses/status
    • networkpolicies
      verbs:
    • '*'
  • apiGroups:
    • metrics.k8s.io
      resources:
    • pods
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • apisix.apache.org
      resources:
    • apisixroutes
    • apisixroutes/status
    • apisixupstreams
    • apisixupstreams/status
    • apisixtlses
    • apisixtlses/status
    • apisixclusterconfigs
    • apisixclusterconfigs/status
    • apisixconsumers
    • apisixconsumers/status
    • apisixpluginconfigs
    • apisixpluginconfigs/status
    • apisixglobalrules
    • apisixglobalrules/status
      verbs:
    • '*'
  • apiGroups:
    • coordination.k8s.io
      resources:
    • leases
      verbs:
    • '*'
  • apiGroups:
    • discovery.k8s.io
      resources:
    • endpointslices
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • gateway.networking.k8s.io
      resources:
    • httproutes
    • tlsroutes
    • tcproutes
    • gateways
    • gatewayclasses
    • udproutes
      verbs:
    • get
    • list
    • watch
  • apiGroups:
    • gateway.networking.k8s.io
      resources:
    • gateways/status
    • gatewayclasses/status
      verbs:
    • get
    • update
    • list

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: apisix-view-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: apisix-view-clusterrole
subjects:

  • kind: ServiceAccount
    name: apisix-ingress-controller
    namespace: apisix

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
name: apisix-ingress-controller
namespace: apisix

代码语言:javascript
复制

vim svc.yml

代码语言:javascript
复制
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: apisix
app.kubernetes.io/name: ingress-controller
app.kubernetes.io/version: 1.6.0
name: apisix-ingress-controller
namespace: apisix
spec:
internalTrafficPolicy: Cluster
ipFamilies:

  • IPv4
    ipFamilyPolicy: SingleStack
    ports:
  • name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    selector:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/name: ingress-controller
    sessionAffinity: None
    type: ClusterIP
应用资源配置清单
代码语言:javascript
复制
kubectl apply -f rbac.yml
kubectl apply -f cm.yml
kubectl apply -f dp.yml
kubectl apply -f svc.yml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixConsumer.yaml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixClusterConfig.yaml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixGlobalRule.yaml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixPluginConfig.yaml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixRoute.yaml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixTls.yaml
kubectl apply -f https://raw.githubusercontent.com/apache/apisix-ingress-controller/master/samples/deploy/crd/v1/ApisixUpstream.yaml

APISIX-Dashboard

准备资源配置清单

ConfigMapDeploymentService

vim cm.yml

代码语言:javascript
复制
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: apisix-dashboard
app.kubernetes.io/name: apisix-dashboard
app.kubernetes.io/version: 3.0.0
name: apisix-dashboard
namespace: apisix
data:
conf.yaml: |-
conf:
listen:
host: 0.0.0.0
port: 9000
etcd:
prefix: "/apisix"
endpoints:
- https://10.1.1.100
- https://10.1.1.120
- https://10.1.1.130
mtls:
key_file: /etcd-ssl/tls.key
cert_file: /etcd-ssl/tls.crt
ca_file: /usr/local/apisix-dashboard/ssl/ca.pem
log:
error_log:
level: warn
file_path: /dev/stderr
access_log:
file_path: /dev/stdout
authentication:
secret: secret
expire_time: 3600
users:
- username: admin # 配置登陆用户名
password: admin # 登陆密码

vim dp.yml

代码语言:javascript
复制
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: apisix-dashboard
app.kubernetes.io/name: apisix-dashboard
app.kubernetes.io/version: 3.0.0
name: apisix-dashboard
namespace: apisix

spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: apisix-dashboard
app.kubernetes.io/name: apisix-dashboard
template:
metadata:
labels:
app.kubernetes.io/instance: apisix-dashboard
app.kubernetes.io/name: apisix-dashboard
spec:
containers:
- image: apache/apisix-dashboard:3.0.0-alpine
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: apisix-dashboard
ports:
- containerPort: 9000
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: http
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
securityContext: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/local/apisix-dashboard/conf/conf.yaml
name: apisix-dashboard-config
subPath: conf.yaml
- mountPath: /etcd-ssl/
name: apisix-dashboard-etcd
- mountPath: /usr/local/apisix-dashboard/ssl/ca.pem
name: apisix-dashboard-etcd-ca
subPath: ca.pem
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: apisix-dashboard
name: apisix-dashboard-config
- secret:
secretName: etcd-tls
name: apisix-dashboard-etcd
- secret:
secretName: apisix-ca
name: apisix-dashboard-etcd-ca

vim svc.yml

代码语言:javascript
复制
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: apisix-dashboard
app.kubernetes.io/name: apisix-dashboard
app.kubernetes.io/version: 3.0.0
name: apisix-dashboard
namespace: apisix
spec:
ports:

  • name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: 31000
    selector:
    app.kubernetes.io/instance: apisix-dashboard
    app.kubernetes.io/name: apisix-dashboard
    type: NodePort
应用资源配置清单
代码语言:javascript
复制
kubectl apply -f cm.yml
kubectl apply -f dp.yml
kubectl apply -f svc.yml

ApisixRoute

现在我们可以为 Dashboard 创建一个路由规则,新建一个如下所示的 ApisixRoute 资源对象即可:

代码语言:javascript
复制
cat > dashboard-ing.yml <<EOF
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: dashboard
namespace: apisix
spec:
http:
  • name: root
    match:
    hosts:
    - apisix.od.com
    paths:
    - '/*'
    backends:
    • serviceName: apisix-dashboard
      servicePort: 80
      EOF

创建后 apisix-ingress-controller 会将上面的资源对象通过 admin api 映射成 APISIX 中的配置:

代码语言:javascript
复制
kubectl apply -f dashboard-ing.yml
kubectl get apisixroutes.apisix.apache.org -A
NAMESPACE NAME HOSTS URIS AGE
apisix dashboard ["apisix.od.com"] ["/*"] 3m30s

可以在本地测试的时候可以使用 kubectl port-forward 将服务暴露在节点的 80 端口上:

代码语言:javascript
复制
# node2 节点暴露 apisix-gateway 服务
kubectl port-forward --address 0.0.0.0 svc/apisix-gateway 80:80 443:443 -n apisix

本次我使用nginx做的转发。

打开http://apisix.od.com

默认登录用户名和密码都是 admin,登录后在路由菜单下正常可以看到上面我们创建的这个 dashboard 的路由信息:

使用 APISIX,也一定要理解其中的路由 Route 这个概念,路由(Route)是请求的入口点,它定义了客户端请求与服务之间的匹配规则,路由可以与服务(Service)、上游(Upstream)关联,一个服务可对应一组路由,一个路由可以对应一个上游对象(一组后端服务节点),因此,每个匹配到路由的请求将被网关代理到路由绑定的上游服务中。