多集群运维(番外篇):SSL证书的管理

概述

在多 Kubernetes 集群环境中,采用泛域名证书管理是一种有效策略。通过申请一个泛域名证书,你能够为同一根域名下的多个子域名提供安全的通信。使用泛域名证书(Wildcard Certificate)和 HashiCorp Vault 对于在多个 Kubernetes 集群中有效地管理证书是一个有效的策略。

下面是一个简单的流程概述:

  1. 申请泛域名证书: 你只需为同一根域名申请一个泛域名证书,该证书可以用于覆盖多个子域名。这可以减少证书的数量和管理成本。
  2. 保存证书到 Vault KV 引擎: 将证书保存到 HashiCorp Vault 中的 Key-Value 引擎。Vault 可以用作安全的中央存储,确保证书的安全性。
  3. 将证书分发到各个 Kubernetes 集群的 master 节点的 /etc/ssl/ 目录。
  4. 在每个集群中使用/etc/ssl/ 目录的证书文件生成 Kubernetes Secret

Demo示例

项目

服务提供商

用途/环境

备注

云服务账号

AWS

通用

云主机

域名

xx云

安全环境

onwalk.net

云DNS服务

阿里云

域名解析

使用xx云的SaaS服务

配置仓库

  1. IAC_code: https://github.com/svc-design/iac_modules.git
  2. Playbook:https://github.com/svc-design/playbook.git
  3. PipeLine : https://github.com/open-source-solution-design/Modern-Container-Application-Reference-Architecture.git

部署vault server

以K3S部署环境,使用 onwalk.net 为例,提前申请好的SSL证书放入部署vault server 环境的节点:

  • /etc/ssl/onwalk.net.key
  • /etc/ssl/onwalk.net.pem

执行shell命令,使用 helm 完成 vault server 的部署

代码语言:yaml
复制
cat > vaules.yaml << EOF
server:
  ingress:
    enabled: true
    ingressClassName: "nginx"
    hosts:
      - host: vault.onwalk.net
        paths:
          - /
    tls:
      - secretName: vault-tls
        hosts:
          - vault.onwalk.net
EOF
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo up
kubectl create ns vault || echo true
kubectl create secret tls vault-tls  --key=/etc/ssl/onwalk.net.key --cert=/etc/ssl/onwalk.net.pem -n vault
helm upgrade --install vault-server hashicorp/vault -n vault --create-namespace -f vaules.yaml

初始化 vault server

1.Vault部署完成后,需要对Vault服务进行初始化, 执行命令

代码语言:yaml
复制
kubectl exec -t -i vault-server-0 -n vault -- sh -c "vault operator init -key-shares=5 -key-threshold=3"

2.记录下返回的密钥, 需要严格私密保存,建议多人分开保存(下面示例key已经作废)

代码语言:yaml
复制
Unseal Key 1: jHeA/olf6URrxshlYqceyuRy8NMx8ICVWaolxfSnRvi+
Unseal Key 2: iAozcZmwczQpkoRwWUm7UO2yi2Ou0dtmsWREyXGaqiIH
Unseal Key 3: 28XG9Gmk/GN3rBJtNhK97dmlLY9jWO9VINAFY+d4lJZe
Unseal Key 4: U4/sW5k+FkIrgnHdBxZUnlg+SU7VRArKkb2Yfjx3qBjz
Unseal Key 5: LN3bp6kkwkeoYCoE7DZ7Y7QQCZPQ7N6NjsIo2PquwgUD

Initial Root Token: s.KJmwUJcHJMF6cUNwhJQpAaAY

当vault服务启动时,它开始是密封(sealed)的状态,需要使用Unseal Key 1-5中的任意3个进行解封(Unsealing )操作,解封后才能vault进行交互。例如认证、管理挂载表,读取等,

3.解封操作,执行命令: kubectl exec -t -i vault-server-0 -n vault -- sh

  • 第一次解封:vault operator unseal jHeA/olf6URrxshlYqceyuRy8NMx8ICVWaolxfSnRvi+
  • 第二次解封:vault operator unseal iAozcZmwczQpkoRwWUm7UO2yi2Ou0dtmsWREyXGaqiIH
  • 第三次解封:vault operator unseal U4/sW5k+FkIrgnHdBxZUnlg+SU7VRArKkb2Yfjx3qBjz
  • 此时检查vault 服务运行状态,执行命令: kubectl get pods -n vault
代码语言:shell
复制
NAME                                           READY   STATUS    RESTARTS   AGE
vault-server-agent-injector-59bd55cb5f-kdcr4 1/1 Running 0 25m
vault-server-0 1/1 Running 0 17m
  1. 登陆Vault UI,确认服务可用

使用 CI pipeline 管理证书

创建一个 GitHub Actions pipeline

使用 ACME 协议从 Let's Encrypt 申请证书,并将结果保存在 Vault Server 中,然后应用集群配置 CertManager 以从 Vault 读取证书,你可以按照以下步骤构建你的 workflow:

  1. 创建 GitHub Repository Secret
    • 在你的 GitHub 仓库中,添加必要的 Secrets,比如 VAULT_TOKENVAULT_URL,以安全地与 Vault 交互。
  2. 设置 GitHub Actions Workflow
    • 在你的 GitHub 仓库中创建一个 .github/workflows 目录。
    • 创建一个新的 YAML 文件,比如 cert-renewal.yml
  3. 编写 Workflow
    • 使用适当的 runner(例如 ubuntu-latest)。
    • 确保你有 ACME 客户端,比如 Certbot,安装在 runner 上。
    • 编写步骤以使用 ACME 协议申请证书。
    • 申请 Let's Encrypt 证书并保存到 Vault

示例 cert-renewal.yml

open-source-solution-design/Modern-Container-Application-Reference-Architecture/.github/workflows/cert-renewal.yml

代码语言:yaml
复制
name: Renew SSL Certs

on:
pull_request:
push:
paths:
- '.github/workflows/app-pipeline-renew-ssl-cert.yml'
workflow_dispatch:
branches:
- main
schedule:
- cron: '0 0 1 */2 *' # 每两个月的第一天执行

jobs:
renew-ssl-certs:
uses: open-source-solution-design/Modern-Container-Application-Reference-Architecture/.github/workflows/use-renew-ssl-certs.yml@main
with:
domain: 'onwalk.net'
secrets:
DNS_AK: ${{ secrets.DNS_AK }}
DNS_SK: ${{ secrets.DNS_SK }}
VAULT_URL: ${{ secrets.VAULT_URL }}
VAULT_TOKEN: ${{ secrets.VAULT_TOKEN }}

open-source-solution-design/Modern-Container-Application-Reference-Architecture/.github/workflows/use-renew-ssl-certs.yml

代码语言:yaml
复制
name: Workflow Call Renew SSL Certs
on:
workflow_call:
inputs:
domain:
required: true
type: string
secrets:
DNS_AK:
required: true
DNS_SK:
required: true
VAULT_URL:
required: true
VAULT_TOKEN:
required: true

jobs:
renew-ssl-certs:
runs-on: ubuntu-latest
env:
DOMAIN: ${{ inputs.domain }}
DNS_AK: ${{ secrets.DNS_AK }}
DNS_SK: ${{ secrets.DNS_SK }}
VAULT_URL: ${{ secrets.VAULT_URL }}
VAULT_TOKEN: ${{ secrets.VAULT_TOKEN }}

steps:
  - name: Checkout code
    uses: actions/checkout@v3
    with:
      submodules: &#39;recursive&#39;

  - name: Pre Setup
    run: |
      curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
      sudo apt-add-repository &#34;deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main&#34;
      sudo apt-get update &amp;&amp; sudo apt-get install vault -y

  - name: Renew SSL Cert
    run: |
      set -x

      # 检查参数是否为空
      check_not_empty() {
        if [[ -z $1 ]]; then
          echo &#34;Error: $2 is empty. Please provide a value.&#34;
          exit 1
        fi
      }
      # 检查参数是否为空
      check_not_empty &#34;$DNS_AK&#34; &#34;DNS_AK&#34; &amp;&amp; export Ali_Key=$DNS_AK
      check_not_empty &#34;$DNS_SK&#34; &#34;DNS_SK&#34; &amp;&amp; export Ali_Secret=$DNS_SK
      check_not_empty &#34;$VAULT_URL&#34; &#34;VAULT_URL&#34;

      rm -fv ${DOMAIN}.key ${DOMAIN}.pem -f
      rm -fv /etc/ssl/${DOMAIN}.* -f
      # Try to issue a certificate from ZeroSSL. If it fails, try Let&#39;s Encrypt.
      curl https://get.acme.sh | sh -s email=156405189@qq.com
      sh ~/.acme.sh/acme.sh --set-default-ca --server zerossl --issue --force --dns dns_ali -d ${DOMAIN} -d &#34;*.${DOMAIN}&#34;
      if [ $? -eq 0 ]; then
          echo &#34;Certificate from letsencrypt successfully issued&#34;
      else
        sh ~/.acme.sh/acme.sh --set-default-ca --server letsencrypt --issue --force --dns dns_ali -d ${DOMAIN} -d &#34;*.${DOMAIN}&#34;
        if [ $? -eq 0 ]; then
            echo &#34;Certificate from zerossl successfully issued&#34;
        else
            echo &#34;Command failed&#34;
            exit 1
        fi
      fi
      cat ~/.acme.sh/${DOMAIN}_ecc/${DOMAIN}.cer &gt; ${DOMAIN}.pem
      cat ~/.acme.sh/${DOMAIN}_ecc/ca.cer &gt;&gt; ${DOMAIN}.pem
      cat ~/.acme.sh/${DOMAIN}_ecc/${DOMAIN}.key &gt; ${DOMAIN}.key
      sudo cp ${DOMAIN}.pem /etc/ssl/ -f &amp;&amp; sudo cp ${DOMAIN}.key /etc/ssl/ -f

  - name: Write certificate into Vault Server
    run: |
      export VAULT_ADDR=${VAULT_URL}
      export CERT_PATH=&#34;certs/${DOMAIN}&#34;

      # Write Certificate and Private Key to Vault

      vault login ${VAULT_TOKEN}
      vault secrets enable -path=certs kv || true
      vault kv put ${CERT_PATH}                             \
          certificate=@/etc/ssl/${DOMAIN}.pem               \
          private_key=@/etc/ssl/${DOMAIN}.key
      echo &#34;Certificate and private key written to Vault at path: ${CERT_PATH}&#34;</code></pre></div></div><p>至此,已经完成SSL Certs 申请的自动化,每两个月执行一次,确保Vault中永远存储有效的证书。</p><figure class=""><div class="rno-markdown-img-url" style="text-align:center"><div class="rno-markdown-img-url-inner" style="width:auto"><div style="width:100%"><img src="https://cdn.static.attains.cn/app/developer-bbs/upload/1722810704618076022.png" /></div></div></div></figure><p>流水线执行成功后,登录 Vault UI 已经看到域名证书已经保存</p><figure class=""><div class="rno-markdown-img-url" style="text-align:center"><div class="rno-markdown-img-url-inner" style="width:auto"><div style="width:100%"><img src="https://cdn.static.attains.cn/app/developer-bbs/upload/1722810704984057984.png" /></div></div></div></figure><h2 id="8uoro" name="%E5%BA%94%E7%94%A8%E9%9B%86%E7%BE%A4%E4%BE%A7%E9%85%8D%E7%BD%AE">应用集群侧配置</h2><h3 id="fcris" name="%E5%B0%86%E8%AF%81%E4%B9%A6%E5%88%86%E5%88%B0%E5%88%B0%E5%BA%94%E7%94%A8%E9%9B%86%E7%BE%A4%E4%B8%AD">将证书分到到应用集群中</h3><p>接下来的的工作就是,如何在IAC流水线中,集成Vault 操作,读取域名证书并写入集群master节点 /etc/ssl/ 目录</p><ul class="ul-level-0"><li>playbook/roles/cert-manager/tasks/main.yml</li></ul><div class="rno-markdown-code"><div class="rno-markdown-code-toolbar"><div class="rno-markdown-code-toolbar-info"><div class="rno-markdown-code-toolbar-item is-type"><span class="is-m-hidden">代码语言:</span>yaml</div></div><div class="rno-markdown-code-toolbar-opt"><div class="rno-markdown-code-toolbar-copy"><i class="icon-copy"></i><span class="is-m-hidden">复制</span></div></div></div><div class="developer-code-block"><pre class="prism-token token line-numbers language-yaml"><code class="language-yaml" style="margin-left:0">- name: Fetch Certificate and Private Key from Vault

script: files/get_certificate.sh {{ domain }} {{ vault_url }} {{ vault_token }}
when: (inventory_hostname in groups[group]) and (vault == true)

  • playbook/roles/cert-manager/files/get_certificate.sh
代码语言:shell
复制
#!/bin/bash

sudo apt-get update
sudo apt install -y software-properties-common
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list > /dev/null
sudo apt-get update
sudo apt-get install -y python3-pip jq vault

check_empty() {
if [ -z "$1&#34; ]; then
echo "$2&#34;
exit 1
fi
}

check_empty "$1&#34; "Please provide DOMAIN" && DOMAIN=$1 check_empty "$2&#34; "Please provide VAULT_ADDR" && VAULT_ADDR=$2 check_empty "$3&#34; "Please provide VAULT_TOKEN" && VAULT_TOKEN=$3

SECRET_PATH="certs/$DOMAIN"

Output paths

CERTIFICATE_PATH="/etc/ssl/${DOMAIN}.pem"
PRIVATE_KEY_PATH="/etc/ssl/${DOMAIN}.key"

vault login

Read certificate from Vault

vault kv get -field=certificate certs/{DOMAIN} &gt; &#34;CERTIFICATE_PATH"

Read private key from Vault

vault kv get -field=private_key certs/{DOMAIN} &gt; &#34;PRIVATE_KEY_PATH"

Set permissions for the private key (modify as needed)

chmod 600 "$PRIVATE_KEY_PATH"

Check if certificate and private key files are non-empty

if [ ! -s "CERTIFICATE_PATH&#34; ] || [ ! -s &#34;PRIVATE_KEY_PATH" ]; then
echo "Certificate or private key is empty. Exiting..."
exit 1
else
echo "Certificate and private key have been written to CERTIFICATE_PATH and PRIVATE_KEY_PATH"

验证 SSL 证书

登陆节点,执行命令:cat /etc/ssl/<domain_name>.* 验证分发的证书

验证 Secret

可以使用以下命令检查 Secret:

代码语言:bash
复制
kubectl get secret my-cert-secret -o yaml

最后使用Curl 命令验证使用和证书的服务或者API接口否生效

代码语言:bash
复制
curl https://your_svc.com

总结

通过以上步骤,你可以建立一个自动化流程,通过 GitHub Actions 自动续订 Let's Encrypt 证书,将其存储在 Vault 中,并更新 Kubernetes 集群的 CertManager 配置以使用这些证书。请注意,这里的示例可能需要根据你的环境和需求进行调整。在部署到生产环境之前,请确保对配置进行充分测试。