概述
在多 Kubernetes 集群环境中,采用泛域名证书管理是一种有效策略。通过申请一个泛域名证书,你能够为同一根域名下的多个子域名提供安全的通信。使用泛域名证书(Wildcard Certificate)和 HashiCorp Vault 对于在多个 Kubernetes 集群中有效地管理证书是一个有效的策略。
下面是一个简单的流程概述:
- 申请泛域名证书: 你只需为同一根域名申请一个泛域名证书,该证书可以用于覆盖多个子域名。这可以减少证书的数量和管理成本。
- 保存证书到 Vault KV 引擎: 将证书保存到 HashiCorp Vault 中的 Key-Value 引擎。Vault 可以用作安全的中央存储,确保证书的安全性。
- 将证书分发到各个 Kubernetes 集群的 master 节点的 /etc/ssl/ 目录。
- 在每个集群中使用/etc/ssl/ 目录的证书文件生成 Kubernetes Secret
Demo示例
项目 | 服务提供商 | 用途/环境 | 备注 |
---|---|---|---|
云服务账号 | AWS | 通用 | 云主机 |
域名 | xx云 | 安全环境 | onwalk.net |
云DNS服务 | 阿里云 | 域名解析 | 使用xx云的SaaS服务 |
配置仓库
- IAC_code: https://github.com/svc-design/iac_modules.git
- Playbook:https://github.com/svc-design/playbook.git
- PipeLine : https://github.com/open-source-solution-design/Modern-Container-Application-Reference-Architecture.git
部署vault server
以K3S部署环境,使用 onwalk.net 为例,提前申请好的SSL证书放入部署vault server 环境的节点:
- /etc/ssl/onwalk.net.key
- /etc/ssl/onwalk.net.pem
执行shell命令,使用 helm 完成 vault server 的部署
cat > vaules.yaml << EOF
server:
ingress:
enabled: true
ingressClassName: "nginx"
hosts:
- host: vault.onwalk.net
paths:
- /
tls:
- secretName: vault-tls
hosts:
- vault.onwalk.net
EOF
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo up
kubectl create ns vault || echo true
kubectl create secret tls vault-tls --key=/etc/ssl/onwalk.net.key --cert=/etc/ssl/onwalk.net.pem -n vault
helm upgrade --install vault-server hashicorp/vault -n vault --create-namespace -f vaules.yaml
初始化 vault server
1.Vault部署完成后,需要对Vault服务进行初始化, 执行命令
kubectl exec -t -i vault-server-0 -n vault -- sh -c "vault operator init -key-shares=5 -key-threshold=3"
2.记录下返回的密钥, 需要严格私密保存,建议多人分开保存(下面示例key已经作废)
Unseal Key 1: jHeA/olf6URrxshlYqceyuRy8NMx8ICVWaolxfSnRvi+ Unseal Key 2: iAozcZmwczQpkoRwWUm7UO2yi2Ou0dtmsWREyXGaqiIH Unseal Key 3: 28XG9Gmk/GN3rBJtNhK97dmlLY9jWO9VINAFY+d4lJZe Unseal Key 4: U4/sW5k+FkIrgnHdBxZUnlg+SU7VRArKkb2Yfjx3qBjz Unseal Key 5: LN3bp6kkwkeoYCoE7DZ7Y7QQCZPQ7N6NjsIo2PquwgUD
Initial Root Token: s.KJmwUJcHJMF6cUNwhJQpAaAY
当vault服务启动时,它开始是密封(sealed)的状态,需要使用Unseal Key 1-5中的任意3个进行解封(Unsealing )操作,解封后才能vault进行交互。例如认证、管理挂载表,读取等,
3.解封操作,执行命令: kubectl exec -t -i vault-server-0 -n vault -- sh
- 第一次解封:vault operator unseal jHeA/olf6URrxshlYqceyuRy8NMx8ICVWaolxfSnRvi+
- 第二次解封:vault operator unseal iAozcZmwczQpkoRwWUm7UO2yi2Ou0dtmsWREyXGaqiIH
- 第三次解封:vault operator unseal U4/sW5k+FkIrgnHdBxZUnlg+SU7VRArKkb2Yfjx3qBjz
- 此时检查vault 服务运行状态,执行命令: kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
vault-server-agent-injector-59bd55cb5f-kdcr4 1/1 Running 0 25m
vault-server-0 1/1 Running 0 17m
- 登陆Vault UI,确认服务可用
使用 CI pipeline 管理证书
创建一个 GitHub Actions pipeline
使用 ACME 协议从 Let's Encrypt 申请证书,并将结果保存在 Vault Server 中,然后应用集群配置 CertManager 以从 Vault 读取证书,你可以按照以下步骤构建你的 workflow:
- 创建 GitHub Repository Secret:
- 在你的 GitHub 仓库中,添加必要的 Secrets,比如
VAULT_TOKEN
和VAULT_URL
,以安全地与 Vault 交互。
- 在你的 GitHub 仓库中,添加必要的 Secrets,比如
- 设置 GitHub Actions Workflow:
- 在你的 GitHub 仓库中创建一个
.github/workflows
目录。 - 创建一个新的 YAML 文件,比如
cert-renewal.yml
。
- 在你的 GitHub 仓库中创建一个
- 编写 Workflow:
- 使用适当的 runner(例如
ubuntu-latest
)。 - 确保你有 ACME 客户端,比如 Certbot,安装在 runner 上。
- 编写步骤以使用 ACME 协议申请证书。
- 申请 Let's Encrypt 证书并保存到 Vault
- 使用适当的 runner(例如
示例 cert-renewal.yml
:
open-source-solution-design/Modern-Container-Application-Reference-Architecture/.github/workflows/cert-renewal.yml
name: Renew SSL Certs
on:
pull_request:
push:
paths:
- '.github/workflows/app-pipeline-renew-ssl-cert.yml'
workflow_dispatch:
branches:
- main
schedule:
- cron: '0 0 1 */2 *' # 每两个月的第一天执行
jobs:
renew-ssl-certs:
uses: open-source-solution-design/Modern-Container-Application-Reference-Architecture/.github/workflows/use-renew-ssl-certs.yml@main
with:
domain: 'onwalk.net'
secrets:
DNS_AK: ${{ secrets.DNS_AK }}
DNS_SK: ${{ secrets.DNS_SK }}
VAULT_URL: ${{ secrets.VAULT_URL }}
VAULT_TOKEN: ${{ secrets.VAULT_TOKEN }}
open-source-solution-design/Modern-Container-Application-Reference-Architecture/.github/workflows/use-renew-ssl-certs.yml
name: Workflow Call Renew SSL Certs
on:
workflow_call:
inputs:
domain:
required: true
type: string
secrets:
DNS_AK:
required: true
DNS_SK:
required: true
VAULT_URL:
required: true
VAULT_TOKEN:
required: truejobs:
renew-ssl-certs:
runs-on: ubuntu-latest
env:
DOMAIN: ${{ inputs.domain }}
DNS_AK: ${{ secrets.DNS_AK }}
DNS_SK: ${{ secrets.DNS_SK }}
VAULT_URL: ${{ secrets.VAULT_URL }}
VAULT_TOKEN: ${{ secrets.VAULT_TOKEN }}steps: - name: Checkout code uses: actions/checkout@v3 with: submodules: 'recursive' - name: Pre Setup run: | curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add - sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" sudo apt-get update && sudo apt-get install vault -y - name: Renew SSL Cert run: | set -x # 检查参数是否为空 check_not_empty() { if [[ -z $1 ]]; then echo "Error: $2 is empty. Please provide a value." exit 1 fi } # 检查参数是否为空 check_not_empty "$DNS_AK" "DNS_AK" && export Ali_Key=$DNS_AK check_not_empty "$DNS_SK" "DNS_SK" && export Ali_Secret=$DNS_SK check_not_empty "$VAULT_URL" "VAULT_URL" rm -fv ${DOMAIN}.key ${DOMAIN}.pem -f rm -fv /etc/ssl/${DOMAIN}.* -f # Try to issue a certificate from ZeroSSL. If it fails, try Let's Encrypt. curl https://get.acme.sh | sh -s email=156405189@qq.com sh ~/.acme.sh/acme.sh --set-default-ca --server zerossl --issue --force --dns dns_ali -d ${DOMAIN} -d "*.${DOMAIN}" if [ $? -eq 0 ]; then echo "Certificate from letsencrypt successfully issued" else sh ~/.acme.sh/acme.sh --set-default-ca --server letsencrypt --issue --force --dns dns_ali -d ${DOMAIN} -d "*.${DOMAIN}" if [ $? -eq 0 ]; then echo "Certificate from zerossl successfully issued" else echo "Command failed" exit 1 fi fi cat ~/.acme.sh/${DOMAIN}_ecc/${DOMAIN}.cer > ${DOMAIN}.pem cat ~/.acme.sh/${DOMAIN}_ecc/ca.cer >> ${DOMAIN}.pem cat ~/.acme.sh/${DOMAIN}_ecc/${DOMAIN}.key > ${DOMAIN}.key sudo cp ${DOMAIN}.pem /etc/ssl/ -f && sudo cp ${DOMAIN}.key /etc/ssl/ -f - name: Write certificate into Vault Server run: | export VAULT_ADDR=${VAULT_URL} export CERT_PATH="certs/${DOMAIN}" # Write Certificate and Private Key to Vault vault login ${VAULT_TOKEN} vault secrets enable -path=certs kv || true vault kv put ${CERT_PATH} \ certificate=@/etc/ssl/${DOMAIN}.pem \ private_key=@/etc/ssl/${DOMAIN}.key echo "Certificate and private key written to Vault at path: ${CERT_PATH}"</code></pre></div></div><p>至此,已经完成SSL Certs 申请的自动化,每两个月执行一次,确保Vault中永远存储有效的证书。</p><figure class=""><div class="rno-markdown-img-url" style="text-align:center"><div class="rno-markdown-img-url-inner" style="width:auto"><div style="width:100%"><img src="https://cdn.static.attains.cn/app/developer-bbs/upload/1722769161232552000.png" /></div></div></div></figure><p>流水线执行成功后,登录 Vault UI 已经看到域名证书已经保存</p><figure class=""><div class="rno-markdown-img-url" style="text-align:center"><div class="rno-markdown-img-url-inner" style="width:auto"><div style="width:100%"><img src="https://cdn.static.attains.cn/app/developer-bbs/upload/1722769161846785000.png" /></div></div></div></figure><h2 id="8uoro" name="%E5%BA%94%E7%94%A8%E9%9B%86%E7%BE%A4%E4%BE%A7%E9%85%8D%E7%BD%AE">应用集群侧配置</h2><h3 id="fcris" name="%E5%B0%86%E8%AF%81%E4%B9%A6%E5%88%86%E5%88%B0%E5%88%B0%E5%BA%94%E7%94%A8%E9%9B%86%E7%BE%A4%E4%B8%AD">将证书分到到应用集群中</h3><p>接下来的的工作就是,如何在IAC流水线中,集成Vault 操作,读取域名证书并写入集群master节点 /etc/ssl/ 目录</p><ul class="ul-level-0"><li>playbook/roles/cert-manager/tasks/main.yml</li></ul><div class="rno-markdown-code"><div class="rno-markdown-code-toolbar"><div class="rno-markdown-code-toolbar-info"><div class="rno-markdown-code-toolbar-item is-type"><span class="is-m-hidden">代码语言:</span>yaml</div></div><div class="rno-markdown-code-toolbar-opt"><div class="rno-markdown-code-toolbar-copy"><i class="icon-copy"></i><span class="is-m-hidden">复制</span></div></div></div><div class="developer-code-block"><pre class="prism-token token line-numbers language-yaml"><code class="language-yaml" style="margin-left:0">- name: Fetch Certificate and Private Key from Vault
script: files/get_certificate.sh {{ domain }} {{ vault_url }} {{ vault_token }}
when: (inventory_hostname in groups[group]) and (vault == true)
- playbook/roles/cert-manager/files/get_certificate.sh
#!/bin/bash
sudo apt-get update
sudo apt install -y software-properties-common
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list > /dev/null
sudo apt-get update
sudo apt-get install -y python3-pip jq vaultcheck_empty() {
if [ -z "$1" ]; then
echo "$2"
exit 1
fi
}check_empty "$1" "Please provide DOMAIN" && DOMAIN=$1 check_empty "$2" "Please provide VAULT_ADDR" && VAULT_ADDR=$2 check_empty "$3" "Please provide VAULT_TOKEN" && VAULT_TOKEN=$3
SECRET_PATH="certs/$DOMAIN"
Output paths
CERTIFICATE_PATH="/etc/ssl/${DOMAIN}.pem"
PRIVATE_KEY_PATH="/etc/ssl/${DOMAIN}.key"vault login
Read certificate from Vault
vault kv get -field=certificate certs/{DOMAIN} > "CERTIFICATE_PATH"
Read private key from Vault
vault kv get -field=private_key certs/{DOMAIN} > "PRIVATE_KEY_PATH"
Set permissions for the private key (modify as needed)
chmod 600 "$PRIVATE_KEY_PATH"
Check if certificate and private key files are non-empty
if [ ! -s "CERTIFICATE_PATH" ] || [ ! -s "PRIVATE_KEY_PATH" ]; then
echo "Certificate or private key is empty. Exiting..."
exit 1
else
echo "Certificate and private key have been written to CERTIFICATE_PATH and PRIVATE_KEY_PATH"
验证 SSL 证书
登陆节点,执行命令:cat /etc/ssl/<domain_name>.* 验证分发的证书
验证 Secret
可以使用以下命令检查 Secret:
kubectl get secret my-cert-secret -o yaml
最后使用Curl 命令验证使用和证书的服务或者API接口否生效
curl https://your_svc.com
总结
通过以上步骤,你可以建立一个自动化流程,通过 GitHub Actions 自动续订 Let's Encrypt 证书,将其存储在 Vault 中,并更新 Kubernetes 集群的 CertManager 配置以使用这些证书。请注意,这里的示例可能需要根据你的环境和需求进行调整。在部署到生产环境之前,请确保对配置进行充分测试。