gitlab runner operator部署配置

背景说明

由于公司管理的git runner资源不足，导致并发的任务比较多时，出现大面积的排队，比较影响效率。基于此问题，我们可以自建一部分Runner给到相应的仓库使用。这里我们有自建的
在k8s集群自建Runner

参考文档
先决条件

• Kubernetes v1.21 及更高版本
  
• Cert manager v1.7.1
  

安装配置Operator

• 安装OLM（Operator Lifecycle Manager）
  

1. curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.32.0/install.sh | bash -s v0.32.0
2. # 如果下载了install.sh，可以直接执行 chmod +x install.sh && ./install.sh v0.32.0
3. root@yuhaohao-10-10-7-19:~# kubectl  get pod -n olm
4. NAME                                READY   STATUS    RESTARTS   AGE
5. catalog-operator-55894c8cf8-d84vw   1/1     Running   0          3d
6. olm-operator-6946d4b877-t55w6       1/1     Running   0          3d
7. operatorhubio-catalog-nvq9q         1/1     Running   0          17h
8. packageserver-64fdd96cbb-ggpnh      1/1     Running   0          3d
9. packageserver-64fdd96cbb-s7r59      1/1     Running   0          3d

复制代码

2.安装Operator

1. kubectl create -f https://operatorhub.io/install/gitlab-runner-operator.yaml

复制代码

3.查看安装的资源

1. root@yuhaohao-10-10-7-19:~# kubectl  get csv -n operators
2. NAME                             DISPLAY         VERSION   REPLACES                         PHASE
3. gitlab-runner-operator.v1.37.0   GitLab Runner   1.37.0    gitlab-runner-operator.v1.36.0   Succeeded
4. root@yuhaohao-10-10-7-19:~# kubectl  get pod -n operators
5. NAME                                                             READY   STATUS    RESTARTS   AGE
6. gitlab-runner-gitlab-runnercontroller-manager-84d6f69dc8-7nsdc   2/2     Running   0          3d

复制代码

安装配置Gitlab Runner

1.安装minio

1. # 这里为了实现gitlab runner的缓存，提高构建效率，可以部署minio作为gitlab runner的缓存
2. # 这里我们安装的是minio 12.9.0版本
3. # 这里我们可以基于主机目录，配置StorageClass，实现动态PVC存储
4. # 配置local-path-provisioner
5. root@yuhaohao-10-10-7-19:~/minio-install# cat /home/yuhaohao/local-path.yaml
6. apiVersion: v1
7. kind: Namespace
8. metadata:
9.   name: local-path-storage
11. ---
12. apiVersion: v1
13. kind: ServiceAccount
14. metadata:
15.   name: local-path-provisioner-service-account
16.   namespace: local-path-storage
18. ---
19. apiVersion: rbac.authorization.k8s.io/v1
20. kind: Role
21. metadata:
22.   name: local-path-provisioner-role
23.   namespace: local-path-storage
24. rules:
25.   - apiGroups: [""]
26.     resources: ["pods"]
27.     verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
29. ---
30. apiVersion: rbac.authorization.k8s.io/v1
31. kind: ClusterRole
32. metadata:
33.   name: local-path-provisioner-role
34. rules:
35.   - apiGroups: [""]
36.     resources: ["nodes", "persistentvolumeclaims", "configmaps", "pods", "pods/log"]
37.     verbs: ["get", "list", "watch"]
38.   - apiGroups: [""]
39.     resources: ["persistentvolumes"]
40.     verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
41.   - apiGroups: [""]
42.     resources: ["events"]
43.     verbs: ["create", "patch"]
44.   - apiGroups: ["storage.k8s.io"]
45.     resources: ["storageclasses"]
46.     verbs: ["get", "list", "watch"]
48. ---
49. apiVersion: rbac.authorization.k8s.io/v1
50. kind: RoleBinding
51. metadata:
52.   name: local-path-provisioner-bind
53.   namespace: local-path-storage
54. roleRef:
55.   apiGroup: rbac.authorization.k8s.io
56.   kind: Role
57.   name: local-path-provisioner-role
58. subjects:
59.   - kind: ServiceAccount
60.     name: local-path-provisioner-service-account
61.     namespace: local-path-storage
63. ---
64. apiVersion: rbac.authorization.k8s.io/v1
65. kind: ClusterRoleBinding
66. metadata:
67.   name: local-path-provisioner-bind
68. roleRef:
69.   apiGroup: rbac.authorization.k8s.io
70.   kind: ClusterRole
71.   name: local-path-provisioner-role
72. subjects:
73.   - kind: ServiceAccount
74.     name: local-path-provisioner-service-account
75.     namespace: local-path-storage
77. ---
78. apiVersion: apps/v1
79. kind: Deployment
80. metadata:
81.   name: local-path-provisioner
82.   namespace: local-path-storage
83. spec:
84.   replicas: 1
85.   selector:
86.     matchLabels:
87.       app: local-path-provisioner
88.   template:
89.     metadata:
90.       labels:
91.         app: local-path-provisioner
92.     spec:
93.       serviceAccountName: local-path-provisioner-service-account
94.       containers:
95.         - name: local-path-provisioner
96.           image: registry.test.com/test/rancher/local-path-provisioner:v0.0.31
97.           imagePullPolicy: IfNotPresent
98.           command:
99.             - local-path-provisioner
100.             - --debug
101.             - start
102.             - --config
103.             - /etc/config/config.json
104.           volumeMounts:
105.             - name: config-volume
106.               mountPath: /etc/config/
107.           env:
108.             - name: POD_NAMESPACE
109.               valueFrom:
110.                 fieldRef:
111.                   fieldPath: metadata.namespace
112.             - name: CONFIG_MOUNT_PATH
113.               value: /etc/config/
114.       volumes:
115.         - name: config-volume
116.           configMap:
117.             name: local-path-config
119. ---
120. apiVersion: storage.k8s.io/v1
121. kind: StorageClass
122. metadata:
123.   name: local-path
124. provisioner: rancher.io/local-path
125. volumeBindingMode: WaitForFirstConsumer
126. reclaimPolicy: Delete
128. ---
129. kind: ConfigMap
130. apiVersion: v1
131. metadata:
132.   name: local-path-config
133.   namespace: local-path-storage
134. data:
135.   config.json: |-
136.     {
137.             "nodePathMap":[
138.             {
139.                     "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
140.                     "paths":["/data"]
141.             }
142.             ]
143.     }
144.   setup: |-
145.     #!/bin/sh
146.     set -eu
147.     mkdir -m 0777 -p "$VOL_DIR"
148.   teardown: |-
149.     #!/bin/sh
150.     set -eu
151.     rm -rf "$VOL_DIR"
152.   helperPod.yaml: |-
153.     apiVersion: v1
154.     kind: Pod
155.     metadata:
156.       name: helper-pod
157.     spec:
158.       priorityClassName: system-node-critical
159.       tolerations:
160.         - key: node.kubernetes.io/disk-pressure
161.           operator: Exists
162.           effect: NoSchedule
163.       containers:
164.       - name: helper-pod
165.         image: registry.test.com/test/busybox
166.         imagePullPolicy: IfNotPresent
167. $ kubectl apply -f local-path.yaml
168. $ kubectl get pod -n local-path-storage
169. NAME                                      READY   STATUS    RESTARTS   AGE
170. local-path-provisioner-6bf6d4456b-f9sbr   1/1     Running   0          3d
171. $ kubectl  get sc
172. NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
173. local-path   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  16d
175. # 创建minio所需的pvc
176. $ cat gitlab-runner-minio-pvc.yaml
177. apiVersion: v1
178. kind: PersistentVolumeClaim
179. metadata:
180.   labels:
181.     app.kubernetes.io/instance: minio
182.     app.kubernetes.io/managed-by: Helm
183.   name: gitlab-runner-minio-pvc
184.   namespace: gitlab-runner
185. spec:
186.   accessModes:
187.   - ReadWriteOnce
188.   resources:
189.     requests:
190.       storage: 100Gi
191.   storageClassName: local-path
192.   volumeMode: Filesystem
193. $ kubectl apply -f gitlab-runner-minio-pvc.yaml
194. $ kubectl get pvc -n gitlab-runner
195. NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
196. gitlab-runner-minio-pvc   Bound    pvc-31430c05-16cc-4441-814b-7aae70b7b134   100Gi      RWO            local-path     <unset>                 15d
198. # 部署minio
199. $ cat minio-values.yaml
200. root@yuhaohao-10-10-7-19:~/minio-install# cat minio-values.yaml
201. image:
202.   registry: registry.test.com
203.   repository: test/bitnami/minio
204.   tag: 2023.11.1-debian-11-r0
205.   pullPolicy: IfNotPresent
207. fullnameOverride: "minio"
208. auth:
209.   rootUser: admin
210.   rootPassword: "minio12345"
211. defaultBuckets: "gitlab"
212. persistence:
213.   ## @param persistence.enabled Enable MinIO® data persistence using PVC. If false, use emptyDir
214.   ##
215.   enabled: true
216.   storageClass: ""
217.   ## @param persistence.mountPath Data volume mount path
218.   mountPath: /bitnami/minio/data
219.   accessModes:
220.     - ReadWriteOnce
221.   size: 100Gi
222.   ## @param persistence.annotations Annotations for the PVC
223.   ##
224.   annotations: {}
225.   ## @param persistence.existingClaim Name of an existing PVC to use (only in `standalone` mode)
226.   ##
227.   existingClaim: "gitlab-runner-minio-pvc"
229. $ helm install -f minio-values.yaml minio ./minio -n gitlab-runner
230. root@yuhaohao-10-10-7-19:~/minio-install# kubectl  get pod -n gitlab-runner
231. NAME                              READY   STATUS    RESTARTS   AGE
232. minio-6d947866f5-rhmwz            1/1     Running   0          3d

复制代码

2.安装gitlab runner

1. # 配置通用的runner配置信息
2. # 注意pre_clone_script 这里的配置是为了解决`Git LFS is not enabled on this GitLab server`的报错
3. $ cat custom.yaml
4. apiVersion: v1
5. data:
6.   config.toml: |-
7.     [[runners]]
8.       environment = ["GIT_LFS_SKIP_SMUDGE=1"]
9.       pre_clone_script = """
10.         echo "Setting up secrets"
11.         ################## run in runner-helper
12.         echo "######### ${GITLAB_USER_LOGIN}"
13.         git config --global lfs.url "https://lfs.test.com"
14.         git config --global credential.helper store
15.         echo "https://test:test@lfs.test.com" >> $HOME/.git-credentials
16.         mkdir -pv $HOME/.ssh >> /dev/null
17.         chmod 700 $HOME/.ssh
18.         echo -e "Host gitlab.testcom\n\tStrictHostKeyChecking no\n" >> $HOME/.ssh/config
19.       """
20.       [runners.kubernetes]
21.         image_pull_secrets = ["inner"]
22.         pull_policy = "if-not-present"
23.         privileged = true
24.         [runners.kubernetes.node_selector]
25.           "ci-node" = "True"
26.         [runners.kubernetes.node_tolerations]
27.           "ci-node=true" = "NoSchedule"
28. kind: ConfigMap
29. metadata:
30.   name: runner-custom-config
31.   namespace: gitlab-runner
33. # 配置minio的secret
34. $ cat minio-secret.yaml
35. ---
36. apiVersion: v1
37. data:
38.   accesskey: YWRtaW4=
39.   secretkey: bWluaW8xMjM0NQ==
40. kind: Secret
41. metadata:
42.   name: minio-secret-for-runner
43.   namespace: gitlab-runner
44. type: Opaque
46. # 配置拉取镜像的secrets
47. $ cat secrets-inner.yaml
48. apiVersion: v1
49. data:
50.   .dockerconfigjson: eyJhdXRocyI6eyjkjkj9ib3QuZG9ja2VyIiwicGFzc3dvcmQiOiJTVGFpMjAyMy5kayIsImF1dGgiOiJkbWx3WlhJdWNtOWliM1F1Wkc5amEyVnlPbE5VWVdreU1ESXpMbVJyIn19fQ==
51. kind: Secret
52. metadata:
53.   name: inner
54.   namespace: gitlab-runner
55. type: kubernetes.io/dockerconfigjson
57. # 配置具体的runner
58. $ cat test-runner.yaml
59. ---
60. apiVersion: v1
61. kind: Secret
62. metadata:
63.   name: gitlab-runner-secret-test
64.   namespace: gitlab-runner
65. type: Opaque
66. stringData:
67.   runner-registration-token: ""
68.   runner-token: "glrt-t2_73uew3jtestEhDM" # 注意，这里是gitlab runner注册的token
70. ---
71. apiVersion: apps.gitlab.com/v1beta2
72. kind: Runner
73. metadata:
74.   name: test
75.   namespace: gitlab-runner
76. spec:
77.   logLevel: debug
78.   # 这里，不要用gitlab-runner-ocp 13.x版本，会有重启runner后，runner无法重新注册连接的问题。
79.   runnerImage: "registry.test.com/test/gitlab-runner-ocp:v17.4.0"
80.   gitlabUrl: https://gitlab.test.com/
81.   buildImage: alpine
82.   helperImage: registry.test.com/test/gitlab-runner-helper:ubuntu-x86_64-v17.5.1
83.   concurrent: 50
84.   token: gitlab-runner-secret-test
85.   tags: "test"
86.   locked: true
87.   cacheType: s3
88.   cacheShared: true
89.   s3:
90.     server: minio:9000
91.     credentials: minio-secret-for-runner
92.     bucket: sumo19
93.     insecure: true
94.   config: runner-custom-config
96. $ kubectl apply -f .
97. # 查看runner
98. $  kubectl  get pod -n gitlab-runner
99. NAME                              READY   STATUS    RESTARTS   AGE
100. minio-6d947866f5-rhmwz            1/1     Running   0          3d1h
101. test-runner-7475ddc7f-nn7xh   1/1     Running   0          15h

复制代码

运行测试

来源：程序园用户自行投稿发布，如果侵权，请联系站长删除
免责声明：如果侵犯了您的权益，请联系站长，我们会及时删除侵权内容，谢谢合作！