https://pastein.ru/t/BxJ
Загрузка данных
Running with gitlab-runner 15.8.0 (12335144)
on gitlab-runner-gitlab-runner-7d64bfd458-xlzsd u1oHxtyM, system ID: r_OKg3w9r0mqgd
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runners
Using Kubernetes executor with image registry.rshbdev.ru/appfarm/infra/images/kube-client-apps:8.21.10 ...
Using attach strategy to execute scripts...
Preparing environment
00:06
Waiting for pod gitlab-runners/runner-u1ohxtym-project-82133-concurrent-0cvttr to be running, status is Pending
Waiting for pod gitlab-runners/runner-u1ohxtym-project-82133-concurrent-0cvttr to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-u1ohxtym-project-82133-concurrent-0cvttr via gitlab-runner-gitlab-runner-7d64bfd458-xlzsd...
Getting source from Git repository
00:03
$ git config --global --add url."https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/".insteadOf "https://${CI_SERVER_HOST}" # collapsed multi-line command
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik/.git/
Created fresh repository.
Checking out 9b7c5eff as 1.6.0-RC1...
Skipping Git submodules setup
Downloading artifacts
00:02
Downloading artifacts for sast_semgrep (23170535)...
Downloading artifacts from coordinator... ok id=23170535 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for sca_scan (23170539)...
Downloading artifacts from coordinator... ok id=23170539 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for init_dojo (23170532)...
Downloading artifacts from coordinator... ok id=23170532 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for sast_ptai (23170536)...
Downloading artifacts from coordinator... ok id=23170536 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for devsecops_antivirus_scan (23170533)...
Downloading artifacts from coordinator... ok id=23170533 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for bca_trivy (23170546)...
Downloading artifacts from coordinator... ok id=23170546 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for dockerfilegen (23170537)...
Downloading artifacts from coordinator... ok id=23170537 responseStatus=200 OK token=64_qPjtK
Downloading artifacts for secrets_gitleaks (23170534)...
Downloading artifacts from coordinator... ok id=23170534 responseStatus=200 OK token=64_qPjtK
Executing "step_script" stage of the job script
10:58
$ ( umask 0077; mkdir -p ~/.kube && echo "$KUBECONFIG_COMBINED" | base64 -d > ~/.kube/config )
$ if [[ ${IS_SENSITIVE_SYSTEM} = true ]]; then export K8S_CLUSTERS=$K8S_CLUSTERS_SENSITIVE; fi
$ echo 'IS_SENSITIVE_SYSTEM:' $IS_SENSITIVE_SYSTEM
IS_SENSITIVE_SYSTEM: true
$ echo 'TARGET_CLUSTERS:' $K8S_CLUSTERS
TARGET_CLUSTERS: rcsdstbl
$ set -x
++ echo '$ for CLUSTER in $K8S_CLUSTERS; do'
$ for CLUSTER in $K8S_CLUSTERS; do
++ for CLUSTER in $K8S_CLUSTERS
++ echo '$ export CLUSTER=$CLUSTER'
$ export CLUSTER=$CLUSTER
++ export CLUSTER=rcsdstbl
++ CLUSTER=rcsdstbl
++ echo '$ if [[ ${CI_ENVIRONMENT_SLUG} = production ]]; then export NAMESPACE=isys-${ISYS_NAME}; else export NAMESPACE=isys-${ISYS_NAME}-${CI_ENVIRONMENT_SLUG}; fi'
$ if [[ ${CI_ENVIRONMENT_SLUG} = production ]]; then export NAMESPACE=isys-${ISYS_NAME}; else export NAMESPACE=isys-${ISYS_NAME}-${CI_ENVIRONMENT_SLUG}; fi
++ [[ rc = production ]]
++ export NAMESPACE=isys-ecp-rc
++ NAMESPACE=isys-ecp-rc
++ echo '$ read -r REVISION STATUS < <(helm --kube-context ${CLUSTER:-default} -n $NAMESPACE history $CI_PROJECT_NAME | tail -1 | cut -f1,3) || true'
$ read -r REVISION STATUS < <(helm --kube-context ${CLUSTER:-default} -n $NAMESPACE history $CI_PROJECT_NAME | tail -1 | cut -f1,3) || true
++ read -r REVISION STATUS
+++ helm --kube-context rcsdstbl -n isys-ecp-rc history ms-core-formatik
+++ tail -1
+++ cut -f1,3
++ echo '$ if [[ "$STATUS" =~ "pending" ]]; then helm --kube-context ${CLUSTER:-default} -n $NAMESPACE rollback $CI_PROJECT_NAME $REVISION || true; fi'
$ if [[ "$STATUS" =~ "pending" ]]; then helm --kube-context ${CLUSTER:-default} -n $NAMESPACE rollback $CI_PROJECT_NAME $REVISION || true; fi
++ [[ deployed =~ pending ]]
++ echo '$ helmfile ${HELMFILE_DEFAULT_NAMESPACE:+--namespace $HELMFILE_DEFAULT_NAMESPACE} --environment ${CLUSTER:-default} -f deploy/helmfile.yaml --log-level info apply --suppress-secrets --set "additionalWorkloadAnnotations.gitlabPipeline=${CI_PIPELINE_URL}" --set "additionalWorkloadAnnotations.gitlabCommit=${CI_COMMIT_SHA}"'
$ helmfile ${HELMFILE_DEFAULT_NAMESPACE:+--namespace $HELMFILE_DEFAULT_NAMESPACE} --environment ${CLUSTER:-default} -f deploy/helmfile.yaml --log-level info apply --suppress-secrets --set "additionalWorkloadAnnotations.gitlabPipeline=${CI_PIPELINE_URL}" --set "additionalWorkloadAnnotations.gitlabCommit=${CI_COMMIT_SHA}"
++ helmfile --environment rcsdstbl -f deploy/helmfile.yaml --log-level info apply --suppress-secrets --set additionalWorkloadAnnotations.gitlabPipeline=https://gitlab.rshbdev.ru/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik/-/pipelines/3324646 --set additionalWorkloadAnnotations.gitlabCommit=9b7c5eff6c2768750cd9c290e94c891d2e2d2433
== /usr/local/link/helmfile: Initialize base deploy hierarchy in /builds/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik
no matches for path: envs/rc/rcsdstbl/helmfile.yaml.gotmpl
Adding repo rshb-charts https://nexus.rshbdev.ru/repository/charts/
"rshb-charts" has been added to your repositories
Comparing release=vault-secrets-ms-core-formatik, chart=rshb-charts/raw
Comparing release=psvc-ms-core-formatik, chart=rshb-charts/raw
Comparing release=platform-database-ms-core-formatik, chart=rshb-charts/raw
Comparing release=links-ms-core-formatik, chart=rshb-charts/raw
Comparing release=ms-core-formatik, chart=rshb-charts/base
isys-ecp-rc, ms-core-formatik, Deployment (apps) has changed:
# Source: base/templates/workload.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ms-core-formatik
labels:
app: ms-core-formatik
fullname: ms-core-formatik
chart: base-1.15.2
release: ms-core-formatik
heritage: Helm
isys: "ecp-rc"
psvc: "ms-core-formatik"
- version: "1.5.0-RC5"
+ version: "1.6.0-RC1"
workload.topology.app.farm/zone: "rumsk1"
annotations:
- gitlabCommit: 5f227a87909e6230a343ba957595dad730555825
- gitlabPipeline: https://gitlab.rshbdev.ru/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik/-/pipelines/3275344
- platform.ckpr.integrations.rshbintech.ru/gitlab-commit-sha: 5f227a87
- platform.ckpr.integrations.rshbintech.ru/gitlab-pipeline-id: "3275344"
+ gitlabCommit: 9b7c5eff6c2768750cd9c290e94c891d2e2d2433
+ gitlabPipeline: https://gitlab.rshbdev.ru/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik/-/pipelines/3324646
+ platform.ckpr.integrations.rshbintech.ru/gitlab-commit-sha: 9b7c5eff
+ platform.ckpr.integrations.rshbintech.ru/gitlab-pipeline-id: "3324646"
spec:
strategy:
type: RollingUpdate
replicas: 1
selector:
matchLabels:
app: ms-core-formatik
fullname: ms-core-formatik
release: ms-core-formatik
template:
metadata:
labels:
app: ms-core-formatik
fullname: ms-core-formatik
chart: base-1.15.2
release: ms-core-formatik
heritage: Helm
isys: "ecp-rc"
psvc: "ms-core-formatik"
- version: "1.5.0-RC5"
+ version: "1.6.0-RC1"
workload.topology.app.farm/zone: "rumsk1"
annotations:
- checksum/configMapsEnv: "6b9bf5f64e9dbe565cd9828b2bb38cde503beee5682848123f947b83e7462562"
- checksum/secretEnv: "deed40c695c47394677c8d8422adc877e9a06e2f61c74bfc31784fda599fe09c"
+ checksum/configMapsEnv: "e57971cf16a12e12ca958cf1e50112b9a177cd7b8e50d5c9f44077bbbc0c5e58"
+ checksum/secretEnv: "93275fefaeb1c6385a3a883e0f68a688616e117c56d2e5049626e9790c21b140"
inject.istio.io/templates: "sidecar,custom"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
sidecar.istio.io/userVolume: "[{\"name\": \"wasmfilters-dir\",\"configMap\": {\"name\": \"ecp-rc-envoy-filters\"}}]"
sidecar.istio.io/userVolumeMount: "[{\"mountPath\":\"/var/local/lib/wasm-filters\",\"name\":\"wasmfilters-dir\"}]"
spec:
automountServiceAccountToken: false
serviceAccountName: default
nodeSelector:
workload.topology.app.farm/zone: rumsk1
priorityClassName: rumsk1
containers:
- name: app
- image: registry.rshbdev.ru/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik:1.5.0-RC5
+ image: registry.rshbdev.ru/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik:1.6.0-RC1
imagePullPolicy: IfNotPresent
envFrom:
- configMapRef:
name: ms-core-formatik-app-cm-env
- secretRef:
name: ms-core-formatik-app-secret-env
resources:
limits:
cpu: 1800m
memory: 1600Mi
requests:
cpu: 800m
memory: 1000Mi
ports:
livenessProbe:
failureThreshold: 5
httpGet:
path: /health/liveness
port: 8080
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health/readiness
port: 8080
initialDelaySeconds: 15
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
- name: app-vault-secrets
mountPath: /secrets
readOnly: true
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
procMount: Default
readOnlyRootFilesystem: false
runAsGroup: 1001
runAsNonRoot: true
runAsUser: 1001
securityContext:
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 1001
runAsNonRoot: true
runAsUser: 1001
tolerations:
- effect: NoSchedule
key: workload.topology.app.farm/zone
operator: Exists
hostNetwork: false
volumes:
- name: app-vault-secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: ms-core-formatik-app-secret-provider
isys-ecp-rc, ms-core-formatik, Service (v1) has changed:
# Source: base/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: ms-core-formatik
labels:
app: ms-core-formatik
fullname: ms-core-formatik
chart: base-1.15.2
release: ms-core-formatik
heritage: Helm
isys: "ecp-rc"
psvc: "ms-core-formatik"
- version: "1.5.0-RC5"
+ version: "1.6.0-RC1"
spec:
ports:
- name: web
port: 80
protocol: TCP
targetPort: 8080
selector:
app: ms-core-formatik
fullname: ms-core-formatik
release: ms-core-formatik
type: ClusterIP
isys-ecp-rc, ms-core-formatik-app-cm-env, ConfigMap (v1) has changed:
# Source: base/templates/cm-env.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ms-core-formatik-app-cm-env
labels:
app: ms-core-formatik
fullname: ms-core-formatik
chart: base-1.15.2
release: ms-core-formatik
heritage: Helm
isys: "ecp-rc"
psvc: "ms-core-formatik"
- version: "1.5.0-RC5"
+ version: "1.6.0-RC1"
workload.topology.app.farm/zone: "rumsk1"
data:
APP_KAFKA_BOOTSTRAP_GROUP_ID: "core.formatik"
APP_KAFKA_CLIENT_ID: "core.formatik"
APP_KAFKA_CLIENT_OFFSET: "true"
APP_KAFKA_CONSUMER_TOPIC: "core.streaming.internal.formatik_request"
APP_KAFKA_PRODUCER_TOPIC: "core.streaming.internal.formatik_response"
APP_KAFKA_RETRY_INTERVAL_MS: "10000"
APP_LIBREOFFICE_TIMEOUT: "300000"
APP_MINIO_BUCKET: "core"
APP_MINIO_HOST: "10.21.108.25"
APP_MINIO_NAME: "core"
APP_MINIO_PORT: "9000"
APP_PLATFORM_KAFKA_BOOTSTRAP_SERVER: "adapter-ecp-internal-cluster.isys-ecp-rc-links:9092"
APP_PORT: "8080"
ENV: "RC"
LOGGING_LEVEL_RU_RSHBINTECH: "INFO"
isys-ecp-rc, ms-core-formatik-app-secret-env, Secret (v1) has changed:
+ Changes suppressed on sensitive content of type Secret
isys-ecp-rc, ms-core-formatik-app-secret-provider, SecretProviderClass (secrets-store.csi.x-k8s.io) has changed:
# Source: base/templates/secret-provider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: ms-core-formatik-app-secret-provider
labels:
app: ms-core-formatik
fullname: ms-core-formatik
chart: base-1.15.2
release: ms-core-formatik
heritage: Helm
isys: "ecp-rc"
psvc: "ms-core-formatik"
- version: "1.5.0-RC5"
+ version: "1.6.0-RC1"
workload.topology.app.farm/zone: "rumsk1"
spec:
provider: vault
parameters:
vaultKubernetesMountPath: rcsdstbl
roleName: "csi-secrets-reader-rc"
objects: |
- objectName: APP_MINIO_KEY
secretPath: manual_kv_rc/data/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik
secretKey: APP_MINIO_KEY
- objectName: APP_MINIO_PASSWORD
secretPath: manual_kv_rc/data/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik
secretKey: APP_MINIO_PASSWORD
- objectName: APP_MINIO_SECRET
secretPath: manual_kv_rc/data/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik
secretKey: APP_MINIO_SECRET
- objectName: APP_MINIO_USER
secretPath: manual_kv_rc/data/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik
secretKey: APP_MINIO_USER
secretObjects:
- secretName: ms-core-formatik-app-secret-env
type: Opaque
labels:
app: ms-core-formatik
fullname: ms-core-formatik
chart: base-1.15.2
release: ms-core-formatik
heritage: Helm
isys: "ecp-rc"
psvc: "ms-core-formatik"
- version: "1.5.0-RC5"
+ version: "1.6.0-RC1"
workload.topology.app.farm/zone: "rumsk1"
data:
- objectName: APP_MINIO_KEY
key: APP_MINIO_KEY
- objectName: APP_MINIO_PASSWORD
key: APP_MINIO_PASSWORD
- objectName: APP_MINIO_SECRET
key: APP_MINIO_SECRET
- objectName: APP_MINIO_USER
key: APP_MINIO_USER
Listing releases matching ^pjob-ms-core-formatik$
Listing releases matching ^exsvc-ms-core-formatik$
Listing releases matching ^data-ms-core-formatik$
Listing releases matching ^assets-config-ms-core-formatik$
Listing releases matching ^ms-core-formatik-raw$
Listing releases matching ^ms-core-formatik-grafana-dashboard$
Listing releases matching ^ms-core-formatik-prometheus-rules$
Listing releases matching ^ms-core-formatik-rumsk1$
Listing releases matching ^ms-core-formatik-rumsk2$
Upgrading release=ms-core-formatik, chart=rshb-charts/base
FAILED RELEASES:
NAME
ms-core-formatik
in deploy/helmfile.yaml: failed processing release ms-core-formatik: command "/usr/local/link/helm" exited with non-zero status:
PATH:
/usr/local/link/helm
ARGS:
0: helm (4 bytes)
1: --kube-context (14 bytes)
2: rcsdstbl (8 bytes)
3: upgrade (7 bytes)
4: --install (9 bytes)
5: --reset-values (14 bytes)
6: ms-core-formatik (16 bytes)
7: rshb-charts/base (16 bytes)
8: --version (9 bytes)
9: 1.15.2 (6 bytes)
10: --wait (6 bytes)
11: --timeout (9 bytes)
12: 600s (4 bytes)
13: --atomic (8 bytes)
14: --kube-context (14 bytes)
15: rcsdstbl (8 bytes)
16: --namespace (11 bytes)
17: isys-ecp-rc (11 bytes)
18: --values (8 bytes)
19: /tmp/helmfile367585299/isys-ecp-rc-ms-core-formatik-values-79756f6c47 (69 bytes)
20: --values (8 bytes)
21: /tmp/helmfile1119567231/isys-ecp-rc-ms-core-formatik-values-6749f9c46 (69 bytes)
22: --set (5 bytes)
23: additionalWorkloadAnnotations.gitlabPipeline=https://gitlab.rshbdev.ru/rshbintech/it-corp/rprul/ecp/core/backend/ms-core-formatik/-/pipelines/3324646 (149 bytes)
24: --set (5 bytes)
25: additionalWorkloadAnnotations.gitlabCommit=9b7c5eff6c2768750cd9c290e94c891d2e2d2433 (83 bytes)
26: --history-max (13 bytes)
27: 10 (2 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
Error: UPGRADE FAILED: release ms-core-formatik failed, and has been rolled back due to atomic being set: timed out waiting for the condition
COMBINED OUTPUT:
Error: UPGRADE FAILED: release ms-core-formatik failed, and has been rolled back due to atomic being set: timed out waiting for the condition
Uploading artifacts for failed job
00:01
Uploading artifacts...
deploy.env: found 1 matching artifact files and directories
Uploading artifacts as "dotenv" to coordinator... 201 Created id=23171717 responseStatus=201 Created token=64_qPjtK
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1