https://pastein.ru/t/j9W
Загрузка данных
Running with gitlab-runner 15.8.0 (12335144)
on gitlab-runner-gitlab-runner-64775b4cff-qpdc8 9cgwHtm5, system ID: r_FsAm41LqwxCr
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runners
Using Kubernetes executor with image registry.rshbdev.ru/appfarm/infra/images/kube-client-apps:8.21.15 ...
Using attach strategy to execute scripts...
Preparing environment
00:05
Waiting for pod gitlab-runners/runner-9cgwhtm5-project-18396-concurrent-05m55x to be running, status is Pending
Waiting for pod gitlab-runners/runner-9cgwhtm5-project-18396-concurrent-05m55x to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-9cgwhtm5-project-18396-concurrent-05m55x via gitlab-runner-gitlab-runner-64775b4cff-qpdc8...
Getting source from Git repository
00:03
$ git config --global --add url."https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/".insteadOf "https://${CI_SERVER_HOST}" # collapsed multi-line command
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards/.git/
Created fresh repository.
Checking out 37271b41 as master...
Skipping Git submodules setup
Downloading artifacts
00:02
Downloading artifacts for build (24185053)...
Downloading artifacts from coordinator... ok id=24185053 responseStatus=200 OK token=64_ZGKz1
Downloading artifacts for devsecops_antivirus_scan (24185046)...
Downloading artifacts from coordinator... ok id=24185046 responseStatus=200 OK token=64_ZGKz1
Downloading artifacts for dockerfilegen (24185050)...
Downloading artifacts from coordinator... ok id=24185050 responseStatus=200 OK token=64_ZGKz1
Downloading artifacts for unit (24185055)...
Downloading artifacts from coordinator... ok id=24185055 responseStatus=200 OK token=64_ZGKz1
Executing "step_script" stage of the job script
16:40
$ ( umask 0077; mkdir -p ~/.kube && echo "$KUBECONFIG_COMBINED" | base64 -d > ~/.kube/config )
$ if [[ ${IS_SENSITIVE_SYSTEM} = true ]]; then export K8S_CLUSTERS=$K8S_CLUSTERS_SENSITIVE; fi
$ echo 'IS_SENSITIVE_SYSTEM:' $IS_SENSITIVE_SYSTEM
IS_SENSITIVE_SYSTEM: true
$ echo 'TARGET_CLUSTERS:' $K8S_CLUSTERS
TARGET_CLUSTERS: devstbl
$ set -x
++ echo '$ for CLUSTER in $K8S_CLUSTERS; do'
$ for CLUSTER in $K8S_CLUSTERS; do
++ for CLUSTER in $K8S_CLUSTERS
++ echo '$ export CLUSTER=$CLUSTER'
$ export CLUSTER=$CLUSTER
++ export CLUSTER=devstbl
++ CLUSTER=devstbl
++ echo '$ if [[ ${CI_ENVIRONMENT_SLUG} = production ]]; then export NAMESPACE=isys-${ISYS_NAME}; else export NAMESPACE=isys-${ISYS_NAME}-${CI_ENVIRONMENT_SLUG}; fi'
$ if [[ ${CI_ENVIRONMENT_SLUG} = production ]]; then export NAMESPACE=isys-${ISYS_NAME}; else export NAMESPACE=isys-${ISYS_NAME}-${CI_ENVIRONMENT_SLUG}; fi
++ [[ dev = production ]]
++ export NAMESPACE=isys-ndbo-dev
++ NAMESPACE=isys-ndbo-dev
++ echo '$ read -r REVISION STATUS < <(helm --kube-context ${CLUSTER:-default} -n $NAMESPACE history $CI_PROJECT_NAME | tail -1 | cut -f1,3) || true'
$ read -r REVISION STATUS < <(helm --kube-context ${CLUSTER:-default} -n $NAMESPACE history $CI_PROJECT_NAME | tail -1 | cut -f1,3) || true
++ read -r REVISION STATUS
+++ helm --kube-context devstbl -n isys-ndbo-dev history owb-ms-prd-salary-cards
+++ tail -1
+++ cut -f1,3
++ echo '$ if [[ "$STATUS" =~ "pending" ]]; then helm --kube-context ${CLUSTER:-default} -n $NAMESPACE rollback $CI_PROJECT_NAME $REVISION || true; fi'
$ if [[ "$STATUS" =~ "pending" ]]; then helm --kube-context ${CLUSTER:-default} -n $NAMESPACE rollback $CI_PROJECT_NAME $REVISION || true; fi
++ [[ failed =~ pending ]]
++ echo '$ helmfile ${HELMFILE_DEFAULT_NAMESPACE:+--namespace $HELMFILE_DEFAULT_NAMESPACE} --environment ${CLUSTER:-default} -f deploy/helmfile.yaml --log-level info apply --suppress-secrets --set "additionalWorkloadAnnotations.gitlabPipeline=${CI_PIPELINE_URL}" --set "additionalWorkloadAnnotations.gitlabCommit=${CI_COMMIT_SHA}"'
$ helmfile ${HELMFILE_DEFAULT_NAMESPACE:+--namespace $HELMFILE_DEFAULT_NAMESPACE} --environment ${CLUSTER:-default} -f deploy/helmfile.yaml --log-level info apply --suppress-secrets --set "additionalWorkloadAnnotations.gitlabPipeline=${CI_PIPELINE_URL}" --set "additionalWorkloadAnnotations.gitlabCommit=${CI_COMMIT_SHA}"
++ helmfile --environment devstbl -f deploy/helmfile.yaml --log-level info apply --suppress-secrets --set additionalWorkloadAnnotations.gitlabPipeline=https://gitlab.rshbdev.ru/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards/-/pipelines/3474002 --set additionalWorkloadAnnotations.gitlabCommit=37271b4140639d570dda53d8a755ccfd3a0ca19d
== /usr/local/link/helmfile: Initialize base deploy hierarchy in /builds/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
no matches for path: envs/dev/devstbl/helmfile.yaml.gotmpl
Adding repo rshb-charts https://nexus.rshbdev.ru/repository/charts/
"rshb-charts" has been added to your repositories
Comparing release=vault-secrets-owb-ms-prd-salary-cards, chart=rshb-charts/raw
Comparing release=psvc-owb-ms-prd-salary-cards, chart=rshb-charts/raw
Comparing release=platform-database-owb-ms-prd-salary-cards, chart=rshb-charts/raw
Comparing release=links-owb-ms-prd-salary-cards, chart=rshb-charts/raw
Comparing release=owb-ms-prd-salary-cards, chart=rshb-charts/base
isys-ndbo-dev, owb-ms-prd-salary-cards, Deployment (apps) has changed:
# Source: base/templates/workload.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: owb-ms-prd-salary-cards
labels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
- chart: base-1.15.2
+ chart: base-1.15.3
release: owb-ms-prd-salary-cards
heritage: Helm
isys: "ndbo-dev"
psvc: "owb-ms-prd-salary-cards"
version: "master"
workload.topology.app.farm/zone: "rumsk1"
annotations:
- gitlabCommit: 3416f0d18c4a379c60a22b68240346f265506b29
- gitlabPipeline: https://gitlab.rshbdev.ru/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards/-/pipelines/3387514
- platform.ckpr.integrations.rshbintech.ru/gitlab-commit-sha: 3416f0d1
- platform.ckpr.integrations.rshbintech.ru/gitlab-pipeline-id: "3387514"
+ gitlabCommit: 37271b4140639d570dda53d8a755ccfd3a0ca19d
+ gitlabPipeline: https://gitlab.rshbdev.ru/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards/-/pipelines/3474002
+ platform.ckpr.integrations.rshbintech.ru/gitlab-commit-sha: 37271b41
+ platform.ckpr.integrations.rshbintech.ru/gitlab-pipeline-id: "3474002"
spec:
strategy:
type: Recreate
replicas: 1
selector:
matchLabels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
release: owb-ms-prd-salary-cards
template:
metadata:
labels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
- chart: base-1.15.2
+ chart: base-1.15.3
release: owb-ms-prd-salary-cards
heritage: Helm
isys: "ndbo-dev"
psvc: "owb-ms-prd-salary-cards"
version: "master"
workload.topology.app.farm/zone: "rumsk1"
annotations:
- checksum/configMapsEnv: "30f710cc8805a4218a8fdf7d3eaf2a4a3e074ab4c31aad55a40ece2792da5305"
- checksum/secretEnv: "93ef754a3c39004c0c8d135d55968a44173d8265708059f6ae171393112b1ea0"
- ci/commithash: "3416f0d1"
+ checksum/configMapsEnv: "59971508adde073bcd5ad9399cc4fcf4920c67259d81a65a5c4083ec463850f5"
+ checksum/secretEnv: "0e83e7ff47b42cf1aecf5a8d7115c0eba2eb1fd55d42b4379ea2004a260e6ace"
+ ci/commithash: "37271b41"
inject.istio.io/templates: "sidecar,custom"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
sidecar.istio.io/userVolume: "[{\"name\": \"wasmfilters-dir\",\"configMap\": {\"name\": \"ndbo-dev-envoy-filters\"}}]"
sidecar.istio.io/userVolumeMount: "[{\"mountPath\":\"/var/local/lib/wasm-filters\",\"name\":\"wasmfilters-dir\"}]"
spec:
automountServiceAccountToken: false
serviceAccountName: default
nodeSelector:
workload.topology.app.farm/zone: rumsk1
priorityClassName: rumsk1
containers:
- name: app
image: registry.rshbdev.ru/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards:master
imagePullPolicy: Always
envFrom:
- configMapRef:
name: owb-ms-prd-salary-cards-app-cm-env
- secretRef:
name: owb-ms-prd-salary-cards-app-secret-env
resources:
limits:
cpu: 800m
memory: 2Gi
requests:
cpu: 500m
memory: 512Mi
ports:
livenessProbe:
failureThreshold: 5
httpGet:
path: /health/liveness
port: 8080
initialDelaySeconds: 80
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health/readiness
port: 8080
initialDelaySeconds: 80
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
- name: app-vault-secrets
mountPath: /secrets
readOnly: true
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
procMount: Default
readOnlyRootFilesystem: false
runAsGroup: 1001
runAsNonRoot: true
runAsUser: 1001
securityContext:
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 1001
runAsNonRoot: true
runAsUser: 1001
tolerations:
- effect: NoSchedule
key: workload.topology.app.farm/zone
operator: Exists
hostNetwork: false
volumes:
- name: app-vault-secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: owb-ms-prd-salary-cards-app-secret-provider
isys-ndbo-dev, owb-ms-prd-salary-cards, Service (v1) has changed:
# Source: base/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: owb-ms-prd-salary-cards
labels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
- chart: base-1.15.2
+ chart: base-1.15.3
release: owb-ms-prd-salary-cards
heritage: Helm
isys: "ndbo-dev"
psvc: "owb-ms-prd-salary-cards"
version: "master"
spec:
ports:
- name: web
port: 80
protocol: TCP
targetPort: 8080
selector:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
release: owb-ms-prd-salary-cards
type: ClusterIP
isys-ndbo-dev, owb-ms-prd-salary-cards-app-cm-env, ConfigMap (v1) has changed:
# Source: base/templates/cm-env.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: owb-ms-prd-salary-cards-app-cm-env
labels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
- chart: base-1.15.2
+ chart: base-1.15.3
release: owb-ms-prd-salary-cards
heritage: Helm
isys: "ndbo-dev"
psvc: "owb-ms-prd-salary-cards"
version: "master"
workload.topology.app.farm/zone: "rumsk1"
data:
APP_POSTGRESQL_HOST: "10.7.27.129"
APP_POSTGRESQL_NAME: "ndbo-dev"
APP_POSTGRESQL_PORT: "5432"
FILE_EXPORT_CODE_PAGE: "cp1251"
JAVA_OPTS: "-XX:MaxRAMPercentage=75.0 -XX:MinRAMPercentage=75.0 -XX:+AlwaysActAsServerClassMachine"
KAFKA_BOOTSTRAP_SERVERS: "10.7.27.128:9192"
KAFKA_SECURITY_PROTOCOL: "SSL"
KAFKA_SSL_ENABLED: "true"
MS_ACCESS_CONTROL_HOST: "http://ms-rb-access-control"
MS_BRANCH_REGISTRY_HOST: "http://ms-branch-registry"
MS_CLIENT_REGISTRY_HOST: "http://ms-org-registry"
MS_CONFIRMATION_HOST: "http://ms-confirmation"
MS_EMPLOYEE_ROLE_MODEL_HOST: "http://owb-plt-ms-employee-role-model"
MS_GENERAL_DICTIONARY_HOST: "http://ms-general-dictionary"
MS_ORG_DELEGATE_AUTHORITY_HOST: "http://ms-org-delegate-authority"
MS_SALARY_CONTRACTS_HOST: "http://owb-ms-prd-salary-contracts"
PUBLIC_KEY_URI: "https://auth.rshbdev.ru/auth/realms/platform/protocol/openid-connect/certs"
REPLICATION_INIT: "false"
SMB_MAX_RETRIES: "6"
SMB_PATH: "smb://10.7.200.11:445/dboul$/impzp/"
SMB_RESEND_BATCH_SIZE: "10"
SMB_RETRY_INTERVAL_MIN: "10"
SUGST_HOST: "http://10.7.27.34:8080"
isys-ndbo-dev, owb-ms-prd-salary-cards-app-secret-env, Secret (v1) has changed:
+ Changes suppressed on sensitive content of type Secret
isys-ndbo-dev, owb-ms-prd-salary-cards-app-secret-provider, SecretProviderClass (secrets-store.csi.x-k8s.io) has changed:
# Source: base/templates/secret-provider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: owb-ms-prd-salary-cards-app-secret-provider
labels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
- chart: base-1.15.2
+ chart: base-1.15.3
release: owb-ms-prd-salary-cards
heritage: Helm
isys: "ndbo-dev"
psvc: "owb-ms-prd-salary-cards"
version: "master"
workload.topology.app.farm/zone: "rumsk1"
spec:
provider: vault
parameters:
- vaultKubernetesMountPath: devstbl
+ vaultKubernetesMountPath: devstbl-secrets-store
roleName: "csi-secrets-reader-dev"
objects: |
- objectName: APP_POSTGRESQL_MIGRATION_PASSWORD
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: APP_POSTGRESQL_MIGRATION_PASSWORD
- objectName: APP_POSTGRESQL_MIGRATION_USER
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: APP_POSTGRESQL_MIGRATION_USER
- objectName: APP_POSTGRESQL_PASSWORD
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: APP_POSTGRESQL_PASSWORD
- objectName: APP_POSTGRESQL_USER
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: APP_POSTGRESQL_USER
- objectName: JWT_SECRET_KEY
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: JWT_SECRET_KEY
- objectName: KAFKA_SSL_CERTIFICATE
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: KAFKA_SSL_CERTIFICATE
- objectName: SMB_USER_DOMAIN
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: SMB_USER_DOMAIN
- objectName: SMB_USER_NAME
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: SMB_USER_NAME
- objectName: SMB_USER_PASSWORD
secretPath: manual_kv_dev/data/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards
secretKey: SMB_USER_PASSWORD
secretObjects:
- secretName: owb-ms-prd-salary-cards-app-secret-env
type: Opaque
labels:
app: owb-ms-prd-salary-cards
fullname: owb-ms-prd-salary-cards
- chart: base-1.15.2
+ chart: base-1.15.3
release: owb-ms-prd-salary-cards
heritage: Helm
isys: "ndbo-dev"
psvc: "owb-ms-prd-salary-cards"
version: "master"
workload.topology.app.farm/zone: "rumsk1"
data:
- objectName: APP_POSTGRESQL_MIGRATION_PASSWORD
key: APP_POSTGRESQL_MIGRATION_PASSWORD
- objectName: APP_POSTGRESQL_MIGRATION_USER
key: APP_POSTGRESQL_MIGRATION_USER
- objectName: APP_POSTGRESQL_PASSWORD
key: APP_POSTGRESQL_PASSWORD
- objectName: APP_POSTGRESQL_USER
key: APP_POSTGRESQL_USER
- objectName: JWT_SECRET_KEY
key: JWT_SECRET_KEY
- objectName: KAFKA_SSL_CERTIFICATE
key: KAFKA_SSL_CERTIFICATE
- objectName: SMB_USER_DOMAIN
key: SMB_USER_DOMAIN
- objectName: SMB_USER_NAME
key: SMB_USER_NAME
- objectName: SMB_USER_PASSWORD
key: SMB_USER_PASSWORD
Listing releases matching ^pjob-owb-ms-prd-salary-cards$
Listing releases matching ^exsvc-owb-ms-prd-salary-cards$
Listing releases matching ^data-owb-ms-prd-salary-cards$
Listing releases matching ^assets-config-owb-ms-prd-salary-cards$
Listing releases matching ^owb-ms-prd-salary-cards-raw$
Listing releases matching ^owb-ms-prd-salary-cards-grafana-dashboard$
Listing releases matching ^owb-ms-prd-salary-cards-prometheus-rules$
Listing releases matching ^owb-ms-prd-salary-cards-rumsk1$
Listing releases matching ^owb-ms-prd-salary-cards-rumsk2$
Upgrading release=owb-ms-prd-salary-cards, chart=rshb-charts/base
FAILED RELEASES:
NAME
owb-ms-prd-salary-cards
in deploy/helmfile.yaml: failed processing release owb-ms-prd-salary-cards: command "/usr/local/link/helm" exited with non-zero status:
PATH:
/usr/local/link/helm
ARGS:
0: helm (4 bytes)
1: --kube-context (14 bytes)
2: devstbl (7 bytes)
3: upgrade (7 bytes)
4: --install (9 bytes)
5: --reset-values (14 bytes)
6: owb-ms-prd-salary-cards (23 bytes)
7: rshb-charts/base (16 bytes)
8: --version (9 bytes)
9: 1.15.3 (6 bytes)
10: --wait (6 bytes)
11: --timeout (9 bytes)
12: 600s (4 bytes)
13: --atomic (8 bytes)
14: --kube-context (14 bytes)
15: devstbl (7 bytes)
16: --namespace (11 bytes)
17: isys-ndbo-dev (13 bytes)
18: --values (8 bytes)
19: /tmp/helmfile349844442/isys-ndbo-dev-owb-ms-prd-salary-cards-values-5bc6696cc (77 bytes)
20: --values (8 bytes)
21: /tmp/helmfile730770810/isys-ndbo-dev-owb-ms-prd-salary-cards-values-54946cfd79 (78 bytes)
22: --set (5 bytes)
23: additionalWorkloadAnnotations.gitlabPipeline=https://gitlab.rshbdev.ru/rshbintech/crft/dboul/ndbo/backend/owb-ms-prd-salary-cards/-/pipelines/3474002 (149 bytes)
24: --set (5 bytes)
25: additionalWorkloadAnnotations.gitlabCommit=37271b4140639d570dda53d8a755ccfd3a0ca19d (83 bytes)
26: --history-max (13 bytes)
27: 10 (2 bytes)
ERROR:
exit status 1
EXIT STATUS
1
STDERR:
E0825 09:16:31.127612 1178 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0825 09:27:48.553312 1178 request.go:1058] Unexpected error when reading response body: read tcp 100.70.152.70:57794->100.68.64.1:6443: read: connection reset by peer
Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: timed out waiting for the condition: release owb-ms-prd-salary-cards failed: unexpected error when reading response body. Please retry. Original error: read tcp 100.70.152.70:57794->100.68.64.1:6443: read: connection reset by peer
COMBINED OUTPUT:
E0825 09:16:31.127612 1178 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0825 09:27:48.553312 1178 request.go:1058] Unexpected error when reading response body: read tcp 100.70.152.70:57794->100.68.64.1:6443: read: connection reset by peer
Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: timed out waiting for the condition: release owb-ms-prd-salary-cards failed: unexpected error when reading response body. Please retry. Original error: read tcp 100.70.152.70:57794->100.68.64.1:6443: read: connection reset by peer
Uploading artifacts for failed job
00:02
Uploading artifacts...
deploy.env: found 1 matching artifact files and directories
Uploading artifacts as "dotenv" to coordinator... 201 Created id=24186450 responseStatus=201 Created token=64_ZGKz1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1