Skip to content

Instantly share code, notes, and snippets.

@mayorova

mayorova/APIcast.adoc

Last active November 19, 2021 12:37
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mayorova/928a67bf31ca29a654115d7be290acca to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mayorova/928a67bf31ca29a654115d7be290acca to your computer and use it in GitHub Desktop.
Download ZIP
3scale migration from templates to operator (in HA mode)
Raw
README.adoc

Migration

Preparation

Export variables that will be used in the instructions.

export OLD_DB_HOST=<old DB host>

export DB_USERNAME=<DB username>
export DB_DATABASE=<DB database>
export DB_HOST=<current DB host>

export WILDCARD_DOMAIN=<new cluster wildcard domain>

Back up

  1. Backup PostgreSQL

    pg_dump -U ${DB_USERNAME} -W -h ${OLD_DB_HOST} -F c -v ${DB_DATABASE} -f system-database.dump

  2. Backing up system-storage

    oc rsync $(oc get pods -l 'deploymentConfig=system-app' --template="{{ (index .items 0).metadata.name }}"):/opt/system/public/system ./local/dir

  3. Backing up backend-redis

    oc cp $(oc get pods -l 'deploymentConfig=backend-redis' --template="{{ (index .items 0).metadata.name }}"):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb

  4. Backing up system-redis

    oc cp $(oc get pods -l 'deploymentConfig=system-redis' --template="{{ (index .items 0).metadata.name }}"):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb

    (skipping restoring Zync database)

  5. Backing up OpenShift secrets and ConfigMaps

    oc get secrets system-smtp -o json > system-smtp.json
oc get secrets system-seed -o json > system-seed.json
oc get secrets system-database -o json > system-database.json
oc get secrets backend-internal-api -o json > backend-internal-api.json
oc get secrets system-events-hook -o json > system-events-hook.json
oc get secrets system-app -o json > system-app.json
oc get secrets system-recaptcha -o json > system-recaptcha.json
oc get secrets system-redis -o json > system-redis.json
oc get secrets zync -o json > zync.json
oc get secrets system-master-apicast -o json > system-master-apicast.json
    oc get configmaps system-environment -o json > system-environment.json
oc get configmaps apicast-environment -o json > apicast-environment.json

Restore

  1. Create a new project

    oc new-project threescale-apim

  2. Create a secret for registry.redhat.io

    oc create secret generic <pull_secret_name> \
    --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
    --type=kubernetes.io/dockerconfigjson>

oc secrets link default <pull_secret_name> --for=pull

  3. Install 3scale operator

  4. Restore secrets

    (skipping system-database, system-redis and backend-redis secrets)

    Note
    		 if the namespace name has changed, update to the new 3scale namespace in the new cluster. 
    oc apply -f system-smtp.json
oc apply -f system-seed.json
oc apply -f backend-internal-api.json
oc apply -f system-events-hook.json
oc apply -f system-app.json
oc apply -f system-recaptcha.json
oc apply -f zync.json
oc apply -f system-master-apicast.json

  5. Restore configmaps

    oc apply -f system-environment.json
oc apply -f apicast-environment.json

  6. Create database secret

    cat << EOF > system-database-secret.yml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  labels:
    app: 3scale-database
  name: system-database
stringData:
  DB_PASSWORD: ${DB_PASSWORD}
  DB_USER: ${DB_USERNAME}
  URL: postgresql://${DB_USERNAME}:${DB_PASSWORD}@${DB_HOST}/${DB_DATABASE}
EOF

  7. Create Cluster Role for Azure File

    cat << EOF > azure-cloud-provider-secrets.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:azure-cloud-provider-secrets
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
EOF

oc apply -f azure-cloud-provider-secrets.yml

  8. Assign role to service account

    oc adm policy add-cluster-role-to-user system:azure-cloud-provider-secrets system:serviceaccount:kube-system:persistent-volume-binder

  9. Get UID range on the 3scale namespace and save it in a variable

    oc describe project threescale-apim | grep uid-range
export OPENSHIFT_UID=<UID>

  10. Create RWX Storage Class for 3scale system-storage

    cat << EOF > azure-file-3scale.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-file-3scale
mountOptions:
- uid=${OPENSHIFT_UID}
- gid=0
- mfsymlinks
- cache=strict
parameters:
  skuName: Standard_LRS
provisioner: kubernetes.io/azure-file
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

oc create -f azure-file-3scale.yml

  11. Create temporary Redis instances

    oc new-app -f template-only-redis.yml

    This template has the following changes with regards to the standard one: * all resources not related to Redis are removed * backend-redis storage increased from 1G to 10G * APP_LABEL (the app label on OpenShift resources) changed from 3scale-api-management to 3scale-redis * removed serviceAccount amp from resources definition

Note
 add MESSAGE_BUS_URL value

  1. Restore Redis databases

  2. Restore PostgreSQL database (the database should exist)

    pg_restore -d ${DATABASE_NAME} -U ${DATABASE_USERNAME} system-database.dump

  3. Create APIManager

    cat << EOF > apimanager.yml
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: apimanager
spec:
  wildcardDomain: ${WILDCARD_DOMAIN}
  resourceRequirementsEnabled: true
  highAvailability:
    enabled: true
  system:
    fileStorage:
      persistentVolumeClaim:
        storageClassName: azure-file-3scale
EOF

oc apply -f apimanager.yml

  4. Update domains to adjust to the new Wildcard Domain

To do a batch rewrite of the domains, use the following steps:

bundle exec rails console

Print the current values for the domains

Provider.all.to_a.each do |p|
  puts p.domain
  puts p.self_domain
end

Replace the wildcard domain with the new value

Provider.all.to_a.each do |p|
  p.domain = p.domain.gsub("apps.CURRENT", "apps.NEW")
  p.self_domain = p.self_domain.gsub("apps.CURRENT", "apps.NEW")
  p.save!
end

Verify

Get admin username and password

oc get secret system-seed --template="{{.data.ADMIN_USER}}" | base64 -d
oc get secret system-seed --template="{{.data.ADMIN_PASSWORD}}" | base64 -d

Verify RH-SSO connections

Upgrade 3scale

Use Operator to upgrade 3scale, one version at a time: 2.8 to 2.9 2.9 to 2.10 2.10 to 2.11

Update number of replicas on the APIManager resource

In the APIManager instance, change all replicas from replicas: 1 to replicas: 2, except APIcast. For APIcast it is required to have staging gateway to pull the policies, but the production gateway can be disabled if not used.

spec:
  imageStreamTagImportInsecure: false
  resourceRequirementsEnabled: true
  system:
    appSpec:
      replicas: 2
    fileStorage:
      persistentVolumeClaim:
        storageClassName: azure-file-3scale
    sidekiqSpec:
      replicas: 2
    sphinxSpec: {}
  appLabel: 3scale-api-management
  zync:
    appSpec:
      replicas: 2
    queSpec:
      replicas: 2
  backend:
    cronSpec:
      replicas: 2
    listenerSpec:
      replicas: 2
    workerSpec:
      replicas: 2
  tenantName: 3scale
  apicast:
    managementAPI: status
    openSSLVerify: false
    productionSpec:
      replicas: 0
    registryURL: 'http://apicast-staging:8090/policies'
    responseCodes: true
    stagingSpec:
      replicas: 1
  highAvailability:
    enabled: true
  wildcardDomain: <DOMAIN>

Restore system-storage

oc rsync ./local/dir/system/ $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system
Raw
APIcast.adoc

Create a secret for connecting with admin portal

oc create secret generic apicast-admin-portal-secret --from-literal=AdminPortalURL=https://<ACESS_TOKEN>@<PORTAL_URL>

Create secret for TLS certificates

oc create secret tls router-certs --cert=server.crt --key=server.key

Production APIcast

apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: 3scale-apicast-production
spec:
  adminPortalCredentialsRef:
    name: apicast-configuration-url-secret
  replicas: 4
  deploymentEnvironment: production
  configurationLoadMode: boot
  logLevel: warn
  pathRoutingEnabled: true
  responseCodesIncluded: true
  cacheConfigurationSeconds: 120
  managementAPIScope: status
  openSSLPeerVerificationEnabled: false
  httpsPort: 8443
  httpsCertificateSecretRef:
    name: router-certs
  extendedMetrics: true
  resources:
    limits:
      cpu: '2'
      memory: 1Gi
    requests:
      cpu: '1'
      memory: 500Mi

Staging APIcast

apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: 3scale-apicast-staging
spec:
  adminPortalCredentialsRef:
    name: apicast-configuration-url-secret
  replicas: 2
  deploymentEnvironment: staging
  configurationLoadMode: lazy
  logLevel: info
  pathRoutingEnabled: true
  responseCodesIncluded: true
  cacheConfigurationSeconds: 0
  managementAPIScope: status
  openSSLPeerVerificationEnabled: false
  httpsPort: 8443
  httpsCertificateSecretRef:
    name: router-certs
  extendedMetrics: true
  resources:
    limits:
      cpu: '2'
      memory: 500Mi
    requests:
      cpu: '1'
      memory: 128Mi
Raw
template-only-redis.yml
apiVersion: template.openshift.io/v1
kind: Template
metadata:
name: 3scale-api-management-redis
objects:
- apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: backend
threescale_component_element: redis
name: backend-redis
spec:
replicas: 1
selector:
deploymentConfig: backend-redis
strategy:
resources: {}
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
deploymentConfig: backend-redis
threescale_component: backend
threescale_component_element: redis
spec:
containers:
- args:
- /etc/redis.d/redis.conf
- --daemonize
- "no"
command:
- /opt/rh/rh-redis5/root/usr/bin/redis-server
image: backend-redis:latest
imagePullPolicy: IfNotPresent
livenessProbe:
initialDelaySeconds: 10
periodSeconds: 10
tcpSocket:
port: 6379
name: backend-redis
readinessProbe:
exec:
command:
- container-entrypoint
- bash
- -c
- redis-cli set liveness-probe "`date`" | grep OK
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 32Gi
requests:
cpu: "1"
memory: 1Gi
volumeMounts:
- mountPath: /var/lib/redis/data
name: backend-redis-storage
- mountPath: /etc/redis.d/
name: redis-config
volumes:
- name: backend-redis-storage
persistentVolumeClaim:
claimName: backend-redis-storage
- configMap:
items:
- key: redis.conf
path: redis.conf
name: redis-config
name: redis-config
test: false
triggers:
- type: ConfigChange
- imageChangeParams:
automatic: true
containerNames:
- backend-redis
from:
kind: ImageStreamTag
name: backend-redis:${AMP_RELEASE}
type: ImageChange
status:
availableReplicas: 0
latestVersion: 0
observedGeneration: 0
replicas: 0
unavailableReplicas: 0
updatedReplicas: 0
- apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: backend
threescale_component_element: redis
name: backend-redis
spec:
ports:
- port: 6379
protocol: TCP
targetPort: 6379
selector:
deploymentConfig: backend-redis
status:
loadBalancer: {}
- apiVersion: v1
data:
redis.conf: |
protected-mode no
port 6379
timeout 0
tcp-keepalive 300
daemonize no
supervised no
loglevel notice
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-disable-tcp-nodelay no
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
activerehashing no
aof-rewrite-incremental-fsync yes
dir /var/lib/redis/data
kind: ConfigMap
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: system
threescale_component_element: redis
name: redis-config
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: backend
threescale_component_element: redis
name: backend-redis-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
status: {}
- apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/display-name: Backend Redis
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: backend
name: backend-redis
spec:
lookupPolicy:
local: false
tags:
- annotations:
openshift.io/display-name: Backend ${AMP_RELEASE} Redis
from:
kind: DockerImage
name: ${REDIS_IMAGE}
generation: null
importPolicy:
insecure: ${{IMAGESTREAM_TAG_IMPORT_INSECURE}}
name: ${AMP_RELEASE}
referencePolicy:
type: ""
status:
dockerImageRepository: ""
- apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: backend
name: backend-redis
stringData:
REDIS_QUEUES_SENTINEL_HOSTS: ""
REDIS_QUEUES_SENTINEL_ROLE: ""
REDIS_QUEUES_URL: redis://backend-redis:6379/1
REDIS_STORAGE_SENTINEL_HOSTS: ""
REDIS_STORAGE_SENTINEL_ROLE: ""
REDIS_STORAGE_URL: redis://backend-redis:6379/0
type: Opaque
- apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: system
name: system-redis
stringData:
MESSAGE_BUS_NAMESPACE: ${SYSTEM_MESSAGE_BUS_REDIS_NAMESPACE}
MESSAGE_BUS_SENTINEL_HOSTS: ""
MESSAGE_BUS_SENTINEL_ROLE: ""
MESSAGE_BUS_URL: ${SYSTEM_MESSAGE_BUS_REDIS_URL}
NAMESPACE: ${SYSTEM_REDIS_NAMESPACE}
SENTINEL_HOSTS: ""
SENTINEL_ROLE: ""
URL: ${SYSTEM_REDIS_URL}
type: Opaque
- apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: system
threescale_component_element: redis
name: system-redis
spec:
replicas: 1
selector:
deploymentConfig: system-redis
strategy:
resources: {}
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
deploymentConfig: system-redis
threescale_component: system
threescale_component_element: redis
spec:
containers:
- args:
- /etc/redis.d/redis.conf
- --daemonize
- "no"
command:
- /opt/rh/rh-redis5/root/usr/bin/redis-server
image: system-redis:latest
imagePullPolicy: IfNotPresent
livenessProbe:
initialDelaySeconds: 10
periodSeconds: 5
tcpSocket:
port: 6379
name: system-redis
readinessProbe:
exec:
command:
- container-entrypoint
- bash
- -c
- redis-cli set liveness-probe "`date`" | grep OK
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 32Gi
requests:
cpu: 150m
memory: 256Mi
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /var/lib/redis/data
name: system-redis-storage
- mountPath: /etc/redis.d/
name: redis-config
volumes:
- name: system-redis-storage
persistentVolumeClaim:
claimName: system-redis-storage
- configMap:
items:
- key: redis.conf
path: redis.conf
name: redis-config
name: redis-config
test: false
triggers:
- type: ConfigChange
- imageChangeParams:
automatic: true
containerNames:
- system-redis
from:
kind: ImageStreamTag
name: system-redis:${AMP_RELEASE}
type: ImageChange
status:
availableReplicas: 0
latestVersion: 0
observedGeneration: 0
replicas: 0
unavailableReplicas: 0
updatedReplicas: 0
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: system
threescale_component_element: redis
name: system-redis-storage
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
status: {}
- apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: system
threescale_component_element: redis
name: system-redis
spec:
ports:
- name: redis
port: 6379
protocol: TCP
targetPort: 6379
selector:
deploymentConfig: system-redis
status:
loadBalancer: {}
- apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
annotations:
openshift.io/display-name: System Redis
creationTimestamp: null
labels:
app: ${APP_LABEL}
threescale_component: system
name: system-redis
spec:
lookupPolicy:
local: false
tags:
- annotations:
openshift.io/display-name: System ${AMP_RELEASE} Redis
from:
kind: DockerImage
name: ${REDIS_IMAGE}
generation: null
importPolicy:
insecure: ${{IMAGESTREAM_TAG_IMPORT_INSECURE}}
name: ${AMP_RELEASE}
referencePolicy:
type: ""
status:
dockerImageRepository: ""
parameters:
- description: AMP release tag.
name: AMP_RELEASE
required: true
value: "2.11"
- description: Used for object app labels
name: APP_LABEL
required: true
value: 3scale-redis
- description: Set to true if the server may bypass certificate verification or connect directly over HTTP during image import.
name: IMAGESTREAM_TAG_IMPORT_INSECURE
required: true
value: "false"
- description: Redis image to use
name: REDIS_IMAGE
required: true
value: registry.redhat.io/rhscl/redis-5-rhel7:5
- description: Define the external system-redis to connect to
name: SYSTEM_REDIS_URL
required: true
value: redis://system-redis:6379/1
- description: Define the external system-redis message bus to connect to. By default the same value as SYSTEM_REDIS_URL but with the logical database incremented by 1 and the result applied mod 16
name: SYSTEM_MESSAGE_BUS_REDIS_URL
- description: Define the namespace to be used by System's Redis Database. The empty value means not namespaced
name: SYSTEM_REDIS_NAMESPACE
- description: Define the namespace to be used by System's Message Bus Redis Database. The empty value means not namespaced
name: SYSTEM_MESSAGE_BUS_REDIS_NAMESPACE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment