tripleo-ui artifacts

There are currently three problems in openstack/tripleo-ui ci:

1. Python wheels in tarballs.openstack.org/tripleo-ui/
2. tripleo-ui-latest.tar.gz isn't updated in tarballs.openstack.org/tripleo-ui/
3. tripleo-ui tarballs use unhelpful commits-since-tag naming scheme

The entry point for openstack ci configuration is the openstack-infra/project-config repository.

Here is the configuration for tripleo-ui:

- project:
    name: openstack/tripleo-ui
    templates:
      - system-required
      - nodejs8-jobs
      - nodejs8-publish-to-npm
      - release-notes-jobs
      - translation-jobs
      - translation-jobs-ocata
      - translation-jobs-pike
    post:
      jobs:
        - publish-openstack-python-tarball

The reader will notice that openstack/tripleo-ui is an npm project and doesn't ship any Python tarballs or wheels. This patch removes the Python tarball job. This solves problem number 1.

Let's focus on the following two jobs:

• nodejs8-jobs
• nodejs8-publish-to-npm

nodejs8-jobs

Here is the definition for nodejs8-jobs in openstack-zuul-jobs:

- project-template:
    name: nodejs8-jobs
    description: |
      Run lint and test jobs using Node 8.
    check:
      jobs:
        - nodejs-npm-run-lint:
            vars:
              node_version: 8
        - nodejs-npm-run-test:
            vars:
              node_version: 8
        - build-javascript-tarball:
            files:
              - npm-shrinkwrap.json
              - package.json
    gate:
      jobs:
        - nodejs-npm-run-lint:
            vars:
              node_version: 8
        - nodejs-npm-run-test:
            vars:
              node_version: 8
        - build-javascript-tarball:
            files:
              - npm-shrinkwrap.json
              - package.json

This job has two sections: check and gate: Check happens on every patchset, and once the patch is approved, the gate happens. You can also have post which runs after a patch is merged.

So, we run three things: lint, test, and build tarball.

Building a tarball only happens when the patch touches the files npm-shrinkwrap.json or package.json. This jobs is redundant, and there is a patch to remove it. The job only tests that npm pack will succeed, and doesn't produce any artifacts.

You can see that nodejs8-jobs doesn't handle any tarball updating/uploading. Let's move on.

nodejs8-publish-to-npm

Here is the definition for nodejs8-publish-to-npm in openstack-zuul-jobs:

- project-template:
    name: nodejs8-publish-to-npm
    release:
      jobs:
        - publish-openstack-javascript-tarball:
            vars:
              node_version: 8
        - release-openstack-javascript:
            vars:
              node_version: 8
        - announce-release:
            dependencies:
              - release-openstack-javascript
              - publish-openstack-javascript-tarball

You will notice that all of the jobs are under the release header. This means that tarballs are published to tarballs.openstack.org only when a release is made. Cool.

But, wait. How are the per-commit tarballs getting there?

Let's go deeper

Let's focus on the tripleo-ui-latest.tar.gz file issue for a moment. Where is that done? Some grepping reveals that it happens here, in openstack-infra/zuul-jobs (note the difference between openstack-zuul-jobs and zuul-jobs):

- name: Rename tarball for uploading
  shell: |
    mkdir -p dist
    cp *.tgz dist/{{ zuul.project.short_name }}-{{ project_ver }}.tar.gz
    cp *.tgz dist/{{ zuul.project.short_name }}-latest.tar.gz

This is in the fetch-javascript-tarball role.

Who uses this role?

A bit more grepping two locations:

1. project-config/playbooks/javascript/post.yaml

... well? Who uses that?

It's release-openstack-javascript in project-config.

This job first runs pre.yaml: installs nodejs, revokes sudo, installs npm packages, etc. Then it runs tarball.yaml which runs npm pack. And then the post.yaml which is the subject of the current heading.

post.yaml fetches javascript output (whatever that means), gets the package version from git, and then fetches the javascript tarball (whatever that means again). Then, finally, it runs the upload-npm role.

So, what do the fetch-javascript-* jobs do? What does upload-npm do?

The more important thing is, do we even care?

That's right, we don't care. The release-openstack-javascript job is only run when a release is made as we saw earlier.

Let's move on.

2. openstack-zuul-jobs/playbooks/javascript/post-tarball.yaml

Who uses this beast?

It's publish-openstack-javascript-tarball in openstack-zuul-jobs.

In pre.yaml, it does npm setup.

In tarball.yaml, it runs npm pack.

In post.yaml, it runs fetch-javascript-output (whatever that means).

And in post-tarball.yaml (subject of current heading) it does version-from-git, and fetch-javascript-tarball.

Who runs publish-openstack-javascript-tarball?

It's the release job in nodejs8-publish-to-npm. So, it's only run when a release is made, again.

Huh? So, let's recap: the fetch-javascript-tarball role which produces the tripleo-ui-image.tar.gz is only run by playbooks/roles that are triggered on releases. Well, dear reader, this is the cause of the problem of updating the latest file. No one runs it in post.

Given that our project inexplicably ships Python wheels, I this this was done by an unsuspecting developer who saw a setup.py file in our project's root.

project-config sha: ad40365e3b9258d191238d52e6ce4c4592cd70f8

openstack-zuul-jobs sha: f63942d86ca86acfdfc670baea04c2fb7ccdf9f0

zuul jobs sha 62836959666e8032738110be854303e52bc954d0