There is another step that I did (which might or might not be necessary):

After creating the PV using this gist, I exec'ed into the rook-toolbox Pod, and mounted the CephFS filesystem root directly. And then I made a folder staticpvs/pv-1 inside the CephFS root to match rootPath in the PV manifest.

It still didn't work for me even with a right creds

Did you base64 encode the admin key when you modify the secret?

BTW, I just realised, since I am using Rook, that I should (conveniently) be using rook-csi-cephfs-provisioner and rook-csi-cephfs-node (both are created by Rook in the rook-ceph namespace) as provisionerSecretRef and nodeStageSecretRef respectively.

BTW, I just realised, since I am using Rook, that I should (conveniently) be using rook-csi-cephfs-provisioner and rook-csi-cephfs-node (both are created by Rook in the rook-ceph namespace) as provisionerSecretRef and nodeStageSecretRef respectively.

The rook created secrets will not have the keys userID and userKey. So reusing the rook created secret as is for static PVs is not feasible.

The secrets being discussed here need to be of the form: https://gist.github.com/ShyamsundarR/1009b79966992470f588a277429ae2e1#file-pod-with-static-pv-yaml-L1-L10

IOW, a user key and value needs to be specified, in exactly the same format as detailed here: https://github.com/ceph/ceph-csi/blob/3e656769b71a3c43d95f6875ed4934c82a8046e7/examples/cephfs/secret.yaml#L7-L10

If using stringData then base-64 encoding the userKey is not required. (if using data instead, then base-64 encoding the same is required)

I just realized that I was using wrong secret (something crewed up when I switched between different deployments). I just put correct one and everything seems working fine for me.

The intention of a static PV is that, the path mentioned as rootPath (here) already exists, and the secret created to access this path is a ceph userID/key with enough credentials to mount and perform required IO on the mount.

So, if using a static PV, the rootPath should be already created against the backing CephFS volume.

Based on the discussions here so far by @elinesterov and @ztl8702 the above needs to be clarified as having been done, and if so please pass on the csi-cephfsplugin logs from the ceph-csi nodeplugin daemonset logs, on the node where the PV is attempted to mounted.

@ShyamsundarR Thanks for the clarification. Manually creating a secret that matches the form using the client.admin key from ceph auth ls worked for me. Do you still want my logs?

I am just wondering if there is a way to more declaratively deploy a CephFS shared filesystem + mount a volume to a Pod that gives access to at a fixed path within cephfs, with ceph-csi. The extra step to get / create a client key in Ceph and then create the correct secret in Kubernetes API bugs me a little bit.
I believe this was previously possible with flexVolume or with provisionVolume: false?

@ShyamsundarR Thanks for the clarification. Manually creating a secret that matches the form using the client.admin key from ceph auth ls worked for me. Do you still want my logs?

No, thank you.

I am just wondering if there is a way to more declaratively deploy a CephFS shared filesystem + mount a volume to a Pod that gives access to at a fixed path within cephfs, with ceph-csi. The extra step to get / create a client key in Ceph and then create the correct secret in Kubernetes API bugs me a little bit.

The step to create a client key and a secret is a choice for security conscious setups. You could use a single secret across all static PVs that are created. This secret can be in any namespace, but the namespace needs to be reflected in here.

The above reduces it to a single secret creation step.

I believe this was previously possible with flexVolume or with provisionVolume: false?

I am not aware of how flexVolume worked, so not commenting on the same.

With provisionVolume: false the method of using static PVs was to request for one dynamically, and the provisioner detecting it as a pre-provisioned PV and hence using the same secret as that used by the CSI plugins. The entire provisioning step (and hence de-provisioning) was superfluous, and the intention was to go with kubernetes based static PV definitions instead.