Azure File may not supported as the system-storage volume for 3scale

Hi,

As reported by rhn-support-apitt, Azure File does not seem to be working correctly as a storage option for 3scale although it is a RWX volume.

The error he reported is getting a 404 page when going to the developer portal:

Hi all.  Trying to debug an issue with 3Scale 2.6 on OCP 3.11 on Azure.  Trying to use Azure File for RWX volume.  I've followed the instructions to create an Azure File StorageClass from the OCP docs.  The RWX PV gets created just fine and 3Scale (mostly) works.  However, when I try to access the developer portal I get the 3Scale 404 page, and I see the following error in system-sidekiq:
Exception -- {:exception=>{:class=>Errno::EPERM, :message=>"Operation not permitted @ chmod_internal - /opt/system/public//system/provider-name/2019/11/14/desk-b2e71e0a8e6bfffa.jpg", :backtrace=>["/opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1241:in `chmod'", "/opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1241:in `chmod'", "/opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:919:in `block in chmod'", "/opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:918:in `each'"]}, :parameters=>{:context=>"Job raised exception", :job=>{"class"=>"SignupWorker::ImportSimpleLayoutWorker", "args"=>[2], "retry"=>true, "queue"=>"critical", "jid"=>"baccb5c46e09d0ab95dbbcc0", "created_at"=>1573697840.6870508, "bid"=>"wmMaZdX9-4wTtw", "enqueued_at"=>1573698185.161446, "error_message"=>"Operation not permitted @ chmod_internal - /opt/system/public//system/provider-name/2019/11/14/desk-b2e71e0a8e6bfffa.jpg", "error_class"=>"Errno::EPERM", "failed_at"=>1573697845.4033988, "retry_count"=>4, "retried_at"=>1573698186.1303504}, :jobstr=>"{\"class\":\"SignupWorker::ImportSimpleLayoutWorker\",\"args\":[2],\"retry\":true,\"queue\":\"critical\",\"jid\":\"baccb5c46e09d0ab95dbbcc0\",\"created_at\":1573697840.6870508,\"bid\":\"wmMaZdX9-4wTtw\",\"enqueued_at\":1573698185.161446,\"error_message\":\"Operation not permitted @ chmod_internal - /opt/system/public//system/provider-name/2019/11/14/desk-ed69f5f389cf38c9.jpg\",\"error_class\":\"Errno::EPERM\",\"failed_at\":1573697845.4033988,\"retry_count\":3,\"retried_at\":1573697966.9995368}"}}

We have verified that the Azure File storage is correctly created as a PersistentVolume in OpenShift, and that is correctly bounded to the 3scale's 'system-storage' PersistentVolumeClaim and it is correctly mounted in the pod, discarding any possible problem related to the deployment:

msoriano@localhost:~/go/src/github.com/3scale/3scale-operator/pkg/3scale/amp (move-system-smtp-to-secret)$ oc get pv | grep -i system-storage
pvc-488dc045-07af-11ea-9306-000d3a8a4eed   1Gi        RWX            Delete           Bound     3scale/system-storage                                           azurefile                 17m

msoriano@localhost:~/go/src/github.com/3scale/3scale-operator/pkg/3scale/amp (move-system-smtp-to-secret)$ oc get pv pvc-488dc045-07af-11ea-9306-000d3a8a4eed -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: azure-file-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/azure-file
  creationTimestamp: 2019-11-15T13:53:20Z
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-488dc045-07af-11ea-9306-000d3a8a4eed
  resourceVersion: "24864"
  selfLink: /api/v1/persistentvolumes/pvc-488dc045-07af-11ea-9306-000d3a8a4eed
  uid: 48bd2db8-07af-11ea-9306-000d3a8a4eed
spec:
  accessModes:
  - ReadWriteMany
  azureFile:
    secretName: azure-storage-account-ocpminrg-secret
    secretNamespace: 3scale
    shareName: kubernetes-dynamic-pvc-488dc045-07af-11ea-9306-000d3a8a4eed
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: system-storage
    namespace: 3scale
    resourceVersion: "24852"
    uid: 488dc045-07af-11ea-9306-000d3a8a4eed
  mountOptions:
  - dir_mode=0777
  - file_mode=0777
  persistentVolumeReclaimPolicy: Delete
  storageClassName: azurefile
status:
  phase: Bound

Two errors have been observed on the sidekiq pod:

• One is the first one reported in the beginning of this comment and it seems related to a problematic configuration of the PersistentVolume (dir_mode and file_mode) that prevents file permissions from being modified, which is not a problem related to the deployment nor code and rhn-support-apitt is taking a look on trying to solve it by changing the configuration of the PV.

• The other problem observed is one where it seems "paperclip" is trying to create symlinks:

  2019-11-15T13:58:48.691Z 1 TID-ouh4hk9t0 ProcessNotificationEventWorker::UserNotificationWorker JID-74bad32bfb71979dc2b75181 BID-gXq7X6g-DuDt3A INFO: done: 74bad32bfb71979dc2b75181 in batch gXq7X6g-DuDt3A
  WARNING: Can't mass-assign protected attributes for User: signup_type
  	lib/fields/fields.rb:129:in `assign_attributes'
  	lib/fields/extra_fields.rb:80:in `unflattened_attributes='
  	lib/signup/signup_params.rb:16:in `build_user_with_attributes_for_account'
  	lib/signup/account_manager.rb:52:in `build_user'
  	lib/signup/account_manager.rb:48:in `build_signup_result'
  	lib/signup/account_manager.rb:12:in `block in create'
  	lib/signup/account_manager.rb:21:in `transaction'
  	lib/signup/account_manager.rb:11:in `create'
  	app/lib/logic/provider_signup.rb:300:in `signup_user'
  	app/lib/logic/provider_signup.rb:291:in `block in ensure_users'
  	app/lib/logic/provider_signup.rb:290:in `times'
  	app/lib/logic/provider_signup.rb:290:in `ensure_users'
  	app/lib/logic/provider_signup.rb:276:in `create!'
  	app/lib/logic/provider_signup.rb:363:in `block in create_sample_data!'
  	app/lib/logic/provider_signup.rb:360:in `create_sample_data!'
  	app/workers/signup_worker.rb:32:in `block in perform'
  	lib/notification_center.rb:30:in `silent_about'
  	app/workers/signup_worker.rb:30:in `perform'
  	app/lib/three_scale/sidekiq_retry_support.rb:54:in `call'
  	app/lib/three_scale/analytics/sidekiq_middleware.rb:5:in `call'
  2019-11-15T13:58:48.769Z 1 TID-ouh4hk9t0 ProcessNotificationEventWorker::UserNotificationWorker JID-74bad32bfb71979dc2b75181 BID-gXq7X6g-DuDt3A INFO: done: 1.926 sec
  [paperclip] Trying to link /opt/system/lib/developer_portal/app/views/developer_portal/images/desk.jpg to /tmp/b506e098cf253ce028b8ba0a5fc7d47320191115-1-1erdqp8.jpg
  [paperclip] Link failed with Invalid cross-device link @ rb_file_s_link - (/opt/system/lib/developer_portal/app/views/developer_portal/images/desk.jpg, /tmp/b506e098cf253ce028b8ba0a5fc7d47320191115-1-1erdqp8.jpg); copying link /opt/system/lib/developer_portal/app/views/developer_portal/images/desk.jpg to /tmp/b506e098cf253ce028b8ba0a5fc7d47320191115-1-1erdqp8.jpg
  

As per the OpenShift documentation symlinks are not supported in Azure File https://docs.openshift.com/container-platform/3.11/install_config/persistent_storage/persistent_storage_azure_file.html

From what I understand from the logs, paperclip is a ruby library that System must be using, probably this one https://github.com/thoughtbot/paperclip. It says it's a deprecated library and recommend using Rails' own ActiveStorage.
Porta seems to be using paperclip 5.3.0 https://github.com/3scale/porta/blob/master/Gemfile.base

Would it make sense not having to use symlinks or the errors handled gracefully?

Please read all comments below. It is not clear that this is a problem with Azure.