Opened 6 years ago

Last modified 2 years ago

#715 in_progress enhancement

Move CloudFiles storage backend over to pyrax

Reported by: joar Owned by: ayleph
Priority: critical Milestone: 0.10.0
Component: programming Keywords: cloudfiles
Cc: Parent Tickets:

Description

python-cloudfiles is no longer being maintained, we should move to pyrax instead: https://github.com/rackspace/pyrax

Subtickets

Change History (19)

comment:1 Changed 6 years ago by joar

Some work has been done over at https://github.com/joar/mediagoblin/commit/8391dc3719a58edd71ee7fc9103d1589f023d9b9. I do not think I will manage to find time to finish it.

comment:2 Changed 6 years ago by Christopher Allan Webber

comment:3 Changed 5 years ago by Christopher Allan Webber

Milestone: 0.8.0
Owner: set to Christopher Allan Webber
Status: newin_progress

claiming and assigning to 0.8.0

comment:4 Changed 5 years ago by Christopher Allan Webber

Milestone: 0.8.00.9.0

comment:5 Changed 5 years ago by ayleph

I've been playing around with pyrax and can so far upload images to and delete images from RackSpace CloudFiles. However, it's not compatible with current MediaGoblin due to restrictions on the python six package.

  File "/path/to/mediagoblin/mediagoblin/tools/common.py", line 34, in import_component
    __import__(module_name)
  File "/path/to/mediagoblin/mediagoblin/storage/pyrax.py", line 28, in <module>
    import pyrax
  File "/path/to/mediagoblin/lib/python2.7/site-packages/pyrax/__init__.py", line 59, in <module>
    from novaclient.shell import OpenStackComputeShell as _cs_shell
  File "/path/to/mediagoblin/lib/python2.7/site-packages/novaclient/shell.py", line 47, in <module>
    from novaclient import client
  File "/path/to/mediagoblin/lib/python2.7/site-packages/novaclient/client.py", line 34, in <module>
    from oslo.utils import netutils
  File "/path/to/mediagoblin/lib/python2.7/site-packages/oslo/utils/netutils.py", line 13, in <module>
    from oslo_utils.netutils import *  # noqa
  File "/path/to/mediagoblin/lib/python2.7/site-packages/oslo_utils/netutils.py", line 170, in <module>
    class _ModifiedSplitResult(parse.SplitResult):
AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'

GMG setup.py requires six==1.5.2. A cursory internet search reveals this is sufficient for pyrax itself, but python-novaclient seems to require six>=1.6.

To test pyrax, I upgraded six (1.9.0) and everything is working okay so far in my limited testing. What's the reason for holding six at 1.5.2 in setup.py?

comment:6 Changed 5 years ago by ayleph

I used Joar's changes as a starting point and created a branch with some additional changes. His branch was so far behind master I decided to create a new repo instead of forking his. I moved the pyrax code into a separate pyrax.py file, but I kept the idea of moving the options into a separate cloudfiles section of config_spec.ini.

The pyrax.py file is a straight copy of the cloudfiles.py with some content changed to enable uploading and to allow setting CORS headers on new objects. I haven't tested all of the methods yet.

https://gitorious.org/mediagoblin/aylephs-mediagoblin/commits/9f62f305bbfd5929a3c4c923a342931d5d3a72cc

comment:7 Changed 5 years ago by ayleph

The requirement on six has been removed in #5011.

comment:8 Changed 4 years ago by ayleph

My branch referenced above, besides being completely out of date, is no longer accessible on gitorious. I need to track down the changes on my local machine, rebase, and make a patch or branch available somewhere.

comment:9 Changed 4 years ago by Christopher Allan Webber

Milestone: 0.9.01.0

comment:10 Changed 4 years ago by ayleph

Owner: Christopher Allan Webber deleted
Status: in_progressaccepted

Just to touch base here, I completely intend to make this work and provide code for it. I've got a version of it in production already. There are a lot of deprecation warnings though, so I need to spend some time cleaning it up first. But for now, I'm going to remove ownership since nobody's actively working on this.

comment:11 Changed 3 years ago by ayleph

I have an implementation of pyrax that I've been using for months, but I've just realized that it doesn't work with python3. It looks like development on pyrax is stopping in favor of a different SDK.

Pyrax is in the process of being deprecated in favor of the OpenStack SDK

https://pypi.python.org/pypi/pyrax

In light of this news, I don't think moving over to pyrax is going to be a viable option.

comment:12 Changed 2 years ago by Boris Bobrov

https://pypi.python.org/pypi/openstacksdk i wonder how hard it will be to use it

comment:13 Changed 2 years ago by Boris Bobrov

Priority: majorcritical

comment:14 Changed 2 years ago by ayleph

I moved away from pyrax. I'm currently using apache-libcloud with RackSpace Cloud Files. However, there's an issue that prevents accessing Cloud Files via SSL. I've added the missing functions on my own server, and I'm planning to offer that as a fix for this issue.

I really like the idea of integrating apache-libcloud, as I think it would give us an easier pathway to other storage providers.

comment:15 Changed 2 years ago by ayleph

I've opened a pull request for the CloudFiles SSL URI issue.

comment:16 Changed 2 years ago by ayleph

Owner: set to ayleph
Status: acceptedin_progress

comment:17 Changed 2 years ago by ShawnRisk

Milestone: 1.00.10.0

comment:18 Changed 2 years ago by ayleph

My pull request was accepted. We should be able to migrate to apache-libcloud when they build a release containing the change.

comment:19 Changed 2 years ago by ayleph

apache-libcloud 2.2.0 currently on pypi includes the necessary changes. I'll provide a patch to allow using this to access Rackspace CloudFiles.

Note: See TracTickets for help on using tickets.