Opened 10 years ago

Last modified 3 years ago

#715 in_progress enhancement

Move CloudFiles storage backend over to pyrax

Reported by: joar Owned by: ayleph
Priority: major Milestone: 0.11.0
Component: programming Keywords: cloudfiles
Cc: Parent Tickets:


python-cloudfiles is no longer being maintained, we should move to pyrax instead:

Change History (22)

comment:1 by joar, 10 years ago

Some work has been done over at I do not think I will manage to find time to finish it.

comment:2 by Christopher Allan Webber, 9 years ago

comment:3 by Christopher Allan Webber, 8 years ago

Milestone: 0.8.0
Owner: set to Christopher Allan Webber
Status: newin_progress

claiming and assigning to 0.8.0

comment:4 by Christopher Allan Webber, 8 years ago


comment:5 by ayleph, 8 years ago

I've been playing around with pyrax and can so far upload images to and delete images from RackSpace CloudFiles. However, it's not compatible with current MediaGoblin due to restrictions on the python six package.

  File "/path/to/mediagoblin/mediagoblin/tools/", line 34, in import_component
  File "/path/to/mediagoblin/mediagoblin/storage/", line 28, in <module>
    import pyrax
  File "/path/to/mediagoblin/lib/python2.7/site-packages/pyrax/", line 59, in <module>
    from import OpenStackComputeShell as _cs_shell
  File "/path/to/mediagoblin/lib/python2.7/site-packages/novaclient/", line 47, in <module>
    from novaclient import client
  File "/path/to/mediagoblin/lib/python2.7/site-packages/novaclient/", line 34, in <module>
    from oslo.utils import netutils
  File "/path/to/mediagoblin/lib/python2.7/site-packages/oslo/utils/", line 13, in <module>
    from oslo_utils.netutils import *  # noqa
  File "/path/to/mediagoblin/lib/python2.7/site-packages/oslo_utils/", line 170, in <module>
    class _ModifiedSplitResult(parse.SplitResult):
AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'

GMG requires six==1.5.2. A cursory internet search reveals this is sufficient for pyrax itself, but python-novaclient seems to require six>=1.6.

To test pyrax, I upgraded six (1.9.0) and everything is working okay so far in my limited testing. What's the reason for holding six at 1.5.2 in

comment:6 by ayleph, 8 years ago

I used Joar's changes as a starting point and created a branch with some additional changes. His branch was so far behind master I decided to create a new repo instead of forking his. I moved the pyrax code into a separate file, but I kept the idea of moving the options into a separate cloudfiles section of config_spec.ini.

The file is a straight copy of the with some content changed to enable uploading and to allow setting CORS headers on new objects. I haven't tested all of the methods yet.

comment:7 by ayleph, 8 years ago

The requirement on six has been removed in #5011.

comment:8 by ayleph, 7 years ago

My branch referenced above, besides being completely out of date, is no longer accessible on gitorious. I need to track down the changes on my local machine, rebase, and make a patch or branch available somewhere.

comment:9 by Christopher Allan Webber, 7 years ago


comment:10 by ayleph, 7 years ago

Owner: Christopher Allan Webber removed
Status: in_progressaccepted

Just to touch base here, I completely intend to make this work and provide code for it. I've got a version of it in production already. There are a lot of deprecation warnings though, so I need to spend some time cleaning it up first. But for now, I'm going to remove ownership since nobody's actively working on this.

comment:11 by ayleph, 6 years ago

I have an implementation of pyrax that I've been using for months, but I've just realized that it doesn't work with python3. It looks like development on pyrax is stopping in favor of a different SDK.

Pyrax is in the process of being deprecated in favor of the OpenStack SDK

In light of this news, I don't think moving over to pyrax is going to be a viable option.

comment:12 by Boris Bobrov, 6 years ago i wonder how hard it will be to use it

comment:13 by Boris Bobrov, 6 years ago

Priority: majorcritical

comment:14 by ayleph, 6 years ago

I moved away from pyrax. I'm currently using apache-libcloud with RackSpace Cloud Files. However, there's an issue that prevents accessing Cloud Files via SSL. I've added the missing functions on my own server, and I'm planning to offer that as a fix for this issue.

I really like the idea of integrating apache-libcloud, as I think it would give us an easier pathway to other storage providers.

comment:15 by ayleph, 6 years ago

I've opened a pull request for the CloudFiles SSL URI issue.

comment:16 by ayleph, 6 years ago

Owner: set to ayleph
Status: acceptedin_progress

comment:17 by ShawnRisk, 6 years ago


comment:18 by ayleph, 6 years ago

My pull request was accepted. We should be able to migrate to apache-libcloud when they build a release containing the change.

comment:19 by ayleph, 6 years ago

apache-libcloud 2.2.0 currently on pypi includes the necessary changes. I'll provide a patch to allow using this to access Rackspace CloudFiles.

comment:20 by Ben Sturmfels, 3 years ago

Priority: criticalmajor

comment:21 by ayleph, 3 years ago

Oh gosh thanks for the poke. I'll try to dig this up and see if it still works.

comment:22 by Ben Sturmfels, 3 years ago


No panic - I'll move this to 0.11.0 for now.

Note: See TracTickets for help on using tickets.