Opened 13 years ago

Last modified 7 years ago

#212 review defect

Use werkzeug as a caching abstraction layer (was: «use beaker»)

Reported by: Caleb Davis Owned by:
Priority: major Milestone:
Component: programming Keywords:
Cc: Parent Tickets:

Description (last modified by Christopher Allan Webber)

There are a few bugs circulating around that block on actually
having an interface to a formal cache. This bug is to build this
cache interface. Then we can use it to do stuff!



Change History (31)

comment:1 by Christopher Allan Webber, 13 years ago

Component: Programming
Owner: set to Christopher Webber
Assigning to me.



comment:2 by Christopher Allan Webber, 13 years ago

[https://gitorious.org/\ :sub:`cwebber/mediagoblin/cwebbers-mediagoblin/commits/i507\_beaker\_cache](https://gitorious.org/`\ cwebber/mediagoblin/cwebbers-mediagoblin/commits/i507\_beaker\_cache)

Added basic functionality. No use and no tests yet, though.



comment:3 by Christopher Allan Webber, 13 years ago

I'm also curious what Elrond thinks about me using [beaker.cache]
as the config section, considering he used [storage:foo] pattern
for storage stuff...



comment:4 by Elrond, 13 years ago

Status: NewIn Progress
1. I already put that on irc a while back: I think, I like
   "``beaker.cache``". dot for fixed things. "``beaker``" and
   "``cache``" are fixed strings, no variations needed. While the
   colon in "``storage:FOO``" splits a fixed "``storage``" from a not
   so fixed string. storage\_from\_config() can be called with "FOO"
   and will happily create a StorageInterface. So the part after the
   colon is dynamic.
2. Instead of ``(u'cache.%s' % key, value)`` I would have used
   ``(u'cache.' + key, value)`` (it's shorter and stops a confusion
   where the right argument to ``%`` ends). Do we even need the ``u``?
   Does parse\_cache\_config\_options expect unicode strings and
   converts them to non unicode keyword-args?
3. If this does not break anything, I'd vote for merging it now
   (despite missing tests)
4. Leave this bug open for writing tests!



comment:5 by Christopher Allan Webber, 13 years ago

Milestone: 0.0.50.1.0

comment:5 by Christopher Allan Webber, 13 years ago

Status: In ProgressClosed
Added a test and merged. We might want a way to disable during
tests though? Not sure. If we do want that, not too hard to add.

Closing this though.



comment:6 by Christopher Allan Webber, 13 years ago

Status: ClosedIn Progress
Re-opening. At least one user (Larissa) has reported errors with
Beaker initializing on OS X. I'd like to figure out why that is. I
think I generally know.



comment:7 by Christopher Allan Webber, 12 years ago

Milestone: 0.1.00.2.0

comment:8 by Christopher Allan Webber, 12 years ago

Milestone: 0.2.00.2.1

comment:7 by Will Kahn-Greene, 12 years ago

The original url for this bug was http://bugs.foocorp.net/issues/507 .
Relations:
#126: related, #110: related, #183: related, #164: blocks

comment:8 by Christopher Allan Webber, 12 years ago

Unless anyone says what's up on OSX, we should just turn this on and start using it.

I guess the main thing to ask is two things:

  • Beaker does caching in a sort of "bucket" system, where you have entire sections of stuff dedicated to being cached.
  • I guess a good question is, "what could really use some caching right now?" Possibly the homepage would be a good first place?

Otherwise, I'm not sure why we would need to delay putting this to use and closing it out.

comment:9 by Elrond, 12 years ago

Milestone: 0.2.10.2.2

We should start to use beaker (or whatever) to cache the markdown conversion.
This is relevant, as we're now converting markdown on the fly.

comment:10 by Christopher Allan Webber, 12 years ago

Description: modified (diff)
Milestone: 0.3.0

We're not going to get this done for 0.3.0 but we need to finish it soon. It's pretty close as-is?

comment:11 by Christopher Allan Webber, 11 years ago

Priority: minormajor

Bumping to major. It's pretty close, we should get it wrapped up.

comment:12 by Christopher Allan Webber, 11 years ago

So this is old... it was pretty close to being done, but I think we should pivot (erk) and switch to using Werkzeug's cashing layer: http://werkzeug.pocoo.org/docs/contrib/cache/

comment:13 by Christopher Allan Webber, 11 years ago

Milestone: 0.3.3
Status: acceptedassigned

It seems like getting this in 0.3.3 wouldn't be too hard. I'm assigning it to myself.

comment:14 by Christopher Allan Webber, 11 years ago

Milestone: 0.3.30.3.4

Not happening for 0.3.3 anyway...
I would like to see it happen soon though.

comment:15 by Christopher Allan Webber, 11 years ago

If someone wants to claim this they can, but leaving claimed to me for now.

comment:16 by Christopher Allan Webber, 11 years ago

Keywords: sprint added
Owner: Christopher Allan Webber removed
Summary: Use beaker as a caching abstraction layerUse beaker as a caching abstraction layer (updated: use werkzeug)

Removing claim from myself

comment:17 by Christopher Allan Webber, 11 years ago

Status: assignedaccepted

comment:18 by Elrond, 11 years ago

Milestone: 0.4.00.4.1

I doubt we get this done for 0.4.0, really.
Proof me wrong!

comment:19 by Amirouche, 11 years ago

Summary: Use beaker as a caching abstraction layer (updated: use werkzeug)Use werkzeug as a caching abstraction layer (was: «use beaker»)

comment:20 by Amirouche, 11 years ago

Can someone provide more insight on what should be done please ?

comment:21 by Christopher Allan Webber, 11 years ago

Milestone: 0.5.00.6.0

comment:22 by Christopher Allan Webber, 10 years ago

Basically, there's a caching system here:

http://werkzeug.pocoo.org/docs/contrib/cache/

Seems there's a few things to consider:

  • How to attach the cache object to the application?
  • How to make it configurable which cache system to use, and under which settings?
  • Which cache system should we turn on by default, if any? Maybe the null one actually, so people explicitly have to set up memcached if they want it?
  • What would the API be like for interacting with it? Might be useful to look at what Flask people do.

comment:23 by Christopher Allan Webber, 10 years ago

Milestone: 0.6.00.7.0

comment:24 by Christopher Allan Webber, 10 years ago

Milestone: 0.7.00.8.0

comment:25 by Christopher Allan Webber, 9 years ago

I took a look into this and whether we should maybe use dogpile.cache instead. As far as I can tell, dogpile doesn't provide a huge gain.

It would be really nice to have this for 0.8.0, but it's a bit lower priority than some others.

As to answer some of my previous questions:

How to attach the cache object to the application?

I think the pattern that currently exists in master is right... the cache system generally can be attached to MediaGoblinApp during the __init__() method, and any specific request-level cache objects can be set up either during the _gen_context() method (if general enough) or the _request_only_gen_context() method, if we find that's for requests only. (I suspect the former will be best, though.)

How to make it configurable which cache system to use, and under which settings?

My suggestion would be to use a hook_handle() method and let a plugin determine what the settings are. It's up to the plugin to determine what the config parameters are. Of course, we should try to provide all the base ones and use a consistent pattern to the setting parameters.

Which cache system should we turn on by default, if any? Maybe the null one actually, so people explicitly have to set up memcached if they want it?

Yes, by default, NullCache can be used if nothing is returned by hook_handle().

What would the API be like for interacting with it? Might be useful to look at what Flask people do.

No answer on this one yet... maybe worth looking at what Flask-Cache does.

comment:26 by Christopher Allan Webber, 9 years ago

Keywords: sprint removed
Milestone: 0.8.0

comment:27 by Ben Sturmfels, 7 years ago

Status: acceptedreview

To answer Chris's last question:

What would the API be like for interacting with it? Might be useful to look at what Flask people do.

The Flask docs recommend the Werkzeug caching that is already being proposed. API would be something like:

  >>> app.cache.set('key', 'value')
  >>> app.cache.get('key')
  value

I've done a prototype integration here, feedback would be appreciated. Happy to add deployment docs if I'm on the right track:

https://gitlab.com/BenSturmfels/mediagoblin/compare/master...212-werkzeug-cache

comment:28 by Ben Sturmfels, 7 years ago

Sorry, I'd skimmed over Chris's suggestion about hook_handle(). Could you give an example of how a plugin might want to change the caching configuration? Would it be to tweak a cache setting such as default_timeout, or would it be providing an alternative cache backend or something else?

Note: See TracTickets for help on using tickets.