Opened 7 years ago

Last modified 2 years ago

#212 review defect

Use werkzeug as a caching abstraction layer (was: «use beaker»)

Reported by: Caleb Davis Owned by:
Priority: major Milestone:
Component: programming Keywords:
Cc: Parent Tickets:

Description (last modified by Christopher Allan Webber)

There are a few bugs circulating around that block on actually
having an interface to a formal cache. This bug is to build this
cache interface. Then we can use it to do stuff!



Subtickets

Change History (31)

comment:1 Changed 7 years ago by Christopher Allan Webber

Component: Programming
Owner: set to Christopher Webber
Assigning to me.



comment:2 Changed 7 years ago by Christopher Allan Webber

[https://gitorious.org/\ :sub:`cwebber/mediagoblin/cwebbers-mediagoblin/commits/i507\_beaker\_cache](https://gitorious.org/`\ cwebber/mediagoblin/cwebbers-mediagoblin/commits/i507\_beaker\_cache)

Added basic functionality. No use and no tests yet, though.



comment:3 Changed 7 years ago by Christopher Allan Webber

I'm also curious what Elrond thinks about me using [beaker.cache]
as the config section, considering he used [storage:foo] pattern
for storage stuff...



comment:4 Changed 7 years ago by Elrond

Status: NewIn Progress
1. I already put that on irc a while back: I think, I like
   "``beaker.cache``". dot for fixed things. "``beaker``" and
   "``cache``" are fixed strings, no variations needed. While the
   colon in "``storage:FOO``" splits a fixed "``storage``" from a not
   so fixed string. storage\_from\_config() can be called with "FOO"
   and will happily create a StorageInterface. So the part after the
   colon is dynamic.
2. Instead of ``(u'cache.%s' % key, value)`` I would have used
   ``(u'cache.' + key, value)`` (it's shorter and stops a confusion
   where the right argument to ``%`` ends). Do we even need the ``u``?
   Does parse\_cache\_config\_options expect unicode strings and
   converts them to non unicode keyword-args?
3. If this does not break anything, I'd vote for merging it now
   (despite missing tests)
4. Leave this bug open for writing tests!



comment:5 Changed 7 years ago by Christopher Allan Webber

Milestone: 0.0.50.1.0

comment:5 Changed 7 years ago by Christopher Allan Webber

Status: In ProgressClosed
Added a test and merged. We might want a way to disable during
tests though? Not sure. If we do want that, not too hard to add.

Closing this though.



comment:6 Changed 7 years ago by Christopher Allan Webber

Status: ClosedIn Progress
Re-opening. At least one user (Larissa) has reported errors with
Beaker initializing on OS X. I'd like to figure out why that is. I
think I generally know.



comment:7 Changed 7 years ago by Christopher Allan Webber

Milestone: 0.1.00.2.0

comment:8 Changed 7 years ago by Christopher Allan Webber

Milestone: 0.2.00.2.1

comment:7 Changed 7 years ago by Will Kahn-Greene

The original url for this bug was http://bugs.foocorp.net/issues/507 .
Relations:
#126: related, #110: related, #183: related, #164: blocks

comment:8 Changed 7 years ago by Christopher Allan Webber

Unless anyone says what's up on OSX, we should just turn this on and start using it.

I guess the main thing to ask is two things:

  • Beaker does caching in a sort of "bucket" system, where you have entire sections of stuff dedicated to being cached.
  • I guess a good question is, "what could really use some caching right now?" Possibly the homepage would be a good first place?

Otherwise, I'm not sure why we would need to delay putting this to use and closing it out.

comment:9 Changed 7 years ago by Elrond

Milestone: 0.2.10.2.2

We should start to use beaker (or whatever) to cache the markdown conversion.
This is relevant, as we're now converting markdown on the fly.

comment:10 Changed 7 years ago by Christopher Allan Webber

Description: modified (diff)
Milestone: 0.3.0

We're not going to get this done for 0.3.0 but we need to finish it soon. It's pretty close as-is?

comment:11 Changed 6 years ago by Christopher Allan Webber

Priority: minormajor

Bumping to major. It's pretty close, we should get it wrapped up.

comment:12 Changed 6 years ago by Christopher Allan Webber

So this is old... it was pretty close to being done, but I think we should pivot (erk) and switch to using Werkzeug's cashing layer: http://werkzeug.pocoo.org/docs/contrib/cache/

comment:13 Changed 6 years ago by Christopher Allan Webber

Milestone: 0.3.3
Status: acceptedassigned

It seems like getting this in 0.3.3 wouldn't be too hard. I'm assigning it to myself.

comment:14 Changed 6 years ago by Christopher Allan Webber

Milestone: 0.3.30.3.4

Not happening for 0.3.3 anyway...
I would like to see it happen soon though.

comment:15 Changed 6 years ago by Christopher Allan Webber

If someone wants to claim this they can, but leaving claimed to me for now.

comment:16 Changed 6 years ago by Christopher Allan Webber

Keywords: sprint added
Owner: Christopher Allan Webber deleted
Summary: Use beaker as a caching abstraction layerUse beaker as a caching abstraction layer (updated: use werkzeug)

Removing claim from myself

comment:17 Changed 6 years ago by Christopher Allan Webber

Status: assignedaccepted

comment:18 Changed 6 years ago by Elrond

Milestone: 0.4.00.4.1

I doubt we get this done for 0.4.0, really.
Proof me wrong!

comment:19 Changed 5 years ago by Amirouche

Summary: Use beaker as a caching abstraction layer (updated: use werkzeug)Use werkzeug as a caching abstraction layer (was: «use beaker»)

comment:20 Changed 5 years ago by Amirouche

Can someone provide more insight on what should be done please ?

comment:21 Changed 5 years ago by Christopher Allan Webber

Milestone: 0.5.00.6.0

comment:22 Changed 5 years ago by Christopher Allan Webber

Basically, there's a caching system here:

http://werkzeug.pocoo.org/docs/contrib/cache/

Seems there's a few things to consider:

  • How to attach the cache object to the application?
  • How to make it configurable which cache system to use, and under which settings?
  • Which cache system should we turn on by default, if any? Maybe the null one actually, so people explicitly have to set up memcached if they want it?
  • What would the API be like for interacting with it? Might be useful to look at what Flask people do.

comment:23 Changed 5 years ago by Christopher Allan Webber

Milestone: 0.6.00.7.0

comment:24 Changed 5 years ago by Christopher Allan Webber

Milestone: 0.7.00.8.0

comment:25 Changed 4 years ago by Christopher Allan Webber

I took a look into this and whether we should maybe use dogpile.cache instead. As far as I can tell, dogpile doesn't provide a huge gain.

It would be really nice to have this for 0.8.0, but it's a bit lower priority than some others.

As to answer some of my previous questions:

How to attach the cache object to the application?

I think the pattern that currently exists in master is right... the cache system generally can be attached to MediaGoblinApp during the __init__() method, and any specific request-level cache objects can be set up either during the _gen_context() method (if general enough) or the _request_only_gen_context() method, if we find that's for requests only. (I suspect the former will be best, though.)

How to make it configurable which cache system to use, and under which settings?

My suggestion would be to use a hook_handle() method and let a plugin determine what the settings are. It's up to the plugin to determine what the config parameters are. Of course, we should try to provide all the base ones and use a consistent pattern to the setting parameters.

Which cache system should we turn on by default, if any? Maybe the null one actually, so people explicitly have to set up memcached if they want it?

Yes, by default, NullCache can be used if nothing is returned by hook_handle().

What would the API be like for interacting with it? Might be useful to look at what Flask people do.

No answer on this one yet... maybe worth looking at what Flask-Cache does.

comment:26 Changed 4 years ago by Christopher Allan Webber

Keywords: sprint removed
Milestone: 0.8.0

comment:27 Changed 2 years ago by Ben Sturmfels

Status: acceptedreview

To answer Chris's last question:

What would the API be like for interacting with it? Might be useful to look at what Flask people do.

The Flask docs recommend the Werkzeug caching that is already being proposed. API would be something like:

  >>> app.cache.set('key', 'value')
  >>> app.cache.get('key')
  value

I've done a prototype integration here, feedback would be appreciated. Happy to add deployment docs if I'm on the right track:

https://gitlab.com/BenSturmfels/mediagoblin/compare/master...212-werkzeug-cache

comment:28 Changed 2 years ago by Ben Sturmfels

Sorry, I'd skimmed over Chris's suggestion about hook_handle(). Could you give an example of how a plugin might want to change the caching configuration? Would it be to tweak a cache setting such as default_timeout, or would it be providing an alternative cache backend or something else?

Note: See TracTickets for help on using tickets.