Ticket #1095: better_extlib.txt

File better_extlib.txt, 11.2 KB (added by Christopher Allan Webber, 3 years ago)

A conversation with Kuno / warp about a better extlib solution

1<paroneayea> kuno: around?                                              [14:07]
2<kuno> yes                                                              [14:10]
3<paroneayea> kuno: you're the one who, in conversation about packaging and
4             etc, expressed "but I like git checkouts for deployment, because
5             complete and corresponding source!" right? :)
6<kuno> yes!
7<paroneayea> kuno: great!  because I have a C&CS puzzle and I want to talk to
8             you about my suggested solution.                           [14:11]
9<kuno> and I love puzzles!
10<paroneayea> kuno: like many web development projects, mediagoblin follows the
11             really bad practice of "just check the javascript libraries into
12             the source tree"
13<paroneayea> kuno: this is bad for a variety of reasons.  Reason 1 is that it
14             bloats the source tree, obv                                [14:12]
15* kuno nods
16<paroneayea> Reason 2 is it's usually a byte-compiled version of the
17             project... actually proprietary if you aren't doing the C&CS side
18             of things when you think about it!
19<paroneayea> it's "standard practice" for webdev, but bad practice.
20<paroneayea> kuno: the problem is made worse though, because most javascript
21             libraries today aren't just "unpack and run" anymore       [14:13]
22<paroneayea> lots of them count on node.js to compile                   [14:14]
23<paroneayea> an onerous requirement for someone doing drive-by contributions!
24             Or really any contributor.
25<paroneayea> kuno: so here's my thought on a solution, let's see what you
26             think.
27<kuno> I disagree on the onerous requirement part, but otherwise agree with
28       your premise.
29<paroneayea> kuno: so I am thinking maybe having a file like:
30             http://pamrel.lu/48764/                                    [14:18]
31<chaosgoblin> Title: Pamrel - 48764 (at pamrel.lu)
32<paroneayea> and some python code
33<paroneayea> that knows how to fetch and extract and move the binary version
34             in to place                                                [14:19]
35<kuno> paroneayea: I think i'd be crazy to ignore the npm and bower
36       repositories
37<kuno> s/i/it/
38<paroneayea> but *at minimum* includes instructions on how to clone the
39             repositories, but maybe even actually can build them
40<paroneayea> with the c_and_cs part
41<paroneayea> kuno: yes, those exist, but the problem is                 [14:20]
42<paroneayea> users already have trouble with running into python's package
43             manager with our stuff
44<paroneayea> having them beat their head against 1000 unique package managers
45<paroneayea> that's onerous :)
46* paroneayea exaggerating of course, we don't have 1000 deps, but you get it
47                                                                        [14:21]
48*** pitanga (~rodrigo@fsf/member/pitanga) has quit: Read error: Connection
49    reset by peer
50*** pitanga (~rodrigo@fsf/member/pitanga) has joined channel #mediagoblin
51                                                                        [14:22]
52<paroneayea> kuno: at minimum, I think this is an improvement over things as
53             they are
54<paroneayea> but it could be much better.
55<kuno> paroneayea: yeah.  I can see the argument that an end-user shouldn't
56       need to have a javascript package manager installed to deploy
57       mediagoblin, but then they also shouldn't need a python package manager
58       really.  You should ship those folks the full thing, all packaged and
59       ready to go.
60<paroneayea> kuno: yeah.  I think the thing is, I want to hit all the major
61             parts of the spectrum:
62<paroneayea>  - just install the package and all packages via system packager
63                                                                        [14:23]
64<paroneayea>  - hack on mediagoblin version: mediagoblin's checkout, but then
65             pull down prebuilt packages for the js deps and etc easy
66<paroneayea>  - full C&CS auditor mode!  have at minimum the ability and full
67             documentation on how to build the whole thing, top to bottom.
68                                                                        [14:24]
69<kuno> paroneayea: Implementing a half-hearted javascript package manager
70       doesn't seem like a great solution.  If you want to stay within the
71       python ecosystem, can you not just package your deps for fanstatic?
72<paroneayea> "for fantastic"?
73<paroneayea> kuno: you mean make python eggs? :P
74<kuno> paroneayea: there's all these javascript libraries on pypi,
75       https://pypi.python.org/pypi/js.jquery/1.9.1
76<chaosgoblin> Title: js.jquery 1.9.1 : Python Package Index (at
77              pypi.python.org)
78<paroneayea> kuno: "yes you can" but it's really quite messy.           [14:25]
79<paroneayea> and they aren't well maintained.
80<paroneayea> I did that at one point with a job's deps
81<paroneayea> it turned into a nightmare.
82*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has joined channel
83    #mediagoblin
84<paroneayea> at the moment I just want something that untars things into a
85             directory mostly
86<paroneayea> since effectively that's what we have
87<kuno> paroneayea: yeah, if you want well-maintained, you need to get these
88       javascript packages from npm and bower :)
89<paroneayea> right
90<paroneayea> so this is intermediate land :)
91<kuno> paroneayea: implementing a half-hearted javascript package manager
92       which just pulls stuff from npm could be an option.  The package
93       description files are straightforward, and I expect the npm API to be
94       similarly so.                                                    [14:26]
95*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has quit: Read
96    error: Connection reset by peer
97<paroneayea> kuno: maybe... but is probably a lot more work than just wget and
98             untar.                                                     [14:27]
99<paroneayea> while checking a hash
100*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has joined channel
101    #mediagoblin
102<paroneayea> kuno: but!                                                 [14:29]
103*** pitanga (~rodrigo@fsf/member/pitanga) has quit: Ping timeout: 246 seconds
104<paroneayea> the thing I suggested above
105<paroneayea> should be able to handle that.
106<kuno> paroneayea: anyway, I need to get back to the dayjob.  I'm going to
107       think about this problem a bit more.                             [14:30]
108<paroneayea> kuno: ok :)
109*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has quit: Read
110    error: Connection reset by peer                                     [14:31]
111<paroneayea> kuno: at the very least, I feel from the conversation we just had
112             that the ~solution I'm putting forward is "adequate enough", if
113             not great, but much better than existing check into git
114             "solutions" ;)
115*** pitanga (~rodrigo@fsf/member/pitanga) has joined channel #mediagoblin
116*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has joined channel
117    #mediagoblin                                                        [14:33]
118*** pitanga (~rodrigo@fsf/member/pitanga) has quit: Ping timeout: 245 seconds
119                                                                        [14:35]
120<kuno> paroneayea: I guess my main issue with it is that you're duplicating
121       metadata.  I love metadata, but for javascript the existing package
122       managers already have all the metadata you need (link to git source,
123       authors and license fields, etc...)                              [14:36]
124*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has quit: Read
125    error: Connection reset by peer
126<paroneayea> kuno: well, in that sense                                  [14:46]
127<paroneayea> the real thing is we should be using a unified package manager
128<paroneayea> and in that sense also
129<paroneayea> I think davexunit is right in that we should maybe all be using
130             guix instead of virtualenv and etc ;)
131<kuno> paroneayea: guix/nix do seem really promising.  I don't think end-users
132       should have to deal with package managers though,..              [14:50]
133<kuno> and for developers it seems perfectly OK to expect a python developer
134       to have a python interpreter installed to run a package manager written
135       in python
136<kuno> and for a javascript developer to have javascript compiler or whatever
137       installed to run a package manager written in javascript         [14:51]
138<paroneayea> kuno: yep, but as you see
139<paroneayea> the problem is when python developers need to run javascript
140             compilers ;)                                               [14:52]
141<paroneayea> and etc
142<kuno> if you're a developer working on a project which has both python and
143       javascript bits, is it really that much to ask that developer to have
144       those package managers?
145<kuno> (it's just a "sudo apt-get install npm")                         [14:53]
146<paroneayea> kuno: it's a lot to ask them to *debug* all those things, and to
147             have to run each package manager in turn.
148<paroneayea> kuno: remember also how often python's package manager breaks
149                                                                        [14:57]
150<paroneayea> start adding more and more breakable language package managers
151<paroneayea> and you're gonna have a bad time
152<paroneayea> (also each one tends to take a really long time to run)
153<paroneayea> so I do think it's onerous still
154<kuno> paroneayea: right, but that's because python packages are notoriously
155       bad about specifying the correct versions for their requirements ;)
156                                                                        [15:00]
157<paroneayea> kuno: not only!                                            [15:01]
158<paroneayea> kuno: we've had a good number of other types of errors:
159<kuno> (this is less of a problem with npm because "npm install some-package
160       --save" does the right thing by default, and you have multiple versions
161       of the same dependency in the tree without conflicts or trouble)
162<kuno> +can                                                             [15:02]
163*** azerus (~badass@unaffiliated/badass) has quit: Quit: Leaving...     [15:04]
164<kuno> paroneayea: in a mixed javascript + python project, it just seems
165       arbitrary to call requiring one of the package managers an onerous
166       requirement, while requiring the other package manager is perfectly
167       fine.                                                            [15:08]
168<paroneayea> kuno: it's a python project with javascript dependencies
169<paroneayea> just as so is it a python project with C dependencies :)
170<kuno> :)                                                               [15:09]
171<kuno> there is a problem with my previous argument that end-users shouldn't
172       be using package managers at all, which is that it increases the
173       distance between developers and end-users.  Which makes it more
174       difficult for a curious end-user to tinker with stuff and become a
175       developer.                                                       [15:10]
176<kuno> in that sense I am in favour of improving our developer tools and
177       package managers to such an extent that they could be used by an
178       end-user.
179<kuno> (but that seems impossible with the current state-of-the-art)    [15:11]
180<paroneayea> agreed there kuno