Ticket #1095: better_extlib.txt

File better_extlib.txt, 11.2 KB (added by Christopher Allan Webber, 9 years ago)

A conversation with Kuno / warp about a better extlib solution

1<paroneayea> kuno: around? [14:07]
2<kuno> yes [14:10]
3<paroneayea> kuno: you're the one who, in conversation about packaging and
4 etc, expressed "but I like git checkouts for deployment, because
5 complete and corresponding source!" right? :)
6<kuno> yes!
7<paroneayea> kuno: great! because I have a C&CS puzzle and I want to talk to
8 you about my suggested solution. [14:11]
9<kuno> and I love puzzles!
10<paroneayea> kuno: like many web development projects, mediagoblin follows the
11 really bad practice of "just check the javascript libraries into
12 the source tree"
13<paroneayea> kuno: this is bad for a variety of reasons. Reason 1 is that it
14 bloats the source tree, obv [14:12]
15* kuno nods
16<paroneayea> Reason 2 is it's usually a byte-compiled version of the
17 project... actually proprietary if you aren't doing the C&CS side
18 of things when you think about it!
19<paroneayea> it's "standard practice" for webdev, but bad practice.
20<paroneayea> kuno: the problem is made worse though, because most javascript
21 libraries today aren't just "unpack and run" anymore [14:13]
22<paroneayea> lots of them count on node.js to compile [14:14]
23<paroneayea> an onerous requirement for someone doing drive-by contributions!
24 Or really any contributor.
25<paroneayea> kuno: so here's my thought on a solution, let's see what you
26 think.
27<kuno> I disagree on the onerous requirement part, but otherwise agree with
28 your premise.
29<paroneayea> kuno: so I am thinking maybe having a file like:
30 http://pamrel.lu/48764/ [14:18]
31<chaosgoblin> Title: Pamrel - 48764 (at pamrel.lu)
32<paroneayea> and some python code
33<paroneayea> that knows how to fetch and extract and move the binary version
34 in to place [14:19]
35<kuno> paroneayea: I think i'd be crazy to ignore the npm and bower
36 repositories
37<kuno> s/i/it/
38<paroneayea> but *at minimum* includes instructions on how to clone the
39 repositories, but maybe even actually can build them
40<paroneayea> with the c_and_cs part
41<paroneayea> kuno: yes, those exist, but the problem is [14:20]
42<paroneayea> users already have trouble with running into python's package
43 manager with our stuff
44<paroneayea> having them beat their head against 1000 unique package managers
45<paroneayea> that's onerous :)
46* paroneayea exaggerating of course, we don't have 1000 deps, but you get it
47 [14:21]
48*** pitanga (~rodrigo@fsf/member/pitanga) has quit: Read error: Connection
49 reset by peer
50*** pitanga (~rodrigo@fsf/member/pitanga) has joined channel #mediagoblin
51 [14:22]
52<paroneayea> kuno: at minimum, I think this is an improvement over things as
53 they are
54<paroneayea> but it could be much better.
55<kuno> paroneayea: yeah. I can see the argument that an end-user shouldn't
56 need to have a javascript package manager installed to deploy
57 mediagoblin, but then they also shouldn't need a python package manager
58 really. You should ship those folks the full thing, all packaged and
59 ready to go.
60<paroneayea> kuno: yeah. I think the thing is, I want to hit all the major
61 parts of the spectrum:
62<paroneayea> - just install the package and all packages via system packager
63 [14:23]
64<paroneayea> - hack on mediagoblin version: mediagoblin's checkout, but then
65 pull down prebuilt packages for the js deps and etc easy
66<paroneayea> - full C&CS auditor mode! have at minimum the ability and full
67 documentation on how to build the whole thing, top to bottom.
68 [14:24]
69<kuno> paroneayea: Implementing a half-hearted javascript package manager
70 doesn't seem like a great solution. If you want to stay within the
71 python ecosystem, can you not just package your deps for fanstatic?
72<paroneayea> "for fantastic"?
73<paroneayea> kuno: you mean make python eggs? :P
74<kuno> paroneayea: there's all these javascript libraries on pypi,
75 https://pypi.python.org/pypi/js.jquery/1.9.1
76<chaosgoblin> Title: js.jquery 1.9.1 : Python Package Index (at
77 pypi.python.org)
78<paroneayea> kuno: "yes you can" but it's really quite messy. [14:25]
79<paroneayea> and they aren't well maintained.
80<paroneayea> I did that at one point with a job's deps
81<paroneayea> it turned into a nightmare.
82*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has joined channel
83 #mediagoblin
84<paroneayea> at the moment I just want something that untars things into a
85 directory mostly
86<paroneayea> since effectively that's what we have
87<kuno> paroneayea: yeah, if you want well-maintained, you need to get these
88 javascript packages from npm and bower :)
89<paroneayea> right
90<paroneayea> so this is intermediate land :)
91<kuno> paroneayea: implementing a half-hearted javascript package manager
92 which just pulls stuff from npm could be an option. The package
93 description files are straightforward, and I expect the npm API to be
94 similarly so. [14:26]
95*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has quit: Read
96 error: Connection reset by peer
97<paroneayea> kuno: maybe... but is probably a lot more work than just wget and
98 untar. [14:27]
99<paroneayea> while checking a hash
100*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has joined channel
101 #mediagoblin
102<paroneayea> kuno: but! [14:29]
103*** pitanga (~rodrigo@fsf/member/pitanga) has quit: Ping timeout: 246 seconds
104<paroneayea> the thing I suggested above
105<paroneayea> should be able to handle that.
106<kuno> paroneayea: anyway, I need to get back to the dayjob. I'm going to
107 think about this problem a bit more. [14:30]
108<paroneayea> kuno: ok :)
109*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has quit: Read
110 error: Connection reset by peer [14:31]
111<paroneayea> kuno: at the very least, I feel from the conversation we just had
112 that the ~solution I'm putting forward is "adequate enough", if
113 not great, but much better than existing check into git
114 "solutions" ;)
115*** pitanga (~rodrigo@fsf/member/pitanga) has joined channel #mediagoblin
116*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has joined channel
117 #mediagoblin [14:33]
118*** pitanga (~rodrigo@fsf/member/pitanga) has quit: Ping timeout: 245 seconds
119 [14:35]
120<kuno> paroneayea: I guess my main issue with it is that you're duplicating
121 metadata. I love metadata, but for javascript the existing package
122 managers already have all the metadata you need (link to git source,
123 authors and license fields, etc...) [14:36]
124*** pitanga1 (~rodrigo@200-153-132-135.dsl.telesp.net.br) has quit: Read
125 error: Connection reset by peer
126<paroneayea> kuno: well, in that sense [14:46]
127<paroneayea> the real thing is we should be using a unified package manager
128<paroneayea> and in that sense also
129<paroneayea> I think davexunit is right in that we should maybe all be using
130 guix instead of virtualenv and etc ;)
131<kuno> paroneayea: guix/nix do seem really promising. I don't think end-users
132 should have to deal with package managers though,.. [14:50]
133<kuno> and for developers it seems perfectly OK to expect a python developer
134 to have a python interpreter installed to run a package manager written
135 in python
136<kuno> and for a javascript developer to have javascript compiler or whatever
137 installed to run a package manager written in javascript [14:51]
138<paroneayea> kuno: yep, but as you see
139<paroneayea> the problem is when python developers need to run javascript
140 compilers ;) [14:52]
141<paroneayea> and etc
142<kuno> if you're a developer working on a project which has both python and
143 javascript bits, is it really that much to ask that developer to have
144 those package managers?
145<kuno> (it's just a "sudo apt-get install npm") [14:53]
146<paroneayea> kuno: it's a lot to ask them to *debug* all those things, and to
147 have to run each package manager in turn.
148<paroneayea> kuno: remember also how often python's package manager breaks
149 [14:57]
150<paroneayea> start adding more and more breakable language package managers
151<paroneayea> and you're gonna have a bad time
152<paroneayea> (also each one tends to take a really long time to run)
153<paroneayea> so I do think it's onerous still
154<kuno> paroneayea: right, but that's because python packages are notoriously
155 bad about specifying the correct versions for their requirements ;)
156 [15:00]
157<paroneayea> kuno: not only! [15:01]
158<paroneayea> kuno: we've had a good number of other types of errors:
159<kuno> (this is less of a problem with npm because "npm install some-package
160 --save" does the right thing by default, and you have multiple versions
161 of the same dependency in the tree without conflicts or trouble)
162<kuno> +can [15:02]
163*** azerus (~badass@unaffiliated/badass) has quit: Quit: Leaving... [15:04]
164<kuno> paroneayea: in a mixed javascript + python project, it just seems
165 arbitrary to call requiring one of the package managers an onerous
166 requirement, while requiring the other package manager is perfectly
167 fine. [15:08]
168<paroneayea> kuno: it's a python project with javascript dependencies
169<paroneayea> just as so is it a python project with C dependencies :)
170<kuno> :) [15:09]
171<kuno> there is a problem with my previous argument that end-users shouldn't
172 be using package managers at all, which is that it increases the
173 distance between developers and end-users. Which makes it more
174 difficult for a curious end-user to tinker with stuff and become a
175 developer. [15:10]
176<kuno> in that sense I am in favour of improving our developer tools and
177 package managers to such an extent that they could be used by an
178 end-user.
179<kuno> (but that seems impossible with the current state-of-the-art) [15:11]
180<paroneayea> agreed there kuno