[Pkg-javascript-devel] script to generate debian/watch for embedding nodejs modules

[Paolo Greppi]

Il 26/11/18 22:15, Xavier ha scritto:
> Le 26/11/2018 à 20:30, Xavier a écrit :
>> Le 26/11/2018 à 19:40, Jérémy Lal a écrit :
>>> Le lun. 26 nov. 2018 à 17:29, Xavier <yadd at debian.org
>>> <mailto:yadd at debian.org>> a écrit :
>>>
>>>     Xavier wrote:
>>>     >> ...
>>>     >> Looks acceptable IMO
>>>     >
>>>     > Policy update:
>>>     >  - components used only during build => not used in version
>>>     >    (except if they inject some code)
>>>     >  - components without major risks    => not used in version
>>>     >  - components that must be followed  => declared as "group" in
>>>     >    debian/watch
>>>     >  - components that must be followed and used in many other packages
>>>     >    => packaged separately
>>>     >
>>>     > Note: I wrote this to help js-team and decrease time to wait in NEW
>>>     > queue. If you feel that "crazy", please simply delete
>>>     > https://wiki.debian.org/Javascript/GroupSourcesTutorial and the
>>>     links to
>>>     > this page.
>>>     >
>>>     > I you don't want to do it by yourself, I can remove the page for you.
>>>     > Just ask me to do it.
>>>
>>> This is certainly not crazy, and while i'm not myself completely
>>> convinced it's the right approach,
>>> (mostly because i did not took the time to review and practice), it's
>>> the closest thing we have
>>> to a usable solution.
>>>
>>> Please don't stop there, it would be such a waste.
>>>
>>> About the bundled packages and the weird version encoding all the
>>> bundled versions:
>>> most upstreams that are serious about their packaging now use a
>>> package-lock file,
>>> effectively locking down a released version to the needed versions of
>>> dependencies.
>>> When it's the case, it means that any change in versions of dependencies
>>> will be reflected
>>> by a new parent package version. In that case, it's useless to encode
>>> dependencies versions
>>> in the parent debian package version ?
>>>
>>> Jérémy
>>
>> You're right, in that case, components must be tagged as "ignore" or
>> best a version number ("0.0.1" for example) directly in debian/watch.
> 
> No! 0.0.1 means "newer than 0.0.1" for uscan. I'm modifying uscan to be
> able to fix a component version using "=0.0.1" syntax.

Hey, when I wrote "crazy" I did not mean to bash your work !
Thanks for all your excellent efforts to automate dependency management.
At least I am glad to have sparked a lively conversation.

I have updated my script to use the new debian/watch syntax:
https://salsa.debian.org/paolog-guest/create-bundled-watch
With that change when I apply it to yarnpkg (I used ignore because which has a yarn.lock) it downloads:

babel-plugin-transform-inline-imports-commonjs-1.2.0.tgz
normalize-url-4.0.0.tgz
tar-fs-1.16.3.tgz
v8-compile-cache-2.0.2.tgz
yarnpkg-0.15.1.tgz

and creates the symlinks:

node-yarnpkg_0.15.1.orig-babel-plugin-transform-inline-imports-commonjs.tar.gz -> babel-plugin-transform-inline-imports-commonjs-1.2.0.tgz
node-yarnpkg_0.15.1.orig-dnscache.tar.gz -> dnscache-1.0.1.tgz
node-yarnpkg_0.15.1.orig-normalize-url.tar.gz -> normalize-url-4.0.0.tgz
node-yarnpkg_0.15.1.orig-tar-fs.tar.gz -> tar-fs-1.16.3.tgz
node-yarnpkg_0.15.1.orig.tar.gz -> yarnpkg-0.15.1.tgz
node-yarnpkg_0.15.1.orig-v8-compile-cache.tar.gz -> v8-compile-cache-2.0.2.tgz

which is clean and totally acceptable. Well done !

With yesterday's news about the event-stream node module being pwned:
https://github.com/dominictarr/event-stream/issues/116
the importance of these matters should be clear to anyone.
Probably there is no component "without major risks", and even if it existed, it would be unfair to lay upon the busy maintainer the task of deciding if it is risky or not.

Better provide a simple and clear policy, here is my attempt:

Should I embed it ?
1. dependency has a non-trivial build step (i.e. needs transpiling or webpack) or is used by 2+ other modules already packaged / to be packaged in Debian
   => package separately
2. dependency is a simple module (just ES5 files that do not need any transformation/transpiling, no build steps required, not depended upon by other packages)
   => embed

How do I embed it ?
1. upstream itself provides node_modules inside their tarball
  => remove the modules already packaged / to be packaged in Debian (repacking with Files-Excluded) and keep the simple dependencies
2. else:
  => embed dependency in source root inside /node_modules

Do I track dependency versions ?
1. dependency is used only during build and does not inject any code (!)
   => do not track version (declare as "ignore" in debian/watch)
2. dependency is used at run time but main module has package-lock or yarn.lock checked in version control (i.e.  any change in versions of dependencies will be reflected by a new main package version)
   => do not track version (declare as "ignore" in debian/watch)
3. else:
   => track version of dependency as a suffix to main module version (declare as "group" in debian/watch)

I am still confused about how the pkg-components / uscan-components workflow described here:
https://wiki.debian.org/Javascript/Nodejs/Npm2Deb#Embedding_some_modules
relates to the uscan group/ignore syntax described here:
https://wiki.debian.org/Javascript/GroupSourcesTutorial
Can some one clarify ? Thanks.

Paolo