[Pkg-javascript-devel] Bug#995722: Not running tests because tests miss source code is not useful

[Pirate Praveen]

On വെ, ഒക്ടോ 8 2021 at 10:31:16 രാവിലെ +0200 
+0200, Thomas Goirand <zigo at debian.org> wrote:
> On 10/7/21 11:40 AM, Pirate Praveen wrote:
>> 
>> 
>>  On 7 October 2021 3:02:55 am IST, Thomas Goirand <zigo at debian.org> 
>> wrote:
>>>  On 10/6/21 6:53 PM, Pirate Praveen wrote:
>>>>  [adding -devel]
>>>> 
>>>>  On ബു, ഒക്ടോ 6 2021 at 12:16:07 വൈകു +0200 
>>>> +0200, Jonas Smedegaard
>>>>  <jonas at jones.dk> wrote:
>>>>>  Quoting Yadd (2021-10-06 11:43:40)
>>>>>>   On Lu, 04 oct 21, 16:40:48, Bastien Roucari�s wrote:
>>>>>>   > Source: src:node-lodash
>>>>>>   > Version: 4.17.21+dfsg+~cs8.31.173-1
>>>>>>   > Severity: serious
>>>>>>   > Justification: do not compile from source
>>>>>>   >
>>>>>>   > Dear Maintainer,
>>>>>>   >
>>>>>>   > The vendor directory should be emptied
>>>>>>   >
>>>>>>   > The debug version is compiled without source (lintian warn) 
>>>>>> and
>>>>>>  moreover the
>>>>>>   > rest of file are already packaged
>>>>>>   >
>>>>>>   > grep -R vendor * gives only a few hit that could be cured by
>>>>>>  symlinking
>>>>>>   >
>>>>>>   > Bastien
>>>>>>   Hi,
>>>>>> 
>>>>>>   this files are used for test only, maybe severity could be 
>>>>>> decreased.
>>>>> 
>>>>>  I find the severity accurate: Relying on non-source code is a 
>>>>> severe
>>>>>  violation of Debian Policy, not matter the purpose of relying on 
>>>>> it.
>>>> 
>>>>  I think we should change the policy here. Running tests helps 
>>>> improve
>>>>  the quality of the software we ship. Many times the vendored code 
>>>> is
>>>>  used to ensure the code does not break in a specific situation. I 
>>>> don't
>>>>  think reducing test coverage in such situations is really helpful.
>>> 
>>>  Right, running tests helps improve the quality of software we ship.
>>>  Which is why you probably need to test using what's shipped in 
>>> Debian
>>>  rather than using a vendored source-less code.
>> 
>>  We are not shipping the source less code.
> 
> You are: Debian also ships source code.

I meant, not shipping in any binary package. Though as Russ mentioned 
in his reply. I will propose a GR.
> 
>>  This is used only during tests. I don't think we are not gaining 
>> anything by removing tests here. Just making it harder for the 
>> package maintainer to run tests.
> 
> You would not gain anything by removing tests, but you would win by
> making these tests completely free software.

I am just saying it increases the work required to run tests and when 
disabling tests is an option, the incentive is to disable tests.
> 
>>>  If we rely on non-free code for tests, that's really bad too, and 
>>> that
>>>  must be avoided just like we're avoiding source-less code 
>>> everywhere
>>>  else in Debian. The policy shall not change, please.
>>> 
>> 
>>  The code is not non-free here, just a specific version of a Free 
>> Software code built outside Debian.
> 
> We build from source...

We build the binary packages from source. I don't think it is useful to 
extend that to tests without considering the tradeoffs involved.
> 
>>  I think tools required for tests should be considered separately 
>> from tools required to compile. I think it should be treated similar 
>> to test data.
> 
> I don't agree.

ok, lets see how the whole project feels via a GR and settle it. I just 
expressed my opinion, you expressed yours and we need to make a 
decision now.
> 
>>  What you are proposing would require the package maintainer to 
>> adapt these tests to versions available (many times with different 
>> API versions) in Debian and the easier choice is disabling tests.
> 
> No. I believe it's ok to have an embedded version of the JS files in 
> the
> upstream code. This is a *very* different issue, please do not mix 
> them.
> What I don't like is using a minified version of the JS files. That's
> *very* easy (hum... trivial?) to add a non-minified version in your
> Debian folder, and use that for tests. You don't care if running the
> tests is a little bit slower (because using a source-full version), 
> do you?

I don't think you really understand the complexities here. Building the 
minified version is not just running the minifier against the non 
minified code. The non minified code itself is generated using many 
other tools (typescript, transpiled using babel, bundled using rollup 
or webpack etc - many times the versions of these tools are very much 
different versions as well).

> 
> However, there's this:
> 
> On 10/7/21 6:17 PM, Richard Laager wrote:
>>  Running tests against vendored dependencies one isn't going to use 
>> at
>>  run-time is of limited usefulness.
> 
> Best is, if you can, use the library packaged separately, in Debian,
> both for tests, and runtime. This way, you do ensure that:
> - patching Debian for security is still a thing
> - the package can run with the Debian version of the lib
> 

You are completely missing the reality here as well. The runtime 
dependencies are already used from the packaged versions. These 
vendored libraries are used only to create specific test cases or 
sometimes using alternative implementations to test the shipped code.

> I think it's less grave than just saying "oh, we don't care about 
> these
> binary blobs, there's just for tests...". It's even worse, because by
> using a different version for tests and runtime, you're faking 
> tests...
> 

See above. All runtime dependencies are packaged and used from packaged 
versions. In many cases the code is used to building dummy apps to 
create specific test cases.

> If the lib are just use for tests and nothing else (ie: not for
> runtime), then back to square one: it's ok to ship the non-minified
> version in your debian folder, and use that for running tests. It's 
> also
> super easy and fast to implement.

See above explanation. It is not as simple as dropping the non minified 
source code.

> 
>>  I think blindly applying a rule without thinking of any 
>> consequences is bad too.
> 
> I think blindly saying "oh, it's ok, it's only test things..." is a
> *very* dangerous path that I would like Debian to avoid.
> 
>>  Just because it is bad in one situation does not mean it will be 
>> bad in every situation. We should evaluate pros and cons of each 
>> situation before making a decision. Blind faith is more suitable for 
>> religions and not for a project like ours.
> 
> Sorry, but using free software from source is *NOT* opened for debate.
> If you would like to do that, choose another distribution. We all
> signed-up for it, when becoming DDs, this is the foundations of 
> Debian.

You are blindly equating the binary packages we ship with tests used to 
check if the built software is fine or not. Lets see how the project 
feels about it via a GR.
> 
>>  I think a nocheck build profile which excludes these files from 
>> build is sufficient to ensure we are not using these to create 
>> binary package.
> 
> What's the problem with using a non-minified version of the files? 
> It's
> not difficult, and it doesn't take too much of your packaging time.
> 

Already explained earlier. You are commenting on a situation which you 
don't understand.

>>  This way we guarantee only packages in main is used to generate the 
>> binary, but still allows to run tests optionally making it easy to 
>> find problems, especially during transitions. Currently when tests 
>> are missing transitions are harder because we can't find breakages 
>> easily since tests are disabled.
> 
> What you're proposing is making the life of anyone willing to modify 
> the
> software harder.

You are simply making false equivalance. Nothing stops anyone from 
writing different tests or even modifying the tests or creating 
different test cases. It just makes modifying one particular test case 
different, which is anyway an arbitrary condition. To test a software 
you don't really need one unique test case.

> 
>>  The current policy is not making Debian better.
> 
> It is. It's ensuring we build from source, and we're able to modify
> what's in Debian with reasonable conditions.

We are just insisting some specific test cases are constructed from 
source, where as there is no real requirement for test cases to be 
built like that. Anyone can build different test cases if required.