[Pkg-javascript-devel] Bug#995722: Not running tests because tests miss source code is not useful

[Thomas Goirand]

On 10/8/21 7:30 PM, Pirate Praveen wrote:
>>>  This is used only during tests. I don't think we are not gaining
>>> anything by removing tests here. Just making it harder for the
>>> package maintainer to run tests.
>>
>> You would not gain anything by removing tests, but you would win by
>> making these tests completely free software.
> 
> I am just saying it increases the work required to run tests

Yes, sure! I'm not contesting this. Just like it increases the work
sometimes to de-vendor minified JS libraries we ship as binaries (which
often we re-minify at build time...).

> and when disabling tests is an option, the incentive is to disable tests.

That's called laziness, and we shall not tolerate this. I've often
packaged some Python libraries *only* to be able to run tests. I very
much think others should at least aim do the same (even if it's not easy).

>>>>  If we rely on non-free code for tests, that's really bad too, and that
>>>>  must be avoided just like we're avoiding source-less code everywhere
>>>>  else in Debian. The policy shall not change, please.
>>>>
>>>
>>>  The code is not non-free here, just a specific version of a Free
>>> Software code built outside Debian.
>>
>> We build from source...
> 
> We build the binary packages from source. I don't think it is useful to
> extend that to tests without considering the tradeoffs involved.

Hang on, let me consider ... done ! :)

I do not think you'll go as far as saying that running unit (or
functional) tests using blobs is superior to do that using source only
tests (or built from source libraries to run tests). We shall have, as a
goal, to ship *every* source code that's useful to contribute / hack /
modify any given piece of software we ship as binary.

It is my opinion that proposing a GR to tolerate blobs when running
tests is a *very* dangerous path that I would strongly recommend
against. Most likely, you will not like the outcome anyways.

>>>  I think tools required for tests should be considered separately
>>> from tools required to compile. I think it should be treated similar
>>> to test data.
>>
>> I don't agree.
> 
> ok, lets see how the whole project feels via a GR and settle it. I just
> expressed my opinion, you expressed yours and we need to make a decision
> now.

Do you understand that what you're proposing is clearly against all
rules we have in Debian since it's inception? We all signed-up for doing
free software, and free software only, without any "tradeoff".

>>>  What you are proposing would require the package maintainer to adapt
>>> these tests to versions available (many times with different API
>>> versions) in Debian and the easier choice is disabling tests.
>>
>> No. I believe it's ok to have an embedded version of the JS files in the
>> upstream code. This is a *very* different issue, please do not mix them.
>> What I don't like is using a minified version of the JS files. That's
>> *very* easy (hum... trivial?) to add a non-minified version in your
>> Debian folder, and use that for tests. You don't care if running the
>> tests is a little bit slower (because using a source-full version), do
>> you?
> 
> I don't think you really understand the complexities here. Building the
> minified version is not just running the minifier against the non
> minified code. The non minified code itself is generated using many
> other tools (typescript, transpiled using babel, bundled using rollup or
> webpack etc - many times the versions of these tools are very much
> different versions as well).

I very much understand all of this. I never contested that it's
difficult. Though I'm very much contesting that the difficulty for
building the binaries you're wishing to embed is a point of argumentation.

The more building these blobs is hard, the more we need the source code
and the recipe for building these blobs. If you believe these are needed
to guarantee the final artifact's quality, then probably they are also
needed for modifying upstream code too, with a good enough insurance not
to break anything. And I don't agree modifying / contributing to any
free software should be allowed using non-free tools.

>> Best is, if you can, use the library packaged separately, in Debian,
>> both for tests, and runtime. This way, you do ensure that:
>> - patching Debian for security is still a thing
>> - the package can run with the Debian version of the lib
>>
> 
> You are completely missing the reality here as well.

I am *NOT* missing ANYTHING here.
Please read carefully once more: I UNDERSTAND THE PROBLEM.

:)

> The runtime dependencies are already used from the packaged versions.

I get that point, you already mentioned it anyways.

> These vendored
> libraries are used only to create specific test cases or sometimes using
> alternative implementations to test the shipped code.

You also wrote that before.

>> If the lib are just use for tests and nothing else (ie: not for
>> runtime), then back to square one: it's ok to ship the non-minified
>> version in your debian folder, and use that for running tests. It's also
>> super easy and fast to implement.
> 
> See above explanation. It is not as simple as dropping the non minified
> source code.

Then you have no choice but to package the test library separately, only
for adding it as build-depends.

Yes, it's more work. Yes, you probably then have the incentive to not
run tests. Yes, it may lower the overall quality of Debian. But then,
it's up to you to decide...

I warmly welcome, and so will everyone in Debian, if you take the
correct path of using the free-software-way when doing the packaging
work for running tests.

I've done in this direction it a lot in the Python world when packaging
OpenStack. Absolutely ALL of the packages from upstream OpenStack run
tests during build. I should also be adding the same test suite as
autopkgtest, probably.

Let me give a few example of packages I am maintaining only so I can do
that (ie: package testing at build time):
- python-ddt
- python-fixtures
- python-nosehtmloutput
- python-nose-parameterized
- python-nose-testconfig
- python-nose-timer
- python-stestr
- python-subunit2sql
- python-testscenarios
- python-testtools
- python-traceback2
- subunit
- testresources
- unittest2

These are general purpose for unit tests. There's also 22 packages
forming the OpenStack functional test suite, which I also ship in Debian:
- barbican-tempest-plugin
- cinder-tempest-plugin
- cloudkitty-tempest-plugin
- designate-templest-plugin
- glance-tempest-plugin
- heat-tempest-plugin
- ironic-tempest-plugin
- keystone-tempest-plugin
- magnum-tempest-plugin
- manila-tempest-plugin
- mistral-tempest-plugin
- murano-tempest-plugin
- neutron-templest-plugin
- octavia-tempest-plugin
- senlin-tempest-plugin
- telemetry-tempest-plugin
- tempest
- tempest-horizon
- trove-tempest-plugin
- vitrage-tempest-plugin
- watcher-tempest-plugin
- zaqar-tempest-plugin

Then OCI [1] can setup an tempest server where all of the above is
installed, and configured automatically, using the Debian package
puppet-module-tempest.

[1]
https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer

I use the tempest functional test suite each time OpenStack is released,
to validate the release functionally. This is once every 6 months.

In some rare cases, I went even as far as packaging dependencies for
running the tests of a single package.

Yes, all of this is a lot of work. No, I do not expect everyone to
invest as much time as I have in software quality (I'm one of the lucky
few paid every day to do that kind of work...). So I do not want to
impose my own standard on others. However, if one want to do things "the
correct way" (tm), then that's the preferred way.

>>>  I think blindly applying a rule without thinking of any consequences
>>> is bad too.
>>
>> I think blindly saying "oh, it's ok, it's only test things..." is a
>> *very* dangerous path that I would like Debian to avoid.
>>
>>>  Just because it is bad in one situation does not mean it will be bad
>>> in every situation. We should evaluate pros and cons of each
>>> situation before making a decision. Blind faith is more suitable for
>>> religions and not for a project like ours.
>>
>> Sorry, but using free software from source is *NOT* opened for debate.
>> If you would like to do that, choose another distribution. We all
>> signed-up for it, when becoming DDs, this is the foundations of Debian.
> 
> You are blindly equating the binary packages we ship with tests used to
> check if the built software is fine or not.

PLEASE ! Not blindly !!! I understand what I'm talking about.

Your world (ie: Javascript and Ruby, am I right?) is probably different
than mine (mostly Python), but the rules and problematic are probably
similar as well. At least, I know that we are both facing the problem of
"hey... too many packages". :)

> Lets see how the project feels about it via a GR.

I very much would prefer if you didn't try to undo what Debian has been
doing for nearly 40 years. We shall stay free. Completely free. If you
wish to add blobs, there's contrib and non-free for that. What makes you
think that you'll succeed convincing a majority, when all of us have
signed-up for fully-free ?

> Already explained earlier. You are commenting on a situation which you
> don't understand.

Again ?!?
Please respect me, and don't assume any lack understanding by default.

This isn't the case that I don't understand: I just have a different
opinion, which I believe is the one everyone has in this community. If
you think otherwise, and you're convince you should go the GR route,
then your GR *must* include changing the social contract & DFSG to add
such exception in there.

Frankly, good luck for convincing everyone that the DFSG shall become
something along the line of:

"Debian does free software, but don't give a shit if software testing is
non-free, so it's ok to do crap non-free blobs there..."

I seriously doubt this will have the success you hope for.

>>>  This way we guarantee only packages in main is used to generate the
>>> binary, but still allows to run tests optionally making it easy to
>>> find problems, especially during transitions. Currently when tests
>>> are missing transitions are harder because we can't find breakages
>>> easily since tests are disabled.
>>
>> What you're proposing is making the life of anyone willing to modify the
>> software harder.
> 
> You are simply making false equivalance. Nothing stops anyone from
> writing different tests or even modifying the tests or creating
> different test cases.

With large software, we cannot just skip unit (and functional) testing.
Some people use to repeat all the time in OpenStack: if it's not tested,
it's broken.

Here, we aren't talking about an hypothetical unit test that someone may
write some day. We're talking about unit testing of most (and preferably
all) components of a big software suite, that interact with a lot of
components. Both you and I know it's vital to run as many tests as
possible to ensure version requirements don't break.

So, all together, these tests are in fact fully part of the source code
you're using to build binaries. Therefore, you cannot just use
source-less artifacts to run them, because that renders the whole thing
non-free.

Cheers,

Thomas Goirand (zigo)

P.S: I very much regret we didn't have the opportunity to discuss this
(loudly) around a few beers, late at night, during a debcamp or
something. I'm eager to meet you in real and have such luck on day.
Please make it to Debconf Prizren, Kosovo next year! :)